The scheduler handed every build `--secret id=GIT_AUTH_TOKEN`, on the reasoning
written above it: private contexts need it, "public repos ignore it". The second
half is false. A host that does not accept the token does not shrug it off -- it
refuses it, and git falls through to a prompt that does not exist inside a build.
MEASURED against our own forge: with the secret present, fetching
http://hanzo-git.hanzo.svc.cluster.local/hanzoai/cloud.git dies at step #1 on
`could not read Username ... terminal prompts disabled`. Without it, the identical
fetch succeeds anonymously in 3.3s. So the flag that exists to make private
fetches possible was the single thing making our own forge impossible.
The credential now goes out under two ids because it answers two questions:
GO_MOD_TOKEN is ours and is what the Dockerfile mounts for the private Go modules
on github.com; GIT_AUTH_TOKEN is BuildKit's reserved name for the CONTEXT host,
and is offered only when the context is somewhere a github credential could
plausibly be right -- which our own forge, by construction, is not.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The Dockerfile mounted its github credential as `id=GIT_AUTH_TOKEN`. That id is
RESERVED: buildkit's git source reads a secret by that exact name and offers it
to whatever host the build CONTEXT lives on. So one name carried two meanings --
"the credential for the private Go modules on github.com" to this RUN, and "the
credential for the context host" to buildkit -- and the two agreed only for as
long as both happened to be github.
They stop agreeing the moment the context is our own forge. MEASURED: with
--secret id=GIT_AUTH_TOKEN present, fetching the cloud repo from
http://hanzo-git.hanzo.svc.cluster.local dies at step #1 on `could not read
Username for '...': terminal prompts disabled` -- buildkit offered a github token
to the forge, the forge refused it, and git fell through to a prompt that does
not exist in a build. The identical fetch with no such secret succeeds in 3.3s,
anonymously, because the forge serves these repos anonymously.
So the fix is not to find a token that satisfies both hosts -- there isn't one,
and looking for it is what the shared name makes you do. It is to stop sharing
the name. GO_MOD_TOKEN says which credential it is and which host it is for;
buildkit finds no GIT_AUTH_TOKEN and fetches the context as itself.
github.com/hanzoai/dashscopego is the only genuinely private module of the 52
(ai, commerce, orm, iam and base all answer an anonymous ls-remote), so this one
secret is the whole reason a cloud build needs github at all.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The comment above this mount already wrote the runbook: "a module resolved while
its tag did not yet exist is remembered as 'unknown revision' forever, so
`go mod download` keeps failing on a tag that now exists and resolves fine from a
clean cache. BUMP THE SUFFIX (-v4 -> -v5) to force a cold module cache the next
time a phantom pin poisons it."
It is poisoned. MEASURED: `go mod download` fails on
github.com/hanzoai/dashscopego@v0.6.0 with "could not read Username for
'https://github.com': terminal prompts disabled", out of the VCS cache at
/go/pkg/mod/cache/vcs/2cb27ec8..., and the entry is remembered per node, which is
why the same commit builds on one runner and dies on another.
dashscopego is the one genuinely PRIVATE module of the hanzoai set -- ai,
commerce, orm, iam and base all answer an anonymous ls-remote -- so it is the
single dependency that keeps this build tied to a GitHub credential, and the
single entry a failed tokenless attempt can poison for everyone after it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The host allowlist learned to TRUST our own forge (dc84b46d). It still could not
FETCH from it. git.hanzo.ai resolves to this cluster's own LoadBalancer, DO's LB
does not hairpin, and so the fetch did not fail cleanly -- it hung. MEASURED from
ns hanzo-build: the public name times out at 25.0s, and a real build sat on it
for 134678 ms before giving up, while the in-cluster Service answers 303 in
0.014s. The object-store lane hit the identical wall at 133755 ms and
artifact-publish-egress.yaml already wrote down the shape of the answer: reach
the internal endpoint, keep publishing the public URL.
So the public URL stays the API contract -- callers POST /v1/runner with
https://git.hanzo.ai/... and validation still judges THAT host -- and only the
fetch is redirected. The two concerns stay apart: hostAllowed decides what we are
willing to build, internalizeForgeURL decides how to go and get it. A test pins
that setting an internal endpoint does not widen what the API accepts.
isSelfGit is one predicate because it answers one question asked twice: trust
(our own forge is always a legitimate source) and routing (our own forge is
reachable in-cluster and must not be dialled by its public name). Two copies
could drift into trusting a host we cannot fetch.
Empty by default, so this changes nothing until an operator says where the forge
lives: CLOUD_PLATFORM_FORGE_INTERNAL_URL=http://hanzo-git.hanzo.svc.cluster.local.
Which endpoint is reachable is a deployment fact, not a code opinion.
The context string was assembled identically at two call sites and the rewrite
has to apply to both, so there is now one gitContext.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The plugin loop is almost pure LINK time, and Go links one binary on one core.
MEASURED: 120 plugins, 119 inter-plugin deltas, mean 2.40s, median 1.80s, 285.1s
total -- 22% of a 21.8m build spent using one of the pod's eight cores while
seven idled. The slowest are commerce 12.1s, ai 12.0s, base 7.4s, zen 7.4s.
-P 4, not more, because the ceiling here is RAM and not CPU: the pod's memory
limit is 16Gi and concurrent links of the heavy binaries are what peak it. Go's
build cache is concurrency-safe, so there is no correctness question, only a
resource one.
busybox xargs was VERIFIED in this exact builder image rather than assumed --
Alpine's xargs is a busybox applet and -P is frequently compiled out. Four 2s
jobs at -P 4 finished in 2s elapsed, not 8s, and a failing child returned 123.
123 is non-zero, so the `set -eu` above still fails the build, and the FATAL
guard still names the plugin it tripped on.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
mode=max exports the intermediate stages on the argument that the Go compiles
live there, so exporting only the final stage would make the expensive steps
rerun anyway. They rerun anyway REGARDLESS. Every expensive step in the cloud
image sits below `COPY . .`; the source is what changed; those layers are
invalidated on entry and no exported copy of them is reachable. The export was
paying, every build, to write layers that cannot be read.
MEASURED on three consecutive cloud builds -- exporting cache to registry cost
265.0s, 263.0s and 275.6s, ~20% of a 21.8m build, and bought SEVEN cached steps
out of ~45: an apk add, an addgroup, a symlink, and three small COPYs. Four of
the seven are in the final stage and survive mode=min. The three that do not are
worth ~10-30s. Net ~240s per build, on every repo that builds this way, plus
~1.9GB of dead layers no longer pushed to the registry each time.
One constant rather than two literals: the registry and the object store were
asking the same question and could drift to different answers. It is named
cacheExportMode so the thing to re-check is findable -- and what to re-check is
the DOCKERFILE, not the backend: this flips back the day expensive work moves
above `COPY . .`.
The two tests asserted mode=max and its rationale verbatim, so they are updated
to assert the measurement rather than the assumption.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/console d761fbc. Clicking "Sign in" on cloud.hanzo.ai appeared to do
nothing: console.hanzo.ai/ served a SECOND copy of the Hanzo Cloud marketing page
wearing the byte-identical @hanzogui/shell header, so the click landed on a page
indistinguishable from the one it left and read as a re-render. Reaching hanzo.id
took three clicks, two of them through pages that only asked "did you mean it?".
The console is the application; the marketing face is cloud.hanzo.ai. `/` is no
longer a special surface — an anonymous visitor STARTS the authorize hop.
Pinned by digest, not the usual sha-<sha7>-amd64 tag: that build published to
:latest, which is the moving target this file already warns about. The digest
names these bytes and no other.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The self-hosted-git allowance existed, was correct in intent, and could never
match. `selfGitHost` was `deps.Domain` verbatim — "api.hanzo.ai" — while
hostAllowed grants that host or a SUBDOMAIN of it. The forge is "git.hanzo.ai":
a sibling. So every native build was refused with
repo.url host "git.hanzo.ai" is not an allowed git provider
and the estate fell back to building from GitHub, which is exactly what broke
when hanzoai/cloud moved to hanzo-inc/cloud and the build credential lost access.
The code's own comment already said APEX; only the assignment disagreed.
selfGitHost is now the registrable apex ("hanzo.ai"), so every sibling the
deployment owns — git., ci., cd. — is a trusted build source by construction,
for hanzo.ai, lux.network, zoo.network and any white-label domain, with no
per-brand list to maintain. publicsuffix rather than "last two labels" so a
multi-label suffix (co.uk) yields the registrable domain and not the suffix
itself, which would trust every domain under it.
Proven both directions: TestSelfForgeIsAllowedFromTheApex admits
git/ci/cd.hanzo.ai and still REFUSES git.evil.com;
TestOldDomainVerbatimRefusedTheForge pins the old behaviour as the defect, so
this cannot silently regress.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
commerceTierToLadder allow-listed only `starter`->trial and `pro`/`enterprise`
->paid, and folded EVERYTHING ELSE to free. The plans commerce actually sells
are go ($9), dev ($19), pro, max, team and business — so a subscriber on
go/dev/max/team/business was scored free, then refused every SKU carrying a
trial or paid floor AND throttled by the free-tier flash cap on top. We took
their money and shut the door.
v1.832.29 inverts the default: a plan is PAID unless it is explicitly free
("", "free", or a `*-free` suffix), with starter/trial the one middle rung.
That direction is safe because A TIER IS NOT A PAYMENT. filter_balance.go has
no exemptions and fails closed — "nothing runs on credit it has not been funded
for" — so an unrecognized plan reaching `paid` still cannot spend a cent it has
not been funded. The OLD default was the dangerous one: it silently cost
revenue every time anyone added a plan slug. Now, add a plan and it works.
Both tests had encoded the defect as intent ("developer"->free,
"mystery"->free) and were corrected to the real plan list. 22 packages ok,
0 FAIL at the module.
Also carries the zen seed fix (v1.832.24) and the object TestMain fix
(v1.832.27), which took that package from ZERO tests executing to 153 PASS —
TestMain called os.Exit(0) before m.Run(). That immediately surfaced a
crawl-storage default still naming the retired MinIO Service, a correction that
had been made twice and could never take effect.
Verified: `go build ./...` clean at this pin.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`p.Status = "error"` on a non-live completion was UNCONDITIONAL, so "this project
is broken" was reachable from any deployment row the org could name — including
one that had already been superseded.
Found by triggering it on hanzo-ai, not by reading it. v5 was live with all 8401
objects correct in the bucket; completing v3 — an older probe deployment left
queued during the Sites-plane migration — as `error` flipped the PROJECT to
"error", and the sites edge stopped serving. Nothing was wrong with the site. The
bytes never moved. A status field on a superseded row took the host down, and
re-completing v5 as live brought it straight back.
The ordinary production shape is the same bug with worse timing: a rebuild that
fails while the PREVIOUS build is still live. The old content is still being
served and the site is up, yet the project gets marked broken — and the "report a
failed build" step that every CI workflow carries (so a dead build cannot leave a
deployment queued forever) is exactly the thing that would send it.
The rule is now named rather than inlined: failureOwnsProject(currentDeploy,
deployID). Error propagates to the project only when the deployment IS the one the
project points at, or when it points at nothing — a first deploy that fails leaves
a project that has genuinely never served, and that one should read as error. The
deployment row is still error in every case and LifecycleDeployFailed still fires,
so the failure stays visible where it belongs: on the deployment, not on the health
of a site that is up.
The `live` branch already recorded p.CurrentDeploy, so the information needed to
make this distinction was there the whole time and simply was not consulted.
TestFailureOwnsProject pins all four cases. Mutation-checked: restoring the old
`return true` fails exactly the two that matter — the superseded deployment and
the failed rebuild — so the test cannot pass against the behaviour it exists to
prevent.
(TestDeleteStays204WithNoBody fails on darwin both with and without this change:
the SQLCipher codec refuses to decrypt without tmpfs, and macOS has no /dev/shm.
Pre-existing and unrelated.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A plan-refused caller was told to request limited-preview access — a
door that cannot open when the plan is the blocker. Each family gate
now speaks its own sentence: upgrade for the subscription floor, paid
capacity for the funding floor, request-access only where a grant is
truly what is missing.
The edge resolved an extension-less path to two candidates — the exact key and
`<rel>/index.html` — and Next.js `output: export` writes neither for a normal
route. Without `trailingSlash` it emits the FLAT file `pricing.html`, so the
homepage served and every other route 404'd.
Measured on the hanzo.ai export (759 pages, 8402 objects) published to
hanzo-ai.hanzo.app:
/ 200 /pricing 404 (/pricing.html 200)
/zen 404 /zen.html 200
/zen/models 404 /zen/models.html 200
That is the shape of a site that reports `status: live`, serves its homepage,
and is unusable — the failure is invisible from the deploy and from the root URL.
`<rel>.html` is added as a middle candidate. Both spellings are legitimate and
both are now tried: Hugo, Jekyll, Vite MPA and `trailingSlash: true` emit the
directory-index form, Next's default emits the flat form. The alternative was
setting `trailingSlash: true` in every repo, which pushes a server limitation
onto each site and rewrites every canonical URL to work around it — this is one
extra HEAD miss on the paths using the other convention, in one place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The embedded family catalog no longer gates any enso SKU, so discovery
stops advertising a waitlist and ai serves every caller without a grant.
The app builder's default model refused everyone outside the launch org;
now it answers.
The envelope's RAM-backed guard now recognises a native macOS tmpfs
(sudo mount_tmpfs <dir>, then HANZO_SQLITE_RAMFS_DIR=<dir>), so the
cek-backed suites can run on a Mac instead of failing on every commit.
Fail-closed is unchanged: anything statfs cannot verify as tmpfs is
still refused.
Record.NeedCommit carried a xorm-style `db:"index"`, and dbx reads that tag
as the column NAME — so the column was called `index` while
ScanNeedCommitRecords queries `need_commit`. Live in this pod: 'no such
column: need_commit' every five minutes, and the record-chain commit task had
never committed a record.
Also carries a guard for the class, placed in routers rather than object
because object's TestMain os.Exit(0)s that package without a seeded database —
a test there reports ok without ever running.
hanzo_service_up is now recorded a second time, through Metrics() — the
registry the framework exports natively — under the exact series name and
label the meter pipeline has always published. Two producers, one probe, one
truth: every rule and reader sees an uninterrupted series whichever road the
sample travelled, which is what lets the old road be retired without a gap
once every reader is confirmed on this one.
The verdict is recorded where it is decided: the probe client's transport,
which already tells each target's reason on change. A nil registry skips
recording — a test that mounts probes without one measures the probing, not
the export.
zip moves to v1.24.6 for Metrics(), which also brings the native span and log
export and the convention that finds a collector with no configuration.
hanzoai/ai is a library this binary links, not a service of its own, so the
crawl4ai leg it dropped stays live in production until this pin moves. v1.832.25
points /v1/crawl at apps/crawl.Fetch — the same guarded dialer the answer
engine's read stage uses, refusing non-public addresses on every hop including
redirects — instead of dialling crawl.hanzo.svc.cluster.local:11235, a name that
does not resolve and returned success:false for every request.
The survey shipped with the loop steerable by the pages it read. Its decision
prompt carries titles and URLs crawled from hosts we do not control, so the move
that comes back is partly authored by them — and nothing checked it. Three
consequences, all reachable by getting one page into the ranked set:
a `read` URL was fetched whether or not the survey had ever gathered it, so a
page could point the cluster's egress at any address and carry the user's
question there in a query string, then have the answer filed in the tenant's
corpus;
a round's `queries` list was unbounded — mode.maxQueries bounded round zero and
nothing else — so one move could issue hundreds of serial meta-searches from a
shared egress that is already bot-challenged;
page text was spliced into the numbered source block unfenced, so a body
containing `[9] Title\nURL\nbody` became a source indistinguishable from a real
one, and nothing validated that the URLs the answer cited were sources at all.
CONTAINMENT, NOT INSTRUCTION. A prompt is advice to a model. These are properties
of the text:
`read` is intersected with the pool the survey itself gathered, and `queries`
and `read` are clipped where the untrusted value crosses into the loop;
ground.go fences each source with a per-request nonce the page cannot know,
because it was written before the request existed;
every markdown link in the answer is checked against the gathered set — on the
stream and in `done`, through the same function — so a citation always points at
a page THIS request fetched. An ungrounded link keeps its text and loses its
target: the sentence still reads.
Two bounds that could not bind now do. tokenCeiling measured the survey's own
subtotal while the plan and the synthesis — the expensive calls — sat outside it,
so research's 400k ceiling was ~50x the reachable total; it now takes the
request's running total. And plan, survey and synthesis shared one 300s clock
with no reservation, so a survey that spent it handed a full corpus to a
completion that could not start and the caller got "the model is unavailable"
after five minutes; the gather now gets 70% and synthesis keeps the rest.
Also fixed, and each one a bug on its own:
unread MARKED every URL a move named while read FETCHED only the first six, so
a move naming ten pages blacklisted four without ever fetching one — and the
round then tripped saturation and ended the survey early, losing evidence on
exactly the runs working hardest. The cap now comes before the marking.
Source.Text carries the fetched page and Source.Snippet stays the search
summary. Reading no longer overwrites what the wire shows, so a research answer
stops re-sending up to a megabyte of duplicate page text across twelve
snapshots, and one frame per round is now the whole story.
A round's queries run concurrently. Serially, at 12s each, a six-query round
spent most of the answer's wall clock waiting.
An unreadable move gets ONE stricter reprompt. A model that opens with "Sure!
Let me look at the JVM next" collapsed research into a single pass, and nothing
downstream could tell that from a model that judged the evidence complete.
A client that hung up ends the run: one research answer costs five minutes,
three dozen fetches and up to eight completions, and a closed tab bought all of
it. The plan's topic headings now reach the client as a planning detail, which
is what makes a three-minute wait legible. Crawl and completion failures are
logged — a degradation nobody can see is a degradation nobody can fix.
The package doc claimed METERED ONCE. It is not true: build.go hands this engine
a metered AI plane, so every internal completion also debits the payer per token
alongside the flat fee. Which layer should price /v1/ask is an open decision,
recorded rather than claimed away.
The SearchEvent union is unchanged — same variants, same keys, same order.
v1.801.445 failed to build, twice, on `go mod download`:
github.com/hanzoai/commerce@v1.49.67: reading go.mod at revision
v1.49.67: unknown revision v1.49.67
v1.49.67 is a PHANTOM. It exists on no remote: github.com/hanzoai/commerce
carries exactly two tag refs (v1.50.0), and git.hanzo.ai/hanzoai/commerce
lists ...v1.49.65, v1.49.66, v1.49.68 — .67 is skipped. It survives only in
warm module caches, which is why .442/.443/.444 built with the same pin: the
BuildKit `cloud-gomod-v4` cache is PER NODE, .444 landed on a node that still
held it (runner-pool-32g-3mn0fk) and .445 landed on one that did not
(runner-pool-32g-3mnls1). The release train was therefore passing by luck of
placement, and any cold node broke it — exactly the failure the Dockerfile
comment above `go mod download` anticipates.
Nothing here can be fixed by retrying. The Dockerfile sets
GOPRIVATE=github.com/hanzoai/* with GOPROXY=...,direct, so hanzoai modules
resolve DIRECT FROM GITHUB, and v1.50.0 is the only commerce version GitHub
has. This is an upgrade, not a workaround: commerce was re-published under
MIT OR Apache-2.0 and cut v1.50.0, and go.mod simply had not caught up.
Verified rather than assumed: `go build ./...` is clean against v1.50.0, and
v1.50.0 carries the same 1875 dirs and 8 billing packages as .67 including the
Cloudflare 502 work (thirdparty/cloudflare, api/billing) — the ~134 fewer .go
files are the tests/enterprise the OSS publish strips.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.832.24 stops seeding an admin/zen provider row that pointed at do-ai's base
https://inference.do-ai.run/v1. familyProvider reads that row as an operator
override of ZEN_URL, and the family paths append their own /v1, so every zen
catalog refresh hit .../v1/v1/models and 404'd — no zen SKU was listed or
served on api.hanzo.ai, once a minute, silently. Sibling enso was never seeded
and never broke.
The seed also self-heals ProviderUrl on every boot and re-creates a deleted
row, so this was not fixable in the database; v1.832.24 drops the seed entry
AND prunes the existing row, scoped to the exact stale shape.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
money.Amount.Minor() rescales, and hanzoai/decimal's Rescale rounds
HALF-AWAY-FROM-ZERO (decimal.go:145). Both balance call sites carried a
comment saying it "truncates toward zero". It never did.
So a balance of 4.995 USD read as 500 cents. hanzoai/ai's transaction check
is `avail < priceCents`, and apps/billing/gpu_charge.go is
`available < req.AmountCents` — both admit a 500-cent charge against a
balance that cannot cover it. The debit that follows is exact, so the
difference lands as a negative balance nobody authorized. Half a cent, every
time, with no error to find.
plane.Money.FloorMinor is that rounding made once, where the wire conversion
already lives, for every caller that COMPARES rather than debits. Minor()
refusing a sub-unit amount is still right for a debit and is untouched.
big.Int.Div is Euclidean, so a negative balance floors away from zero rather
than drifting back toward solvency the way truncation would, and the scale
comes from the currency exactly as Minor() takes it — cents are a property
of USD, not of money.
The guard that existed for this GREPPED ai.go for `a.Minor()` and for the
sentence claiming truncation, so it confirmed the wrong claim was still
written down. A test that reads the source cannot notice the sentence is
false. plane/money_test.go asserts the arithmetic instead — including the
4.995 case that was admitted, the real prod balance, and both signs. What is
left in apps/ai is the one thing only that package can say: that its gate
still asks for the floored figure.
Also corrects apps/billing/balance.go's comment calling this number "a
DISPLAY, nothing is billed from it". It is `available` on
/v1/billing/balance, which is what ai's balance gate reads over the S2S HTTP
path, and what gpu_charge.go compares against a GPU's price.
Verified: money.Amount.Minor()=500 vs FloorMinor()=499 for 4.995.
apps/billing's TestGPUCharge_* fail identically at origin/main -- they refuse
to run without a tmpfs (/dev/shm), which macOS has not; unrelated to this.
Deep research was the one thing the answer engine could not do. `research` planned
sub-queries, searched them ONCE, read six pages, and wrote. A question whose answer
is only visible after the first round of reading — which is what "research" means —
got a search with a longer prompt.
SURVEY is the value that was missing: search and read applied to a plan, and
ITERATED. Each round asks the model for one compact JSON move (`next`, `queries`,
`read`, `done`), runs it, and discards the prose that came with it — gathering is
not writing. No tool plane is needed or used, and a model that answers in prose
ends the survey instead of derailing it.
`rounds == 0` is the single pass, byte for byte. search/news are untouched: same
queries, same one-per-host set, same 90s, same 2c. One code path, parameterized by
a mode value — not a second engine, and not a second route. /v1/ask stays the one
door and `mode` stays a value handed to it.
Bounded four ways, because an unbounded agent loop cannot be gated by Bill.Gate
before it runs and an ungated loop on a per-org ledger is a money bug:
rounds the mode's budget, hard-capped at maxRounds=8
deadline per-mode wall clock (search 90s, research 300s)
tokenCeiling per-mode token spend (120k / 400k)
saturation a round that found no new source and read no new page
Saturation replaces the `len(srcs) >= maxSources` guard the design called for.
That test cannot do the job it was written for: rank() already caps the set at
maxSources, so it fires on the FIRST productive round and collapses research back
into the single pass the survey exists to iterate. "No new evidence arrived" is
the bound that was meant, it cannot be satisfied vacuously, and it also terminates
a model that keeps proposing a query it has already run.
Every exit lands on the same return, and the caller always goes on to synthesize
whatever was gathered — the envelope reaches `done` from every path.
The contract did not move. status | sources | text | follow_ups | done, four
stages, no fifth: the round's next step rides as `status{planning, detail}`, and
per-page reading progress as `status{reading, detail:host}` — which the union has
always declared and nothing used to emit. `sources` stays a CUMULATIVE snapshot
because all three SDK consumers replace their list on it.
Alongside, four things the port made necessary:
rank takes a per-host cap. One page per host is right for a six-source answer,
where breadth is the value, and wrong for research, where three pages from
an authoritative domain are the point. hostCap<=1 reproduces the old set.
rank cleans `[PDF]`/`(Official Site)` furniture off a title so a citation reads
as a document name — keeping the original when a title is entirely
bracketed, which would otherwise cite the page by bare hostname.
read takes the url list and the clip from its caller instead of a mode's top-N,
and reports progress per source. An iterated survey clips to 3000 runes:
many sources x a long page is the one way this loop could overrun a window.
synth streams through a joiner that holds a markdown link until its closing
paren, so a citation never renders as `[Rich Hickey](htt` and rewrite
itself. Delivery only — the finished text is identical.
research is repriced 10c -> 25c. It now gathers across rounds, decides between
them, and reads more than once; the price follows the work.
Not ported, deliberately: code interpreter, X search, chart artifacts, images, and
every external-SaaS leg (Tavily, Exa, Firecrawl, Notte, Supadata, Daytona). Search
is apps/websearch in-process and keyless; reading is apps/crawl, SSRF-guarded, with
its archive->fetch->headless ladder already standing in for Exa->Firecrawl->metadata.
Provider reasoning tokens are never mapped to `text`: every consumer appends delta
into answer and would corrupt both the answer and the persisted training sample.
Round decisions want temperature 0. cloud.ChatRequest carries no temperature field
and inventing one would fork the AI contract for a single caller, so the prompt
buys the determinism instead. Flagged, not faked.
Tests: survey_test.go proves rounds==0 is the old loop, that the gather actually
iterates (a later round searches and reads what round zero did not, and round
zero's page survives the re-rank), that the plan is carried verbatim into every
round, that snapshots are cumulative, that a round's prose never reaches the
answer, and that each of the five exits leaves the frame sequence intact.
stream_test.go adds the wire ordering invariant and proves the server never emits
the union's `error` variant.
v1.832.24 stops seeding an admin/zen provider row that pointed at do-ai's base
https://inference.do-ai.run/v1. familyProvider reads that row as an operator
override of ZEN_URL, and the family paths append their own /v1, so every zen
catalog refresh hit .../v1/v1/models and 404'd — no zen SKU was listed or
served on api.hanzo.ai, once a minute, silently. Sibling enso was never seeded
and never broke.
The seed also self-heals ProviderUrl on every boot and re-creates a deleted
row, so this was not fixable in the database; v1.832.24 drops the seed entry
AND prunes the existing row, scoped to the exact stale shape.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The rail's "could not generate a deposit address" was a 502, and Cloudflare
replaces an origin 502 with its own HTML interstitial — so the customer read
"request failed" while a clear JSON message sat at the origin, unreachable.
v1.49.68 answers 503 (which passes through, as the wire rail's own 503
already proves) with a sentence naming the rail, saying it is temporary, and
pointing at the alternatives.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`g := r.Group(p); g.Use(mw)` is the form that panicked — the group carried
middleware, the routes were registered on the root, and zip refused to compose
a guard that could never run. It is fixed, and it was the only one of the three
Use idioms with no test. Eleven production apps write it (books, captable,
commerce, company, compliance, dataroom, framework, git, legal, risk,
validators); the tests covered the other two.
Reverting scope.Group to hand back the raw router reproduces the original panic
verbatim and turns TestGroupThenUseGuardsOnlyItsSubtree red, which is the point:
asserting only that the panic is gone would also pass on a scope that silently
dropped the middleware, and a dead guard is worse than a panic because the panic
is honest. Each case asserts both halves — the guarded path answers 401, its
unguarded sibling answers 200 — across three subsystems, one with declared
prefixes and one nesting a group inside a group.
Addressing is asked separately from gating, because a gated route answers 401
whether or not it exists, so a test that asked both at once could not tell "the
route is where I said" from "the guard refused a 404".
And the residue is written down: a scope installs at the root, so its middleware
runs for what follows it. Use before the routes guards them; Use after them
guards nothing, and it composes either way because the root always has routes.
apps/company depends on exactly that — its fundraise/deck leaf sits above
g.Use(limitBody) on purpose, since the deck is document bytes and a JSON body
cap on it would be wrong. That deliberate exemption rested on an unwritten rule.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
INERT IN PRODUCTION. CLOUD_WRITER_LEASE is unset everywhere and this change
does not set it. With it unset every path here is a no-op, the lock file is
never created, and boot is byte-identical to what is running now. The env flip
and the strategy change remain a separate, human-gated step. Nothing to watch
on deploy.
The lease states a POD-level fact — "this pod owns this volume" — with a
PROCESS-level primitive, flock(2) on {DataDir}/.writer.lock. Those are the same
sentence only when exactly one process per pod reaches for the lock, and cloud
stopped being one process: cmd/cloud is a router that spawns every subsystem as
its own child, all sharing one DataDir.
The acquire lived in cloud.Listen — the body every PLUGIN binary runs, and the
one thing the router never calls. So turning the lease on aimed a single-holder
lock at the siblings instead of at the other pod. kms won the flock; pubsub and
kafka waited out the 90s fail-closed budget; nothing bound :8000; the liveness
probe killed the pod and its replacement deadlocked identically. api.hanzo.ai
served 503 from 08:52Z to 08:56Z on 2026-08-04.
The repair after that (aa416cc6) made the children skip the lock and stopped
the deadlock. It also left the lock in nobody's hands, because the router does
not run that code at all — an interlock that logs as armed and holds nothing.
That is the worse of the two states: the deadlock announced itself, whereas a
lock held by nobody is quiet until someone believes the log line and switches
to RollingUpdate. With S3_ADMIN_* armed the hydrate path renames over the live
DB, so the quiet failure is the expensive one.
So the duty is decided once, in internal/writerlease, from the only thing that
can distinguish these processes — their position in the tree:
Take the pod root, which takes the lock BEFORE it spawns anything and
releases it AFTER the last child is gone
Inherit a child, which is handed the answer and never contends
Off no lease configured — today, and correct under Recreate
cmd/cloud takes it before its mount loops and stamps CLOUD_WRITER_LEASE_HELD
with its own pid; zip already spawns children with append(os.Environ(), …), so
each is born knowing the volume is claimed. The stamp carries the pid rather
than a bare flag so a child CHECKS it — it counts only when it names that
child's own parent — which is what stops a stray value in a manifest from
talking a fresh pod root out of taking the lock. cloud.Listen makes the SAME
call: in the fleet it inherits, and run standalone (`hanzo kms` on its own
volume) it is a pod root and takes the lease itself. One rule, one reader of
CLOUD_WRITER_LEASE; the Config bool is gone, since a bool parsed per process
is true in all of them alike, which was the original misreading.
Also: Acquire rechecks that the inode it locked is still the file at that path,
because flock locks an inode and an unlink+recreate under it yields two holders
who both believe they are alone.
Tests run real processes, since the defect was a property of the process tree:
TestLegacyRule_SiblingsDeadlock reproduces the incident — 3 subsystems, one
DataDir, nobody above them: 1 serves, 2 spend the whole budget waiting for
a handoff that cannot come
TestFixedRule_SiblingsAllServe same topology, root holds first: all 3 serve,
duty=inherit, zero contention
TestFixed_VolumeStillDefended the one that fails against main as it stands
("a second pod OPENED the volume while this pod holds the lease") — proves
the fix did not simply disconnect the alarm
TestStampCannotBeForgedByConfig a hand-placed stamp cannot disarm a pod root
go build ./... green; go test ./... introduces no new failures (29 packages
fail on origin/main before and after, all unrelated).
Follow-ups NOT in this change, deliberately: flock is a lie on NFS/CIFS, so a
shared-filesystem refusal is still worth having (see blue/writer-ha); and
cto/writer-interlock-honest argues the exclusive-store premise is now stale,
which is a question about whether to keep the mechanism at all, not about
whether it should be correct.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Not crashing was not the fix. The ai router maps ANY non-2xx from
/v1/billing/tier to TierZenFree, so the 400 that replaced the 500 downgrades a
paying customer exactly as the crash did — 60 rpm against 500 for pro, silently,
with no error the customer or we would ever see. The tier has to RESOLVE.
Its caller is a service, not a person, and the two resolve an org by different
doors. IAMTokenRequired admits only a gateway-validated user identity (ownerID
AND X-User-Id AND email) and deliberately falls through on a bare X-Org-Id,
because admitting on that alone once let an off-gateway caller name any victim
org. The router sends precisely that shape — verified in ratelimit.go, a service
bearer plus X-Org-Id and no user — so it arrived with a nil org.
TokenRequired is the door for a caller that IS a credential: it verifies the
service token first and only then calls ensureIAMOrg to resolve the header.
Credential before trust. Same reason the catalog CRUD and the recharge poke each
carry their own TokenRequired rather than riding an IAM chain.
The commerce v1.49.67 handler fix stays and is still right: a tier that genuinely
cannot be read is refused rather than answered Free. This makes it readable.
The saved-card family had no way in. A customer could list cards and remove
one; adding required POST /v1/billing/methods, which the billing app
forwards to commerce's /v1/billing/methods — an address commerce does not
serve co-resident, and which the in-cluster `commerce` Service resolves back
to these pods, so the forward re-enters the forwarder. It never got that
far: CLOUD_COMMERCE_HTTP_URL is unset, so the proxy is unconfigured and
every saved-card call, GET and POST alike, answers 501 "billing is not
configured". Measured live on pay.hanzo.ai. Nothing could bill a monthly
plan to a card on file because no card could be put on file.
The portal face is where its siblings already live, so the POST goes there
too: no HTTP hop, nothing to self-dispatch, same gate. commerce's
CreatePaymentMethod vaults the Square nonce as a reusable card-on-file and
stores the billing address with it — that vault is what a renewal charges.
PinBillingSubject is load-bearing here rather than decorative: this handler
takes its subject from the BODY (customerId), and the pin rewrites the
subject keys there while preserving card/type/sourceId, so a caller can only
attach a card to its own account whatever the body claims.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The route registration landed in v1.801.440 and /v1/billing/tier still answered
500: the chain was never the problem, the handler was. GetTier type-asserted the
organization, and IAMTokenRequired resolves one only from a gateway-validated
user identity — so every S2S caller, which is this route's main caller, arrived
with nil. Fixed upstream where the handler lives, not worked around here.
Outside apps/iam no app file imports what a bearer is hand-rolled from
(crypto/hmac, a JWT library) or joins apps/team/token, except the pinned
entries: foreign auth wires (LiveKit, SigV4, webhook schemes, the mpc ring),
non-identity seals (OAuth state, unsubscribe links, a KDF), and the condemned
team token with its complete reader set, which only shrinks. The root's
deleted second authority failed permissively — key confusion, machine
principals as humans — and apps/ had no guard against growing another.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud/apps/<name> is plugin-mode wiring for github.com/hanzoai/<name>; the
product owns the functionality and ships its own standalone daemon. The
boundary now counts itself: an app either mounts its product or holds a pin
naming which debt it is (mismatched / unwired / unextracted), and the lists
only ratchet down. Measured at pin time: 140 apps — 16 mounted, 1 mismatched
(deploy mounts hanzoai/cd), 27 unwired, 96 unextracted.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The status page published three incidents for services that were up. Not one
bug — one structural defect: a hand-maintained address table drifted from the
fleet, and the prober faithfully published three wrong addresses. iam was
probed on a path it serves on a different port; kms was probed at a deployment
scaled to zero years after the fold into cloud; hanzo-mpc named a Service that
selects nothing.
fleetTargets is now the one registry of where a service answers, and both
availability reads take their address from it. iam is probed at the OIDC
discovery document — what its own readiness reads, and the first thing every
client fetches, so answering it means a customer can sign in. kms is probed at
cloud's embedded /v1/kms/health, which still fails closed without the master
key, so the API can be up while KMS honestly is not. hanzo-mpc is removed
rather than retargeted: the real ring answers 307 to every path including
absent ones — a probe that can only say yes trades a false outage for a false
all-clear.
The second address table is deleted. productmap synthesized addresses by
convention — port 80, try /health then /healthz — wrong for 19 of 27 products
and burning two timeouts per miss. An unwatched workload now has no address
and is probed not at all.
A failure names its target, address and reason, edge-triggered: one line when
it breaks, one when it recovers, and a changed reason reports again because
that is a different chase.
Verified from inside the live pod: the shipped list answers 20/20. A red test
turned green with it — the scoped-status read that blocked a full second
dialing a service that was never there.
/v1/search and /v1/vector belong to provisioning; product owns exactly
four routes inside them. Hanging requireKey on the two parent groups
claimed both subtrees, so the confinement gate refused the boot — and it
was right to: in the unified binary that middleware would have gated
provisioning's routes with product's key.
The credential is a REQUEST fact, so each op now declares it: keyedIn
carries the Authorization header as a typed input field (zip's stated
replacement for exactly this middleware) and requireKey opens every
handler. Same statuses in the same order — unset key 503s, wrong key
401s, search and vector keys never cross — and the four addresses are
byte-identical; the document gains only the header parameter each op
always required but never published.
Verified: apps/product suite green, bin/product boots (was the one
compose failure in 120), openapi weave green with no path added, moved
or removed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every program now emits its request log natively — method, path, status,
duration, trace and span, the caller when the environment parked one — through
its own logger, so the per-app Logger install is deleted along with the three
test copies of it. The framework also propagates trace context and serves
/metrics from the one registry.
Kept deliberately: the OTel meter pipeline (metrics_http.go, installMeter) and
the span path. The datastore pipeline they feed is what /v1/summary and
availability READ, and the framework's export has not yet been proven to land
where those readers look. Two instruments briefly is safe; a blinded status
page is not. The compiler found the callers a text search missed — the span
middleware records the RED metrics — which is exactly why they stay until the
export path is measured.
Also carries luxfi/metric v1.9.1 transitively pinned by zip — the registry
that records by default.
failRedirect is the one place a failed OAuth return lands, and it logged
nothing, so every failure was indistinguishable from a request that never
arrived. It now logs the opaque public reason together with a precise
internal cause; the browser still gets exactly one reason, so no oracle is
offered to a caller probing states.
verify() wraps errBadState with that cause. errors.Is still matches, so
control flow is unchanged. The distinction it makes visible is between a
state signed by a different key -- what happens when the signing key is
absent and each boot invents its own, breaking every flow that spans a
restart -- and one that simply expired.
Folds the two ad-hoc warns into the funnel, carrying org in the cause so
no detail is lost.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The module-side registrations (commerce v1.49.65+) serve nothing in the
fleet by themselves — commerce's api.Route() bundle is never compiled here,
and the host hands an app only the prefixes its manifest row names. So the
four rails get what every self-service billing address before them got:
co-resident registrations on the pinned-subject IAM chain, and their
prefixes named deeper than the bare /v1/billing stem nobody claims.
- GET /v1/billing/wire — the serving brand's receiving bank
details, caller's billing key in the payment reference (attribution is
why it sits on the pinned chain; an unpinned wire is unattributable).
- GET /v1/billing/crypto/options — chains+tokens from the live MPC
processor; the pay SPA's asset picker renders exactly this.
- POST /v1/billing/crypto/deposit — per-payer custody address via the
signer fleet's /keygen; payer is the PINNED subject, never a body value;
open intents are reused so a refresh cannot spray keygens.
- GET /v1/billing/crypto/deposit/:id — caller-scoped intent state.
Nothing mints on any of these: wire settles via the admin wire/credit verb
on bank receipt, crypto via the chain watcher on real confirmations.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
geometry is a closed sum and family derives from the geometry's own type, so
"half-space parameters attached to a transformer" does not compile and this
package carries no check for it. family leads the content address, domain
separated, because every term after it is one family's arithmetic — two
families' numerically identical masses can no longer be named as one value.
The detector seam is the six methods the learner already calls. The half-space
counters are the first implementation and behave as before. legacy() is the one
remaining braid, at the disk boundary, and refuses a family the resume column
cannot carry rather than recording a shape for a model that is not one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.49.65's crypto rail called an MPC API the deployed fleet never served
(vaults routes, all 404). v1.49.66's GenerateAddress speaks the live
luxfi/mpc contract: POST /keygen {"org_id"} → all-chain addresses in one
threshold keygen. Env to arm it: MPC_ENDPOINT + MPC_API_KEY (KMS-synced
into commerce-secrets).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestTheThreeAddressesAreToldApartNotShared has been red on main since
"identity is the composer's" moved cloud.Bridge out of every subsystem and into
cloud.App, which installs it at the root ahead of every typed route. surfaceApp
was not updated, so it mounted o11y onto a bare zip.App with no Bridge — and a
typed op with no Bridge has no validated org on its context, so it answers 403
to everything, including the RED read this test asks for its OWN datastore
refusal.
That reads like a live outage of the o11y surface and is not one: the real
process composes through cloud.App. A harness that stops composing the way the
process does does not measure the process, and this one was reporting a
middleware it never installed as a broken handler.
scopeApp already says exactly this, one file over. surfaceApp is where it was
missed. Package goes green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
event.trace is the summary the read plane resolves a trace id against, and it
is fed by the SPAN WRITER — one partial row per trace per batch, folded by an
AggregatingMergeTree — not by a materialized view. hanzoai/o11y's own driver
does exactly this (pkg/datastoretraces/writer.go, traceSQLTmpl). This sink is
the production span writer and implemented event.span without it, so the
summary stayed empty while 710k spans piled up beside it.
An empty summary is not a missing feature, it is a silent outage.
TraceTimeRangeFinder resolves every trace_id predicate against this table
first, and a lookup that returns no row makes the querier SHORT-CIRCUIT THE
TRACE QUERY TO EMPTY (pkg/querier/builder_query.go, narrowWindowByTraceID) —
a missing summary row means "no spans exist", which is the one thing it did
not mean. The detail read, the waterfall, the flamegraph and every funnel
answered empty over a complete span table.
So: traceRowsOf folds the rows the span writer ALREADY built — one row-building
path, not two, with the four column indices pinned by a test so a reorder goes
red instead of corrupting the summary. end is max(start+duration), not
max(start): the longest span need not start last. The summary write fails SOFT
— a derived rollup must never be able to take down the fact ingest it is
derived from.
And GET /v1/o11y/traces, the org's trace LIST: the one address in the family
the module leaves open. It declares the detail (/traces/{traceId}), the field
catalog and three per-trace projections, every one of which needs an id this
read is where you get — the detail was reachable only by someone who already
knew the answer. Claiming the collection and nothing under it keeps that a
composition rather than a second declaration at an address that has an owner.
Typed, declared at the ROOT at its full path so zipdoc can resolve it, org
pinned as a bound parameter on the table's leading sort key, no admin
widening: a trace list is a tenant's records, not a rollup over them. It
aggregates on read because a merge is asynchronous and never a promise —
skipping that reports a BATCH as a trace.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Brings GET /v1/billing/wire (brand receiving-bank details, payer-attributed
reference), GET /v1/billing/crypto/options and POST /v1/billing/crypto/deposit
(per-payer MPC custody addresses) onto the served billing surface, plus the
$5 pay-as-you-go floor (topupBounds min 100→500 cents). v1.49.65 is the merge
of the forge lineage (zip v1.24.2, these rails) with the GitHub lineage
(dual-license, billing payment/invoice cores) — both remotes now converge.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The gate refused ANY apps/* import from cmd/cloud as a stand-in for "links no
subsystem graph". The stand-in went wrong the moment an app's EDGE had to run in
this process: cmd/cloud/sites.go serves <slug>.hanzo.app, and it must, because
the image runs this binary on the public port — a published site mounted anywhere
else is served nowhere, which is the outage that put it there.
MEASURED on origin/main: cmd/cloud links 392 packages, inside the ~395 the gate's
own prose names, and apps/sites pulls 2 cloud packages. Nothing grew. The gate has
been red for a day over a graph that never changed — and because every later car
declares `needs:` on it, that is the entire release train stopped by its own
approximation. Live is v1.801.431 while tags reach v1.801.436.
So the exception is NAMED with its reason, the discipline go-unit's -skip list
already follows, and the property it approximates is measured directly beside it.
Two checks, because they are two questions: the name catches a subsystem leaking
in, the count catches an ALLOWED leaf that quietly grew a graph. A widened pattern
would have answered only the first and silently given up the second.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud.App(name, cfg, deps, tools) now builds every program — the host and all
125 plugin programs — and installs the canonical chain through identity in one
place. Identify states the order invariant once: the boundary that mints the
validated org runs before the enrichment that parks it, and neither exists
without the other. Middleware order at the host is proven byte-identical
before and after (18 entries), so a refused request is still audited.
Every subsystem self-install of the enrichment is deleted — 74 sites across 67
packages. A subsystem asserting identity for itself repeats a claim it cannot
check; two of those copies sat on nodes owning no routes, which zip refuses to
compose, and that is the outage that took /v1/o11y and /v1/integrations down.
The childless spelling is gone everywhere: a gate passed TO Group is installed
at the root bounded by its prefix, and the two-step form that slipped past
that (a bare Group then Use on it) is collapsed at its last holdouts —
referrals' three gates and the audit fixtures.
Tests stop rebuilding the composer by hand: each package that mounts on a
bare app owns one compose helper, so anonymous cases still refuse and
principal-carrying cases reach the handler exactly as production does.
Also fixed, found by the constructor's own test: the markdown negotiation
replaced the Vary header CORS had written, so every CORS response on every
program advertised Vary: Accept — a shared cache could hand one origin's body
to a different origin. The later writer is additive now.
Census: zero non-test enrichment installs under apps/, zero childless chains.
Build and vet clean over the whole tree in the shipping mode.
The workspaces + members tables were 443 lines of hand-written database/sql:
positional placeholders, a hand-rolled scanWorkspace whose Scan matched wsCols by
POSITION, and a prefixed() that rebuilt the projection as a string. They now go
through hanzoai/orm's relational plane — orm.Select/orm.Typed for the reads, the
dbx builder for the writes.
Not orm.Model. That is the document plane over the JSON _entities table, and
these tables are typed and indexed with composite uniqueness ((owner_org, slug),
(workspace_id, user_id)) that a json_extract store cannot hold; orm's own guidance
names this file as a dbx target for exactly that reason.
The handle still comes from cek and orm is ADAPTED onto it. orm's SQLiteDBConfig
carries no master key, so orm.OpenSQLite would have written this store — every
workspace, membership and display name — as a plaintext `SQLite format 3` file.
That is the regression apps/iam documents removing, and it is not reintroduced
here: encryption at rest is a property of who opens the file.
The positional Scan is gone with it. wsCols and the struct now agree by `db` tag,
so a column added without a field is a mapping that does not resolve rather than a
silent shift of every value one position left.
THREE STATEMENTS STAY VERBATIM, each because the builder cannot say the thing that
makes it correct, and each now running through orm's own NewQuery rather than a
database/sql handle — so the file has ONE data path:
migrate — CreateUniqueIndex emits neither IF NOT EXISTS nor a WHERE, so it can
express neither the idempotence nor the PARTIAL (owner <> '') index; dbx.Sync
writes no indexes at all.
the EnsureWorkspace create — dbx's Upsert only ever emits DO UPDATE SET, and
there is no seam for a conflict target carrying the partial index's WHERE. As a
DO UPDATE the meaning INVERTS: the racing loser would overwrite the winner
instead of yielding to it, and the converge-to-one-workspace property is gone.
AddMember — Upsert fans EVERY inserted column into the SET list, so a re-invite
would overwrite joined_at and is_bot. joined_at is the order GuestRank ranks by
and the guest cap admits by, so that is not a cosmetic overwrite: it silently
reshuffles which guests keep access.
TestAddMemberPreservesJoinOrderAndBotFlag pins that last one at the four columns
the statement treats differently. Swapping in db.Upsert turns it red on three of
them (joined_at, is_bot, display_name) — checked, not assumed.
The test fixtures moved to the builder too, so no `?` placeholder survives in the
package.
apps/team + apps/analytics + apps/meet green, CGO_ENABLED=0 -tags sqlite_fts5,
against the ten pre-existing zip-compose failures already red on this commit's
parent.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zen composing was never the same as zen running, and only the first had been
fixed. It is Coresident: it answers no path and works by wrapping ai's "/v1",
routing zen-SKU requests and Next()ing the rest. A middleware cannot be a
separate process, so the light host deliberately skips Coresident apps
(cmd/cloud mount returns early) and plugin/zen exists only for the gen-app-cmds
bijection. plugin/ai linked only ai. The Claim was therefore mounted in NO binary
the fleet runs — which cmd/cloud had already recorded as "zen's child never saw a
request" — so every zen SKU served through ai's catch-all with zen's gate and its
meter never consulted.
zen mounts FIRST in plugin/ai, ahead of the greedy All("/v1/*") ai registers:
zip scopes middleware to the entries that follow it, so a Claim installed after
that catch-all would sit behind the very route it exists to gate.
MEASURED, on the real zen.Mount through the real MountAll:
zen5 -> 402, reached host catch-all = false (claimed and gated)
gpt-4o -> 200, reached host catch-all = true (falls through untouched)
Both halves matter. The first is the gate doing its job; the second is the proof
zen did not become a wall in front of the whole model API.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v1.4.4 billed every streamed call from an accumulator no chunk had touched.
c.SendStreamWriter hands its closure to fasthttp, which spawns it on its own
goroutine (stream.go:43) and returns immediately, so the recordCtx that followed
read usage.tally() before a byte arrived: completion -> 0, cached -> 0 (the whole
prompt re-billed as fresh) and ResponseID -> "", which is the join key /v1/feedback
and the learning ledger need to tie streamed traffic to its own feedback.
Reproduced here before bumping, on the exact shape: `go test -race` reports the
write from the writer goroutine against the handler's read, and the value visible
at metering time is 0 where the streamed total is 50. It is not merely racy — the
closure genuinely has not run, so a mutex would silence -race and change no number.
Both same-dialect paths were affected (proxy.go stream, sse.go anthropic-native);
streamTranslated, buffered and ultra meter synchronously and were always correct.
The drift runs BOTH ways, which is why this is not just our lost revenue:
answer-heavy shapes under-collect 62-85%, and cache-heavy short answers OVERCHARGE
a live ledger by up to 4x.
v1.4.9 was cut for this — the fix existed on zen's main and sat in no released
tag, so every consumer was still on the defect. MEASURED at the boundary:
v1.4.8 fails `go test -race` at proxy.go:509; v1.4.9 exits 0 with no races.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The rebase produced a program that does not compile — `undefined: sv1`, so the whole
apps/commerce test binary failed to build, a package that was GREEN on main.
Both sides fixed the same zip v1.24 composition rule and picked different halves of
it. This branch dropped the /v1 group and moved the middleware to Use; main kept the
group and moved the ROUTE onto it, so the chain guards something. The auto-merge took
this branch's deletion of `sv1 := app.Group("/v1")` together with main's
`sv1.Get("/store/current", ...)` that still names it.
Main's shape is the better statement of what the test is for: the case is an envelope
that must apply to commerce's own route and must NOT clobber a sibling's, and putting
commerce's route back on the group is what production does (Mount's storeV1 group).
So this file returns to main's version byte for byte — `git diff a4a1f998 --
apps/commerce/errorscope_test.go` is empty — and the branch keeps no opinion about it.
No commerce money logic is touched, on this branch or in this commit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The zipdoc -check gate at the top of `make test` was red on three packages, so the
suite aborted before a single test ran. The lift is regenerated from source; all 99
directories carrying the directive now answer -check clean.
What the regeneration is, measured rather than assumed:
marketing prose only. Operation set identical, field-doc set identical.
account prose, plus doc comments for onboardResp.accessKey/accessSecret.
projects prose, plus a doc comment for projectsProject.key, plus the lift for
POST /projects/resolve-key (an internal cloud.Plane op whose file
landed without a regeneration).
The extra entries are DESCRIPTIONS for fields that already exist: AccessKey and
AccessSecret are account.go:604-605 and Key is projects.go:159, all three already
carrying the comments lifted here. zipdoc supplies prose; a schema property comes
from struct reflection at describe time. So this adds documentation to the surface
and no field to it.
THE PUBLISHED ARTIFACTS ARE DELIBERATELY NOT INCLUDED. Regenerating
plugin/{account,marketing,projects}/openapi.json from this same source does NOT
come out prose-only, and it is not this commit's business to land it quietly:
- plugin/account: onboardResp gains accessKey + accessSecret as published
properties, and THREE operationIds are renamed — post_v1_orgs -> v1.post_orgs,
post_v1_keys -> v1.post_keys, delete_v1_keys -> v1.delete_keys.
- plugin/projects: projectsProject gains a published `key` property.
The rename is zip's scheme change, not ours, and the fleet is mid-migration: the
committed account subset still carries the old form while marketing and projects
already carry the new one. Every SDK is generated from these files, so completing
that migration renames methods for callers and is one deliberate fleet-wide act
with an owner behind it, not a side effect of unblocking a test gate. The same
change is what TestTargetOpsProjectEverywhere has been red about (it asserts
post_v1_agents_targets and zip now emits v1.agents.post_targets).
Verified identical on a pristine a4a1f998 worktree, so none of the drift above is
this branch's: it is the published document catching up to source that already
landed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
main answered the red ratchet by RECORDING apps/risk/policy_wire.go in
allowedRequestUses (f345ac70). This branch answered it by removing the second call
site instead — caller() moved beside ops.gate, which already reads that same
X-User-Id header, so the package reaches for the raw request in one file and the
pin stays one entry.
Both landed, so the rebase left both: the entry describes a call site that is no
longer there. The gate catches that in the direction people forget — it walks
allowedRequestUses and fails on any file that no longer calls cloud.Request, so a
pin cannot outlive the code it justifies. Removing it is what makes the ratchet
green, not an exemption.
Net effect against main: the escape hatch does not grow. apps/risk/typed.go states
both reasons on the one entry it already had.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/networks answered 403 "X-Org-Id required" to a validated caller. Bridge was
installed on the group, twice — ng.Use and mg.Use — and middleware on a group wraps
what is composed beneath it. The two collection ROOTS are declared on the App with
their whole path on purpose: joining "/v1/networks" with an empty leaf yields
"/v1/networks/", a different address from the one they have always served. So the
group's Bridge covered /v1/networks/routers and /:id and missed /v1/networks itself,
the op read no parked org, and the tenant gate refused a request that was fine.
One Use replaces both. On a scope it is bounded by the prefixes the manifest already
declares for zt (/v1/networks and /v1/mesh/services), which is every route here and
nothing else; on a bare app — what this package's tests mount on — it is app-wide,
which is what the tests want. The groups stay as what they are, path prefixes.
TestBridgeIsInstalledOnEveryPrefix already stated this and had been red: it asserts a
validated caller is SERVED on all four routes and an anonymous one still refused.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two more of the shape the previous commit fixed in nineteen places, found by the
gate rather than by reading: apps/deploy panicked the fleet's own surface-check
(`the group "/v1/deploy" declares middleware at deploy.go:316 and no routes
anywhere beneath it`), and apps/agents failed 54 of its own tests.
deploy is the clearer one. deploy.go built app.Group(dashPrefix) to hold
cloud.Bridge + bounce, and dashboard.go builds ANOTHER app.Group(dashPrefix) to
hold the routes. Group returns a new definition per call, so those are two nodes
at one path: the middleware sat on the empty one and NEITHER the bridge NOR the
sign-in bounce ever ran for a single route of that surface.
agents is the same mistake with a subtler tell, because its group is not empty —
/metrics, /activity and the :ref leaves ARE beneath it, so nothing panicked. The
rest of the surface is not: the collection root, mountSessions and mountTargets
all register on the Router by absolute path. So Bridge parked no org for
/v1/agents/targets or /v1/agents/sessions, and every op under them answered 403
"X-Org-Id required" to a request that carried one.
It only ever showed up in tests, and that is the part worth keeping in mind:
Serve installs a Bridge app-wide, so serving was unaffected and only a bare Mount
could see the hole. TestHTTPTargetRejectsOversizeGPUList is the example — it
asserts 400 for an oversize GPU list, got the 403 first, and so had never once
exercised the bound it is named after. It passes now for its stated reason.
No bound widens. Every prefix either subsystem declares is under its own group's
path (manifest/apps.go), so a scope confines this to exactly what the group named
and the plugin binaries serve nothing else.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestRequestEscapeHatchIsPinned was red on main: apps/risk/policy_wire.go called
cloud.Request from a second file in a package that already had one, and the pin knew
nothing about it.
The fact caller() needs is the validated user id (X-User-Id) a policy version is
recorded against, and nothing parks that in the context — principal parks the org and
validated-ness, so principal.OrgFrom and principal.ValidatedFrom cannot answer it, and
an attribution the caller could state in a body is not an attribution. So the request
is genuinely required. What was not required is a second call site for it: ops.gate,
in typed.go, already reads that exact header for the meter's actor.
caller moves there, beside gate. Two functions, one seam, one pin — which is the shape
the map's own entries describe for wallets, tools, deploy, usage and o11y, and the
reason it is a per-FILE list. The pin's existing apps/risk/typed.go entry now states
both reasons; no entry is added, and the escape hatch does not grow to make a test
green. The function itself moves byte for byte.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nineteen sites carried the same defect scope.Use had, at the other door: a group
created with middleware, its routes registered on the App with their whole paths.
Middleware on a group wraps what is composed BENEATH that group, so every one of
those groups was empty — the middleware never ran, and since zip v1.20 the program
does not compose at all, which is a panic at mount for the plugin and an aborted
test binary for the package.
TWO GATES HAD SILENTLY STOPPED RUNNING, and that is the part that matters more than
the panic. apps/referrals put requireOrgOnWrite on a /v1/referrals group and
requireAdmin on each of the two /v1/admin/referrals boards, and registered all three
leaves on the App: the write gate never ran on POST /v1/referrals/claim and the admin
gate never ran on EITHER board. apps/team installed Bridge on one /v1/team group
while every file builds its own group from the same constant — the same routing
subtree, a different definition — so the bots, files, blob and cookie planes ran with
no validated org and answered 403 to valid requests (eight tests, all of which were
unreachable behind the package's own panic).
The identity and error-shape middleware moves to Use, which a scope bounds to the
prefixes the manifest declares for the subsystem and which is app-wide under a bare
Mount — admin, admission, auditlog, catalog, guide, integrations (x2), label,
marketing, prefs, projects, team, templates. referrals keeps three different bounds
and states each as the predicate a group prefix was standing in for (under), the same
shape apps/commerce already uses for its error envelope. o11y installs on its own
App, which its routes are already beneath. In team the install moves ABOVE the group
it used to sit on: fiber runs middleware in registration order, so one added after a
subtree never wraps it.
Five test harnesses mirrored the invalid shape on a bare app and move the same way.
No address changes and no gate widens: every bound here is the one the group prefix
named.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The door description still promised "NO CREDENTIAL IS ALSO ADMITTED ... filed
under the reserved $public tenant" — and it goes into the OpenAPI document
customers read. The code has refused since the key work landed: 401
ingest_key_required with nothing presented, 403 ingest_key_unknown for a
credential that names no project. A contract that promises the opposite of the
code is worse than no contract, because a client writes against it.
The projection survives for the reduced principal it was narrowed for — a
workspace token writing into its own org.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The escape-hatch ratchet has been failing on origin/main, naming
apps/risk/policy_wire.go and then apps/commerce/invoices.go. Both are
legitimate; neither was recorded, so the guard could not tell them from a new
one and main stayed red.
A red main is not a cosmetic problem here. Three of today's near-misses got as
far as they did because "the tests pass" had stopped meaning anything, and the
next reader has to re-derive whether each failure is theirs. So the fix is to
answer the pin, not to loosen it.
apps/risk/policy_wire.go caller() reads c.User() for an attributable policy
record. No ctx helper answers it — OrgFrom gives
the tenant, not WHO changed the appetite bounds.
Off the HTTP path it returns empty and plane.enact
refuses, so a change is never recorded anonymously.
apps/commerce/invoices.go eventsFrom/kmsFrom lift two request-scoped side
channels out of c.Locals(), which no ctx helper
exposes. Both optional by design: a missing
analytics collector must never fail a money move,
and a missing KMS client is the dev/test posture.
The root package is green again.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
event.fact is not a rename of event.event, it is a merge — five signals in one
table. Moving the source without naming the signal counted errors and spans as
device observations, and this is the one cross-tenant reader, so a phantom
identity inflates both k-anonymity floors: 425 identities unfiltered against
423 real ones on the live plane.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
config.go read CLOUD_CORS_ORIGINS and fell back to GATEWAY_CORS_ORIGINS, the
pre-rename gateway spelling, "shared with the gateway so both trust boundaries
agree on one list". Measured across every chart in universe, GATEWAY_CORS_ORIGINS
is set ZERO times — by cloud's chart, by the gateway's, by anyone — so the shared
name agreed on nothing. cloud's own values file sets CLOUD_CORS_ORIGINS.
Two spellings for one security-relevant allowlist is a way to half-configure it:
set the dead name and the browser silently gets no ACAO. No test pinned the
fallback, and the direction of failure on removal is restrictive — an origin that
is not listed is refused, never admitted.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
docs/consolidation.md was the execution queue for merging the standalone Go
services into one binary. Nothing references it, and every mechanism its Method
section instructs the next wave to use has since been deleted:
apps/apps.go no such file
apps.Wire no such symbol
cloud.Register no such symbol (only unrelated seam registrars remain)
"order < 150" there are no orders; manifest/apps.go is the table
clients/<svc>/Mount an app is plugin/<name>/main.go, one binary per app
Its inventory has drifted the same way — "37 native apps/*" against today's 120
manifest rows, three of the apps it names (paassvc, console, prompt) no longer
exist, "Wave 1 — THIS build" shipped long ago, and its CGO_ENABLED=0 "production
parity" rule is the opposite of what the Dockerfile does for plugins
(CGO_ENABLED=1 + libsqlite3 + sqlite_fts5 + sqlite_math_functions). A doc that
tells the next engineer to write code against a registration mechanism that does
not compile is worse than no doc.
The durable half — which tiers stay out of this binary and why — is not lost: the
edge/data-plane split is in README.md, and the isolation reason for identity is
stated where it binds, in apps/iam/iam.go. If that table is wanted as a fleet-wide
statement it belongs in LLM.md, which is the one doc this repo maintains, not in a
migration queue.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
scripts/smoke-runtime.sh said it was "intended for CI smoke gating", and CI
never called it — nothing in this repo, in .hanzo/workflows, in the Makefile or
in universe named it. The role it claimed is already served twice over, by
things that ARE invoked: `make smoke` runs plugin/smoke, the release gate's
functional prober, and `make e2e` runs e2e/run.sh, which builds the host and its
plugins and boots them for the Playwright suite.
It was also the weakest of the three. Five hardcoded probes against
plugin/smoke's one read per subsystem, with none of the contract that makes that
prober worth running — no 402-on-a-read rule, no 5xx rule, no authed/tolerant
distinction. And it reached for `curl -fsS`, the flag that turns a failing probe
into a silent one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
migration/ + plugin/migrate-pg-to-sqlite ported a `hanzo_cloud` PostgreSQL
database into per-org SQLite. That database belonged to the deployed cloud-api
PYTHON service, which this repo replaced; the runbook the tool cites
(CLAUDE_PG_TO_SQLITE_MIGRATION.md) no longer exists, no Makefile target, CI job,
compose file, chart or sibling repo invokes it, and it is absent from
manifest/apps.go so the image never built it. Its only two mentions were the
generator's exemption list and a comment naming it as an example. A one-shot
import for a service nobody runs is not part of v1.
The bijection test derives tool-ness by PARSING for a cloud.Listen call rather
than matching a name list, so it needed nothing; only gen-app-cmds' notApps map
did.
LLM.md's release section described an architecture that was deleted with the
fused binary: a `hanzo.yml binaries:` lane publishing ./cmd/cloud to S3, every
app resolving to "the multi-call binary with a different --enable", pinned by
TestRemote_DedicatedBeatsMultiCall. hanzo.yml now states NO binaries lane and
gives the reason; remote() looks up exactly name/os/arch with no multi-call
fallback; and the cited test was replaced by TestRemote_NoMultiCallFallback,
which asserts the opposite of what the doc claimed. The line numbers it quoted
had come to land on the comment saying the lane was removed. Corrected to what
the code does, and CLOUD_PLUGINS named for what it is: a supported input with no
producer.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The catalog shipped 66 entries and 45 of them pointed at repositories that do
not exist. github.com/hanzo-templates/<slug> 404s for all of them, authenticated
or not, and their demo URLs 404 too — a gallery where two thirds of 'fork this'
leads nowhere.
The work was never missing, only misfiled. Those templates live in hanzo-apps
under a template- prefix, which is the org convention for a static site; the
catalog was written against a hanzo-templates layout that only 21 of them ever
moved to. 66 template-* repos in hanzo-apps, 66 entries here — the sets match.
21 kept real in hanzo-templates (expo-*, flutter*, swiftui*, desktop-*)
42 fixed repointed to hanzo-apps/template-<slug>
3 dropped innovise, kalli, unfixed — no repo in any org, and a catalog entry
that cannot be forked is worse than an absent one
All 63 remaining sources verified 200 against the GitHub API before this landed,
not assumed from the naming rule.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The plane's product-event table was renamed event.event -> event.fact. Writers
moved; these four readers did not, and nothing errored because the old table
still exists — it just stopped receiving rows on 2026-08-02. A frozen table
answers every query, so the failure was stale numbers, not an outage: the GTM
funnel was still reporting $13.37 of revenue from July.
event.fact holds every signal in one table (act, clip, error, log, span), so
the rename alone is not the fix. Each read pins signal='act', the predicate
apps/analytics already carries in scope(). Without it the funnel counts 85
visitors where 83 acted, and risk folds a person's error rows into things
they did.
risk binds the signal as a literal rather than a positional arg: all four
rollups share one 5-arg storeExec call, and the SQL already spells 'person'
and 'session' the same way.
Verified against the live warehouse — every query returns growing data where
it returned frozen, and apps/risk tests pass.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/o11y went native and now declares all 367 of its routes by name.
There is no wildcard left for a host route to shadow, so the three
addresses cloud also declared stopped being a silent precedence artefact
and became a refusal to compose — the plugin crash-looped and every
/v1/o11y read 503'd.
o11y.Claimed(...) made it boot, but a claim SUPPRESSES the module's
declaration. It did not resolve the collision, it just picked a winner and
wrote the choice down: each of the three quietly cost the fleet the
module's real read at that address, and one of them was cloud's forward
into a runtime path that no longer exists. What the addresses needed was
to be told apart.
GET /v1/o11y/logs — DELETED. Not a stub (it was a real two-view read
over event.log/event.span) but it had NO caller: the console reaches
logs through the query engine, and nothing in cloud calls it either.
An address nobody calls is not a contract. The module's real
log-record read answers there now.
GET /v1/o11y/metrics — MOVED to /v1/o11y/product/metrics. The module's
is the metric-NAME CATALOG; ours is one product's RED window keyed by
?product=. Two questions, so two names — ours says which one it
answers, and the module keeps the bare name. Live in the console
(App Platform drawer + canvas sparklines); patched there on the same
branch.
POST /v1/o11y/query_range — DELETED, along with POST /v1/o11y/query and
the whole of query.go. Both pinned /api/v3/<resource> to reach the v3
engine. The runtime registers every route at its full public path and
dropped prefix-stripping, so it serves no /api/* route at all, and
queryRangeV3 has no caller left — the forward reached the terminal /*
catch-all, not an engine. A route that forwards to an address nothing
serves is dead. The module's v5 querier answers there now.
Cloud therefore claims nothing: o11y.Mount(a) takes no options and the
boot log reads claimed:0. A host that has to name the addresses it takes
is a host that took addresses it did not own.
STILL OWED, and it is a console change: the trace/log explorers send a v3
composite that v5 refuses with unknown field "queryType". They were
already broken before this (the v3 pin reached the catch-all), so nothing
regressed — an opaque non-answer became an honest 400. The v5 migration
is recorded in apps/o11y/LLM.md.
The route table is unchanged in size: 389 method+path pairs before and
after. Only two lines move — /v1/o11y/product/metrics arrives and the dead
POST /v1/o11y/query leaves — because the module reclaimed exactly the
addresses cloud vacated.
openapi.yaml is the whole-fleet weave and cannot be regenerated per app,
so it also catches up on two other lanes' already-committed subsets
(billing +5 paths, payments +2) that it was stale against.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
IAM's error envelope rode on HTTP 200, so an SDK checking the transport before
the body read a refused signup as a successful one. The envelope is unchanged;
the status now agrees with it.
cloud's own IAM client is unaffected by construction: iamClient.do parses the
envelope whatever the status and decides on status != "ok", so it reached the
same verdict before and after. The bump is what makes the two agree at the wire
as well as in the body.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The unclassified branch rendered err.Error() verbatim as the 500 body, so a
customer's first fault told them about ZapDB migrations, CLOUD_KMS_MASTER_KEY_REF,
http://iam.hanzo.svc and features that are "not yet implemented". That text is
written for an operator; putting it on the wire published our internals to
whoever tripped it.
The rule is provenance, not status. An *HTTPError or a *fiber.Error was
constructed by a handler that chose a status AND a sentence — a decision the seam
does not second-guess, which is what keeps "Billing temporarily unavailable"
readable. An error that chose neither now renders a stable sentence naming the
request id instead.
The detail is not lost, it is relocated: ErrorHandler logs every 5xx whole, with
method, path and the X-Request-Id the response carries, so the log line and the
response a caller holds identify each other. The wrapped chain reaches the
operator, which is more than the client ever saw, since the client only ever got
the outermost sentence.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Federated sign-up files a brand-new user under the sign-up application's own
organization (iam internal/oidc/federation.go: `org := app.Organization`), so
the first request a new customer ever makes already carries an X-Org-Id. onboard
read that as "already has an org": the first org they asked for took the
ADDITIONAL branch, which creates an org and leaves the founder outside it, and
`personal: true` answered 409 "you already have an organization" — true of the
landing org, and useless to someone thirty seconds past signing up.
The orgs a sign-up can land in are exactly the ones this package already refuses
to hand to a customer (onboarding.go's reservedOrgs). One list, one fact, asked
twice: an org no customer may CREATE is one no customer can be said to OWN.
Naming the set rather than a single brand constant keeps a white-labelled
deployment correct, where the landing org is that brand's own.
Standing beats the landing. A SuperAdmin IS a member of the reserved `admin`
org, so treating that as a landing and moving them out would strip the privilege
it defines; an org admin therefore always counts as owning their org, read from
the authoritative IAM row rather than a header, since the answer is the one that
MOVES a user. A caller already in a real tenant is spared the read entirely, so
an invited member creating a second org is never yanked out of the team that
invited them.
The provisioning response now carries the credential it minted. IAM stores the
argon2id digest and blanks the plaintext, so the secret is readable exactly once
— in the answer to the call that mints it. Dropping it left a customer holding
an account whose credential had been issued and could never be obtained; a
replay, which mints nothing, reveals nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Merging the three analytics branches leaves publicTenant named where it no
longer exists: the heatmap tests normalize under it, and five comments describe
a lane the key work deleted. The tests take a real org like the rest of the
file; the comments say what the door does now — the org is the reduced
principal's own, resolved from its credential.
apps/reference read event.event, which stopped receiving rows on 2026-08-02
while the plane moved to event.fact. Columns are identical, so the device
aggregate was grouping a frozen table.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The golden is DERIVED from plugin/*/openapi.json, and the merge of origin/main
into forge/main brought a subset the forge-side golden predates: 1969abfc gave
POST /v1/runner's description its release: true SUPERADMIN paragraph. Weaving
the committed subsets reproduces every other byte, so the whole delta is those
three lines of prose.
No route moves — 1695 paths before and after, none added, none removed, and the
floor ratchet is untouched at 179 products. TestFleetIsTheWeaveOfItsApps and
TestTheServedDocumentIsTheArtifact both read this file and both go green.
Only the weave step ran. `make describe` also regenerates the subsets from
code, and that step is broken on BOTH mains independently of this merge — forge
fails first in zipdoc on apps/o11y/annotation_queues.go (which origin's dc013156
fixes), and past that the authors app is refused for installing middleware at
/v1/admin/authors, outside the /v1/authors prefix it owns. Neither is this
merge's to decide, and neither touches the derivation performed here.
zip removed App.Shadow — the verb that let an op be declared in one scope and
answered in another. Cloud never referenced it, so nothing here changes shape.
o11y v1.5.58 carries the same removal plus one narrowing of under().
Proven route-neutral: the woven fleet document is byte-identical before and
after (1695 paths, 3676621 bytes, cmp clean), so no SDK repo sees a route move.
Suite: 161 packages ok, 143 failing tests, and the failing set is byte-identical
by NAME to main's — regression set empty.
`release: true` on POST /v1/runner publishes releaseImage — ghcr.io/hanzoai/cloud,
the binary every service in every org runs. The gate read
principal.IsSuperAdmin(c) ||
(imageInOrgRegistry(releaseImage, org) && principal.IsOrgAdmin(c))
and the second term is not a narrower platform predicate, it is a self-service one
wearing a registry map. `isAdmin` is org-scoped and an org's OWN admin sets it on a
member of THEIR org, so every `hanzo` admin could enrol any `hanzo` member into the
gate — and `hanzo` is the deployment's brand org in every deployment, and the one
orgRegistryNamespaces gives `hanzoai`. So the far side of the gate chose its own
callers, and what it chose them for was the tag the whole fleet rolls onto at the
next reconcile: iam, kms, gateway and every customer app, in every tenant.
Owning the registry namespace is the right bound for an ordinary PUSH, and it stays
there. A push lands ONE tenant's artifact in the namespace that tenant owns, so the
caller's own org is exactly what should bound it. A release lands OURS on everyone,
so no property of the caller's own org can be what admits it. The two lanes part
company at that one fact, and nowhere else on this endpoint.
SuperAdmin <=> `owner == "admin"` is the ONE platform predicate, so the gate is now
cloud.Super and nothing conjoined to it:
func mayRelease(c *zip.Ctx) error {
if !cloud.Super.Admits(cloud.AuthorityOf(c)) { ... }
}
Validated is part of that scope, which collapses the MACHINE arm into the same
expression rather than a second rule beside it: PLATFORM_BUILD_CALLBACK_TOKEN mints
no principal, so a leaked build token still enqueues an ordinary build and still
cannot cut a release — the property TestRunnerRelease_SharedTokenCannotRelease
already pinned, now held by the scope itself.
READING a release took the same org-scoped gate (mayReadReleases), mirrored by
comment. Both are now the one function, so "the 202 hands back an id its caller may
ask about" is true by construction. Nothing was widened: the published contract for
both GETs already read "SuperAdmin only — this is the platform's own publishing
record, not a tenant surface", and the code did not do that. The /v1/runner prose
said cutting a release was "IAM's decision alone", which was true of the credential
and silent on the scope; it now names SuperAdmin.
The production release path does not go through this gate at all — a merge to
cloud's own main dispatches launchRelease in-process (push.go, isReleasePush), so
what tightens here is only the hand-cut, and the CLI cannot even send the flag
(cli.BuildReq has no Release field).
TestReleaseSurfacesTakeOneAuthority drives the REAL handlers across all three doors
— cut, list, read-one — over the role axis, and asserts they agree principal for
principal. Against the parent the brand-org ADMIN case fails with the escalation
verbatim, on every door:
cutting a release: admitted=true, want false (HTTP 502 — resolve main: ...)
listing releases: admitted=true, want false (HTTP 200 — {"data":[]})
reading one release: admitted=true, want false (HTTP 404 — no such release ...)
The 502 is the tell: the seams are stubbed to 500, so a request only reaches the
pipeline by having been authorized. It holds the SuperAdmin row on all three doors
too, so merely disabling the path would not pass, and TestRunnerRelease_-
OwningOrgAdminRefused pins the cut on its own.
It replaces TestReadingAReleaseIsNotStricterThanCuttingOne, which read runner.go
and release.go as TEXT and asserted both mentioned the same predicate NAMES.
Spelling was all it could ever see: it was green while both surfaces contradicted
the published contract, and it would have stayed green had the two admitted
different callers under the same names.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every release since v1.801.427 has failed. The image build runs
'go generate -run zipdoc ./...' and it exits 1:
zipdoc: apps/o11y/annotation_queues.go:94:10: zip.Get: cannot resolve the path
prefix of the router this op registers on
That is why seven plugins — templates, integrations, admin, guide, marketing,
prefs, o11y — answer 503 'no instance running' in production while the fix for
them sat on main unable to ship.
The cause was under{}, introduced to declare o11y's typed ops at the ROOT so
their ids stay unqualified and their schemas keep the app's origin. The goal is
right; the mechanism was a composite literal, and zipdoc resolves a router's
prefix only from a .Group() call or a variable assigned from one. A custom
OpTarget is invisible to it, and zipdoc treats what it cannot resolve as an error
rather than assuming an empty prefix — correctly, since a wrong prefix files
prose under the wrong identity.
Same intent, statically visible: register on the app with the full path,
o11yPrefix+"/reviews". The router is the *App (root, no prefix) and the path is a
constant expression the type checker folds, so zipdoc reads both. Declaration
stays at the root, so the ids and schema names under{} protected are unchanged.
Addresses are byte-identical — the regenerated zipdoc_gen.go changes by pure
ADDITION: GET /v1/o11y/sessions is documented now, because zipdoc could not reach
it before. under.go is deleted; nothing else used it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A project mints a publishable key at create — the ONE thing that attributes its
site's events. The key resolves to (org, project), so it names the site as well
as the tenant, and a key no project holds resolves to nothing: delete the
project and recording stops, structurally.
Removes the two attribution paths that were not that:
- the keyless lane. A beacon carrying no credential was ACCEPTED into a
reserved `$public` tenant and answered {"accepted":1}. No org could read
that partition, so every such caller lost everything it sent behind a 200.
Three first-party properties shipped keyless without one failed build.
handle now refuses: 401 ingest_key_required with no credential, 403
ingest_key_unknown with one that names no project.
- the site-host carve. A POST to <slug>.hanzo.app routed into the anonymous
lane with a host-derived tenant — a second mechanism, and the one that could
not be checked, since that middleware runs before the identity boundary. A
site's beacon carries its project key to the ingest door instead.
The projection, its bounds and the opt-out gate survive for the reduced lane (a
team guest writes into the org that invited it, at reduced capability).
CLOUD_ANALYTICS_PUBLIC_CAPTURE gated the deleted lane and is gone with it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Group handed back the raw zip router, so `g := app.Group(p); g.Use(mw)` walked
past every gate in this file and hung middleware on a node whose subtree is
necessarily empty — typed ops register through ZipApp, on the ROOT, so a
subsystem's routes never land beneath the group. zip v1.24 refuses to compose
that, correctly, and crash-looped fifteen plugins on v1.801.425/.426.
This file's own header said that idiom "needs no policing: the group already
bounds it". It bounds the MIDDLEWARE and says nothing about where the ROUTES
went, which is the half that mattered.
Group now returns a child scope, so all three idioms are ONE install — at the
root, gated by path — and no node is left that can be empty. The child carries
`at`, its path prefix, because a group PREFIXES what is registered through it:
without that, `zip.Get(app.Group("/v1"), "/bots", h)` (bots, entitlements)
registers /bots — a route silently MOVED, which still composes, so no compose
check could have caught it. OpScope carries the same prefix for the same reason.
Also, three subsystems that were escaping their bounds silently, because a bare
Group used to skip the check entirely:
- team answers /collaborator; the manifest says so and the plugin did not.
- label answers /v1/risk/labels; same.
- zt installed one bridge at /v1/mesh, a level ABOVE the only path it
serves there. One app.Use, gated by scope to what zt declares.
And OwnsHealth on authz/domain/experiments/metrics, which serve their own
health — so serve.go's generic liveness route was a second declaration of it.
Verified three ways, because a weaker check let a broken build reach production
twice today. The binaries were exiting on `mkdir /var/lib/cloud/orgs: permission
denied` BEFORE composing, and I read that silence as a pass:
1. survival — a compose panic is fatal, so rc 124 under timeout is the only
honest signal; "zip new" is logged before composition and proves nothing.
17/17 affected plugins survive, each with private ports and a writable dir.
2. route projection — `describe` diffed before vs after across 34 plugins:
34 identical, 0 changed. deploy/label/referrals now project where they
previously panicked. This is what caught the moved-route defect above.
3. go test -run 'Scope|Mount|Prefix|Route' green.
o11y is NOT fixed here: its three routes (logs, metrics, query_range) are
duplicate declarations against upstream o11y@v1.5.55, and zip dedupes on the
resolved path, so moving the node cannot help. Separate change.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The anonymous lane projected the property bag down to the @hanzo/observe
annotation and dropped everything else, so a $click crossed it naming WHICH
element was clicked and never where on the page it sat. Element identity cannot
be drawn as a heat map, and logged-out traffic is the bulk of what one is made
of — so the position survived only on the signed-in lane, for a minority of
clicks.
The position is the second declared family. It does not get the annotation's
free ride: nothing lifts these into a column, so each really does add a key to
the attributes dictionary. What bounds it is that the set is CLOSED and spelled
by this server — five keys, never a caller's own vocabulary, values that are
numbers and a boolean, so the dictionary grows by five and stops.
boundedPosition filters rather than clamps, for the same reason
boundedAnnotation does: a clamped coordinate is a click somewhere the visitor
did not click, and a heat map is a picture of exactly that.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/event.js is the install path for a surface with no bundler:
<script defer src="https://api.hanzo.ai/v1/event.js" data-key="pk-…"></script>
One line, and the same line for a Hanzo property and for a customer's own
page. It autocaptures pageviews (initial and SPA) and uncaught errors onto
the canonical {batch:[…]} wire, carrying the key as a bearer on fetch and as
?ingest_key on the sendBeacon drain that cannot set a header. Identity uses
@hanzo/event's own storage keys and session TTL, so a page carrying both
clients resolves to one person.
WITHOUT A KEY IT SENDS NOTHING. The keyless beacons this replaces named
their site in a body field and carried no credential, so their events were
accepted 200 into $public — a reserved tenant the owning org cannot read.
The tenant comes from the publishable key, never from a body, so an unkeyed
page is silent rather than reporting success into a tenant nobody reads.
It is served beside the door because a tag that drifts from its wire is a
tag that 400s: asset and ingest ship in one binary and version together.
openapi grows the response half it had deferred until a route asked: Bytes
declares a body under the media type the handler sets, mirroring Binary on
the request side, so the document states JavaScript instead of claiming
JSON. Register copied the body half field by field and would have dropped
it; the copy now carries it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip refuses a program whose middleware has no routes beneath it, and the refusal
was right about three surfaces here. Nothing in this repo called Build() from a
test, so the refusal could only ever surface as a startup panic — or, for a
subsystem no test drove, as silence.
auditlog and catalog declared Bridge (and audit's noStore) on a group at their
own prefix while declaring the op on the App with its WHOLE path. The op is
therefore that group's SIBLING, not its child, so the group's subtree was empty
and the middleware could not run. Both packages' entire test suites were
panicking out of app.Test, which builds. Use is the verb that says the true
thing, and it needs no second copy of the prefix: cloud's scope bounds it to the
subsystem's declared subtrees, and a bare *zip.App treats root middleware as
live. The ops keep their exact paths — moving them onto the group with an empty
leaf would publish /v1/audit/ and /v1/catalog/, which neither API has served.
zen was worse and not the same defect. Its Claim gates ai's "/v1", it declares no
prefix of its own (Coresident), and plugin/zen fed manifest.PrefixesFor("zen") —
a ROUTING answer — into cloud.Plugin.Prefixes, which is the MIDDLEWARE grant. One
field was answering two questions, so dropping "/v1" from zen's row (right, for
routing: it duplicated ai's claim) silently revoked the gate, and MountAll refused
the mount outright. manifest.App.Gates states the second fact where the first
cannot, GrantFor reads it, and TestGrantMatchesPrefixesForRoutedApps keeps it from
becoming a second list.
crm was the same failure wearing an ordering convention. Its intake limiter used
to cover "everything registered after this line" — the public form plus the three
staff routes. Under lexical scoping it narrowed to the one route chained onto it
and the staff routes lost cover with no error anywhere. The covered routes are
composed BENEATH the limiter now, which states the coverage instead of implying
it. TestIntakeRateLimitScope was red before this and is green after.
compose_build_test.go is the gate that was missing: Build() through BOTH routers,
because MountAll hands a scope that can HIDE a seam a bare app refuses. It also
pins the rule itself, so a zip that stopped refusing the shape could not let the
seam rot back in green.
Deps: base v1.5.15 (the sqlite_math guard is a probe now, so `go build ./...`
needs no tags) and zip v1.24.2, plus iam v1.34.19 and o11y v1.5.57 restored —
394ba2e8 downgraded all three and left go.sum missing commerce v1.49.64, so
origin/main did not build at all.
guide's blueprint assertion matched a flat tool-naming zip no longer uses, so it
selected nothing; it matches the subject now, not the scheme.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
394ba2e8 said "base v1.5.15" and did not bump base. What it actually did was
roll THREE dependencies BACKWARD:
iam v1.34.19 -> v1.34.18
zip v1.24.2 -> v1.24.1
o11y v1.5.57 -> v1.5.56
base v1.5.11 unchanged <- the only change it advertised
Cause, so it is not repeated: `go get` was run against a working tree that was
BEHIND origin/main. go get pins the versions it is handed and leaves the rest at
whatever the tree already had, so every dependency a concurrent lane had already
advanced got written back to the older pin. The commit message described the
intent; the diff recorded the accident. Nothing verified the two agreed.
This restores all three and lands the bump that was missed:
zip v1.24.2, base v1.5.15, iam v1.34.19, o11y v1.5.57, commerce v1.49.64
Each is the newest tag on its remote, read with `git ls-remote` — GOPRIVATE
means proxy.golang.org cannot serve hanzoai/*, so the proxy is not an authority
for these and `@latest` silently answers from a stale public view.
base v1.5.15 is the one that pays for itself immediately: v1.5.11 carried a
guard whose symbol was defined nowhere, so `go build ./...` failed at default
CGO_ENABLED=1 and every build in this repo needed -tags sqlite_math_functions by
hand. Verified after this change: `go build ./...` with NO tags exits 0.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestMergedAndChainedAreIndistinguishable... pinned a fact so that a zip bump
which changed it would fail here. A zip bump changed it, so it failed — this is
the test working, not breaking.
The old fact: a duplicate registration and a middleware chain produce the SAME
observable route, so the generator must never read the handler count. From zip
v1.24 the duplicate half is not constructible at all — a second registration of
one pattern is REFUSED at composition time instead of merged, and the refusal
arrives as a panic out of Registry(), which took the whole package down before
any assertion could run.
The conclusion is unchanged and now rests on something stronger: a handler count
above one can ONLY be a chain, because a collision can no longer reach the
registry. Both halves are still pinned — the chain projects to exactly one
operation, and the duplicate is still refused — so if either moves, the
generator's assumption is forced back into review.
The chain's shape is now MEASURED rather than assumed. zip registers a GET as
GET plus an automatic HEAD companion, so one registration is two route entries;
the old test asserted one and would have failed on that alone. The GET is what
carries the chain and the HEAD is fiber's own, dropped from the projection.
openapi.yaml is regenerated because this branch adds six payment/invoice path
keys. Nothing is removed and no other app's operations move.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`tools/list` at api.hanzo.ai answered 570 tools and not one of them could take
money. Searching payment / charge / checkout / invoice returned nothing; the only
"pay" hit was an admin payout. An agent could incorporate a company, issue its
cap table, open a data room and sign a contract, and then had no way to be paid
for any of it.
The rails were never the problem. Square is live in the commerce module compiled
into this binary — official SDK, real charges, per-org KMS credentials, first in
the fiat priority list — and POST /v1/billing/topup/token has been charging cards
through it all along. What was missing was SHAPE: that route is a raw
func(*zip.Ctx) error, so it is a route and nothing else. No registry entry means
no OpenAPI operation, no MCP tool, no SDK method, no CLI command. The same was
true of the whole invoice lifecycle, of which this binary mounted only the list
and the PDF — an org could read invoices it had no way to create.
So commerce v1.49.64 lifted the money logic into cores that take values instead
of requests, and this adds seven typed ops over them:
takePayment POST /v1/payments
getPayment GET /v1/payments/:id
raiseInvoice POST /v1/billing/invoices
getInvoice GET /v1/billing/invoices/:id
issueInvoice POST /v1/billing/invoices/:id/issue
collectInvoice POST /v1/billing/invoices/:id/collect
voidInvoice POST /v1/billing/invoices/:id/void
They are TYPED, which is the whole point: each publishes a real JSON Schema with
per-field prose lifted from the handler's doc comment, down to a $defs for an
invoice line item. A payment tool with no parameters is useless to an agent — it
can be listed and never called — and this estate already has 469 untyped writes.
These add none.
THERE IS STILL ONE WAY TO TAKE A PAYMENT. Every op delegates to the same core
commerce's own HTTP handlers now delegate to, so the server-side amount bounds,
the idempotency derivation, the processor selection and the ledger credit are
shared rather than reimplemented. A second charge path would be a second set of
bounds to drift and a second idempotency key to disagree, which is a double
charge waiting for the right retry.
IDENTITY IS NEVER AN INPUT. The paying org is read from the validated principal
cloud.Bridge parks on the context; there is no org field and no subject field on
any of these ops, so a caller cannot steer money to an account it did not prove.
Mode is not an input either — sandbox versus live follows the org's credentials,
and the answer STATES which bucket it credited, so a sandbox receipt can never be
read as live money. Tests pin both: the schemas must not publish org/subject/test,
and every op must refuse a caller with no validated org rather than defaulting.
Also fixes errorscope_test's own empty group node — it declared middleware on a
/v1 group and registered every route on the app, so from zip v1.24 the group
guarded nothing and Registry() panicked before any assertion ran. The store route
moves onto the group, which is what production does and what makes the test a
valid program; its address and every expectation are unchanged.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
base v1.5.11 carried a guard referencing cgoBuildNeedsSQLiteMathFunctions, a
symbol defined NOWHERE — its only mechanism was the compile error it produced.
So the plainest command in Go failed in this repo and in every other importer of
base/core, reporting a missing symbol rather than the actual problem, and the
`-tags sqlite_math_functions` workaround had to be passed by hand on every
build. It was also a false negative in the config the driver's own docs call
production, because the tag was a PROXY for a capability rather than a
measurement of it.
v1.5.14 replaced the guard with a probe that asks the ENGINE — it runs the real
expression once against a throwaway in-memory DB and refuses the connection when
the answer is no, so the check reports a measured absence instead of a failure
to measure. v1.5.15 carries that plus zip v1.24.2.
Verified here: `TMPDIR=... go build ./...` with NO tags exits 0.
zip v1.24.2 and commerce v1.49.63 were already on origin/main when this landed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Fifteen plugins refused to compose and every route they own answered 503,
including /v1/event — the telemetry front door.
zip v1.24 refuses a group that declares middleware with no routes beneath
it, because that middleware would silently never run. Six apps wrote
app.Group("/v1/x").Use(mw) and then registered their routes on the app at
full paths, so the group node was always empty. f28bf6bd fixed exactly
this for scope.Use; these six bypassed it by reaching for Group directly.
app.Use is now the one way: scope.Use already gates a subsystem's
middleware to the prefixes it owns, by path, so the group node is not
needed and cannot go empty. One verb, one rule, no second mechanism.
Verified by running all sixteen affected plugin binaries: 0 compose
panics.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
tools/list proves an agent is TOLD a data room can be opened. It does not prove
one can open it, and the gap between those is a real failure mode with its own
shape: over MCP there is no URL, so zip passes the arguments object as the body
with a NIL path map, and an op whose address reaches it only from the path is
addressable over REST and NOWHERE else. Every existing test here drives HTTP,
where the path always binds, so all of them would stay green while the agent got
not-found — and three of these ops carry an id.
So the demo is driven the way the agent will drive it: open a room, grant a party
access naming the room by argument alone, read it back by id, and list it. The
negative half is what makes the positive half mean anything — the same read with
no id must NOT resolve a room, or 'found' proves only that something answered.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Tranche D listed dataroom 17. It is 10 typed and 7 refused now, and the note
records the two things a later reader needs: that the shared bundle-backed kit
(Scalar, SizedIn, BundleErr, Envelope) moved to apps/goja on its SECOND use
rather than becoming a second copy, and that ScalarList exists because a wrong
type on an access-control list is a SILENT failure — the room discards a
non-array and reports success, so a link meant for one investor would admit
everyone.
The tranche's remaining-count column is left alone: it already disagrees with
its own row (104 declared, 51 remaining after pricing/ml/automations/compliance
were struck without it), so reconciling it here would be a second lane's edit
wearing this one's commit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zip v1.24.2 stops composition renaming a DECLARED operation id by the prefix it
is included under. That mattered here more than anywhere: this binary is the host
that includes o11y under /v1/o11y, and doing so was renaming 217 of its 353
declared ops — every cached MCP tool name, operationId, CLI command and generated
SDK method moved as a side effect of one wiring line.
With commerce v1.49.63, iam v1.34.19 and o11y v1.5.57, every subsystem this
binary mounts is published on the same framework version. That agreement is the
point: Router is the type a decorator implements, so a host on one version and a
subsystem on another is a decorator that cannot be written.
Measured against a clean tree on this host: 97 failing packages before, 97 after —
zero new. macOS SQLCipher, unrelated and unchanged.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/dataroom/* has served fourteen routes and reached no agent. An untyped route
appends no op to zip's registry, and the MCP door renders tools FROM that
registry — so the fleet answered tools/list with 570 tools and not one of them
opened a data room. The hole was never the mount (dataroom mounts, and
/v1/dataroom/health has been answering in production); it was that every route
was an untyped relay.
Ten of them are typed ops now — every JSON route on the admin surface, which is
the whole demo surface: open a room, put documents in it, grant a party access,
and list what exists. Each still relays the bundle's own (status, body); what
changed is that it now carries an In and an Out, so the same declaration yields
the tool, the OpenAPI operation, the SDK method and the CLI command.
The package doc claimed NONE of these could be typed, on two premises the shared
kit answers: that a relayed answer is opaque (it is not — the bundle's shapers
are total and schema.go types them, which is what the models are), and that a
typed error path would overwrite the bundle's envelope (it does not — BundleErr
carries the bundle's status and BYTES). captable had already disproved both;
this makes that the second use rather than the second copy, so Scalar, SizedIn,
BundleErr and Envelope move to apps/goja, beside the bundle seam they serve.
ScalarList is the one piece captable did not need. A bundle substitutes an EMPTY
list for anything that is not an array, so an agent told allowList is a `string`
sends one, the room discards it, and the call SUCCEEDS having ignored the access
control — a link meant for one investor admitting everyone, reported as success.
Declaring the array is what puts that failure out of reach.
Four routes stay relays for reasons in the wire, each named at its registration:
the upload takes the file itself as the body, the two /file routes answer with a
byte stream, and the three public viewer routes have no principal to read.
Proven: the demo flow end to end over the typed routes; the reads byte-identical
to the bundle they replace; the cross-tenant link index still written, so a
granted link still opens for an anonymous visitor; org scoping; the room's own
refusal envelope intact; and the ten tools present with descriptions and schemas.
The regenerated captable subset is operationId-only (40 lines, 0 schema changes)
— pre-existing drift between the committed spelling and what zip v1.24.1 derives,
corrected by regenerating from source rather than by hand. The same drift had
left one captable test asserting a tool name nothing produces; it now asserts the
derived one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two independent ways a host WE run could serve a tenant's site. Both are the same
mistake: a set or a test that answers "is this ours" was allowed to shrink.
FIRST — the self-domain set was a default, not a floor.
self := list("CLOUD_SITES_SELF_DOMAINS")
if len(self) == 0 { self = defaultSelfDomains(apex, domain) }
An explicit list WON OUTRIGHT, so an operator adding a vanity domain silently
subtracted the derived ones. That set is the sole unconditional gate on the
site_hosts table (projects Store.bindHost asks sites.Ours) and the serve gate's
self-host exclusion, so dropping hanzo.ai makes every `<label>.hanzo.ai` a tenant's
to claim — a first-come row on api.hanzo.ai that denies us our own production host
for good, verbatim the defect SetSelfDomains was added to close. Nothing would have
caught it in review either: hanzo.ai reaches the set by DERIVATION from CLOUD_DOMAIN,
so no deployment states it and no deployment diff would show it leaving. The
reserved LABELS have had this floor all along ("trimming the env only ever ADDS ...
never subtracts"); the half guarding the more dangerous decision did not. Config now
adds to selfFloor and can never subtract from it.
In practice the brand domain also arrived a second way, via FirstPartyApex, so
BOTH had to be misconfigured before it actually vanished — which is why the new
test moves the first-party apex to prove the floor holds on its own.
SECOND — requestHost decided "is the parsed host real" by asking "is it a host we
would SERVE": siteSlug, else customCandidate. Those are different questions, and
the gap between them is exactly OUR OWN domains. api.hanzo.ai names no site, and
customCandidate excludes it BY DESIGN (IsSelfHost), so neither arm fired and the
client-supplied X-Forwarded-Host won:
Host: login.hanzo.ai
X-Forwarded-Host: <any bound custom domain>
resolved and served that domain's site — a tenant's content returned for a request
addressed to our auth apex, needing no site_hosts row on hanzo.ai at all. The
file's own comment already stated the property correctly ("a request that HAS a
host ignores the header completely"); the code did not have it. The two tests that
look like they pinned it both send a host that IS a site, so the early return fired
and the header was never read — they proved the property only where it already
held.
Whether we serve a host has nothing to do with which host was asked for, so
requestHost no longer asks: a parsed name with a dot is a hostname and is final.
That is the same set the three-way test actually admitted, minus the coupling that
made our own names the exception — and it leaves the header exactly the job it was
added for, the ingress case where fiber parses no host at all, plus bare internal
names no client addresses us by.
TestForwardedHostNeverOverridesOurOwnHost and TestSelfDomainsAreAFloorNotADefault
both fail on the parent:
login.hanzo.ai: the binding resolver was asked about [attacker.example]
hanzo.ai held only via the first-party apex — it must come from the floor
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Bind, verify and release must agree on what a hostname IS, and they did not. Bind
required fqdn.Valid; verify and release required only fqdn.Clean, and release did
not even require that much beyond non-empty.
site_hosts holds TWO shapes: a project's custom FQDNs, and its BARE SLUG — the
structural row deploy.go binds so `<slug>.<apex>` serves. A bare label is not Valid
(nameRE wants labels, a dot and a TLD), so release accepted a row bind could never
re-create. The domains panel renders that row like any other claim, as a live
`https://<slug>`, with a delete control beside it. A tenant tidying away the
odd-looking entry drops its OWN subdomain, and the domains API cannot put it back:
resolution falls back to ResolveUniqueLiveSlug, which refuses once two live
projects share the slug, so the subdomain 404s for everyone — and the next tenant
holding that slug to deploy takes the freed row, and the subdomain, for good. An
add-only asymmetry in a first-come global namespace is a transfer primitive.
hostOf is now the ONE reading of a hostname off a request — canonical form, then
the syntax this surface deals in — and all three ops ask it. A blank entry inside a
bind LIST is still skipped rather than refused; that is a list semantic, not a
disagreement about names.
TestReleaseOnlyAddressesNamesBindCouldHaveMade drives the real DELETE at its real
path. Against the parent it fails with the takeover verbatim:
release of the bare slug = 204, want 400
EXPLOIT: the site's own subdomain row was DELETED through the domains API
It also holds the positive case — a real custom domain still releases 204 and the
row goes — so refusing everything would not pass.
TestVerifyDomainPromotesOnlyOnProof closes a coverage hole rather than a defect:
Store.VerifyHost was tested and fqdn.Verify was tested, but the HANDLER that joins
them had no test at all, so nothing pinned that this surface requires the proof. It
injects a fake resolver — the projects package had none, so any test that reached
verifyDomain would have hit real DNS — and drives no-record, wrong-token,
right-token, already-verified, unclaimed and un-addressable in one pass. The token
is read from the ROW, never from the request, and that is what the wrong-token case
pins.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Both halves of an authorization comparison must be the same value. The
registry-ownership lookup folded one of them:
orgRegistryNamespaces[strings.ToLower(strings.TrimSpace(org))]
while the org it compares came from principal.Org, which returns the validated IAM
owner VERBATIM and never lowercases — deliberately, because "acme" and "ACME" are
DISTINCT tenants in IAM and a fold would let a member of one select the other
(principal.go; TestMembershipMatchIsByteExact pins it).
So a tenant who self-serves an org named `Hanzo` — a different owner from `hanzo`,
and one whose own RoleOwner makes IsOrgAdmin true INSIDE it — folded onto the
`hanzo` key and inherited the `hanzoai` namespace. That is both lanes at once:
push over another brand's production images on the build path (runner.go
imageInOrgRegistry, repoOwnerInOrg), and satisfy the org term of the RELEASE gate
on ghcr.io/hanzoai/cloud, the binary every pod in the fleet runs. A fold applied to
one side of a comparison is not a normalization, it is a collision, and here the
collision IS a cross-tenant privilege grant — the same defect 26f69224 removed from
the custom-domain operator set, in a lane with the fleet downstream of it.
ownedBy is now the ONE lookup, keyed verbatim, and both callers ask it. Trimming
stays: whitespace is not an identity. repoOwnerInOrg keeps EqualFold on the OTHER
side — the forge owner parsed out of a repo URL, which genuinely is
case-insensitive — because that side is not an identity this system issues.
TestRegistryOwnershipIsVerbatim drives both lanes over both directions. Against the
parent it fails with the grant verbatim:
tenant "Hanzo" folded onto a brand's REGISTRY namespace — it could overwrite
that brand's production images and cut a release of the binary the fleet runs
tenant "Hanzo" folded onto a brand's FORGE owner
It holds the positive cases and the cross-brand refusal too, so narrowing the map
to nothing would not pass.
NOT addressed here, and flagged for review rather than changed: the release gate
admits `imageInOrgRegistry(releaseImage, org) && principal.IsOrgAdmin(c)` at all.
runner.go:255-267 argues that deliberately and at length — publishing your own
org's artifact is not cross-tenant — but the artifact is the platform's own binary,
`hanzo` is a customer org like any other, and its admin bit is one that org's own
admins set on their own members. fleet.go:254-261 reaches the opposite conclusion
for the mutation next door and gates it on cloud.Super. Overturning a documented
decision in this app is a call for its owner, not for this change.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Binding a custom domain WITHOUT proving control of it — the bind lands VERIFIED
and routes immediately, and the "a host we operate" refusal does not apply — is
platform authority. The gate read
principal.IsSuperAdmin(c) || (operatorOrgs[org] && principal.IsOrgAdmin(c))
and that second term is not a second platform predicate, it is a self-service one
wearing config. `isAdmin` is org-scoped and an org's OWN admin sets it on a member
of THEIR org, so every `hanzo` admin could enrol any `hanzo` member into the gate
— and operatorOrgs defaults to the deployment's brand org in EVERY deployment
(config.go getenv CLOUD_BRAND, brand.Default "hanzo"). Naming the org in config
does not repair that: config grants a capability TO a tenant, but the tenant still
decides who inside it holds the role, so the deployment ends up delegating a proof
bypass to an authority it does not administer. A gate whose far side can enrol its
own callers is not a gate.
SuperAdmin <=> `owner == "admin"` is the ONE platform predicate. A second, weaker
spelling of platform authority IS the escalation, whatever conjunction dresses it
up, so vouches() is now that predicate alone:
func vouches(c *zip.Ctx) bool { return principal.IsSuperAdmin(c) }
It is cross-tenant by construction, so it still vouches in ANY org — the operator
switched into a customer's org to bind the domain it manages DNS for, which is how
a customer domain is onboarded. That path is unchanged and stays pinned.
Everything the org term reached goes with it: state.operatorOrgs,
operatorOrgsFromEnv, and CLOUD_PLATFORM_OPERATOR_ORGS. The variable appears
nowhere in the universe manifests, so no deployment states it and none needs
editing — a deployment must never state a variable nothing reads. The gate now has
no configuration at all, which is also why a test can no longer under-configure
it: there is nothing left to pass, so the tests drive exactly what production runs.
TestVouchIsSuperAdminOnly drives the real handler over the ROLE axis in the
deployment's own brand org. Against the parent, with the state Mount actually
builds — operatorOrgsFromEnv("hanzo"), the default {hanzo} — the brand-org ADMIN
case fails with the exploit verbatim:
EXPLOIT: brand-org ADMIN bound a bank's login host with NO proof:
{Host:login.example-bank.com Status:live Verified:true
URL:https://login.example-bank.com Records:[]}
It holds the SuperAdmin case too, so merely disabling onboarding would not pass.
TestVouchDoesNotTurnOnTheOrg keeps the ORG axis the verbatim pin covered: four org
names spanning the classes that used to matter — the brand org, its case fold, an
unrelated tenant, and the reserved `admin` org's own NAME — each asserted twice,
org-admin never vouches and SuperAdmin always does. The fold collision that pin
existed for is now dead by construction rather than by comparison, and THAT is the
property it now pins.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The table declares every route it owns by name, so there is no wildcard left
for a host's own route to win against, and two declarations at one address is
a refusal to compose rather than a silent shadow. cloud declares three of them
natively — GET /v1/o11y/logs, GET /v1/o11y/metrics, POST /v1/o11y/query_range
— and used to win them only by registering first.
o11y v1.5.56 gives that fact a spelling: Claimed names the addresses the HOST
serves, and the table then declines to declare exactly those. Say it here.
It is worth being explicit about WHY these three are the host's, because the
ordering that used to decide it decided it invisibly. This package's handlers
are tenant-scoped: handleLogs resolves the caller's org and pins the read to
it. The table's relay hands the call to the runtime with no org on it. Both
answered the same address, the first one registered won, and the two could
have drifted apart forever without anyone being told. The claim is that
decision written down where it can be read — and read the same way on both
sides of the seam, since the string that names the conflict in zip's own
diagnostics is the string that resolves it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/integrations and /v1/connectors answered 503 for the same reason o11y did:
the subsystem exited before it listened, because it declared middleware on two
group nodes that owned no routes.
app.Group("/v1/integrations").Use(cloud.Bridge(), zip.H(bridgeFacts))
app.Group("/v1/connectors").Use(cloud.Bridge(), zip.H(bridgeFacts))
Every op below both lines registers on zapp at an ABSOLUTE path, so neither
group ever received a leaf. A group's middleware wraps the routes in its own
subtree, and an empty subtree means the bridge could never run — which zip
refuses to compose rather than serve ungated.
scope.Use is the install that already solves this: once at the root, gated by
`owns`, confined to exactly the prefixes the manifest declares for this
subsystem. The manifest lists /v1/connectors and /v1/integrations both, so one
install covers what two group nodes were reaching for, and connectorRoutes
needs no bridge of its own.
The test fixture had drifted from the thing it reproduces. installV1Flatten
still hung commerce's error envelope on a Group("/v1"), while commerce itself
moved to gating by path (commerceErrorScope) precisely because a shared /v1
node wraps every subsystem mounted after it. Same shape, same empty subtree,
same refusal — it aborted the package on origin/main before this change. It
now installs the way its twin does.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The subsystem's ops were declared through a.Group(o11yPrefix), and zip
qualifies an op's id by the prefix of the occurrence it is declared under.
That rule exists so one definition included twice cannot publish one
operationId for two operations, and it is right — but o11yPrefix is not that
kind of prefix. It is not a composition point a host chose; it is this
subsystem's own address. Read through a Group it looked like one, and every
published id came out "v1.o11y.get_logs": the OpenAPI operationId, the MCP
tool, the CLI command and the generated SDK method, all renamed, with no
opt-out — an explicit WithOperationID is qualified the same way.
The same Group also cost the ops their origin, which is the app an op is
declared in and the thing that qualifies published TYPES. A Group is not that
app, so 23 schemas went out bare — logsResponse where the weave expects
o11y.logsResponse, and a bare name is one another subsystem can collide with.
Declaring at the root gets both, because an occurrence there is unqualified
and carries the app's own origin. OpScope.Prefix then prepends to the op's
path exactly as a Group's does, so every ADDRESS is byte-identical: same
method, same full path, same middleware. Only the two names change, and they
change back to what the declaration wrote.
hanzoai/o11y's own table reaches this conclusion for the same reason in its
relay.go; this is that shape on the cloud-native half. mountAlerts is left
alone — it registers raw routes, which publish no id and no type.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/o11y, /v1/sentry, /v1/event, /v1/errors, /v1/analytics, /v1/insights,
/v1/integrations and /v1/summary all answered 503 with
mount /v1/o11y: no instance running
while the pod stayed Ready with 0 restarts. The subsystem runs as its own
child process, and it was exiting before it listened: zip refuses a program
that does not compose, and this one declared middleware at two addresses it
had no leaf beneath.
a.Group(o11yPrefix).Use(cloud.Bridge()) // o11y.go
a.Group("/v1/summary").Use(cloud.Bridge()) // summary.go
Neither group ever received a route. The o11y leaves register through a
SECOND Group(o11yPrefix) in scope.go and annotation_queues.go, the module's
353 typed ops register on the app at a root prefix, and /v1/summary's own
leaf registers on the app at the group's address rather than beneath it. A
group's middleware wraps the routes in its OWN subtree, so all three sat
outside the thing meant to guard them — which is the defect the walk names,
and it named it correctly.
Install it once, on the app, where serve.go already says it belongs: a
subsystem whose routes are spread across several top-level nouns owns no
single prefix to hang it on. This one owns eight. That also keeps the module's
353 ops wrapped, which a prefix group silently would not have done — the org
a typed op reads off its context has to be parked for every leaf, not for the
fraction that happens to share a prefix.
The tests carried the same shape and are moved with it, so they exercise what
serving installs rather than a composition only the test builds.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The avatar write failed for the whole session with two different errors, and
both were this one door mis-reading a reply it had assumed the shape of:
iam non-envelope response (400) an error body it could not unmarshal
iam: iam status 200 a SUCCESS it rejected
Some IAM routes answer the {status,msg,data} envelope this client was written
for. Others — /v1/iam/users/get among them — answer the RESOURCE DIRECTLY, and
their errors are {"status":404,"error":"…"} where `status` is a NUMBER, not the
string "ok". Measured against the running service:
GET users/get?owner=hanzo&name=z
-> {createdAt,updatedAt,deleted,id,owner,name,…} no envelope at all
So a raw row parsed with Status "" and was rejected as `iam status 200`, and an
error body failed to unmarshal and became `non-envelope response`. Every read
through this client was one of those two.
The HTTP STATUS now decides and the body is only read for what it carries: a
non-2xx yields its `error` or `msg`, a 2xx envelope keeps its old meaning, and a
2xx that is not an envelope IS the resource.
`go get github.com/hanzoai/cloud@latest` fails outright:
module source tree too large (max size is 524288000 bytes)
v1.801.413 is the last consumable version; .420 and everything after cannot be
downloaded by anyone. That is every consumer of this module, not just ours — ai
hit it trying to take the release it had been waiting on.
The cause is build output committed at the repo root. `go build ./apps/gateway`
drops the binary HERE by default, and five landed that way: gateway 53M,
account 33M, authz 30M, smoke 8M, gen-app-cmds 4M. They are ELF x86-64, ELF
aarch64 AND Mach-O arm64 — three different people's machines, over three weeks,
each an accident nobody could see because the repo already had them. `account`
arrived today and is what crossed the line.
Untracked and ignored by exact path. Every one is built from a package whose
source is untouched (apps/gateway, apps/account, plugin/authz, plugin/smoke,
plugin/gen-app-cmds), so nothing is lost and `go build ./...` is unchanged.
Tracked tree: 174MB → 40MB.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
commerce v1.49.62, iam v1.34.18, o11y v1.5.55, ai v1.832.21 — every subsystem now
published on zip v1.24.1, so cloud and everything it mounts agree on one framework
version rather than four.
That agreement is the point. Router is the type a decorator implements, and the
verbs a decorator must satisfy changed in v1.23 (Use is the one composition verb,
taking a Component); a host on one version and a subsystem on another is a
decorator that cannot be written.
Measured against a clean tree on this host: 97 failing packages before, 97 after —
zero new. That is the macOS SQLCipher limit (no tmpfs for the pure-Go codec),
unrelated and unchanged.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The seam's doc said every app is its own process and read that as the reason
SetRiskScorer cannot reach the gateway. The premise is false as stated.
plugin/campaign, plugin/integrations and plugin/guide each link three or four
sibling apps/* into ONE process and wire process-global seams across them
(seams.go in each), under no build tag, and the image builds every plugin
directory. Co-residency is a per-plugin composition choice: one import in one
composition root is the whole distance between the two arrangements. As
composed today plugin/risk links risk alone and plugin/gateway links gateway
alone, which is true and is all that should have been claimed.
The remedy stands for a simpler reason that holds whatever the topology is:
apps/risk exports Mount and Shutdown and nothing else. There is no scoring
function to install, so the seam has no producer because none can be spelled,
not because a boundary forbids one. Giving it one means EXPORTING a scorer and
then deciding what it answers for — this global answers for its own process,
while arming asks whether the risk plane can answer for the fleet, which is a
cross-process ask.
The obs event door stays as the precedent it is, in its own clause rather than
as the justification.
Comments only; no behaviour changes and nothing is armed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two independent holes let a name the platform holds enter the global
site_hosts table.
SHAPE. Store.bindHost asked sites.IsReserved with its argument, but that
argument is a bare slug on the deploy path (deploy.go siteHost) and a full
hostname on the custom-domain path, while the reserved set holds bare LABELS.
So every FQDN compared against a set of labels matched nothing: login.hanzo.ai
sailed past the only guard behind the claim gate and took a first-come row on
our own auth apex. The storage invariant that is supposed to make the
serve-time gate a mere backstop contributed nothing at all for half the table.
sites.Ours splits on shape — a bare label asks the reserved policy, a hostname
asks the self-domain set — so one predicate answers both, and the claim gate
(domains.go ours) and the host table now ask the SAME question. Note what the
split avoids: `www` and `login` are reserved labels AND the two most common
custom domains a customer brings, so a backstop keyed on "the first label of
this FQDN is reserved" would refuse www.example.com. The label policy must
never reach a name a customer owns.
COMPOSITION. The self-domain set was published only by sites.New. Which apps
share a process is a per-plugin choice, and as composed today plugin/projects
links apps/sites for exactly these two predicates and never calls New — the
edge is not in that process. So the set was EMPTY in the very process that
enforces the claim gate and the host table, and full in the one that serves:
IsSelfHost answered false for everything there. Measured on the parent commit,
in a process that constructs no Server:
IsSelfHost("hanzo.ai") = false
IsSelfHost("api.hanzo.ai") = false
IsSelfHost("login.hanzo.ai") = false
IsSelfHost("hanzo.app") = false
which is the exact defect SetSelfDomains was added to close, defeated by
composition rather than by logic. No test could see it: they publish the set
themselves, in one binary.
The policy is now DERIVED from config wherever it is asked. ConfigFromEnv is
the one reader of that config and every process reads the same environment, so
the answers cannot drift the way a hand-off between processes does, and it
stays right however the plugins are later composed. An explicit publication
still wins and is never re-read over. Production needs no deployment change:
hanzo.ai is derived from CLOUD_DOMAIN's registrable domain, not configured.
selfOf folds the first-party apex in itself, so New's published set is complete
by construction rather than by statement order — the ordering hazard the
previous fix had to hold by hand.
Tests: the Server-less derivation, the shape split over both halves including
the customer hostnames that must stay bindable, and a concurrent first touch
under -race.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Binding a custom domain WITHOUT proving control of it — the bind lands
VERIFIED and routes immediately, and the "a host we operate" refusal does not
apply — is platform authority. The gate read
vouched := c.IsAdmin() || s.State.operatorOrgs[org]
and that second term is bare MEMBERSHIP. operatorOrgs defaults to the
deployment's own brand org (config.go getenv CLOUD_BRAND, brand.Default
"hanzo"), so the set is {hanzo} in every deployment, and `org` is the
IAM-validated effective org, gated upstream by isMember alone. Every staff
account whatever its role, plus anyone a brand-org admin ever invited, could
bind login.example-bank.com live with no DNS-01 proof: attacker content served
at any custom-domain customer whose DNS already points at our edge, and the
name denied to its rightful owner for good, since a verified row is first-come
and global.
vouches() now names the two grants, both admin-scoped:
SuperAdmin platform sudo (owner == the reserved admin org).
Cross-tenant by construction, so it vouches in ANY
org — the operator switched into a customer's org to
bind the domain it manages DNS for. Unchanged.
operator-org ADMIN the deployment named this org an operator AND IAM
says the caller administers it.
The second is a conjunction of two independently administered facts: a
capability the deployment grants to an org, and the role IAM grants inside it.
The set names an org; it never names an authority. The org-admin bit is asked
of the EFFECTIVE org — the same value SanitizeIdentity keys X-User-IsOrgAdmin
on — so the pair reads "admin OF this operator org" and never "admin of some
org I switched out of". Both bits are stripped on ingress and re-minted only
from validated claims.
Fail-secure: an issuer that stops signing the org role drops the operator-org
grant to a PENDING claim carrying the DNS challenge, the same self-service path
every other tenant takes. SuperAdmin onboarding never depended on that claim.
TestOperatorVouchNeedsAdminScope drives the real handler over four identities.
On the parent commit it fails with the exploit verbatim — a plain member's bind
of login.example-bank.com comes back {Status: live, Verified: true}. It also
holds the positive cases, so a fix that merely disabled operator onboarding
would not pass. TestOperatorVouchIsVerbatimEndToEnd keeps the verbatim-owner
pin and now has both callers be org admins, leaving the org name as the only
axis; it went from panicking to passing because it no longer builds the whole
surface.
The harness registers the ops it drives rather than calling routes(), which
composes only on the Router production gives it: routes() declares middleware
with Group(prefix, mw) and registers its typed ops on the App with full paths,
so on the bare *zip.App a test holds the prefix node has no routes beneath it
and zip refuses to compose. cloud.Listen mounts on a *scope, whose Use and
Group install at the root and gate by request path, so the same registration
composes there. What is driven is the production chain at the production paths:
cloud.Bridge parking the request, siteOf resolving the tenant, bindDomains
deciding.
The lifted prose and both published specs carry the corrected contract; the old
text told customers that membership of the operator org was the vouch. zipdoc
regenerates; the OpenAPI subsets cannot be projected for this app yet, so those
two files take the identical substitution zipdoc made.
apps/projects: 55 -> 58 tests passing, no test that passed on the parent fails.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The pin sat at sha-9da3984 while console main moved four commits ahead. Since
console.hanzo.ai IS this binary, none of them was reachable by any customer:
a0a4899 onboarding: Continue keeps its place, and no step can strand you
06e4163 retire the ML Pipelines (Kubeflow) product — its backend is gone
da26897 auth: a refusal is not always a failure
2271c29 profile: a photo you can change, instead of one you can only look at
The embed image for sha-a0a4899 is published and was probed before moving the
pin (200, against a known-good positive and a bogus negative control).
855 files, 360.7 MB, 192 of them real .rlib/.so/.a binaries: a complete cargo
build tree committed to git. Every published hanzoai/cloud module version has
carried it, so every consumer downloads 360 MB of another project's build output
to compile Go. The local module cache alone holds ten copies.
It is orphaned, not merely misplaced. native/flags/ contains NOTHING ELSE — no
Cargo.toml, no src/, no crate at all — so there is not even a project here to
rebuild it. No .go file, Makefile, Dockerfile or shell script references the
path. It arrived as collateral in 56c1b003 and nothing has needed it since.
.gitignore has listed `native/flags/target/` since before this commit; an ignore
rule does not untrack what is already tracked, which is exactly how 360 MB stays
in a tree everyone believes is ignored. `git rm --cached` is the part that was
missing.
Files stay on disk; only the index changes. History still carries the blobs, so
this shrinks FUTURE module versions rather than past ones.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nine plugins failed to start and every route they own answered 503 — account,
catalog, analytics, o11y and the rest — on one panic:
panic: zip: the group "/v1/avatar" declares middleware at scope.go
and no routes anywhere beneath it
THE NODE WAS THE BUG, NOT THE PREFIX. scope.Use installed middleware with
s.app.Group(p).Use(...), which creates a node AT p; scope.Get and its siblings
delegate straight to s.app, so the routes land on the ROOT node. Same paths, two
nodes. zip >= 1.23 checks the subtree of the node the middleware is on, finds it
empty, and refuses to compose a program whose middleware could never run. It was
right to.
Declaring the prefixes did not fix it — it moved the panic from /v1/account to
/v1/avatar — and analytics failed identically while already declaring them.
Both gates now install ONCE at the root and test the request path: Use against
the subsystem's prefixes, Group against its own. The root always has routes, so
nothing is empty, and `owns` does on the request what the per-prefix node was
there to do on the tree. `under` is that one meaning of "inside my subtree",
shared by both.
Confinement is unchanged and still proven by the tests that were red:
ScopeConfinesUseToTheSubsystem, ScopeHonoursDeclaredPrefixes and
ScopeAllowsGroupInsideItsPrefixes all pass. The root package now reports ZERO
composition panics; its 43 remaining failures are two macOS-only causes (40 cek
"no RAM-backed scratch", 3 unix-socket path length), identical before and after.
THE SUITE ALREADY REPRODUCED THIS. `go test ./` was red on main throughout the
outage, with the exact production panic, in under a second and with no cluster.
It was read as environmental noise while the fix was hunted in production.
SetRiskScorer has no production caller. The seam's own doc said to follow the
observability plane's event door "(obsevents.go)" — a file deleted in 4a6f3918,
by the commit that established the opposite: "A plugin is a process;
cloud.SetObsEventIngest / SetObsErrorIngest could never have worked across that
boundary." That door read nil in the process that needed it and answered 503 to
every Sentry SDK until it became a plane op (apps/o11y/obs_rpc.go).
SetRiskScorer is the identical shape, and the fleet is one process per app:
cmd/cloud mounts each subsystem as its own binary, plugin/risk lists risk alone
and plugin/gateway lists gateway alone, and no binary links both. So the wire
this seam invites — call SetRiskScorer from the risk app's Mount — would arm the
risk process and nothing else, while apps/gateway's RiskScorerInstalled, running
in the gateway binary, stayed false and PUT /v1/gateway/config went on refusing
every arming request. It would read as wired and change no outcome.
The tests cannot show this: they link cloud and the app into one binary, where
the handoff always works. That is why 43 references pass over a seam with no
producer. Both comments now say so — what the seam reaches, and that the
gateway's refusal is currently unconditional and fail-SAFE, answering "is a
scorer linked here" rather than the question arming asks, which is whether the
risk plane can answer for the fleet.
Comments only; no behaviour changes and nothing is armed. Reaching the fleet
answer is a plane op, and that is a decision to take deliberately, not a wire
to restore.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
New() published the self-domain set and THEN folded the first-party apex into
the slice it had just handed over. SetSelfDomains copies its argument
(reserved.go), so that append could never reach IsSelfHost — the one predicate
that answers "is this host ours", read by both the serve gate and the claim
gate.
Under a config that names the apex only as FirstPartyApex — not also in
CLOUD_SITES_SELF_DOMAINS — every non-allowlisted <label>.<fpApex> was therefore
a custom-domain CANDIDATE on the apex that carries api/login/console: a claim
row a customer could take, and a per-request binding lookup on the hot path the
exclusion exists to keep clear. Production lists hanzo.ai in BOTH, which is why
TestSelfDomainsCoverTheBrandApex passes either way and never saw it.
The publish now runs after the fold, so one set leaves the constructor.
Server.selfDomains went with it: it was written once here and read nowhere,
a second copy of a set whose only reader is the package global in reserved.go.
TestFirstPartyApexReachesThePublishedSet
fix reverted, test kept → RED (IsSelfHost false, customCandidate true
for hanzo.ai / api.hanzo.ai / login.hanzo.ai)
fix restored → GREEN
apps/sites and cmd/cloud green; apps/crm unchanged from its baseline failure
(TestIntakeRateLimitScope, pre-existing on origin/main).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/risk/search ranked model spaces against a tenant's own history and
answered with the one that fit. Nothing could promote it, and nothing could:
adoption refused a shape change, and a winner is a different shape by
definition. The stated replacement for Katib produced advice nobody could take.
A model value now carries the SPACE its masses describe, not only the masses,
and install REPLANTS — it builds that space, restores into it, and swaps it in.
Every gate the refusal used to carry still holds: the tenant comes from the
validated principal, the geometry seed is checked against the model already
running before anything is rebuilt, and the recorded shape is rebuilt and
compared by digest rather than trusted. A value recording no shape is refused
rather than defaulted, and a failed replant leaves the residency untouched.
The winner arrives as one of the organisation's own published values: the run
fits it once more after the grid — a sixty-fifth pass, gated and metered as one
— under that organisation's OWN geometry, because the grid's reference partition
is a constant and a model an organisation runs must partition the space in a way
an outsider cannot predict. Keeping all sixty-four fitted stores instead would
hold 21 MiB for sixty-three shapes nobody adopts.
Also:
- the shape is its own value, and the half of a config that belongs to the
state. The grid's candidate embeds it, the residency records it, a published
value stores it: one spelling, so adopting a shape cannot restate a policy.
- the resume row's shape is what the next process plants. Without it an
adopted shape was silently lost on every rollout and the organisation went
back to the default, warming, deciding nothing.
- the fold watermark travels with an adopted value. It is in the address for a
reason; leaving it behind meant a rollback would never re-read the fold it
skipped.
- snapshot+restore collapse to POST and PUT on /v1/risk/state/model: one
address for one kind of thing, the same collapse GET and PUT on
/v1/risk/policy already made. openapi/floor.json loses one PATH (the risk
product keeps all 31 operations).
- the value bound is sized on the widest shape the grid declares and MEASURED
at full occupancy (2,214,100 bytes), because a fitted sample understates a
busy organisation's tree by an order of magnitude.
- open() no longer reads the state believing the read plants a tenant's trees.
Measured against the pinned engine: it does not.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The two Kubernetes submit paths in ai/cluster no longer name a vendor in a
scheduler contract. The TrainJob's resourcesPerNode and the KServe
InferenceService's predictor both take the device-plugin resource name the
CLUSTER advertises, and a cluster that advertises no accelerator is refused at
submit time rather than accepting a workload that can never be scheduled --
which is what the KServe path did, silently, with a live controller
reconciling the result into a permanently-pending predictor and a model
registered on api.hanzo.ai whose every call fails.
Pairs with agents.Need/Spec.Satisfies landed here: one vocabulary for what a
job requires and what a machine advertises, and no vendor field in either.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Spec already carried what a machine IS -- os/arch/cpus/memory and each
accelerator's vendor/model/memory. Nothing carried what a job WANTS, so the
only way to ask for a GPU was a scheduler contract that named a vendor:
resourcesPerNode.limits."nvidia.com/gpu". That is not a requirement, it is one
vendor's name for a requirement, and it made a job unroutable to an AMD or
Apple machine that could have run it.
Need is the other half of Spec, in the same vocabulary, and Satisfies is the
one place the two meet -- a pure function of two values, so the dispatch gate,
a scheduler and a UI preview cannot disagree. Need has no vendor field: which
vendor clears an accelerator requirement is the machine's business, and
hanzo-kernel lowers one kernel source to CUDA/ROCm/Vulkan/Metal so a job never
has to care.
Unknown memory does not clear a memory floor. Every unified-memory
accelerator advertises 0 today (nvidia-smi answers "[N/A]" on a GB10,
system_profiler emits no VRAM line on Apple Silicon, lspci carries no memory
at all), so a VRAM floor currently refuses all three -- fail-closed, and the
reason the probe should report the memory an accelerator can actually address.
The three boxes are fixtures here, carrying their measured values, so that
change shows up in one place.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
account went down because its plugin declared no Prefixes and MountPrefixes
falls back to /v1/<name> — a path it does not serve — so the scope guarded a
subtree with no routes and zip refused to compose. Same fault, same fix, in
every other plugin that carries it.
Found by predicate rather than by waiting for each outage: undeclared Prefixes
AND a manifest row without /v1/<name> AND an app that calls app.Use(). All three
are needed — 25 plugins match the first two and serve fine, because the panic
only fires when a subsystem actually installs middleware at its scope root.
admission bot dataset do graph knowledge leaderboard treasury
Each now declares what the manifest already says it answers, which is what the
host routes to it either way — so this changes no address, it only stops the
scope guarding one that was never served.
Every account route answered 503 in production — /v1/keys, /v1/csrf, /v1/avatar
— because the plugin panicked at mount:
panic: zip: the group "/v1/account" declares middleware at scope.go:134
and no routes anywhere beneath it (via root -> /v1/account)
The plugin declared no Prefixes, and undeclared is not "no prefixes":
MountPrefixes falls back to the /v1/<name> convention (subsystem.go:78). Account
answers at NONE of /v1/account — its routes are /v1/keys, /v1/csrf, /v1/avatar,
/v1/orgs, /v1/embed and /v1/commerce/topup/*. So scope.Use installed the
subsystem's Bridge on a path with nothing beneath it, and zip refuses to compose
a program whose middleware can never run.
The same fallback bit analytics and entitlements before this; both carry the
same one-line fix, and this is it.
WHY THE TESTS DID NOT CATCH IT, which is the part worth keeping: they mount on a
bare zip.App, where Use attaches at the root and the root HAS routes. Production
mounts through a SCOPE. Same code, opposite outcome — so a green suite proved
nothing about the composition that actually ships.
manifest/clients_test.go's ledger of addresses a first-party client asks for and
the fleet does not answer held three entries. All three are closed; the ledger is
empty.
GET /v1/billing/portal/methods — this is what made "card save is broken" true END
TO END. cloud's billing app serves the saved-card list by proxying here, and
nothing in the fleet served it, so the proxy forwarded a 404 verbatim and the list
rendered as "no cards saved" no matter how many cards were vaulted. Claimed on
commerce's manifest row and registered co-resident.
THE GATE, which is the actual work: PortalPaymentMethods keys tenancy on a
?customerId QUERY PARAM, so the chain has to pin the subject for both principals
that arrive. TokenRequired (not IAMTokenRequired) authenticates and resolves the
ORG from the gateway-pinned X-Org-Id for an IAM member AND for the raw service
token the proxy presents; PinBillingSubject is the IDOR control — it overwrites
every billing-subject key with the validated caller's own account.Payer subject
and drops ?org, passes the query through only for a bearer that constant-time
matches COMMERCE_SERVICE_TOKEN, and fail-closes anyone who is neither. The tenant
is never a caller-supplied field on either path.
DELETE /v1/billing/methods/{id} — 405 at the live edge: a customer could ADD a
card and never REMOVE one. billing registers the sub-resource on the same router
as the collection (the host claims a prefix for ONE app across every method) and
proxies it to commerce's DELETE /v1/billing/portal/methods/{id}, the target that
does not self-dispatch. The org comes from principal.Org — the VALIDATED
principal only, never readerOrg's service-token admission, because this is a
mutation and that is the rule createPaymentMethod and gpuCharge already follow.
The id is escaped into the upstream URL: it names a resource, not a route.
PATCH stays unserved — no client edits a card.
POST /v1/billing/payment — DELETED, not served. No app in either server repo has
ever registered it, in any commit, so the crypto top-up's recording step always
failed and a customer who had already sent USDC to the treasury got a 502; it was
501 besides, since TOPUP_RAILS is configured in no environment. There is nothing
to point it at: money-IN has one door (commerce's mint-gated POST
/v1/billing/deposit) and the fleet routes NO mint address at the edge — the only
two money-in paths manifest.Apps hands to an app are the card ones, both with a
server-authoritative amount. What the browser client sends here is a
client-supplied amount and a client-supplied subject, which is the exact shape the
mint gate exists to refuse. So the surface that existed only to call it goes with
it, and openapi/floor.json records the two-operation reduction next to its reason.
Tenant isolation: TestDeletePaymentMethod_TenantIsolation proves org A cannot aim
a delete at org B through the org, the id, or any subject key; the far side is
proved in commerce (payment_methods_tenant_test.go) for both handlers and both
caller profiles.
Needs hanzoai/commerce fix/billing-methods for the portal detach it proxies to.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The lease is a single-holder flock on one inode under DataDir. Between two POD
GENERATIONS rolling over the same PVC that is exactly what is wanted — it is the
thing that stops a surge writer double-opening the exclusive-lock ZapDB and audit
stores. Between the sibling processes of ONE pod it is a deadlock.
cloud is a plugin host: kms, pubsub and kafka are separate processes in one
container, sharing this DataDir by design. Every one of them called
acquireWriterLease on the same path, so the first to start won and the rest
blocked to the 90s fail-closed deadline. Nothing bound :8000, the liveness probe
killed the pod, and the replacement deadlocked identically — api.hanzo.ai served
503 in a restart loop produced entirely by the safety mechanism.
`CLOUD_WRITER_LEASE` was reverted in the values file to stop the bleeding
(universe 47e195f1, "the binary is not one process"). This is the other half: the
variable can be set again without taking the deployment down.
The guard is underRouter(), which already exists and already means this — ZIP_ADDR
is proof of a parent that owns a plugin table. A child is not a second pod; it is
part of the writer that is already holding the lease.
The test pins all three facts: the host takes it, a second host-shaped process is
still refused, and a child never asks.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The wire was V1-shaped and ours in the places it was not the spec's, so a
compliant @x402/fetch or x402[httpx] client could not pay us at all: it sends
PAYMENT-SIGNATURE and we read X-Payment; it signs `payTo`/`asset`/`amount` and we
quoted `payee`/`token`/`amount`; it speaks CAIP-2 and we published a network label
beside a separate chainId. Every field name, header name and error reason on the
wire is now the one x402-specification-v2.md prints.
header X-PAYMENT / X-PAYMENT-RESPONSE -> PAYMENT-SIGNATURE / PAYMENT-RESPONSE
(plus PAYMENT-REQUIRED for the challenge), all base64 JSON
version "1" (string) -> 2 (number)
scheme "erc3009" -> "exact" (+ extra.assetTransferMethod)
network "hanzo" + chainId 36963 -> "eip155:36963", chain id read OUT of it
challenge bare PaymentRequirements -> PaymentRequired{resource, accepts[]}
payment flat Proof -> PaymentPayload{accepted, payload{
signature, authorization}}
answer Receipt on a header -> SettlementResponse, on success AND failure
The V1 shapes are deleted, not aliased. A header a client may send under either
name is two wires, and the one the server forgot to read is the one where a payer
pays and is never served.
Two things the alignment forced, both real bugs:
* The client's echoed `accepted` is now checked field-by-field against what we
offered (spec 6.1.2 step 5). `extra` IS the EIP-712 domain, so a payload free to
restate it would sign a message of its own devising and verify against itself.
* settleLedger documented a hole and left it: debit lands, credit fails, nothing
served, and the ONLY recovery was the client re-presenting an authorization that
expires in 300s. A client that gave up for five minutes was permanently debited
with nothing delivered.
Fixed by ordering, not by compensation (a reversing entry refunds a payer who
was correctly charged when the failure was a timeout). The settlement row is now
CLAIMED before any money moves and flipped to settled after both halves land, so
an interrupted settlement is a durable row naming the payer, the payee subject
and the amount — keyed on the id both money writes are idempotent on. And the
time window gates ACCEPTING an authorization, not COMPLETING one already
accepted: EIP-3009's validBefore bounds when a transfer may be submitted, not how
long submission takes. Two independent paths now converge — the client is served
whenever it comes back, and Reconcile finishes it from the claim if it never
does.
The payee's ledger SUBJECT is on the claim rather than re-resolved, so completing
a settlement pays the payout wallet the listing named. Re-resolving credited the
seller ORG, which no balance assertion could catch (finance aggregates at the
org) — the test now asks the ledger where the money went.
Also: POST/GET /v1/wallets answered 403 "sign in" to everyone. zip v1.23 scopes a
definition's middleware to its own subtree, and the collection root was declared on
the app, outside the group carrying cloud.Bridge — so it reached its handler with
no validated principal. Declared on the /v1 parent now. This is what blocked all 8
x402 and 4 marketplace tests from running at all.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A version was allocated by READING — max(registry tags, git tags) + 1 — and a
read reserves nothing. Two lanes publish ghcr.io/hanzoai/cloud (this repo's
.hanzo/workflows/cicd.yml, and apps/platform/release.go behind POST /v1/runner);
both computed the same next number, both built for ~20 minutes, and both pushed.
GHCR TAGS ARE MUTABLE, so the second push silently REPLACED the first's bytes
under a name the first had been told was its own:
v1.801.361 overwritten at 04:40:53
v1.801.410 overwritten at 08:17:12, twelve minutes after the real release,
by an image whose revision label read `unknown`
The loser found out at its tag step — three quarters of the way through, after
it had already corrupted the winner's image, which the winner then smoked,
pinned and shipped. A check twenty minutes downstream of the act it guards is
not a check.
Creating refs/tags/v<N> is the one operation in either lane the server performs
as a COMPARE-AND-SWAP: 201 if absent, 422 if present, decided under its lock.
So the claim IS the allocation, and it moves to the FRONT of both lanes. A
number that cannot be claimed was never ours to build; the loser walks to the
next one in a single HTTP call, before building anything. The tag step that used
to mint now VERIFIES the claim still names this commit before the pin.
Cost: a failed build leaves a hole — a tag with no image. That is the cheap
direction. A hole is inert and visible (pin.sh refuses a tag that does not
resolve); a reused number is invisible and serves the wrong bytes.
Also:
- resume is decided by the REGISTRY, not the tag. With the claim moved before
the build, a tag on HEAD no longer implies an image exists, so the old
`git tag --points-at HEAD` resume would have skipped the build of a release
that had none. The tag says what we own; the image says how far we got.
- the pushed image is verified to BE the commit we built, read back off the
registry rather than trusted from our own build output. This is what makes a
clobber by any lane — including one that ignores the claim — loud instead of
a green pin onto foreign bytes.
- REVISION is passed as a build-arg by both lanes. The Dockerfile declares
`ARG REVISION=unknown` and stamps the label from it; buildFrontendCmd never
passed it, so every platform-built image was untraceable to a commit. That
is why the two v1.801.410 images could not be told apart without diffing
layers. Branch refs are refused (isCommitSHA) — a label that says "main" is
populated and useless.
- .hanzo/scripts/image-revision.sh reads an image's commit, descending an
index to its amd64 child so multi-arch images are not silently unchecked.
- the two 422s are distinguished. "Reference already exists" is the collision
this loop is for; "Object does not exist" means our own commit is not on
github.com — reachable, since these lanes run on git.hanzo.ai and claim
against GitHub — and no amount of walking forward fixes it.
Tests: TestClaimReleaseVersion_IsExclusive reproduces the race (a number another
commit holds is skipped and never overwritten; our own claim is a resume that
mints nothing), and TestTagRelease_VerifiesTheClaim asserts the tag step only
READS. claimFrom is split from computeReleaseVersion so the exclusion property
is testable without the registry.
`make openapi` was renamed to `make describe` in e247e255 — correctly, since
the target now projects every app rather than only the spec — and 27 places
were left naming the old one. Two of them are FAILURE MESSAGES: a developer
whose golden is stale is told "run `make openapi`", which prints "No rule to
make target". A gate that says how to fix it, and names a command that does
not exist, is a gate that reads as broken tooling.
Comment-only apart from those two t.Fatalf strings; no target, no behaviour and
no artifact changes. `go build ./openapi/` and the four apps with the most
edits build with the tags hanzo.yml's own go-unit gate uses.
openapi/fleet.go also gains the projection it was missing. It lists four
projections of this API compared against each other by test; there is a fifth,
downstream and in another repo — hanzoai/openapi's hanzo.yaml, which every
published SDK is generated from — and it refutes itself against the LIVE
endpoint in that list, because that is the only one of the four a repo with no
checkout of this one can read.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Upstream landed the v1.23 verb migration (Graft/Add/Mount folded into Use). This
is the half that was missing, and it is the half that made tests lie.
App.Test used to skip prepare, which installs the deferred projections — /mcp, the
OpenAPI document, the op-call plane, the plugin route. So those four addresses
answered 404 under test and 200 in production, and the papering-over was an
exported Prepare each caller had to remember. zip v1.24.1 makes Test prepare;
apps/ai's MCP door test passes because of that, not because of anything here.
414 call sites move from app.Fiber().Test(...) to app.Test(...) with
zip.TestConfig. That is the point of the escape hatch living on the concrete type:
reaching through it bypasses what App.Test does, so the tests most wanting to
exercise the real program were the ones that did not. Sites whose receiver is a
raw fiber app keep fiber's type — the two are not interchangeable and pretending
otherwise is how the first sweep broke things.
Also: the multi-line `Use(func(c *zip.Ctx) error {…})` literals in tests, which
the verb migration missed because they fail vet rather than build; and the last
`.Prepare()` calls, now that it is implicit.
iam v1.34.11 → v1.34.12.
Measured against upstream on the same host: 103 failing packages before, 97 after
— ZERO new, 6 fixed. The remainder is the macOS SQLCipher limit (no tmpfs for the
pure-Go codec), unrelated and unchanged.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
With the read shape fixed, the avatar write moved one step and stopped: the
lookup now reaches IAM and asks for user `hanzo/2d4d67ab-…`, which does not
exist under that spelling. IAM addresses a row by its NAME — the row whose id is
2d4d67ab-… is named `z` — and on the direct-Bearer path the only user handle a
token carries is the UUID `sub`, because X-User-Name is not stamped there.
So a failed direct lookup now resolves the id against the org's roster
(get-users?owner=…, which carries both id and name) and retries once. Measured:
that roster returns 268 rows for hanzo, each with both fields, so the mapping is
available exactly where it was needed.
It runs ONLY after the direct read has already failed — the gateway path, where
username == name, never pays for it.
zip v1.23 deletes graft.go outright. Graft(children ...*App) error was the third
way to attach something to an app, alongside Add and Use; v1.23 keeps exactly
one, and an *App IS a Component, so a child is included by reference through the
same verb as a middleware.
Two call sites, both the "include a whole subsystem's app" case Graft existed
for: apps/iam grafting iamserver.NewApp(db), apps/o11y grafting its assembled
app. Both now Use it.
ONE SEMANTIC CHANGE, recorded because it is not visible at the call site: Graft
refused an address conflict EAGERLY and returned the error to the caller, having
checked every child address against the parent's router and its siblings before
mutating anything. Use appends and defers that verdict to Build, where the whole
program is known. Same refusal, later and with more information — but a mount
that used to fail at its own line now fails at seal, so read Build's error, not
the mount's.
Verified: `go build -tags sqlite_math_functions ./...` returns ZERO across the
whole tree with iam v1.34.11, o11y v1.5.54 and commerce v1.49.61 — the three
dependencies that had to publish first. Graft appears in no .go file in any repo
in the estate.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Cloud pinned zip v1.18.23 while latest was v1.23.0 — five minor versions of
drift on the framework every subsystem composes through. v1.23 removes the
second and third ways to attach something to an app and leaves exactly one:
Use(cs ...Component), where a Component is a Handler or an *App included by
reference.
Cloud's own four adaptations:
- Router.Use widens from ...zip.Handler to ...zip.Component, mirroring
zip.Router exactly. That mirroring is load-bearing: ZipApp's type switch asks
whether a *zip.App satisfies this interface, and a narrower Use made that case
IMPOSSIBLE — the compiler rejected the switch outright rather than silently
taking the wrong branch.
- scope.Use forwards Components unchanged, still once per declared prefix.
- (*zip.App).Add is gone. zip.Load already returns the leaf *App and an *App IS
a Component, so both call sites capture the leaf and Use it. The eager and
lazy rungs of the plugin ladder keep their existing error handling; only the
attach verb changed.
- Five bare closures passed to Use now go through zip.H. Go will not implicitly
convert func(c *zip.Ctx) error to an interface that only the named Handler
type implements. Route methods still take ...Handler, so no route registration
changed — only Use sites.
Verified: with zip at v1.23.0, `go build -tags sqlite_math_functions ./...`
reports ZERO errors from apps/, cmd/, clients/ or internal/. The only remaining
failures are in three dependencies that must publish first — commerce
(mintRouter implements the old Use, and reaches Fiber() through the Router
interface, which no longer exposes it), iam and o11y (both call the now-
unexported app.Prepare; the public replacement is app.Build, which returns an
error Prepare did not). Those are in flight; this commit is the cloud half and
does not build green until they land.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The site edge is mounted in two processes — the light router that owns the
public port (cmd/cloud) and cloud.Listen in every per-app child — and each
resolved sites.Config for itself, from two spellings of the first-party keys
(CLOUD_SITES_FIRSTPARTY_* against CLOUD_SITES_FIRST_PARTY_*) with two sets of
defaults. Neither spelling is set in production, so the two processes booted
different policy off their DEFAULTS alone: the router resolved no first-party
apex, so hanzo.ai never entered its self-domain set, so every hanzo.ai host —
api.hanzo.ai included — was a custom-domain candidate and took the per-request
binding lookup the self-domain exclusion exists to keep off that path.
The reserved denylist was NOT affected. Both spellings read the same
CLOUD_SITES_RESERVED key, the operator value only ever ADDS via
SetReservedExtra, and the labels an attacker wants are baked into reserved.go —
so an empty value cannot un-reserve anything. Measured at the live edge:
www/api/app/admin/stg/login/wallet.hanzo.app all fall through with no
X-Hanzo-Site, against quest.hanzo.app which answers with one.
The env keys and their defaults now live in apps/sites, the package that owns
the type, and both call sites read that one function. Config keeps only Domain,
which feeds the self-domain set.
Also removes exports with no caller in this repo or any that import it:
OKList/OKRaw (envelope.go, whose note about a clients/admin/core delegator was
stale — that package is gone), BrandInfo, DegradedNames/IsDegraded (the release
smoke reads /v1/health over HTTP; the in-process accessors had only tests, whose
coverage of the live Degraded/Degradations pair is kept), and the four inert
Stage-0 control-plane fields with their NODE_ID/PEERS/ROLE/CONTROL_PLANE_QUORUM
reads.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every op on /v1/engine was dialling a refused connection in production. The
upstream default named :1234; svc/engine in namespace hanzo exposes ONE port and
it is 36900 (name http, port 36900, target 36900), and the cloud Deployment sets
no ENGINE_UPSTREAM — so the wrong default WAS the production value. status
answered reachable:false, and models, model and system answered 503. The
subsystem was honest about being unreachable, which is why nothing alarmed: a
truthful report of a broken configuration reads exactly like a runtime that is
down.
1234 is standalone `hanzo serve`'s default. 36900 is the port the engine binds as
the node's engine, and 36900 is what is deployed — so the comment claiming 1234
was "the in-cluster Service of the engine deployment" described a process this
cluster does not run. The 1234-vs-36900 confusion is already on record from the
desktop build, where a frontend discovered models at one port while the engine
that answers ran at the other; this is the same mistake on the cloud side.
MEASURED against the deployed engine over a port-forward to svc/engine:36900,
which answers exactly the three endpoints this plane calls, all 200 —
GET /health ("OK"), GET /v1/models (the model list, with a loaded model carrying
"status":"loaded") and GET /v1/system/info (the SystemInfo document: os, kernel,
cpu, memory). So the upstream is present and correct and only the port was wrong;
this is one token, not a redesign.
ENGINE_UPSTREAM stays the override, and 1234 remains right where it is right — a
dev box running `hanzo serve`. Behaviour only: no route, no schema, no operation
id and no published byte changes, and regenerating the subset produces no diff.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An operation id is the generated SDK METHOD NAME and the CLI COMMAND, and a
summary is what the MCP tool list shows a model choosing between tools. This
plane stated neither, so both were defaults, and both defaults were wrong in a
way only a caller sees.
zip derives an unstated id from the path, so the four ops published
`get_v1_engine_status`, `get_v1_engine_models`, `get_v1_engine_model`,
`get_v1_engine_system` — path mangling where the rest of the fleet publishes the
product and the noun. The risk product's thirty-one operations are `riskScore`,
`riskState`, `riskDatasets`, `riskLabelCoverage`; these are now `engineStatus`,
`engineModels`, `engineModel`, `engineSystem`, which is also the rule this
package already applied to its own SCHEMA names and only to those.
A summary defaults to the first sentence of the Go doc comment, and a Go doc
comment opens with the Go IDENTIFIER — so the published summaries read "Status
reports whether the engine deployment is reachable", "Models lists the models
the engine serves", "Model reads one model's load state". A Go symbol name was
the first word a CLI user, an SDK reader and a model picking a tool saw. Each op
now states a summary written in the imperative for the person calling it, and
the doc comment stays a Go doc comment that zipdoc still lifts as the
description: two audiences, two sentences, one declaration.
FORWARDS-ONLY, and it costs nothing: this plane has no customers on it. No
alias, no redirect, no compat shim.
Regenerating from source changes exactly four operation ids. No path is added or
removed, no (path, method) pair moves, no schema changes, and openapi/floor.json
is byte-identical because the operation count did not move.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
openapi/floor.json is the ratchet: "a published surface may grow and may not
quietly shrink". It carries a global path and operation count AND a per-PRODUCT
count, and the per-product half is the half that binds — floor.go compares
`f.Products[p]` against the regenerated count and refuses a product that lost
operations. The file's own code states the failure mode out loud: "an absent floor
makes every shrink legal".
The merge 94df22e1 resolved this generated file to
`{"paths":1684,"operations":2336}`: the entire `products` map — 180 entries —
GONE, and both global counts rolled backwards. Every commit before it carries the
map. So on main right now every product may shrink without the gate saying
anything, and the global floor sits five paths and five operations below the
surface actually published.
IT ALREADY COST SOMETHING, within hours. Deleting the /v1/train facade removed ten
operations, and a ten-operation shrink is exactly what this ratchet exists to make
an author state deliberately — the dataset move had to lower `ml: 14 -> 7` in the
same commit that moved the routes, and said so. The train deletion needed no such
line, because there was no per-product floor left to lower. The gate was not
merely stale; it was switched off, and a shrink walked through it unremarked.
Restoring it is `make describe` and nothing else: 1689 paths, 2341 operations, 178
products. `train` is simply absent from the restored map rather than floored at
zero, because the map is regenerated from the document that exists — the ratchet
resumes from the current truth, which is the only honest baseline available once
the old one has been discarded.
Committed on its own because it belongs to no surface change: these counts are
identical with or without the engine renaming beside it, since an operation id
does not move a count.
The lesson is the fleet's own about derived artifacts, with a sharper edge: two
derived files can agree with each other while both are wrong, and a MERGE is a
third way for a derived file to become wrong — resolved by hand toward whichever
side the conflict presented. A ratchet resolved toward the weaker side does not
look stale. It looks fine, and it is off.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/label addressed /v1/ml/labels, apps/reference addressed /v1/ml/reference and
apps/dataset addressed /v1/ml/datasets. All three are the RISK product — the
labels a decision is adjudicated with, the lookup data it consults, and the
snapshot its model was fitted on — and all three were corrected to /v1/risk/*.
The reason is the same each time: openapi.Product takes a product from the FIRST
/v1 segment and a per-op tag cannot override it, so an address IS a published
product membership, and /v1/ml is the live serving product. Filing a second
product there makes /v1/ml/models mean two things at once.
apps/dataset's address_test.go says the third time "stops being a recollection
and becomes this gate" — and builds it, for apps/dataset. A per-app gate cannot
catch the fourth time, because the fourth time happens in a fifth app that does
not have one. So the invariant is stated once, on the side every app must pass
through: the ROUTING GRANT. A plane cannot publish under /v1/ml without a
manifest row saying so, which makes the row the one place the mistake is always
visible.
It refuses a second OWNER, not a second app, and that boundary is measured
rather than assumed: fourteen products here are answered by more than one app and
27 apps publish into more than one product, so neither "one app per product" nor
"one product per app" is a fleet invariant, and asserting either would invent a
rule the fleet does not keep. What all three mistakes actually broke is narrower
and true — the serving prefix has one owner.
The second test is the half the ai incident argues for: a gate that only refuses
intruders stays green when the owner itself vanishes. Narrowing ai's row to
/v1/ai once took the entire inference surface off the wire with every probe
green, so the owner's own leaves are asserted individually — a count cannot say
WHICH address stopped routing.
BOTH DIRECTIONS ARE MUTATION-PROVEN, because a gate nobody made fail is a gate
nobody knows works. Re-addressing dataset to /v1/ml/datasets fails with the
ROUTING GRANT message naming the app, the prefix and the product it would join;
narrowing ml's row to /v1/mlops fails BOTH tests — the non-vacuity check (which
also proves `under` is segment-aware, since /v1/mlops is correctly not under
/v1/ml) and each unrouted leaf by name.
/v1/train is deliberately NOT fenced: it was just deleted because the Kubeflow
CRDs behind it are not served. A gate must fence prefixes that exist, and naming
a deleted one would trip the non-vacuity check while saying nothing true.
It also does not judge the NAME. `ml` being the vague word is why this keeps
happening, but renaming a prefix that is published is a wire change with its own
cost; until that is worth paying the ambiguity is fenced, not resolved.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/train/* was a thin proxy onto two Kubeflow CRDs that this cluster does not
serve: trainer.kubeflow.org/trainjobs and kubeflow.org/{experiments,trials}.
GET /v1/train/health answers 503 degraded in production right now
({"crds":{"experiments":false,"trainjobs":false},"status":"degraded"}) with
nothing in cloud changed — the CRDs were retired underneath it. Ten operations
go, and with them the degraded door.
Measured before deleting, cluster-wide: zero TrainJobs, zero Experiments, zero
Trials, and no ClusterTrainingRuntime for a TrainJob to reference. The katib
half could not have worked at all — katib's admission webhook requires the
namespace label katib.kubeflow.org/metrics-collector-injection=enabled,
ensureNamespace writes only {managed-by, hanzo.ai/org}, and no namespace in the
cluster carries it. So POST /v1/train/experiments took the billing gate, created
an Experiment, and katib never admitted a Trial.
There were also TWO doors onto one TrainJob CRD: this one and the hanzoai/ai
broker at /v1/finetune/*, which has the product around it (presets, HF pickers,
status polling, deploy-to-serving). One door survives.
KServe STAYS. /v1/ml/models is the only path in the estate that serves a
classical artifact end to end, and it is proven: POST /v1/ml/models with an
sklearn joblib -> 201, the storage-initializer pulls the model, then POST
/v1/ml/models/{name}/predict -> 200 with correct predictions on the
kserve-mlserver runtime. GET /v1/ml/health is 200 today. Its runtime-capacity
clause and every serving test are untouched.
openapi/floor.json needs NO edit on this base: the shrink guard is a FLOOR, and
main's committed floor (1684 paths / 2336 operations) is already below the
post-deletion document, measured at 1689 / 2341. The guard still bites — raising
the floor above the real count fails TestFleetIsTheWeaveOfItsApps with the same
"THE PUBLISHED SURFACE SHRANK" report, which is how these numbers were read.
(An earlier pass here lowered a floor that ALSO carried a per-product map; main
has since dropped that map, so the rebase takes main's shape unchanged.)
The billing-gate integration tests keep
their coverage by exercising the surviving create (POST /v1/ml/models) — the
gate is the shared create() body, not a per-kind one.
Also repointed every pointer that named the deleted route, so none dangles:
apps/engine's intentRefused reason and LLM.md (now /v1/finetune/jobs), spend.go's
routing-union example, apps/platform/drift.go's analogy, and the mutation in
scripts/mutate.py whose anchor line and target test are both gone (it would have
reported ANCHOR-MISS).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
getUser sent the `<owner>/<name>` composite as `id`, and IAM wants the two as
separate fields. Measured against the running service with the console client:
?id=hanzo/2d4d67ab-… 400 field "owner" is required
?id=hanzo/z 400 field "owner" is required
?owner=hanzo&name=z 200
So NO caller of this ever read a user row. The avatar write is where it became
visible — "photo stored but the profile could not be updated", with
`iam non-envelope response (400)` behind it — but moveUserToOrg carries the same
fault silently, and its failure mode is worse: the whole-row re-submit it feeds
would have nothing to re-submit.
`name` is the USERNAME (the row's own `name`, "z"), never the UUID that `sub`
carries — the row this now reads has id 2d4d67ab-… and name z, which is why
addressing it by the uuid found nothing.
The composite is KEPT for the one caller that has no owner to send: a first-run,
org-less user, where resolving their authoritative (owner, name) is the entire
purpose of the read. Refusing that here would break onboarding to fix the avatar.
The fake IAM now insists on the same shape, so a client that regresses to `id`
for a caller that HAS an owner fails in the suite instead of in production.
A sibling dissolved PUT /v1/risk/state/appetite while this branch was open (the
decision regime has one address now), so typed.go conflicted on the op that used to
sit between `state` and `snapshot`. Resolved by taking MAIN'S typed.go whole and
re-applying this branch's five changes onto it, rather than by editing the conflict
markers: the incoming change deletes an op, and a hunk-level resolution is how a
deletion gets silently un-deleted.
zipdoc_gen.go is generated — resolved by regenerating from the merged source, never
by merging the artifact. Same for openapi.yaml and plugin/risk/openapi.json.
Verified after the merge: 11 operations before and after, floor.json identical to
main (risk 31, paths 1695, operations 2351), and main's own new gates — verb_test
and learn_cost_test — pass beside this branch's.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Regenerating every subset from source is the drift gate, and it turns up drift in two
apps beyond the ones whose prose this branch fixes. Both are real, and both are in the
direction of the published document being AHEAD of the code: it advertises money fields
the runtime stopped returning.
7b4ddd9e removed the mint from the referral read and 3a8be85b removed it from the promo
redemption. Both regenerated their zipdoc_gen.go, so the PROSE moved; neither
regenerated plugin/<app>/openapi.json, so the SCHEMAS did not. The result is 15
referral fields and 4 promo fields published as response properties no handler can
populate -- refereeGrantCents, referrerGrantCents, creditsEarnedCents, refereeBonusCents,
referrerBonusCents, grantedCents, creditsCents, the credited counters and the txn ids on
the referral side; creditCents and creditEntryId, plus the plan and seats request fields,
on the promo side. apps/referrals' assertNoMoneyKeys already fails the runtime response
if any of them reappears, so source and test agreed with each other and only the artifact
disagreed. Two fields the code does return -- Redemption.discountCents and
sweepResult.qualified -- were missing for the same reason.
Every SDK, the MCP tool list and the CLI are projections of this file, so those were dead
money fields in every generated client's types.
The operation-count ratchet did not and could not catch it: openapi/floor.json counts
paths, operations and per-product operations, and none of those move when a response
schema loses a field. It ratchets UP here -- billing 26 -> 27, operations 2351 -> 2352 --
for the newly described GET /v1/billing/tier.
Proved structurally rather than by diff, over the parsed documents with description and
summary elided at every depth: across openapi.yaml the only identity change is that one
operation GAINED, no operation was lost, no operationId moved, and all 22 contract
differences are the marketing and referrals schema fields named above. The commerce prose
entry contributes none of them.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The repo's own gates could not pass on main. Both failures are the same shape --
prose that drifted from the routes it describes -- and both are fixed at the one
place the prose is written.
GET /v1/billing/tier published an operationId and nothing else, so openapi.Complete
refused commerce's document outright and `make -f mk/fleet.mk surface-check` died
there. Its handler is commerce's, in another module, and its path reaches the router
through a table rather than a literal, so there is no doc comment here for zipdoc to
lift -- which is what describe.go exists for, and what every sibling on that same
registration loop already uses. The entry states what the caller gets, that the
subject keys are pinned to the validated caller before the handler runs, that tier
is derived from active and trialing subscriptions, and the two rules a reader
otherwise gets wrong: gate on effectiveAvailable rather than prepaidAvailable,
because granted credits spend too and an account funded only by a grant reads zero
prepaid while holding real spendable credit; and a subscription-store error answers
500 rather than downgrading a paid subscriber to free.
POST /finance/starter was the inverse -- prose describing an operation that no
longer exists. The op, its middleware and its tests went with the automatic
money-mint (41b23f12) and the lift was never rerun, so commerce's registry still
explained a route the router does not carry. Regenerating drops it; nothing here
re-adds it.
POST /sites/resolve and /sites/resolve-org are live typed ops in
apps/projects/sites_rpc.go whose doc comments had never been lifted at all.
Verified per package, the way the generator loads: zipdoc -check is clean across all
100 directories that declare it, where commerce and projects were stale.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
THE MODEL WAS A PLACE. The `model` table is keyed on the tenant and written ON
CONFLICT DO UPDATE, so it is ONE CELL per organisation and every write destroys
the state before it. That single fact was three separate open problems: there was
nothing to roll back TO, so rollback needed an op that shipped the masses out to
the caller and an op that took them back in; two fitted models could not both be
named, so champion-and-challenger had nowhere to live; and no decision could say
which model produced it.
A content address ends all three. The value's name is a pure function of what
makes two models answer the same event differently — the shape, the geometry seed,
the position in the window, the threshold, the masses as IEEE-754 bits, and the
FOLD WATERMARK, which is not redundant with the mass count: two models with
identical masses reached by different routes disagree about what is left to fold,
and one will re-teach history the other will not.
THE IN-PROCESS MODEL STAYS MUTABLE, and that is the design. Measured: one encoded
value is 466 KiB at the reachable shape — 25 trees of 511 nodes over two windows,
every mass carrying a full mantissa because folding blends them. The sweep writes
every 500 events or every 30 seconds, so "a value per write" is 56 MiB an hour per
active organisation to record a counter going up. So identity is a SUCCESSION OF
STATES: the `model` row stays a place and its whole job is resuming a killed
process, and `published` is append-only, addressed by content, retained under a
budget stated in BYTES (10 values, following policyBudget's reasoning).
WHAT THIS DELETES. riskSnapshotBody is gone from the wire in BOTH directions, with
its two conversions. Publishing answers with a NAME; adopting takes one. The
masses never leave the organisation's own store, so the caller stops being the
custodian of a tenant's model — the engine's own Restore asks for exactly this
("it belongs where the tenant's own data belongs, and sealed if it travels"). Two
tests changed from refusing a threat to proving it unreachable: a caller can no
longer describe a model instead of naming one, so there is no body to compose and
no seed to choose. Rollback is naming a prior address; what the working state
descends from is DERIVED from its mass count, so rolling backward is right for
free where a stored head pointer is exactly what would fall out of step.
A SCORE NOW NAMES THE MODEL SPACE IT RAN IN. The shape is cached on the residency
and read in the same critical section as the verdict, so it costs nothing and
cannot cite a space the score did not run in. It is deliberately NOT an address:
the masses at the instant of a score are counters between two published values, so
citing one would claim that value produced this score. The shape, the policy
version and the event's own time are what IS true, and the value history's clock
brackets the decision from there.
ISOLATION, MEASURED RATHER THAN ASSERTED. The address deliberately omits the
organisation — a name that must be unguessable is obscurity, not isolation. Two
mutations were run and two drafts of the test comment were wrong before they were:
the per-organisation FILE and the `tenant = ?` row predicate are two layers and
EITHER ONE ALONE HOLDS, so a foreign org handed a real address resolves nothing
through all three doors with either layer removed. The single-layer regression is
caught where the file layer is already absent by design — two brands' identically
named organisations share one file — and that test fails on that one mutation
alone. The finding underneath inverts the obvious reading: the per-org file is not
what makes this isolated; the row predicate holds on its own.
11 operations before, 11 after: no path moved and the floor is untouched. The two
paths SHOULD collapse into one (/v1/risk/state/model, POST to publish and PUT to
adopt) — that is the prefix plane's call, not this commit's.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/risk/learn recorded a batch, trained on it AND answered the model's
verdict on every event. Three things under one name: you could not observe
without training, and could not train without being answered.
An observation is a VALUE the plane records; learning is a TRANSFORMATION over
observations; a verdict is a QUERY against the result. POST /v1/risk/score is
already the query and is already pure. So learn drops the verdict and answers
`learned` — how many events the model actually learned from.
WHAT IT COST TO CARRY. plane.learn called the engine twice per event, Inspect for
the response's verdict and Assess for the counters, and both enter the engine's
judge: two projections of the point at three aggregate reads each, and two walks
of the forest. Above the cut both also ran the counterfactual attribution, a
further walk per dimension over nine dimensions. Assess's own return was
discarded, so the attribution was computed twice and thrown away once.
MEASURED, and stated as measured (learn_cost_test.go, BenchmarkLearn):
before after
batch 8 31.5 µs/event 27.8 µs/event -12%
batch 128 20.0 µs/event 17.5 µs/event -12%
batch 128 6717 allocs 6169 allocs -8%
A tenth, not a half: the durable record and the aggregates are the larger part of
what a caller waits for, and the attribution is reached only by the share of the
stream the appetite admits — one per cent by default.
NOTHING DEPENDED ON THE SYNCHRONOUS VERDICT. No CLI command references risk, no
SDK carries a risk client, and the one in-process consumer of a verdict is
cloud.Decide, whose scorer is never installed outside tests (SetRiskScorer has no
non-test caller), so every question it asks answers {allow, scorer-absent}. The
live plane reports one resident model built once. Observe-and-judge in one round
trip is now a COMPOSITION and the published prose says which order: score first,
then learn, so the verdict is the model's opinion of an event it has not yet
learned from.
A DUPLICATE IS NOW WHOLLY INERT. It moved nothing before and was still judged;
now it costs no model work at all, is not counted in `learned`, and is not
metered — the meter runs on what was DONE, which is this app's own stated rule
for the gate/meter pair. The gate still bounds the batch the caller stated,
because how much of it is new is unknowable until the record is written.
TestPlane_TheVerbsStaySeparate (verb_test.go) walks the package and fails if
learn calls Inspect or score calls Assess. It is structural because the braid is
invisible behaviourally: with Inspect put back, every other test in this package
still passes.
zipdoc regenerated; plugin/risk/openapi.json and openapi.yaml rewoven
mechanically (make -f mk/fleet.mk openapi-weave, exit 0). riskScoreOut, riskCause
and riskValue stay in the document — score still answers them; they are simply no
longer reachable from riskLearnOut. Package green under -race.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/risk/state answered seven kinds of thing at once and PUT
/v1/risk/state/appetite answered all of them back to a call that changes three
numbers. This lands the half whose home already exists: the decision regime is
read and written at ONE address, /v1/risk/policy, and both verbs answer
riskPolicyOut. The published surface loses a path and keeps its operation count.
plane.appetite no longer computes the model value — r.mod.State and r.vel.strain
are gone from the policy write. What the model IS stays for the model-value track;
the telemetry kinds (refusals by reason, blind by feature, the realised share,
saturation, aggregate strain) stay on GET /v1/risk/state until they are emitted on
the fleet's one metric road, because deleting them before that would remove an
organisation's only view of its own refusals.
Per-organisation isolation is proven through the new address by the two policy
tenancy tests. Mutation table in the commit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The regime was written at PUT /v1/risk/state/appetite and read at GET
/v1/risk/policy — two addresses for one plane, the writing one named after the
mutable spot the regime happened to sit in. `state` is literally the word for
that spot; a regime is not a spot. It is three numbers an organisation adopted,
at a time, by a named identity, kept under a version forever after.
Worse than the second address is what the write ANSWERED: riskModelState, fifteen
fields describing the whole MODEL — its shape, what it had learned, whether it
was warm, the threshold in force, the realised share beside the stated one, every
refusal by reason, every feature that read blind, the fold coverage of the event
surface, and the aggregate strain. A call that changes three numbers knows none
of that. It was reporting a model it happened to be holding a lock on, which is
the same braid, one level up, as the regime living on the learned state's row —
the defect the versioned policy record was cut to fix.
So the plane has ONE address and both verbs answer riskPolicyOut. plane.appetite
returns the VERSION it left in force and nothing else: the two calls to
r.mod.State and r.vel.strain are gone from the policy write, so a policy write no
longer computes the model value at all. policyOut is the one projection both verbs
render through, and the version in force is a PARAMETER to it — a write knows what
it enacted from inside the lock it enacted under, and re-reading it there would let
two concurrent restatements each report the other's version.
There is no `minted` flag. plane.enact is idempotent on the regime, so a
restatement answers the version already in force and equality of the VALUE is the
signal. A boolean about the operation is a second thing to keep true beside the
value that already says it.
The published surface SHRINKS by one path and holds its operation count: the write
joined the address that already existed instead of keeping a second one. Mechanically:
paths 1696 → 1695 (-/v1/risk/state/appetite)
operations 2351 → 2351 (PUT moved onto /v1/risk/policy)
openapi/floor.json is lowered in this commit because the weave refuses a shrink
otherwise, which is the gate working: a reduction is reviewed next to its reason.
The op's published schema description loses 27 lines of riskModelState /
riskSurface / riskAggregates prose it had no business publishing.
Nothing consumed the old address. Swept every checkout under ~/work: the only
references outside apps/risk's own tests were the derived specs. Live, PUT
/v1/risk/state/appetite is routed and GET /v1/risk/policy is 404 — the policy
plane is merged but not yet deployed — so this is the one moment the consolidation
costs a deployed client nothing.
Per-organisation isolation is untouched and proven through the NEW address:
TestPolicy_HistoryIsPerTenantOverTheWire and
TestPolicy_TwoBrandsShareAFileAndNotAHistory both state their regimes at PUT
/v1/risk/policy and both still see nothing of the other organisation.
Mutation table — each named test RED under the defect, GREEN after revert:
M1 re-register PUT /v1/risk/state/appetite TestPolicy_HasOneAddress RED
M2 put a model field back on the answer TestPolicy_AWriteAnswersThePolicyAndNotTheModel RED
Verified: go build -tags "sqlite_purego sqlite_math_functions sqlite_fts5" ./... =
0; go vet apps/risk = 0; go test apps/risk = ok; -race = ok (23s); zipdoc -check
./apps/risk/ = 0. Fleet-wide zipdoc -check reports the same 16 stale/missing
generated files before and after this change (measured by stashing it), so this
adds no drift: apps/risk is not among them.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/iam/* was forwarded to a separate identity origin by an edge in cloud:
a tenant gate, an org pin, a read/write allowlist and a sign-in passthrough,
all re-deciding what IAM already decides for itself.
It mounted on `!cfg.Enabled("iam") && iamHost() != ""`. An empty enable list
mounts everything (Config.Enabled), and no deployment in the fleet sets one,
so Enabled("iam") is true everywhere and the condition never held: the iam
app owns /v1/iam through its graft, on every deployment, and has since the
graft landed. The edge answered no request anywhere.
Deleting it removes the second reader of the identity address and the second
place a tenant scope is decided. The prefixes are unchanged — manifest's iam
row already routes them — and the org pin the edge applied is IAM's own
authorize, in the process that owns the store.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Removing CLOUD_ENABLE/--enable (ecafb31c) deleted the flag from the binary but
left three launchers passing it. Go's default flag.CommandLine is ExitOnError,
so this is not a warning:
$ ./bin/cloud --enable=iam
flag provided but not defined: -enable
`make run` and the documented compose quickstart both died before boot, and the
four README brand examples were copy-paste instructions to do the same. The
removal was deliberate and test-enforced (cmd/cloud/mount_test.go asserts the
flag's absence) — the launchers were simply never updated with it. My miss.
RUN_ENABLE is renamed RUN_PLUGINS because it never mounted anything: it is the
list of plugin BINARIES to build so local dev does not build all 106. The host
mounts what manifest.Apps lists and resolves each plugin as a file beside
itself; a lazy one with no binary never starts, a Required one fails loudly.
Keeping the name would have said the binary takes a mount list, which is the
thing that took devnet down twice.
compose.yml's HANZO_ENABLE goes with it. Its environment: block is inert
anyway — HANZO_BRAND/DOMAIN/DATA_DIR have no Go read sites and work only
because the same file passes the flags, which main.go forward() republishes as
CLOUD_*. That is a separate cleanup, not this one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/risk answered ONE refusal in TWO shapes, with the same status and the same
sentence in both. Measured on this package's priced ops:
this app's own copy of the rule 403 {"status":403,"error":"no validated principal"}
the fleet's money door 403 {"error":{"code":"forbidden","message":"no validated principal"}}
The nested one is canonical — it is what the edge gate and every other Hanzo
surface emit, so a client that reads `error.code` reads it everywhere. The flat one
was zip rendering a returned error, which meant `error.code` was absent from
exactly one product's refusals.
[ops.gate] held its own `if ledger == ""` copy of the rule from before the fleet
door had one. [cloud.ResourceMeter.Gate] now refuses an empty org above both of its
branches as [cloud.ErrNoLedger], and [cloud.denial] renders that as the 403 with
this exact sentence, so the copy decided nothing except the shape. Deleted; the
twenty-four lines of prose explaining a defect that is fixed elsewhere are replaced
by a citation of where.
Deleting it exposed a residual the report did not name, because a status assertion
cannot see a shape: POST /v1/risk/search reaches [ops.admit] BEFORE it prices
anything, so its refusal came from [tenantOf] — still flat. Eight ops nested and
one flat is worse than nine flat, so tenantOf's no-principal branch answers with
the same fleet value. A malformed org is a different fact and keeps its own
sentence.
And the hole the deletion could have opened, closed: ResourceMeter.Gate returns
EARLY at zero cost, before its own empty-org refusal, and the price is an operator
knob where 0 is legal. So "the money door refuses an unidentified caller" holds
only while somebody is charged; the app's deleted copy had covered that by
accident, running before the price was computed. tenantOf covers it on purpose, and
a test sets the price to zero and requires the same 403.
E1 risk holds its own copy again TestPricedOps_RefuseAnUnidentifiedCallerInTheFleetsOwnEnvelope RED
E2 tenantOf answers flat again TestPricedOps_RefuseAnUnidentifiedCallerInTheFleetsOwnEnvelope RED
E3 tenantOf answers flat again TestPricedOps_RefuseAnUnidentifiedCallerEvenWhenTheOperatorPricesThemAtZero RED
E4 the fleet door stops refusing TestPricedOps_ResolveTheTenantBeforeTheyAskForMoney RED
E5 identity takes a money status TestPricedOps_ResolveTheTenantBeforeTheyAskForMoney RED
All GREEN after revert. E4 and E5 mutate cloud's own money door, which is where
gate_order_test.go's mutation note now points — the note said "delete the
empty-ledger guard from ops.gate and every op reports 503", and that had silently
stopped being true.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The label row said a specific prefix MUST be registered before the bare /v1/risk
or "every label op lands on the decision plane", and cited
TestSpecificPrefixesPrecede as the gate that pinned it. Neither half was true.
No test of that name exists anywhere in the tree, and the rule is not the
router's: registering the bare /v1/risk FIRST and all three specific prefixes
after it, the live router still delivers /v1/risk/labels to label,
/v1/risk/reference to reference and /v1/risk/datasets to dataset, and the router
oracle stays green. zip resolves nested static prefixes by specificity; mount
order decides only between EQUAL patterns, which is what the ai-before-zen note
is actually about.
Says that, and names the gate that does hold —
TestEveryServedPathReachesTheAppThatServesIt builds the real router from these
rows and asks it, per published path, which app receives the request. An oracle
over every row at once is why no row needs a rule of its own to remember.
Prose only; no row moves and no behaviour changes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The dataset plane addressed /v1/ml/datasets, beside the KServe model-SERVING
plane's own /v1/ml/health and /v1/ml/models — one prefix, two products.
openapi.Product reads an operation's product off the FIRST /v1 segment and
nothing else, so `ml` published 14 operations that were seven serving ops with
customers on them and seven dataset ops whose rows feed the risk model, which
learns in-process from the org's own events and is never served by KServe.
Moves the five paths to /v1/risk/datasets and renames the seven operation ids
and nine schema names to the risk face, so the SDK method, the CLI command and
the generated type each name the product they belong to. Fourteen of the fifteen
Go types take the bare risk<Noun> name; the dispose pair carries the noun it
disposes of, because apps/label already publishes riskDisposeIn for LABEL
disposal and the fleet's schema namespace is flat.
The tag was never the product: openapi.Fold assigns op.Tags from the router
projection, so zip.WithTags("ml") had been inert. It now reads "risk" too, so a
declaration and a projection do not disagree.
/v1/ml is untouched. Its four serving paths (7300 bytes) and its three schemas
(1533 bytes) are byte-identical, and regenerating the fleet document from source
moves exactly the seven dataset operations and the nine dataset schemas: 2350
operations, 1695 paths and 2060 schemas before and after, with every other
triple unchanged.
No alias and no redirect from the old prefix, because nothing calls it: a sweep
of every git checkout under work/hanzo finds /v1/ml/datasets named only in this
repo's own generated artifacts and prose. No client, SDK, CLI or MCP consumer
names it.
The floor ratchet refuses a shrink, so `ml: 14 -> 7` is lowered here in the same
commit that moves the routes; `risk` rises 23 -> 30 on the same regeneration.
address_test.go makes the address a gate rather than a third recollection after
label and reference: the product, the operation-id prefix and the schema prefix
are each asserted off the published projection, and each assertion fails when
the projection is empty, so none of the three can pass by examining nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
kserve admits an InferenceService whose model format no ClusterServingRuntime
supports and then never schedules it, so /v1/ml/health answered 200 while every
deploy hung. A served CRD is not capacity, and the probe read only the CRD.
health now takes the cluster-scoped coordinate the plane needs at least one of
(the zero GVR for a plane with no such fact — training carries its own images)
and reports the count as its own field. An unreadable list reports the read
error instead, because a missing grant is a broken probe and not an empty
cluster, and the two call for different acts.
scripts/mutate.py carries four rows: dropping the clause, folding the read error
into the count as zero, asking capacity of training, and reading the runtime kind
at the InferenceService's v1beta1 instead of v1alpha1.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The bounds on `review` and `sample` were spelled twice — once at [ops.appetite]
and once in [admitRegime]. Collapsing them to one spelling is right. But the
SURVIVING spelling had no test: admitRegime was made to `return nil`
unconditionally and the WHOLE package stayed green, so the published contract
(`review ∈ (0, 0.5]`, `sample ∈ [0, 1]`) was held by code that could be deleted
without one failure. Same shape as a control switching itself off, one layer down:
the bound is present and nothing measures it.
Both halves are asserted, because a refusal that half-applies is worse than no
bound: six out-of-contract appetites are refused 400 with the field named, AND the
regime in force is untouched afterwards — no version minted, no history row, the
model still deciding under 0.02/0.10. Over the WIRE, because that is where the
contract is published and where the op's deleted copy used to answer.
A1 admitRegime returns nil for every regime named test RED, whole package RED
A2 plane.enact stops calling admitRegime named test RED, whole package RED
Both GREEN after revert.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An organisation that took its model out of shadow BEFORE the model had learned
anything was told live=true and had nothing written down. The regime lived on the
same row as the learned state, and that row's writer declines to write while the
snapshot holds no learned mass — correctly, because there is no state to lose. So
PUT /v1/risk/state/appetite answered 200, reported live, and persisted nothing;
this binary deploys Recreate at one replica, so the next rollout rebuilt from
defaultConfig — shadow — and the model decided nothing. No error, no log, nothing
to alert on: a model silently disarmed, on a routed door.
Two conflicts, both resolved by REGENERATING rather than choosing a side:
openapi.yaml and openapi/floor.json are derived, and the branch was cut when the
fleet had 1684 paths. `make -C apps/risk describe` + the weave produce 1696 paths
and risk 24 operations — main's 1695/23 plus this branch's one op, GET
/v1/risk/policy. apps/risk/zipdoc_gen.go and plugin/risk/openapi.json regenerate
byte-identical to the branch's, so the projection did not drift.
One semantic conflict: `plane.close` gained the shutdown window on main
(cda5a031), so the branch's four `close()` calls in policy_test.go take
context.Background() like the other eighteen in the package.
Mutation table re-run against THIS merge, since main moved under the branch —
each named test RED under the defect, GREEN after revert:
M1 plane.enact writes no durable row TestPolicy_GoingLiveSurvivesTheRollout RED
M2 verdict drops the version citation TestPolicy_EveryScoreCitesTheRegimeItWasDecidedUnder RED
M3 no value-equality short circuit TestPolicy_ARestatementOfTheSameRegimeMintsNoVersion RED
M4 the 24-per-24h bound deleted TestPolicy_TheRateBoundBindsAndIsNamed RED
M5 a pre-existing regime is not adopted TestPolicy_ARegimePredatingTheRecordIsAdopted RED
M6 the history read loses its predicate TestPolicy_TwoBrandsShareAFileAndNotAHistory RED
M7 retention stops disposing TestPolicy_RetentionIsCountedAndNotSilent RED
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Four defects on the LIVE /v1/risk plane, each closed so the wrong thing is
unrepresentable rather than merely unlikely, plus the harness rows that hold
them.
THE ORG STORE. CloseAll emptied its maps, which made "closed" indistinguishable
from "nothing opened yet" — so a request still in flight during a rollout
reached For() after shutdown and opened the file again, re-hydrating it and
re-claiming the fence lease the SUCCESSOR pod was claiming. Closing is now
terminal and every door answers ErrStoreClosed.
THE ROLLOUT. close() was `stop(); wg.Wait()` with every save queued BEHIND it.
The wait was unbounded inside a 30-second window while ONE search's durable
Sync is bounded at 30 seconds on its own, so the process was killed with not
one tenant's model written down — every tenant back to warming, and a warming
model refuses to score, which reads as clean. The saves now run
unconditionally; the drain gets what is left of the caller's own window; a
drain that did not finish is ErrDrainIncomplete and not a silence.
THE SEARCH BOUND. "ONE RUN PER TENANT" checked the slot and set it two
warehouse operations later, so sixteen concurrent callers each rolled the
tenant's source planes and read its whole history before any of them claimed
anything — and the ledger gate was the LAST thing in the sequence, so all of it
was free. The slot is claimed atomically with the check, and a search is priced
in TWO halves, each before the half it prices: the surface on the window the
caller stated, the grid on the measured history.
THE STRAIN REPORT. velocity caps PER SHARD, so the store drops a tenant's
subjects long before the flat census notices: 200 subjects in, the store holds
198, and the report said forgotten=0, saturated=false. Two of that
organisation's own subjects read as "has done nothing" and nothing said so.
reconcile now measures the store/census shortfall as a high-water mark, and the
report describes the bound that actually binds.
THE RESIDENT BOUND had no test at all: evict() could be made to return nil —
the bound fully disarmed, an OOM on a one-replica Recreate deployment — with
the whole suite green. Its operating point is now four properties held
together: SERVED past the bound, BOUNDED at it, LOUD on the probe, LOSSLESS on
the way out.
Mutation-proved: scripts/mutate.py "risk:" — eighteen rows, each reintroducing
one defect and going red on the named test.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The ceiling was the wrong way round. maxOverrides was a chosen number (10,000)
and the byte figure was computed beside it from `row` — the SUM OF THE WIRE
TERMS, 1,152 bytes. A row does not cost what it carries: measured on a real
per-org store, the widest row this door admits costs 1,952 bytes once the
implicit index over the primary key, page slack and the encryption are counted.
So the published 120 MiB per-organisation ceiling was understating the truth by
1.69x, on the ONE volume every organisation's store shares.
Inverted, so the bound is in the dimension that binds:
ownBudget = 128 MiB the primary figure — what one org may occupy
rowBytes = 2048 MEASURED (1,952) plus a page of room
maxOverrides() = ownBudget / (sets x rowBytes) = 5,957 per set
The count is now the division rather than a number standing next to one, so
count x rowBytes IS the byte bound. Adding a set lowers the count instead of
quietly multiplying the volume.
`row` is renamed `stated` and keeps its old meaning — the WIRE width — because
the two figures are different quantities and conflating them is what caused
this. TestTheVolumeOneOrgMayOccupyIsStated now checks the quotient (one more
entry per set must not fit) instead of pinning a constant, and
TestOneOrgsOverridesCostWhatTheyArePublishedToCost fills a real store with
worst-case rows and fails if one costs more than the figure the count is divided
from — so rowBytes can never drift from a store again.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The header claimed "nothing held the mechanism at all — the whole suite stayed
green". That was true when it was written and is not true now: hold_test.go
landed the cross-tenant reclaim tests on main, and MEASURED with
scripts/mutate.py, TestEviction_IsCountedOnTheProbe and
TestEviction_WritesTheVictimsStateDownFirst kill all three mutations that disarm
this bound. A test whose stated reason for existing is false is a test the next
reader deletes for the wrong reason, or keeps for one.
What it actually adds is two things the pair cannot see: the bound asserted at
every one of maxResident+8 arrivals rather than once just past it, and a lossless
leg with no t.Skip exit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The surface read — rolling up to four source planes into the organisation's own
feature surface and reading the window back — ran BEFORE ANY GATE AT ALL. A
caller with no balance drove the whole warehouse cost, was refused at the very
end, and paid for none of it, as often as it cared to ask. Two rows: the gate
moved back below the work it prices, and the surface half deleted so only the
grid is paid for. Each goes red on the test that names the property.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Sixteen rows for the org-store lifecycle, the rollout, the search bound, the
strain report and the resident bound — so every assertion this cycle added is
held by a mutation that breaks the thing it guards, in the harness rather than in
a transcript.
AND THE HARNESS COULD DESTROY ITS OWN EVIDENCE. The restore is in a finally,
which does not run when the process is killed — a CI timeout, a ^C — so an
interrupted run leaves the mutant in the tree and the original in .mutbak. The
next run then copied over that backup, destroying the only clean copy, and
reported ANCHOR-MISS on a tree it had silently corrupted. That happened here and
cost a file. A leftover backup is now a hard refusal that says how to recover.
One row is written against the CONDITION rather than as an early return, because
`return nil` after the guard is unreachable code: go vet rejects it, the row
scores NO-COMPILE, and a mutant that never ran proves nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The run slot now bounds one tenant's CONCURRENCY. Nothing bounded its RATE: the
setup ran before any gate at all — the ledger check was the LAST step of begin —
so a caller with no balance drove a roll of up to four source planes and a
full-window read on the one warehouse every tenant shares, was refused at the
end, and paid for none of it, as often as it cared to ask.
Pricing the grid on its upper bound would close the same hole and leave no
viable operating point: the upper bound is maxHistory x the whole grid whatever
that organisation's history holds, so a tenant with two hundred events would be
refused unless it could cover twenty thousand.
So a search is priced TWICE, each half before the half it prices. The surface
from the WINDOW, which the caller states and which is therefore a number before
anything runs — the same unit and the same price ops.features already pays for
the same work, now spelled once in windowScreens. The grid from the MEASURED
history, where it already was. Each meters on what it actually did, so a run
cancelled by a rollout is still billed for the trials that ran.
The plane takes ONE money seam (charge: gate n, get the meter for what was done)
instead of a gate parameter and a book parameter, which is what makes "gate
before, meter after" the same shape at both halves rather than a convention.
And the debit test now waits for BOTH debits. With a synchronous surface debit
landing first, waiting for one was satisfied without the background meter ever
running — the fixture would have made the property it exists for unobservable.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
velocity applies its cap PER SHARD — MaxKeys/shards+1, which is FIVE keys against
a published census ceiling of 320 — and subjects hash unevenly, so a shard fills
long before the total does and the store drops that shard's least-recently-used
key. The census is a flat count and saw none of it. Measured:
subjects=200 storeKeys=198 censusLen=200 forgotten=0 SATURATED=false
subjects=320 storeKeys=290 censusLen=320 forgotten=0 SATURATED=true
At 200 subjects two of that organisation's own subjects are gone and the model
reports itself healthy; each one reads as "has done nothing", scores as
unremarkable, and raises nothing. At 320 the state is right by luck and the COUNT
an operator acts on still says zero while thirty subjects are gone. The published
8 MiB budget buys 320 subjects and the bound that binds starts biting near 200 —
a bound stated in one dimension and enforced in another.
reconcile already read the store's own key count and threw the comparison away.
It now measures the shortfall against the census under ONE lock, so the two
numbers describe one moment, and keeps it as a HIGH-WATER MARK: a dropped subject
is re-admitted the moment it is active again, which closes the shortfall but does
not un-blind the window in which that subject read as inactive. A gauge would
report that a control which switched itself off never did.
ALSO: the resident bound had no guard at all. `evict` could be made never to trip
and the whole suite stayed green — the bound fully disarmed, residents growing
without limit, which is 64 x (8 MiB of rings + its model) against a 9 GiB
GOMEMLIMIT on a ONE-replica Recreate deployment. A bigger constant was never the
fix; the operating point is four properties at once, and each is a way to be
wrong: SERVED past the bound (a refusal is a cliff, not an operating point),
BOUNDED at every step, LOUD on the probe (eviction is lossless, so the count is
the only sign), LOSSLESS (otherwise one tenant's arrival costs another its model,
which is the cross-tenant defect wearing a capacity hat).
Mutation-proved: scripts/mutate.py "strain"/"bound" — seven rows. Dropping the
store's half of Saturated reports "the store holds 142 of this organisation's 144
subjects and strain reports saturated=false". Refusing past the bound reports
"organisation 64 of 72 was refused a residency". Dropping the evicted tenant's
save reports "an evicted organisation came back having learned 0 of 120".
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TWO DEFECTS, both "a bound that is not on the dimension that matters".
THE ROLLOUT LOST EVERY TENANT'S MODEL. close() was `p.stop(); p.wg.Wait()` with
every save BEHIND that wait, and the wait had no bound at all — while Shutdown
was handed the shutdown window as a context and threw it away. The process gets
30 seconds (serve.go) inside a 60-second grace period, and ONE search finishing
after cancellation writes its result to a shelf whose durable Sync is bounded at
durableOpTimeout — thirty seconds, the whole window, on its own. So the wait
outlived the window, the pod was killed, and not one resident model had been
written down. Every tenant returned to warming, from one tenant's search, once
per deploy — and a warming model REFUSES to score, which reads as clean to
anything that does not check the refusal.
The saves now run unconditionally and never behind the drain; background work
gets whatever is left of the caller's own window, bounded by it and by
drainBudget; and a drain that did not finish is ErrDrainIncomplete — a named
state joined into the error, with the count of models written down anyway, rather
than a silence.
THE SEARCH BOUND BOUNDED NOTHING EXPENSIVE. begin() checked "is a search already
running for this organisation?" and set the flag that answers it two warehouse
operations later — check-then-act with the entire cost of a search setup in the
window. Measured: 16 concurrent callers for ONE organisation were all accepted,
all rolled that tenant's source planes, and all read its whole history, against
the one warehouse every tenant shares; 16 background grids then raced for one
shelf row. The slot is now claimed atomically with the check that grants it,
before any of that work, and released on every path that does not reach the run.
claim/settle/unclaim are one fact in one place — the background goroutine's
hand-rolled delete is gone with them.
Mutation-proved: scripts/mutate.py "rollout"/"search" — six rows. The original
close() ordering does not merely fail, it HANGS to the test timeout, which is the
production SIGKILL. Making the saves conditional on a clean drain reports "tenant
A came back having learned 0 of 300". Restoring the check-then-act reports "16 of
16 concurrent searches were accepted".
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
OrgStore.CloseAll closed every handle and then installed fresh empty maps, which
made "closed" indistinguishable from "nothing opened yet". Every one of the
fifteen callers is a Shutdown path, so a request still in flight during a rollout
reached For() after it and opened the file again — on a durable deployment
re-hydrating it and re-claiming the fence lease the SUCCESSOR pod was claiming at
that moment. Two live writers for one org, through a handle nothing thought was
reachable.
Closing is now terminal: the flag is set before the maps are cleared, so there is
no window in which the store is empty and still openable, and every door that can
open a file answers ErrStoreClosed. An arrival after shutdown is a fact worth
surfacing, so it is an error rather than a silent no-op.
The risk plane reaches this path by construction — its own close() empties the
resident map, so an in-flight score rebuilds a residency and asks for the shelf.
Mutation-proved: scripts/mutate.py "orgstore" — three rows, one per half of the
mechanism (the flag, the For guard, the Each guard). Each goes red on
TestOrgStoreCloseAllIsTerminal at a distinct assertion.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ownVolume() is what one org may occupy on the volume every org's SQLite file
shares: maxOverrides x len(Catalog()) x row, where row = maxKey + maxNote + 128.
The key and the note were bounded at the wire door. The third term was not: the
writer is actor()'s reading of the X-User-Id the request carries, so it was a
caller-sized value stored on up to 10,000 rows in every set the catalog
publishes. A count over caller-sized values is not a byte bound, and the stated
120 MiB ceiling was a figure nothing held to.
maxActor = 128 (three times the UUID IAM mints), enforced at the store door
rather than the wire op — the row is what the budget is about, so bounding it
where a row is written covers every path that reaches the store and not only the
one a reviewer read. row now DERIVES from it, so raising the bound moves the
ceiling instead of silently detaching it.
Refused, never shortened: the writer is who an adverse action is attributed to,
and a truncated identity names someone who does not exist. errActor separates
the two refusals put() can produce — the row cap is a 409 (the org already holds
what it holds), an over-long writer is a 400 (a value that arrived on this call).
TestTheWriterOnARowIsBoundedLikeEveryOtherTerm rides the wire, because the
defect was in what the wire hands the store: a unit test on put() would have
passed against a handler that never bounded the header. call() is now callAs()
with the default principal, so there is one request builder and not two.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/ is where mounted, billable subsystems live: 116 of its 139 directories
declare `func Mount(`. apps/money declares none, serves zero routes, and is 216
non-test lines that already wrap github.com/hanzoai/money (imported as `hz`) to
pin ONE unit — creditUSD, dollars at 18 decimals. That is a library, and filing
it under apps/ said it was a product.
Pure path move: apps/money -> money. No API change, 44 import lines rewritten,
0 residual references. Full tree builds (`go build -tags sqlite_math_functions
./...` rc=0).
Not deleted and not folded upstream: hanzoai/money is the general library
(multi-currency, rates, min/max/sum) and this is the platform's credit-asset
facade over it. creditUSD is Hanzo-specific and does not belong in a public
money library. One library, one facade, each in the right place.
Unrelated, found while verifying: `go build ./...` fails on
hanzoai/base@v1.5.11/core/sqlite_math_required.go (undefined
cgoBuildNeedsSQLiteMathFunctions) on a CLEAN tree too — it needs
-tags sqlite_math_functions. The default build command does not work in this
repo and that is worth fixing separately.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ResourceMeter.Gate is the one pre-create gate all 23 priced surfaces call, and
handed an empty org it asked the money plane to price a spend for a nameless
subject. The money plane then answered a question about IDENTITY in the
vocabulary of MONEY, in two measured shapes:
co-resident ledger — metering refuses an empty org fail-closed, which is not a
4xx, so the wire's fallback renders 503 "Billing temporarily unavailable":
the caller is told the biller is broken.
peer ledger — gatePeer ships AuthorizeIn{Subject:""}, commerce's own
validate:"required" rejects it, and because that refusal IS a 4xx the wire
preserves it verbatim: 400 `field "subject" is required` — a field that
appears in no published request schema on any of these surfaces, so no
caller can ever satisfy it.
apps/risk already carried this guard at its own door; the same disagreement is
reachable wherever a handler's tenant check and principal.Ledger differ.
provisioning is the live instance: create() admits an admin with no org, and an
org over MaxOrgLen, through tenant() — and Ledger answers "" for both. Its seven
routes serve on api.hanzo.ai today.
An empty org is now refused as ErrNoLedger before either branch runs, and denial
— the one decision both renderings read — renders it as the tenant gate's own
403 rather than as a fault of the biller. This changes no allow/deny outcome on
a named caller: an empty org already failed on both branches. It does close a
worse case than the wrong status, proven by the mutation below: under fail-OPEN
with the biller down, an unidentified caller was allowed, so the surface was
free rather than gated.
Mutation-proven via scripts/mutate.py:
gate: ask the money plane to price a spend for a nameless subject KILLED
gate: render an identity refusal as a fault of the biller KILLED
risk: widen the empty-ledger guard until it refuses every caller KILLED
Two rows are deliberately absent and say so in place: "guard below fail-open"
is a semantic no-op (fail-open lives inside Authorize/gatePeer, so no placement
in Gate can follow it), and "delete ops.gate's guard" now SURVIVES — with Gate
refusing, the risk-level guard is no longer load-bearing for the status or the
sentence, only for the envelope.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
plugin/<app>/mcp.json stopped being a generated artifact when the door started
asking each subsystem for its tools at the moment it is asked (package fleet,
mk/plugin.mk describe). The committed copies were the exact hazard that change
removed: plugin/o11y/mcp.json held 12 tools while the o11y binary served 365,
and nothing compared them.
plugin/dataset/mcp.json was the last one left — no go:embed names it, no Go
source opens it, and every remaining mention in the tree is prose recording that
the mechanism was retired. Removing it leaves 0 of 124 app dirs carrying one, so
"the tool catalogue is not an artifact" is a property of the tree rather than a
sentence in a comment.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The last attempt at this served a plane op from deps.AI, which in the `ai`
process is the HTTP gateway to api.hanzo.ai — so it added a hop instead of
removing one, and was reverted. The premise was right and the mechanism was
wrong. zip states the mechanism itself:
"Because it is an ordinary route on the app, it rides EVERY transport the
app Listens on with no extra wiring — ZAP over a unix socket is simply the
address the caller dialed."
`ai` already answers its whole /v1 surface on $ZIP_RUNTIME_DIR/ai.sock. Nothing
in hanzoai/ai has to change and no second inference API has to exist: a sibling
speaks the SAME OpenAI-compatible wire to the SAME routes, over the socket.
before sibling --HTTPS--> Cloudflare --> ingress --> api.hanzo.ai --> ai
after sibling --unix----> ai
AIHTTPOn / AIHTTPM2MOn state the TRANSPORT separately from the address, so
there is still ONE inference client; only the route it travels differs. The
M2M token exchange keeps the default transport — IAM is a different peer and
naming it is a separate question, still open.
aiRoute is the one decision both pickers share, so completions and embeddings
cannot disagree about where the peer is. It is answered by WHAT THE PROCESS IS:
!Enabled(ai) means this process does not carry the app, which is exactly when
`ai` is a sibling. The ai process and the host keep the configured address —
routing inference back through the picker there would be a self-call, and
TestTheAIProcessDoesNotDialItself pins it.
The socket transport WAKES the peer through the same reach() every plane call
uses, so a cold lazy app is "not started yet" rather than "not deployed here".
CLOUD_AI_ZAP_ADDR is deleted with its field: it made a deployment state where
its own code lives, which is the thing being removed. CLOUD_AI_BASE_URL now
only describes inference that is genuinely elsewhere — and on the Hanzo
deployment it is no longer consulted, because that pod carries `ai`.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The app was renamed to /v1/risk/reference — openapi.Product reads the product
off the first /v1 segment, and these six operations are the risk product's —
but the committed projection and the woven document were generated before the
rename and still published /v1/ml/reference. manifest's
TestEveryServedPathReachesTheAppThatServesIt is the gate that caught it: four
paths the fleet published and routed to the /v1 catch-all instead.
Regenerated plugin/reference/openapi.json from the app's own live router and
re-wove openapi.yaml from the subsets. The delta is exactly the four reference
paths; no neighbour moved.
plugin/reference/mcp.json goes with it. The tool catalogue stopped being a
generated artifact when the door started asking each subsystem for its tools at
the moment it is asked (package fleet) — 122 of 124 apps ship no such file, and
a stale one beside a regenerated projection is a second answer to a question
that has one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The coordinates are the part worth proving, and ai's own tests cannot reach them.
ai declares the store structurally (object.SecretStore) precisely so it needs no
KMS dependency — which means its tests can only use a fake, and a fake proves the
logic while proving nothing about where the bytes land.
A bare ref resolves to path "/", which fileOrg treats as the FACADE and lands in
the deployment's system partition. The REST surface folds the caller's org and
lands the same name at /orgs/{org}. Those are different databases. Write through
one door and read through the other and the secret is simply not there: no error,
no warning, just a key the gateway cannot find. This asserts the admin seal and
the completion-path resolve use one door.
Second test pins the migration's direction: env serves until a key moves into KMS,
and KMS wins after. Without it "migrate the key" could be a no-op that looks
complete.
Verified on Linux, where libsqlcipher is linked and the store actually opens.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The commit claimed a sibling would stop reaching `ai` through Cloudflare. It
does not. In the `ai` process cfg.Enabled("ai") is true, so deps.AI is the HTTP
gateway pointed at https://api.hanzo.ai/v1 — the pod's own public address — and
the plane handler I added served inference THROUGH it. The path became
sibling --UDS--> ai process --HTTPS--> api.hanzo.ai --> airouters
where before it was one HTTPS call. Strictly worse, and the opposite of what
the message said.
The premise was right and the wiring was wrong: `ai` exposes its inference as
HTTP ROUTES (airouters), with no in-process ChatCompletion to call, so serving
the plane op from deps.AI just re-entered the transport. Closing this for real
means an in-process entry point in hanzoai/ai that the plane op can call
without a socket — a change in that module, not a rewiring in this one.
Reverted whole rather than patched so main does not carry a half-measure that
reads, from the commit log, as if the round trip were gone.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`ai` is a plugin of this same binary running as its own process, and the only
way a sibling could get a completion was its PUBLIC address. So the pod left
through Cloudflare, performed a client_credentials exchange to authenticate to
its own deployment, and came back — to reach code one socket away. Its own
startup line said so:
deps.AI (completions) -> HTTP gateway (IAM M2M)
base_url https://api.hanzo.ai/v1 <- the pod's OWN public address
token_url http://iam.hanzo.svc/... <- a token to call itself
Every part of that was a consequence of addressing a peer by URL. plane.AIChat
and plane.AIEmbed are addressed by APP NAME — zip.SocketPath resolves it and
reach() starts the app if it is not listening — which is how the meter already
debits commerce and how the gate already reads the ledger. There is no address
to configure, no credential to mint, and no second answer to where `ai` is.
Which client a process gets is decided by WHAT IT IS, not by an env:
!cfg.Enabled("ai") means this process does not carry the app, which is exactly
when `ai` is a sibling. The process that IS `ai` (and the host, which carries
everything) falls through to the real transport — asking the plane there would
be this process calling itself, and the test pins that.
CLOUD_AI_ZAP_ADDR is deleted with its field: it was the previous attempt at
this, and it still made a deployment state where its own code lives. The
gateway stays for inference that is genuinely REMOTE — a different fact, not a
second way to reach the same thing.
Billing is deliberately not on the servant side: the caller reserves before it
asks and settles on the reported usage, so pricing it again there would bill
one completion twice. MaxTokens rides the request so the ceiling the caller
reserved against is the ceiling the servant honors.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The upload reached S3 and the profile did not move: 502 "photo stored but the
profile could not be updated", and the log said why — `iam non-envelope
response (400)` for id hanzo/2d4d67ab-….
IAM parses a user id as `<owner>/<name>` through GetOwnerAndNameFromId, and on
the direct-Bearer path X-User-Id is a UUID subject, so `<owner>/<uuid>` names no
user. keyID() is the composite that resolves — the key ops (mint/revoke) already
use it for exactly this reason, and the comment explaining it sits in the file I
edited. On the gateway path username == name, so the two are the same value and
nothing changes there.
Found by uploading a real PNG to production rather than by reading the route
back. The honest failure is what made it findable in one step: the handler
reported that the bytes had landed and the record had not, instead of a 500 or a
success the user could not see.
The branch carried a second one: a root `cloud.Qualify` returning `type Tenant
string`. Two spellings of the key the dataset plane writes, the risk plane reads
and this plane joins on — and the two did not agree.
It did not canonicalise the brand. A deployment started with CLOUD_BRAND=Hanzo
filed `Hanzo/acme` while apps/dataset asked for `hanzo/acme`. Silent,
permanent, no error: one business, two key spaces.
It did not ask the registry. A brand nothing vouches for minted a key the
writers of these surfaces refuse to produce.
`Tenant` is a STRING type, so it can be written as a literal in any package and
decoded straight out of a request body. `tenant.Key` is a struct with one
unexported field: no literal, no JSON, no caller-asserted tenancy. Only
tenant.Mint and tenant.Of produce one.
tenant.Of additionally compares the token's verified issuer brand against the
deployment's — a token minted by lux.id cannot become `hanzo/acme`.
So the root file goes and apps/label takes tenant.Key/tenant.Of throughout. The
door shrinks: tenantOf no longer re-checks the shape it just minted, because the
mint is the check.
Mutation-tested at the ONE place the property now lives: drop the case fold in
tenant.canon and TestTheMintAgreesWithTheWriterOfTheSourceTable reports
`Mint("Hanzo","acme") = "Hanzo/acme"` against a writer that writes `hanzo/acme`,
RED; restored, green. 67 label tests pass on the ported key.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Eight of the ten declared /v1/risk paths answered a 400 that named a field the
caller cannot send. Measured on api.hanzo.ai with each operation's own declared
body: score, learn, features, state, state/appetite, state/snapshot,
state/restore and search/{id} all returned
{"error":{"code":"","message":"field \"subject\" is required"}}
`subject` is plane.AuthorizeIn's field, not any risk operation's. It appears in
no published request schema on this surface, so no caller could ever satisfy it
— a door that answers and cannot be opened.
ops.gate re-derived the caller's ledger with principal.Ledger, which answers ""
for exactly the requests the tenant gate refuses (it composes the same Validated
check), and then handed that empty ledger to the money plane. Asked to price a
spend for a nameless subject, the money plane answered in the vocabulary of
money about a question of identity, in two shapes:
co-resident ledger — metering refuses an empty org fail-closed, which is not a
4xx, so the money wire's fallback renders 503 "Billing temporarily
unavailable": the caller is told the biller is broken.
peer ledger (what deploys) — the gate ships AuthorizeIn{Subject:""} over the
internal plane, commerce's own `validate:"required"` rejects it, and because
that refusal IS a 4xx the money wire preserves it verbatim as the 400 above.
ops.search never had the defect, for the one reason that it reaches ops.admit
before it prices anything. This makes that ordering general: an empty ledger is
an identity refusal, answered with the tenant gate's own sentence from the one
function that owns it, before the money plane is asked anything.
The route itself was never missing. GET /v1/risk/score is a 405 here — the route
is declared, the verb is not — and the report of a 404 traces to the deployed
edge flattening that to plain-text "not found", byte-identical to an unregistered
path. Named in the report; not reachable from this suite.
TestTypedOpsRefuseAnUnvalidatedPrincipal asserted this exact property and passed
throughout, because it mounts deps with no metering client: with no client the
money gate is a no-op and the op fell through to the honest 403 the test wanted.
The new tests mount through mountBilled, the only fixture in which the ordering
is observable. A companion assertion counting calls to the ledger fake was
written, found unfalsifiable — an empty org is refused inside AuthorizeVerdict
before any HTTP request, so the counter reads zero either way — and deleted
rather than shipped; the reason is recorded where it would have gone.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`oversize` is on the wire, so it is in the document: the app's own subset, the
prose map and the woven openapi.yaml every SDK repo pulls. Generated by
`make -C apps/dataset describe` + `make -f mk/fleet.mk openapi-weave OUT=openapi.yaml`
— no hand edit. No new operation, so the floor is unmoved.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A COUNT OVER CALLER-SIZED VALUES IS NOT A BOUND. maxRows capped how many rows a
materialisation holds and `page` capped how many an export returns, and neither
bounded a single byte. A row's coordinates are float64s this plane counts, but its
SUBJECT is a string this plane does not write: the rollup lifts it from
`distinct_id`, `session_id` and `user_id`, which arrive on /v1/event from the
caller. distinct_id is capped at 256 bytes on the anonymous lane and REPLACED by
the token's own subject on the signed one — but session_id, which the `session`
rollup files as a subject verbatim, is capped nowhere. So "200k rows of ten
float64s plus a subject key is tens of megabytes" and "eight jobs is a few hundred
megabytes" were arithmetic over an unknown, and one tenant's traffic decided the
real number for the whole process.
maxSubjectBytes bounds the one caller-sized value a row carries, at 256 — what the
identified lane already states for a subject, for the reason that carries over
unchanged: a minted id is a uuid, the value is KEYED, and something longer is not
an id. With it, count times max IS the byte bound, and every byte figure the
package states is now DERIVED from it rather than written down beside it:
maxRowBytes, maxResidentBytes, maxProcessBytes, maxPageBytes. The stale prose
claims are gone rather than corrected — that was the other spelling.
ONE ENFORCEMENT POINT. `representable` is the predicate and the only place it is
written. It is applied on the way OUT of the source, which is the way IN to this
process and to the rows table, so no read path needs a second check: an export
page is bounded because every row it can return already came through it.
AND THE DEGRADATION IS NAMED. A bound that quietly drops rows is worse than no
bound — the dataset that comes back looks complete, and a model fitted on it is
blind to a population nobody can see was missing. The census measures both halves
on ONE pass (conditional aggregates, not a filter), the excluded subject count
rides on the version, the manifest, the lineage and the wire, and it is in the
source fingerprint, so `reproducible` is measured over it too: a window that grew
a subject too large to carry is a window that MOVED, and lineage now says so.
census and facts cannot disagree about the population, because the predicate is
one expression used by both — measured against, then read with. Two spellings
would sample the share from rows the read never returned.
Three gates, each mutation-tested (defect reintroduced, named test RED, reverted,
green):
TestEveryReadOfTheSourceIsBoundedInBytes the package's own AST — a
function that reads the source without the bound fails, the same shape as the
admission gate beside it, because it is the same failure: a new op skipping a
property nobody checks. Carries the same anti-vacuity floor.
TestAnUnrepresentableSubjectIsExcludedAndCounted end to end: the bound binds,
the representable rows all survive, the count is on the version AND the
lineage, and it is falsifiable.
TestTheByteBoundIsDerivedFromTheValueBound the arithmetic, so no byte
figure can be asserted independently again.
The fake store EVALUATES the bound rather than ignoring it — a fixture that
ignored it would let every test above pass with the predicate deleted. It also now
binds `?` positionally across the WHOLE statement, as a driver does, which is what
the census's conditional aggregates require.
TestTheTenantLeadsEveryPredicate was reading args[0] and calling it the tenant.
That was a positional coincidence, true only while no statement carried
placeholders before its WHERE; it now reads the argument bound to the leading
`org = ?` itself, which is the property it always meant.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
(cherry picked from commit b851446a2a)
CLOUD_ENABLE / --enable named the subsystems to mount. The binary already
knows: manifest.Apps is the host's set, and a plugin IS its app (Listen sets
cfg.Enable from what plugin/<app>/main.go was built as — that stays; it is
the binary stating itself, not a deployment restating it).
A second source of truth can only add disagreement, and it did. Both devnet
outages on 2026-08-02 were this list: one named "plans", an app the manifest
does not have, and one omitted "kms", the credential broker every other child
pulls its data-plane key from — so every child failed closed at its first
store open. Production has never set it.
Removing the input removes the failure class and five branches with it: the
broker precondition, the unknown-name guard, and three `on != nil` selections
that only existed to police a list nobody should have been writing. Empty
already meant all, which is what production runs.
Values move in the same change — devnet and testnet drop the list, so no
deployment is left naming a variable the binary no longer reads. Docs and
cloud-probe.sh follow; the probe had been STRIPPING the variable, so it
already agreed.
TestTheAppSetIsNotNamedTwice replaces TestAnAllowlistWithoutTheBrokerIsRefused
— that test policed the list, and the guard is now that no source states the
set again.
Also fixes a pre-existing red on main, unrelated to this: apps/catalog's
cloud.Request call site was never added to allowedRequestUses, so the escape
hatch ratchet failed on clean origin/main. It is a legitimate use (the
published corpus reads as PublicOrg for everyone, so the tenant is re-pointed
while the caller's authority travels whole) and is now recorded with that
reason.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An hourly goroutine in authors deposited credit with nobody in the loop, and a
GET on three surfaces accrued-and-paid on read. Both are gone, along with the
capability that made them one line each.
- apps/authors/scheduler.go, sweepAndPayout, autoPayoutAuthor — the unattended
hourly accrue+pay loop, default ON, no env var, no route, no human.
- the lazy sweep on GET /v1/authors, /v1/affiliates, /v1/affiliates/me and
/me/earnings. Reads read; the admin POST sweep still accrues.
- payout settlement in both programs. A payout RECORDS what is owed, for every
method including credits; a human settles it. Accrual — the product — stays.
- treasury.Reserve/Credit returned backed=true when unmounted, and `mounted` is
a package global, so in one-binary-per-app it was ALWAYS nil in callers: every
"reserve-backed" payout was an unbacked mint that logged itself as reserved.
With settlement gone it has no callers, so it is deleted rather than fixed.
- POST /v1/admin/credits — a second admin mint with no cap and no positivity
check, whose audit did not fail closed. core.ApplyGrant is the one door: it
caps, rejects non-positive amounts, checks the org, and refuses without a
durable audit store. The relay and its wire client are deleted.
- payout.Client.Deposit, the ONE money-in primitive all three programs shared,
and the deposit method on each program's seam. The seams now carry a single
read, matching referrals: reviving a mint has to start by re-declaring the
capability, in front of a test that says no.
- the published POST /finance/starter op, advertising a grant deleted in
41b23f12.
openapi.yaml, plugin/admin/openapi.json and openapi/floor.json drop the deleted
route in this commit, so the reduction is reviewed next to its reason.
Tests assert the guarantee rather than the old behaviour: a GET grants nothing
and the ledger receives zero deposits, proven at the wire against a commerce stub
that fails on any write.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/ws/query_progress is in openapi.yaml and in plugin/o11y/openapi.json, so it is a
method in every generated SDK — and no manifest row named it, so the composed
binary routed it nowhere. Its own prose says the address "was unreachable from the
composed binary until the route table named it, because the old wildcard covered
only the o11y prefix"; the wildcard went and the row never grew the prefix back.
TestEveryServedPathReachesTheAppThatServesIt has been RED on main for this one
path. It reads 1673-of-1684 now against 1672-of-1684 before, and the recorded
ledger is unchanged — nothing else changed hands.
Mutation-tested: prefix removed, the named test reports UNREACHABLE and fails;
restored, green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Both avatar routes answered 404 in production. The routes were registered and
the handlers were fine; the HOST routes by manifest prefix, and account's row
did not name /v1/avatar — so `ai`, which owns the /v1 remainder, won it.
manifest's own router test said so in as many words on the tree that shipped:
UNREACHABLE: account /v1/avatar -> ai
UNREACHABLE: account /v1/avatar/{org}/{user}/{digest} -> ai
I did not run it. Targeted package tests passed and I shipped on those, which
is how a route can be complete, tested, deployed and unreachable at once.
With the prefix named, both paths mount to account (zip mounting … addr=account)
and the ledger drops from three unreachable paths to one — o11y
/ws/query_progress, which is not this change's and stays as it was.
iamurl.go opens by saying the IAM-address policy "existed as three inlined
copies ... so there is exactly one now". There were four, and they had grown
three different fallbacks:
iamurl.go IAM_URL, else the public issuer (the policy)
auth_apikey.go IAM_URL, else IAM_INTERNAL_URL, else NOTHING
apps/account/iam.go IAM_URL, else a hardcoded cluster address
apps/platform re-read IAM_URL to answer a different question
A copy of a policy does not disagree until one is edited. These had already
diverged: auth_apikey read a fifth env name no cloud deployment sets
(IAM_INTERNAL_URL is on admin-guard and chat only), so a single-process deploy
that knew only its issuer resolved "" and API-key auth stayed silently
unconfigured; apps/account reached a cluster address that a non-cluster deploy
does not have.
iamurl.go now answers both questions the estate actually asks, over ONE env
read: IAMBase() for the address, IAMExternal() for whether a separate IAM is
named. They are separate because conflating them IS the bug — a deployment
with only a public issuer resolves a real address while naming no external
IAM, and a caller inferring one from the other reaches for a store that is
not there. TestIAMAddressHasOneReader walks the tree and fails on a second
reader, so this cannot drift back.
Also: pickCompletionsClient preferred the PUBLIC gateway over the in-cluster
peer. `ai` is a plugin of this same binary running as its own process, so that
ordering sent the pod out through Cloudflare and back, minting an OAuth token
to authenticate to its own deployment, to reach code one socket away. The peer
is now first. Ordered, not gated: nothing sets CLOUD_AI_ZAP_ADDR today, so
this is inert until an address is named and the gateway keeps answering.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
main was pinned to hanzoai/ai v1.832.18, which is RETRACTED. That version seals
the secret MASK as a provider key: the admin API returns "***" for a stored
secret, so saving a provider form without touching the key field seals the literal
"***" into KMS under the provider's own name — and because ai resolves KMS-first,
the store then answers "***" for every read, outranking the env var that was
serving the real key. The provider stops authenticating while its row still looks
correct. v1.832.19 carries the guard.
.19 also brings: secrets resolved from the EMBEDDED in-process KMS (this binary's
own apps/kms) instead of over HTTP to the standalone deployment, which had never
worked — 404 on the path it used, 401 on the correct one; /v1/provider-flags
renamed to /v1/models/providers and derived from the served catalog, so the
provider set and the model list cannot disagree; and a model family is now
controlled from admin.hanzo.ai rather than only from deployment env.
apps/referrals did not compile on main: payout.Deposit gained a required `ref`
(commerce guards on it so a retried payout credits AT MOST ONCE) and this caller
was not updated with it. A referral pays TWO wallets, so the referral id alone
would name both and commerce would dedupe the second against the first — the
referee's bonus would silently never land. bonusRef(id, side) makes each credit
its own event while staying stable across retries.
The published spec is regenerated: /v1/provider-flags is gone from openapi.yaml
and plugin/ai/openapi.json (the .18 pin never regenerated it, so the drift gate
was red), and the retired provider-flags product is dropped from the floor.
Verified on Linux (macOS has no tmpfs for the pure-Go SQLCipher codec, so the
store-backed suites cannot run there): apps/referrals, apps/ai, apps/kms and
openapi all green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The plane fix shipped and /v1/catalog answered 500 "index: no org on the call"
— the public browse reaching the index as nobody.
cloud.For() states a caller, but zip's forwardIdentity prefers an INBOUND
request over a stated one ("an inbound request always wins over what it said"),
and a typed handler's ctx carries the in-flight request. So For() was silently
overridden and the call went out as whoever asked. For a signed-out visitor
that is nobody, and the index refuses a call with no org — correctly.
cloud.As() is the form for this: it re-points the tenant on a context with no
request behind it, which is the one place zip reads what we stated. The caller's
authority still travels whole; only the tenant moves — which is the point, since
the published corpus is read as PublicOrg by everyone, signed in or not.
Caught by asking production rather than by trusting the deploy: the previous
commit was verified live and answered 500, not rows.
payout.Deposit sent no idempotency key at all, so every retry of an author
royalty or affiliate commission credited the wallet again. Commerce now
requires the key (it is the reference to the money event), and both callers
already hold the right value: the payout row's own id.
- Deposit takes ref and sends it as X-Idempotency-Key, via a post helper
kept beside do because only a WRITE carries a reference — a read has no
event to name.
- ErrNoRef refuses an unnamed deposit HERE rather than at commerce, so the
failure names the caller's missing value instead of arriving as a status,
and a new payout path cannot quietly ship without one.
- authors and affiliates pass "payout:"+payoutID through their commerce
seams.
Tests: an unnamed payout never reaches commerce (the fake server records
that it was not called), and the ref travels as the key commerce guards on.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/referrals ran a "lazy qualify sweep" before listing, and that sweep
reached commerce.Deposit. Loading your own referrals page minted platform
credit — $10 to you, $5 to your referee — on a read. The middleware in front of
the prefix gated writes by asking `method != POST`, so it waved the GET through;
the two safeties behind it did not hold either. treasury.Reserve returns
backed=true when treasury is unmounted, so the "backed by the reserve fund"
claim was a passthrough in any deploy without it, and the at-most-once latch
bounds the mint per referral, not in total.
The precedent is set twice on main: 41b23f12 deleted the $5 starter grant and
45b3b5cf deleted finance.Deposit along with its imports. Same here. The deposit
path is DELETED, not disabled — a flag-disabled money mint is one flag from an
enabled one.
What goes: the two bonus constants, the ledger currency + grant:referral tag,
grant(), the treasury reservation and its import, LatchCredit, SetTxns, the
grant/txn/credited_at columns, the `credited` status, and every cents field on
the wire. A field that can only ever report zero is a lie about what the surface
does, so creditsEarnedCents and the two bonus amounts go rather than freeze at 0.
The commerce seam keeps ONE method, spendCents — it is a question, not an
instruction — and TestCommerceSeamIsReadOnly fails if it grows a write.
What stays, because it is the actual product: who referred whom, the stable
code, the share link, and qualification. Qualification is a WRITE, so it now
happens only on POST /v1/admin/referrals/sweep. A GET reports; it does not
transition.
The gate is fixed at the defect class, not the instance: it waves through GET
and HEAD by name and requires an org for every other verb, including ones this
package does not serve. "Not POST" meaning "harmless" is the reasoning that let
a read reach a deposit.
What a qualified referral is WORTH is not this package's question. That is an
affiliate payable in hanzoai/commerce, settled by wire or to a connected wallet
— never minted as platform credit.
Tests: a GET in the exact state that used to pay leaves the row byte-identical
and touches the money plane zero times; the real payout client against a stub
commerce that fails on any write proves zero deposits at the wire.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The reservation landed with a 32768 constant standing in for "the most a
completion can be". That number is wrong per model by construction: it caps a
1M-context model at whatever was typed, and it is one release out of date the
moment a model ships. This estate has already paid for that shape twice —
ai/model's name-matching table gave deepseek-v4-pro 16384 and 402'd every long
prompt, and glm-5.2 dead-ended /compact on a stale 16K fallback. Both were
fixed by moving the number into models.yaml. This does the same for billing.
- completionCeiling(model) resolves through SetCompletionCeiling, installed
in apps/ai from ModelConfig.MaxOutput (ai v1.832.18), falling back to the
model's context window — still a true bound, since prompt + completion can
never exceed it. The constant survives only as a FLOOR for a model the
catalog does not declare, and is documented as never a per-model answer.
- the seam exists because hanzoai/ai/controllers imports hanzoai/cloud, so
the catalog is a CYCLE from cloud's root, not merely weight. apps/ai links
both, which is where every other cross-module hook is installed.
- atMost no longer writes the ceiling onto the request. What we reserve is a
billing fact; req.MaxTokens is the CALLER's, and forwarding a limit they
never asked for silently truncates their answer. The reservation is sound
regardless — a model cannot exceed its own max output — so the bound holds
whether or not we restate it on the wire. Only a caller-set MaxTokens now
reaches the provider.
TestCeilingComesFromTheModelNotAConstant pins all four: a 1M model reserves
>=1M, an undeclared model takes the floor, a caller's MaxTokens wins, and
atMost never mutates it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzo.app's Community page rendered "ERROR: CATALOG: 503" under an otherwise
fully-drawn page. api.hanzo.ai/v1/catalog answered
{"status":503,"error":"catalog: index not mounted"} on EVERY request, and
nothing was down.
browse() guarded on index.Ready(), and Ready() reports whether the index is
mounted IN THIS BINARY — its own doc says so. That was true and harmless while
cloud was one fused process. It stopped being harmless when the fleet became
one process per app: `index` and `catalog` are two manifest rows, index.Mount
is only ever called by plugin/index, so inside the catalog process that global
is nil and always will be. An in-process dependency survived a process split,
and the only symptom was a status code.
The index is now ASKED, on the internal plane, exactly as visor asks tasks for
its activities and as commerce serves its ledger. Not opened: the index store is
one encrypted SQLite with a single writer (MaxOpenConns(1), keyed through cek),
so a second process opening the same file to read it is the collision, not the
cure.
Both legs are kept and neither is dead: a fused binary that mounted both apps
still reads in-process, because a wire hop to something in the same address
space is pointless; the deployed fleet takes the plane. The write side does not
move — Reconcile stays in the process that owns the file.
apps/search carries the same dependency and is mounted by no plugin at all, so
it is unreachable rather than broken. Left alone here; naming it so the next
reader does not mistake this fix for having covered it.
Moving imageType into internal/magic took the only use of "bytes" in files.go
with it and left the import behind — which does not compile, and it is what
turned the release train red twice (the image job exited 1 at ~6m18s and no
cloud image published, while v1.801.399 had built fine from the commit before).
It was invisible locally for a reason worth writing down: `go build ./apps/team`
on macOS fails first on hanzoai/base's `cgo && !sqlite_math_functions` guard, so
the package was never type-checked and my error hid behind someone else's. The
build CI and the Dockerfile use is CGO_ENABLED=0 — under that, the package
compiles and the mistake is immediate.
Verified the way CI does: CGO_ENABLED=0 go build ./... and go vet ./... both
clean. The apps/team test failures that remain on this machine are all one
environmental cause (75 of 75: "cek: no RAM-backed scratch", macOS has no
/dev/shm) and fail at store setup before any assertion.
The drift gate regenerates every app's spec subset FROM SOURCE and fails on any
diff, so adding /v1/avatar without re-running it turned CI red — the release
train's image job exited 1 after 6m29s and no cloud image was published.
Regenerated, not hand-edited: zipdoc lifts the doc comments into zipdoc_gen.go,
`describe` projects account's own subset, and the weave proves the fleet spec
equals the sum of the subsets. floor.json moves by exactly what was added —
+2 paths, +2 operations, one new `avatar` product with 2.
zip dials every plugin with zaphttp's DEFAULT transport, whose readTimeout is
30 seconds, and nothing overrode it. That deadline is armed ONCE, just before
the response head is read, and for a STREAMED response it is never re-armed:
the head arrives, SetBodyStream returns, and the whole body must then land
within what is left of the original 30 seconds.
So it is not an idle timeout. It is a hard cap on the TOTAL DURATION of a
response — applied to the plane whose responses are model completions, because
`ai` owns the /v1 remainder (/v1/chat/completions and the rest of the
OpenAI-compatible surface) and runs as a plugin behind this wire.
Every completion longer than 30 seconds was cut:
- streamed: the body stops mid-token at ~30.1s with NO finish_reason, which
every layer downstream reads as a complete answer. Measured against the
live gateway, a long-page request to claude-opus-4.8 returned 7056 bytes,
no </html>, total 30.1s — while the identical request straight to the
upstream streamed 389s and finished properly at 15661 bytes. Reproduced
in-cluster too, so it was never the edge.
- unstreamed: 502 `zaphttp: read response: read unix …: i/o timeout`.
This is why hanzo.app's builder could produce a small landing page but not a
real app: an app is just a longer generation, and the truncation was silent. It
also made a slow-to-first-token model look broken rather than slow — enso
spends ~19s thinking, so most of its budget was gone before it emitted a token.
The ZAP scheme is re-registered before any plugin mounts (the transport is
resolved at dial time from a process-global registry, so a later registration
would leave already-dialed plugins on the default). BOTH halves are supplied: a
Dial-only Transport would silently remove the host's ability to LISTEN on its
own default scheme.
15 minutes, not zero: zero is no deadline at all and a wedged plugin would hold
the connection forever. The honest shape is an idle timeout — time BETWEEN
frames — which this transport does not offer, so it stays a total-duration cap
set past anything a real completion reaches.
There was no way to set one. IAM carries an `avatar` on every user row and the
console renders it, but the only writers were federation (a GitHub avatar_url,
an OIDC `picture` claim) and SCIM — so a user who signed up with a password had
a monogram and no way to replace it, and the console's Profile card answered the
attempt with "Edit in IAM", which links to an IAM that cannot do it either.
Production agreed: /v1/avatar was a 404 while /v1/keys was a 403.
POST /v1/avatar stores the image and records its URL on the caller's IAM row, so
every surface that already reads `avatar` picks it up with no further call.
GET /v1/avatar/:org/:user/:digest serves it.
Storage is deps.VFS — the existing S3 seam (SeaweedFS via clients/s3vfs), which
was chosen for exactly this: "an adapter+crypto is needless complexity for small
avatars". No new store.
Three properties make it safe to serve an uploaded file back from an API origin:
- THE FORMAT IS DECIDED BY THE BYTES. png/jpeg/gif/webp by magic number;
anything else is 415 on the way in and 404 on the way out. A filename and a
part Content-Type are the client's to choose, so neither may decide what
this origin later serves — an SVG is a program, not a picture.
- THE ADDRESS IS THE CONTENT. The key ends in the sha256 of the bytes, so a
new photo is a new URL rather than a stale cache of the old face, and the
read caches for a year. A replaced photo is deliberately NOT deleted: the
old URL is already inside issued tokens and rendered pages, and an object
store costs bytes where a broken face costs a person their profile.
- THE READ TAKES NO CREDENTIALS, AND MUST NOT. Its whole job is to be an <img>
from console.hanzo.ai, a different origin that sends no cookies and cannot
set a header. So the 64 hex of sha256 IS the capability — producible only by
someone who already has the image. The org and user segments are REFUSED
unless they are plain identifiers rather than sanitized: apps/team's seg()
folds "a/b" and "a_b" onto one key, and a fold in a tenancy key is two
identities sharing an address.
internal/magic is the one magic-byte allow-list, now shared with apps/team's
files plane instead of copied. (The three `seg` functions are NOT duplicates —
same name, three different concepts — so they stay where they are.)
The oversize test mounts the app at production's 16 MiB edge body limit. Left at
zip's 4 MiB default the framework refuses the request first and the handler's
own 413 is unreachable and untested — the shape of the bug where studio's 4K
sources could not enqueue.
POST /v1/marketing/promos/:code/redeem was a self-service money mint. The only
gate was tenant(ctx) -- ANY validated principal. `plan` and `seats` came off the
REQUEST BODY unvalidated and were multiplied into a finance.Deposit, so
{"plan":"team","seats":10} deposited 10 x $179.10 = $1,791.00 of real spendable
credit into the caller's own org. Nothing ever collected the charge the discount
was supposedly against; the charge was computed, returned to the caller, and
discarded. `instrument` was the anti-farming key, but instrumentUsed("")
returned false, so OMITTING the field skipped the guard entirely. With the
1,000-org cap, open signup and a personal org per account, that is ~$1.79M of
self-serve credit. The seed shipped active=1 in v1.801.398, which is live.
THE CAMPAIGN WAS NEVER AUTHORIZED. It became live because a schema migration
INSERTed it on every boot -- a business decision arriving as a side effect of a
code change. The seed is deleted, and because deleting an INSERT does nothing
for a database that already ran it, migratePromos now DELETEs the row on every
boot. Redemption history is deliberately KEPT: it is the evidence of what
happened while the campaign was live, and destroying it would destroy the audit
trail exactly when it matters.
CREDIT INTO AN ORG IS AN ADMIN DECISION -- deliberate, through the admin
surface, against an auditable ledger. So the deposit is not fixed, it is GONE,
along with the finance/money/types imports that made it reachable: reviving a
mint here would have to start by reviving an import. This follows 41b23f12,
which deleted the automatic $5 starter grant rather than switching it off, for
the same reason -- a money-mint left disabled is one flag away from enabled.
The subsystem now ships OFF (campaignsLive=false, read from no env var, no
platform switch, no column; TestCampaignsShipOff asserts the shipped value).
The guards are hardened anyway, so a REVIVED campaign cannot resurrect the hole:
- Plan is DERIVED from the org's live ACTIVE/TRIALING paid subscription via
cloud.PlanChecker -- the same seam SpendGate resolves -- and RedeemInput no
longer HAS plan/seats fields. A field that does not exist cannot be trusted
by the next reader. No qualifying subscription means no redemption.
- FAIL CLOSED on an unreadable plan authority. SpendGate deliberately fails
OPEN on this same read, because refusing on an outage 402s every paying
customer at once. Here the asymmetry inverts: an outage must not be able to
manufacture a claim that money is later granted against.
- instrumentUsed("") now returns TRUE. An absent instrument is not evidence of
a fresh card, it is the absence of evidence, and the partial unique index
(WHERE instrument <> '') means the database will not catch it either.
- maxClaimCents bounds every recorded claim, checked under the same lock as
every other guard, and REFUSES rather than clamps -- a silent clamp would
record a wrong figure and hide the bug that produced it.
- Seats is the single-seat floor, never a caller-supplied multiplier.
Redemption.CreditCents/CreditEntryID become DiscountCents: the row records a
claim, not a balance, and a field named for credit that credits nothing is how
the next bug gets written.
Tests drive the shipped routes through the real router, because the hole was a
handler that trusted its input -- a store-level test would have proved the store
fine and missed it. The hardening tests force a campaign live (revivedPromoRoutes)
to answer the question that matters if the decision is ever reversed. Proven: the
exploit body cannot move the recorded figure; a live ledger receives ZERO
deposits; a redemption while closed is refused; nothing is seeded; migrate purges
an already-seeded row while preserving its redemptions.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A prepaid gate reads a SETTLED balance and a completion's cost is not known
until it finishes. Two ways an org spent money it did not have sat between
those facts:
- the gate priced only EstTokens(prompt), so the completion was never
covered. A 1c org asking for a million-token completion was allowed and
settled at $2.00, ending at -$1.99.
- N calls in flight each read the balance before any of them debited, so
each was authorized for the whole of it. Twenty simultaneous callers
against 5c spent 20x the balance.
meteredAI now commits a call's worst case before weighing it and releases
that commitment when the debit reaches the ledger:
- types.ChatRequest gains MaxTokens, and atMost() resolves the ceiling ONTO
the request so the transport forwards the very number the gate priced.
Reserving a ceiling nobody enforces leaves the completion just as unfunded,
only less visibly, so clients/aihttp sends it on both the buffered and the
streamed path.
- commitments tracks per-org committed-but-unsettled cents. commit() returns
the RUNNING TOTAL, which is what the balance must cover, so a second
concurrent caller must clear the first one's commitment. Nothing else has
to know reservations exist.
- the release runs inside the recording goroutine (meterUsage's new posted
hook, threaded through the peer path too). Releasing when the call returns
would let the next gate read a balance that still contains money already
being spent — the very window this closes. It runs on every exit: a hold
that leaks is a paying customer locked out of their own balance.
Holds stay per-pod. apps/finance owns the settled truth and says so
("transient holds are the caller's in-pod concern, never persisted here"),
so this never becomes a second ledger.
MeterUsage keeps its signature — one debit verb, twelve callers untouched.
Tests drive the real meteredAI against a wallet whose balance MOVES; the
existing fixtures answer every gate from a fixed body, which is why neither
gap showed up before. A barrier holds the concurrent callers at the balance
read so the TOCTOU is deterministic rather than a race won by luck.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The starter grant minted $5 into a wallet from middleware, on first credential
contact, with no human in the loop. Credit into an org is an ADMIN decision --
made deliberately, through the admin surface, against an auditable ledger --
so an automatic path that creates money is not a feature to fix but a mechanism
to remove.
DELETED RATHER THAN SWITCHED OFF. A disabled money-mint is one flag away from
an enabled one, and the flag is the kind of thing a later reader flips to
"unblock" something. There is no starter code left to re-enable: the middleware,
its mount in serve.go, the cross-process plane op (finance_starter / StarterIn /
Granted) that let a non-ledger binary ask for it, and their tests are gone.
Note this also removes the shared-signup-org exclusion that lived in the gate.
It was sound anti-abuse for a grant that no longer exists, and keeping half a
mechanism to guard the other half is how dead code survives.
The paywall consequence is deliberate and is NOT taken here: SpendGate stays
behind its kill switch. With no automatic funding, enforcing it 402s every new
account from its first request -- an honest paywall, and a product decision that
deserves its own change rather than arriving as a side effect of this one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
sha-846069c predates the fix. The console SPA is baked into this binary by
//go:embed, so nothing about a console release reaches production until this
line moves — which is exactly what the pin is for, and why it is a sha and not
`:latest`.
sha-9da3984 carries three commits: the token store moved off sessionStorage
(a second tab started signed OUT while the first was still signed in), the
landing CTA starts the sign-in instead of routing to a page that asks again,
and the @hanzo/iam bump to 0.21.6 without which the static export dies on
`ReferenceError: sessionStorage is not defined` while prerendering
/auth/callback.
Verified before pinning: the console CI run for 9da3984 is green and pushed
console-embed:sha-9da3984-amd64. The previous run, on b9d31aa, was RED — so the
image for the storage fix alone never existed, and pinning to it would have
failed the CONSOLE-GATE here rather than shipping anything.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two corrections to what shipped in v1.801.397.
TIER. It was registered on the bare public chain plans uses, but GetTier opens
with middleware.GetOrganization, so it panicked on a nil interface conversion and
answered 500 — the route went from unreachable to reachable-and-broken. A tier is
org state; the org has to be resolved first. Moved onto the billingRead loop,
which supplies the IAM leg its six siblings already rely on.
ROUNDING. The balance now reads (502 -> 200, $149,913.08), but it rounds UP, not
down as the comment claimed. hanzoai/decimal's Rescale rounds half-away-from-zero
(decimal.go:145) and Minor() is a Rescale — measured live, …078983985999994361
served 14991308 cents, a tenth of a cent above the true balance. The comment is
corrected rather than the behaviour: this number is a display, nothing is billed
from it, and the spend gate reads the exact decimal itself. A debit that must not
overstate has to round down deliberately instead of reusing this.
apps/ai carries the same "truncated toward zero" claim on the same call and is
wrong the same way. Noted in place; its gate compares > 0, so a sub-cent rounding
cannot change its verdict.
GET /v1/billing/balance answered 502 "billing upstream unreachable" on every
call and the console rendered "Unavailable" while the ledger held the money.
Nothing upstream was involved — that label is wrong and it is why this looked
like a connectivity problem for days. plane.Money.Minor() REFUSES a value finer
than a cent rather than round behind the caller, and the ledger keeps eighteen
decimals because per-token charges are routinely finer than a cent. So a REAL
balance broke the read: live the org held $149,913.078983985999994361 and the
error was "is finer than its minor unit; round explicitly". It got worse as
usage accumulated, since a longer history makes a sub-cent tail likelier.
Minor()'s own doc says a caller that wants a rounded figure — "a display, a
summary" — should round explicitly, where the choice is visible. This view is
exactly that and never did. It now rounds DOWN, the same choice apps/ai
documents for the same value: a displayed balance must never exceed what the
account can actually spend, and truncation understates by under a cent. Nothing
is billed from this number.
Also reverts a wrong fix from earlier in this session: registering balance
co-resident in apps/commerce. The manifest gives /v1/billing/balance to the
`billing` app, not commerce, so that registration was in a subtree it does not
own and could never have run.
Two things this exposed, both left alone deliberately:
- The 502's message names an upstream that is not in the path. Renaming it is
a behaviour change to an error contract callers may match on.
- apps/billing's test binary needs libsqlcipher and cannot compile on a
laptop, so `go build` passing there proves nothing about the test file. That
masked a first version of this patch which called a Minor() that does not
exist on the local money type; caught by type-checking with go vet instead.
GET /v1/billing/balance answered 502 "billing upstream unreachable" on every
call, and the console renders that as "Unavailable" while the ledger holds real
money.
The cause is the one apps/account/billing_coresident.go already documents in its
header: co-resident there is NO standalone commerce — the in-cluster service
selects the cloud pods themselves — so the /v1/billing/* bridge forwards to a
default base that is the public edge, and the read re-enters the same bridge in
an unbounded self-dispatch loop. Six sibling reads (invoices, subscriptions,
alerts, payouts, settings, credits) were already registered co-resident to
shadow that wildcard. Balance was not one of them.
It now registers through the identical chain — RequestContext, IAMTokenRequired,
PinBillingSubject — so the subject pin is byte-for-byte what the bridge applied
and a read scopes to exactly the account the spend gate debits, never wider.
balance/all rides the same prefix, since a prefix owns its whole subtree.
The apps/commerce test binary does not compile on a laptop (hanzoai/base needs
libsqlcipher, which the build image links and macOS does not); verified by
stashing that the identical failure predates this change.
Every AI request logged
tier_cache: Commerce lookup failed for key=...: commerce returned 404
(defaulting to zen-free)
and that default is 60 rpm against 500 for pro, 2000 for team, 50000 for
enterprise. So the failure did not give anything away — it silently served every
PAYING customer the most restrictive tier in the table.
Cause is the exclusive-subtree rule again: account-bridge owns /v1/billing, and
commerce's row named seventeen sibling paths but not tier, so
GET /v1/billing/tier never reached the handler that answers it
(commerce api/billing/handlers.go:43 registers it; live it 404s).
The UNREACHABLE ledger in manifest/router_test.go did not catch this and is not
wrong to have missed it: it fires on paths the fleet PUBLISHES, and cloud does
not publish tier — commerce serves it and only the ai router calls it. A
cross-binary caller is outside what that gate can see from here.
The pinned console-embed was sha-147ecd3, built 2026-07-27. console main is 63
commits ahead of it, so a week of console work — including the onboarding fix
below — has never reached production: the pin does not track main BY DESIGN (a
floating tag made a release silently bake the PREVIOUS console), so it only moves
when someone moves it, and nobody had.
sha-846069c is console main HEAD. Verified published before pinning:
ghcr.io/hanzoai/console-embed:sha-846069c-amd64 resolves to
sha256:f2849a31b082da0d690bc5403a41ad995624fdfd6595089008821c07718a9255, and
console-embed:latest now points at the same digest. Probed with all four Accept
types against a bogus-tag control — a two-header probe returns a FALSE 404 on an
image that is present, which is how a healthy registry can be misread as an
outage.
What this carries to production, beyond 62 other commits: a refused org create no
longer reads as a complaint about the NAME. /v1/iam/onboard answers 409 for two
opposite reasons — the first-run gate (this account already admins an org;
founding a second would orphan it) and a name genuinely held by another tenant —
and the console showed the server's organization-level message immediately after
the customer typed a name. It now reads the account to tell the two apart, and
offers the way into the org the identity is actually in instead of dead-ending on
a form that can never submit.
cloud pinned ai v1.832.16 while five commits sat on ai's main untagged, so a fix
merged hours ago was in no release and no binary. That is the third instance
today of merged-and-unshipped, each with a different cause: an image published
before its own fix landed, a build job silently skipped by a stale generated-doc
gate, and now a module change in no tag at all. The symptom is identical from
outside — the code is right and production is wrong — which is what makes it
expensive to notice.
Carries: the ok-with-no-user key refusal (IAM answering status=ok with a null
user used to fall through to a bare "invalid API key" that named no cause), the
openrouter seed URL carrying a /v1 the family appends itself, a family's kms://
key resolved before it becomes an Authorization header, family control from
admin.hanzo.ai rather than env alone, and /v1/provider-flags becoming
/v1/models/providers derived from the served catalog.
No release cut here. The next cloud build carries it: cloud is one replica with
strategy Recreate, so every release is a total outage of inference, billing and
auth, and a dependency bump does not justify one on its own.
The site edge has never run in production. It was mounted in serve.go — the
FUSED composition root — but the image runs cmd/cloud (the light router) on the
public port with one process per app beside it, and none of those is serve.go.
So every <slug>.hanzo.app fell through to the console SPA, and the whole /v1
surface answered on the customer's own hostname.
Proven at the pod rather than inferred: the process holding :8000 is /cloud (a
separate 23MB binary from /plugins), and `grep -c sites_resolve /cloud` is 0 —
the code was not in the running binary at all, even at the tag that contains it.
The two earlier fixes this session were both real and both invisible for this
reason: the host resolution fix (8b729f8ef) and the plane resolver (5f0b74fe9)
were compiled into a composition root nothing runs.
The router "deliberately links none of" the fleet's package graph, and that
holds: `go list -deps ./cmd/cloud` still contains ZERO of the root package.
apps/sites is a leaf, and the cross-app call is made here with zip.DialApp —
the same door wake.go already uses to publish one op without cloud.Plane().
apps/sites exports the wire types so the caller restates no mapping.
Mounted BEFORE webui, which owns "/" for every unclaimed path and would
otherwise answer first for every site host.
The test reads run()'s own source for the call site, because the defect was
never a logic error — the package was always correct — it was a middleware that
ran nowhere. My first version called mountSites directly and PASSED with the
call site deleted, which is exactly the mistake this file is about. Both failure
modes are now negative-controlled: no call at all, and mounted after the console.
Every published site served the console SPA. The host fix in 8b729f8ef was
necessary and not sufficient: with the host resolving correctly the edge still
found no site, because sites.SetResolver writes a PACKAGE-LEVEL registry inside
`projects` and the edge middleware reads it inside whichever process fronts
:8000. The pod boots ~25 single-app processes ("enabled":["<one>"], 25 distinct,
zero multi-app — measured on the live pod), so those are never the same process
and the registry is always nil where it is consulted.
A nil registry is a clean MISS, not a fault. So every lookup failed silently,
every request fell through to the API pipeline, and no error was logged anywhere
because nothing had failed. Proven at the pod with the ingress bypassed:
wget --header="Host: app.maxpower.hanzo.app" http://127.0.0.1:8000/
-> <title>Hanzo Cloud Console
for a site that is genuinely published and has its own Ingress.
The fix is the seam this repo already uses for exactly this shape:
FinanceScopeRules is on the plane, in its own words, because "the READER is a
cloud EDGE middleware" and the fact belongs to another app. projects now
publishes sites_resolve / sites_resolve_org from the one process that owns the
store, and the edge falls back to them. Co-residence still wins with no hop —
currentResolver prefers the in-process registry and only then asks.
Not-found stays a clean 404; a failure to ASK stays an error, so the edge can
render 503. Collapsing those would serve 404s for live customer sites during any
transient failure of the owning app, which is indistinguishable from deletion.
The comment on SetResolver said "until it is set, every site request is an honest
404 (the projects subsystem is not mounted)". That premise was the bug: projects
IS mounted, just elsewhere, and 404 is not honest when the site exists.
Negative-controlled: removing the fallback fails the new test with the
fall-through this commit is named for. The second test pins that a co-resident
store is still used without the hop.
The relational engine wrapped an identifier in the dialect delimiter but wrote
the body verbatim, so a delimiter inside the identifier closed the quote the
writer opened and the rest executed — Desc("name`,(subquery)--") emitted two
quoted identifiers plus bare SQL rather than one identifier. xorm v1.4.5 doubles
the delimiter, the standard SQL identifier escape, so a name that smuggled one
becomes a single identifier that does not exist and fails closed.
orm is the only place that version is chosen — consumers name hanzoai/orm and
the engine arrives underneath — so this is the hop that makes it live here.
go list -m confirms xorm resolves to v1.4.5 in this build.
Bump only. The four failing packages are the standing pre-existing set
(apps/iam, apps/plan, apps/pricing, manifest); the bump adds none.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
plugin/dataset/main.go called cloud.Serve, which does not exist. Every other
plugin calls cloud.Listen. The image build enumerates the manifest and compiles
each plugin in turn, so this failed the whole build at the dataset step:
plugin/dataset/main.go:29:18: undefined: cloud.Serve
That means no cloud image has been buildable since ae3a30994 landed. `go build
./...` at the repo root does not catch it — the plugin mains are only reached by
the Dockerfile's per-plugin loop, so the gap between "compiles locally" and
"produces an image" is exactly one word wide.
Every published site served the CONSOLE, and mounted the whole cloud API under
the customer's own hostname. Measured live 2026-08-03:
quest.hanzo.app/ -> <title>Hanzo Cloud Console
quest.hanzo.app/v1/billing/plans -> 200
fiber parses the request URI once, and behind the ingress the parsed host is
empty — so siteSlug("") failed, customCandidate("") failed, and every request
fell through to c.Continue() into the API pipeline. The site edge was mounted and
configured correctly the whole time; it just never learned which host was asked
for. Same accessor and same failure as commerce's tenant resolver, in a second
codebase, the same night.
The parsed host still WINS whenever it names something this server serves — that
ordering is the security property, not a detail, because the host picks the ORG
here and a client able to override a real host could serve itself another
tenant's site. X-Forwarded-Host is consulted only when the parsed host is not a
host we can serve, which is the ingress case and never a direct request.
TestMiddlewareTenantKeyedByHostNotPath (pre-existing, unchanged) still passes.
Negative-controlled: reverting to Hostname() alone fails the new test with the
fall-through this commit is named for. One correction on the way: the first
version of the new test used req.Host="" to express "no parsed host" — httptest
synthesizes "localhost" for that, so it proved nothing until measured.
studio could not enqueue a single render. `POST /v1/tasks/.../activities` answered
403 "identity required", so thirteen jobs sat `queued` in its worklog for up to
nineteen hours while both GPUs polled an empty namespace every two seconds and
reported themselves healthy. Nothing in the queue, the worker logs, or the studio
UI said why.
The cause is one branch. homeOrg already knows that a machine cannot mis-attribute
its org the way a human choosing an app can — that is why the KMS sync identity
reads `owner`. But KMS is recognised by audience, which works only because its
client id is DERIVED from its org ("<org>-platform-kms"). No other app's id is,
so every other client_credentials principal fell through to the estate rule, found
the empty `orgs` a machine correctly carries, and resolved nothing. SanitizeIdentity
then minted X-User-Id with no X-Org-Id, and each org gate refused it.
So the recognition comes from the token's SHAPE, which a human token cannot wear: in
a client_credentials token the client IS the subject — IAM sets sub to "<org>/<app>"
— and azp equals the sole audience, because the app asked for a token for itself. A
human's subject is the user and azp is whichever app they signed in through, which is
the exact mis-attribution homeOrg exists to prevent. Every field read is IAM-signed;
none is a header a caller sets.
This does not widen who may cross tenants. It resolves an org for a principal that
has exactly one and can no more choose it than an sk- key can: `owner` is the
application's own organization, and getting the token requires that application's
client secret. A human with no `orgs` still resolves nothing and still fails closed —
tested, along with each half of the shape, because a partial match is a human token
that merely resembles a machine.
An organisation that took its model out of shadow BEFORE the model had learned
anything was told live=true and had nothing written down. The regime lived on the
same row as the learned state, and that row's writer declines to write while the
snapshot holds no learned mass — correctly, because there is no state to lose. So
PUT /v1/risk/state/appetite answered 200, reported live, and persisted nothing;
this binary deploys Recreate at one replica, so the next rollout rebuilt from
defaultConfig — shadow — and the model decided nothing. No error, no log, nothing
to alert on. A model silently disarmed, on a routed door.
The two facts are decomplected. The regime is now its own append-only versioned
record on the tenant's own shelf, written BEFORE anything in memory moves, so a
policy that cannot be written down is refused rather than answered from state the
next rollout will undo.
A regime is a VALUE: a version is minted only when the numbers CHANGE, so a
version means "the Nth distinct policy this organisation adopted" rather than "the
Nth time somebody pressed save", and a client that restates its config on every
deploy is free rather than the cheapest way to fill a disk.
Every score now cites the version it was decided under. Cut is derived from the
appetite that version states, so without the citation a restated appetite made
every earlier decision unreconstructible — the threshold it was measured against
no longer existed anywhere. GET /v1/risk/policy reads the history back.
Bounds, both per tenant and both on the tenant's own table:
RATE at most 24 distinct regimes per rolling 24h, refused past it with the
organisation's own bound named and the regime in force untouched.
TOTAL 381 versions, which IS 256 KiB divided by a measured worst-case row.
At the ceiling the oldest is disposed of and the number disposed of is
REPORTED, derived from the lowest surviving version so it cannot drift.
A regime that predates this record is ADOPTED as version 1 on first residency.
Without that, resolving the regime only from the new record would have returned
every already-live organisation to shadow on the first rollout after this ships —
causing the very defect being fixed, to every tenant at once.
The bounds on review and sample had two spellings, one at the op and one in the
plane. The op's copy is deleted; admitRegime is the one door, at the same
strictness the published contract always had.
Two test gates widened, because both could have been passed by looking at
nothing:
- TestOps_EveryOpIsAdmittedAndPriced parsed typed.go alone, so an op declared in
any other file was admitted and priced by nobody's assertion. It now parses
the package, matches on the RECEIVER TYPE (the plane carries score/learn/
state/appetite too), and fails when a registered op is declared nowhere.
- the per-tenant history assertion over the wire cannot observe the query's
tenant predicate: two orgs are two FILES. The predicate is load-bearing where
two brands share one file, and TestPolicy_TwoBrandsShareAFileAndNotAHistory is
the test that fails when it is dropped.
Eleven mutations applied, each named test red under the defect and green after
revert.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
THE ADDRESS IS THE PRODUCT. openapi.Fold takes an operation's product tag from
the first /v1 segment of its path and nothing else (openapi.Product); a per-op
zip.WithTags names a different axis and cannot override it. The seven ops were
addressed /v1/ml/labels, so seven compliance operations — their own writers
(commerce adjudicates the dispute, the compliance face closes the case, an
analyst files the review), their own five-year retention floor, their own
per-tenant file — would have been published as part of the KServe model-SERVING
product, which is four paths and live with customers on it. Nothing in the fleet
would have said so: the floor ratchet reads `ml: 7 -> 14` as growth, because it
refuses a shrink and only a shrink. It is the same mistake apps/risk's own
manifest row already records having made and corrected once, one layer up.
So: /v1/risk/labels, tag risk, every operation id and schema name risk-prefixed
(riskLabelEvent and not riskEvent — apps/risk publishes a riskEvent already, and
it is a scored decision rather than a judged one). floor.json returns ml to 7 and
raises risk to 17. The manifest row precedes risk, whose prefix is the bare
/v1/risk, and TestEveryServedPathReachesTheAppThatServesIt proves all six paths
reach label over the real fleet router rather than a comment claiming they do.
address_test.go walks the live projection — the same openapi.FleetSpec that writes
the committed subset — so a route re-addressed into somebody else's product fails
at the plane.
A BOUND ON COUNT OVER CALLER-SIZED VALUES IS NOT A BOUND. maxResolve capped a
resolve at 500 named events and nothing capped a subject: the rows were bounded
and the bytes were bounded only by the edge's BodyLimit, which is a fact about the
deployment. Each subject is then amplified below the door — a dedupe key, a
grouping key, one bound parameter per event in a statement against a single-writer
file. The write door had the ceiling all along (admit, subjectMax); the read doors,
added after, did not, and nothing compared them.
There is now ONE spelling of each ceiling — admitSubject, admitKind, admitSource,
admitEvidence, and instantMax inside stamp(), which is the one parser every time
field passes — and every door asks it. So `count × ceiling` IS the byte bound of
everything this plane binds, holds and stores. An unknown kind or source is
refused on the READ path too: it can only ever match zero rows, so refusing says
so instead of charging for the scan. bound_test.go proves it twice: reflect walks
every In type and fails on a caller-sized field with no declared ceiling (the
structural half — a new field cannot arrive unbounded), and every declared ceiling
is refused over the wire with a refusal that does not carry the value back.
A LITIGATION HOLD THAT ARRIVES MID-SWEEP KEEPS THE RECORD IN BOTH PLANES OR IN
NEITHER. dispose sweeps the derived copy FIRST so nothing is orphaned in the
warehouse, then deletes from the record re-asserting `hold = 0`. That protected
the record and silently corrupted the copy: a record the delete declines to remove
has already been swept, its seq is behind the delivery cursor, and deliver() asks
the cursor rather than the world — so no retry re-sends it, pending() answers zero,
and the row is present in the compliance record and permanently absent from the
answer key a training join reads. A missing fraud label reads as an honest
customer, and the row is the one somebody is litigating. remove() now reports what
it kept, the sweep writes those back from the record, a repair that fails refuses
the request rather than acknowledging a short copy, and `restored` is a NAMED state
on the response. `disposed` counts what was disposed of rather than what was
identified: a compliance report that says it deleted a record it is still holding
is the wrong answer to the only question the report is asked.
THE PUBLISHED PRECEDENCE RULE NAMES THE FIELD THE RESOLVER READS. The op exists so
a caller holding a contested resolution can reproduce it, and its second term said
`seen` while stronger() compares `knowable`. The two are equal for a live pipeline
and differ for exactly the backfilled history the derivation exists to hold back,
so a caller reproducing the rule got a different winner and no way to see why. The
test counted the terms, which made the only property that matters unobservable; it
now pins each term to the field at its position.
Also: plugin/label/mcp.json is deleted. It is the only mcp.json in the tree, no
app on main has one, the generator that wrote them was retired with its gate
(manifest/mcp_test.go says so), and it declared seven tools under the old
operation ids with nothing left to regenerate or compare it.
25 mutants in scripts/mutate.py, 11 of them new, 25 KILLED: each reintroduces one
of these defects and the named test goes RED.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
resolve could spend the whole process on one authenticated request. Two
amplifiers composed. A key had a count bound and no BYTE bound, and the domain
matcher split a host into labels and re-joined every tail, so an L-label host
allocated a copy of each of its L suffixes: one 8 KB dotted key materialised
16.8 MB and 100 of them 1.7 GB, measured over the router. And `sets` had no
bound and no dedupe, so naming one set N times ran N times the answers.
- maxKey bounds one key in bytes at every door it crosses — looked up,
written, removed — and REFUSES rather than truncating, because a shortened
key is a different key.
- a domain suffix is now a slice of the host, not a join of its labels: the
same answers in O(L) headers over one backing array.
- a call may name each published set once, and a set named twice is consulted
once.
The override write had the same hole from the other side: maxOverrides bounded
rows and nothing bounded a row, so 10,000 entries x 11 sets was gigabytes of
attacker-chosen bytes on the one volume every other organisation's store lives
on. The same maxKey closes it; an over-long note is refused rather than trimmed;
and what one organisation may occupy on that volume is now a figure the code
computes and a test pins, so raising rows, a key, a note or the catalog is an act
with its consequence next to it.
An override is a record, so it is now shipped before it is acknowledged, the way
apps/research and apps/books do it — this deployment is one replica with a
recreate rollout, and an unshipped write is a control an operator believes is in
force and is not.
prune spared ONE version: the call site passed the current version for both of
the statement's two placeholders, deleting the rows behind every citation taken
in the window before a refresh. What a take supersedes is now decided by
sweepOld over what the plane held before it, and proved against a warehouse
rather than against the text of the statement.
The publisher's end of the same amplifier is closed with the same door. A take
is refused whole if it carries a member longer than maxKey — one no lookup could
ever reach, so it is only weight in the warehouse, in every hydrate and in the
snapshot every request reads — or more members than a published set holds: swing
measures GROWTH against the version a take replaces, and a first take has nothing
to measure against, which after a cold start is every take. maxBody comes down to
six times the largest source in the catalog (measured: 2.6 MB), because the parse
allocates before any later gate can look at what it made. A publisher's redirect
must keep the two properties its origin already had, TLS and a destination
outside this network: this process runs in the cluster, where "wherever the
publisher says" reaches the pod network and the metadata address.
Also: a take whose size swings past 4x is refused and the previous version
stands (force is the operator lever); the disposable list is refused whole if it
names a mailbox provider, which is the one-row attack the size gate cannot see
on the one unpinned source; an attest receipt with no version or no designations
is recorded as a refusal instead of a current, fresh list; the plane sweeps at
cold start instead of refusing every set for six hours after a deploy; the one
cross-fleet aggregation states its own memory and time budget; every source
states a typed redistribution Basis from a closed vocabulary, on the wire, so
the licence position is an audit rather than a sentence; refresh reads the ONE
SuperAdmin predicate rather than restating it, and an admin of their own org is
refused there, because this route writes the baseline every org reads; and the
bridge sits on this app's own leaf, not on the /v1/ml parent two other products
answer under.
65 tests green under -race (one skipped: it dials the real publishers). Every fix
has a regression test that fails when that fix alone is reverted: 24 mutants, 24
killed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/ml/reference publishes ten sets — disposable email domains, hosting and
Tor address ranges, crawler user-agent patterns, delegated autonomous system
numbers, card-scheme prefixes, browsers the fleet sees everywhere, and the
freshness of the designation lists the screening engine holds. Six typed ops.
The unit of version and freshness is the SOURCE, not the set, so one publisher's
outage neither blocks the others' updates nor silently shrinks the set. A
version IS the content digest of the sorted entries, which makes a re-take of an
unchanged publisher a no-op that says so, and a half-landed version resumable
from its cursor without depending on it — the primary key already deduplicates.
Two planes, two stores. The baseline tables carry no tenant column, so a
cross-tenant write is unrepresentable rather than refused; a tenant's own
allow and deny entries live in that organisation's own store. Resolution is
override then baseline, both through one candidate function.
The baseline carries only published data under terms we hold — every source
states its licence — and aggregates above a k-anonymity floor no single
organisation can reach. Sources we may not redistribute are declared seams that
refuse, because an absent set and an unlicensed one look identical from outside.
A set that never loaded refuses rather than answering "not listed", and a stale
set answers and says so: every answer carries the version, its as-of, its age
and whether it is past the bound.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/ml/labels is the answer key the model plane cannot build for itself: what
turned out to be fraud, who said so, and when they could first have said it.
Chargeoff, dispute, case, refund, review and the below-the-line sample all file
into one append-only record per tenant — its own encrypted SQLite file, no org
column, so a cross-tenant read is not forbidden but inexpressible — with a
derived ClickHouse copy for joining at training scale.
Seven typed zip ops, so the REST route, the OpenAPI operation, the MCP tool, the
CLI command and every SDK method are projections of one declaration: mlLabel,
mlLabels, mlResolveLabels, mlLabelCoverage, mlLabelVocabulary, mlDisposeLabels,
mlHoldLabels.
THREE PROPERTIES, EACH THE POINT OF THE PLANE, NONE OF WHICH THE FIRST CUT HAD.
DURABILITY. Nothing in the package called OrgStore.Sync, so the only ship was
CloseAll on a graceful shutdown. cloud deploys strategy Recreate at one replica:
an ungraceful termination lost every acknowledged record since process start,
and the successor hydrated the older durable snapshot OVER the local file — an
acknowledged compliance record was not merely at risk, it was overwritten by an
older copy of the tenant's own history. state.ship is now the ship-before-ack
step every write path calls before it answers, and an unacked ship fails the
request rather than acknowledging a divergent local copy. That covers BOTH
shapes: a replica that never held the lease (ErrNotOwner) and one deposed
between the write and the ship, whose fenced Put is refused at a stale round
with no error at all. The two sibling durable planes hold the same contract
(apps/research shipFor, apps/books shipLedger).
DELIVERY. The cursor was the pair (wrote, id) over a write clock truncated to
the second and a content digest — an order the writer never took. A write that
commits after a concurrent delivery has read, whose digest sorts lower inside
the same second, was already behind the mark: never mirrored, unreachable by any
retry, and pending() answered zero because it asked the same predicate. A hole
in the answer key reads as an honest customer. The cursor is now the store's own
AUTOINCREMENT position, allocated inside the insert on the single connection
every statement for a tenant runs on (sqlpool.Single), so cursor order is commit
order by construction.
LEAKAGE. `seen` is whatever the caller sent, bounded only by At <= Seen <=
now+skew, and nothing tied it to any fact the server observed — a dispute filed
today with seen == at was knowable a year before the record existed, and a
backtest standing two days after the event resolved it. Fact.Knowable is derived
server-side as the later of `seen` and the clock at the write, it is the only
time visible() and stronger() read, and it is a column in the derived copy so
the warehouse applies the same predicate the record plane does. A live pipeline
is unaffected: Wrote is within minutes of Seen and the derivation changes
nothing.
Also: the coverage window now ends where maturity begins, so the gate on
training no longer answers zero on its own defaults; Group returns the whole
matured cohort so `matured` counts what matured and `unlabelled` says why
`judged` is low; a litigation hold is a fact about the record with its own op
that can also release one, instead of a digest-excluded flag silently dropped on
any record that already existed; the warehouse partitions by month like every
other table in the fleet rather than by tenant, whose cardinality is the
customer count; and resolve answers one row per distinct event, so an event
named twice is no longer its own conflict.
Every one of those carries a mutant in scripts/mutate.py that reverts it and a
test that goes RED when it does: 14 rows, 14 killed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An org name is unique within an issuer and not across issuers: acme on
hanzo.id and acme on zoo.ngo are two unrelated businesses. A per-org SQLite
file never confuses them, because DataDir is per deployment and the brand is
the directory the file is in. A COLUMN in the shared columnar warehouse has no
such directory — org = 'acme' there is a predicate over both — so every row a
brand-shared table carries has to be keyed on <brand>/<org> and every read of
it has to bind the same form.
Qualify joins SanitizeOrg and OrgNamespace as the third org-naming door and the
only one for a shared plane. Qualified is derived from Qualify rather than
restated, so a change to the shape cannot leave a validator behind. Tenant
carries no JSON tags and has no constructor but Qualify, so no In struct can
decode one and no caller can assert a tenant for itself.
The brand half comes from Deps.Brand, never a header: a caller that can choose
its brand has chosen which tenant space its org lands in.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-02 11:27:31 -07:00
1643 changed files with 55801 additions and 18260 deletions
# TWO DIFFERENT FAILURES SHARE THIS STATUS, and treating them alike
# would turn one of them into a ten-attempt loop that ends in the
# wrong diagnosis. "Reference already exists" is the collision this
# loop is for. "Object does not exist" means OUR OWN COMMIT is not
# on github.com — which is a live possibility, because this workflow
# runs on git.hanzo.ai and claims against GitHub, so a commit that
# reached the forge and not the mirror lands exactly here. It is not
# a name to skip past; it is a repo that has not been published, and
# the next number would fail identically.
WHY=$(jq -r '.message // empty' /tmp/claim.json)
if [ "$WHY" != "Reference already exists" ]; then
echo "::error::claiming v${CAND} was refused with: ${WHY:-unknown}. If this is 'Object does not exist', commit ${SHA} is on the forge but not on github.com — the release lane claims versions against GitHub, so the mirror must carry the commit first."
exit 1
fi
# Taken. By us, or by somebody else? The distinction is the whole
# difference between a resume and a collision, and it is one GET.
echo "v$NEXT is already published from our sha — skipping the build"
RESUMED=1
else
echo "::error::v$NEXT is a version this run OWNS (tag at ${SHA}) but the registry already serves bytes built from '${REV:-an unlabelled commit}'. Another lane pushed onto a name it did not hold. Nothing here may overwrite it — publish the intended bytes under a new number and delete the foreign image."
exit 1
fi
elif [ "$CODE" != "404" ]; then
echo "::error::manifest probe for v$NEXT returned $CODE — cannot tell whether the name is free"
exit 1
fi
echo "version=$NEXT" >> "$GITHUB_OUTPUT"
echo "highest seen v$LAST (registry + git tags) -> building v$NEXT (document sha256:$SPEC_SHA)"
echo "resumed=$RESUMED" >> "$GITHUB_OUTPUT"
echo "highest seen v$LAST (registry + git tags) -> v$NEXT, claimed at ${SHA} (document sha256:$SPEC_SHA)"
- uses:docker/setup-buildx-action@v3
if:steps.ver.outputs.resumed == '0'
@@ -379,8 +475,17 @@ jobs:
# VERSION is what the binary reports as X-Api-Version. Without it the
# ldflag falls back to the `dev` default and a released image cannot
# say which release it is — and every car below keys off that header.
# REVISION is passed as a BUILD-ARG, not only as a label, because the
# Dockerfile declares `ARG REVISION=unknown` and stamps the label from
# it. A builder that sets only the outside label leaves that ARG at its
# default, and a builder that sets neither publishes an image whose
# commit is unrecoverable — which is precisely what the platform lane
# did for every image it ever pushed. Feeding the ARG makes the label
# truthful no matter which builder runs the Dockerfile, instead of
# truthful only in the lane that remembers to override it afterwards.
echo "::error::${img} resolves, but the bytes behind that tag were built from '${REV:-an unlabelled commit}', not ${{ github.sha }}. Another lane pushed over the tag this run owns. Nothing downstream may pin it."
exit 1
fi
echo "$img is built from ${{ github.sha }} — tag, commit and image agree"
# THE SMOKE GATE. Boot the image that was actually pushed and require it to
# reach "zip listening" without a crash signature, on release.go's boot env
@@ -460,37 +587,32 @@ jobs:
steps:
- uses:actions/checkout@v4
# The receipt, minted only now: build pushed it, smoke proved it boots. A
# tag can therefore never name an image that did not start.
# THE TAG IS NOT MINTED HERE ANY MORE — it was claimed before the build, as
# the compare-and-swap that made this version this run's to build (see the
# `image` job's claim step). Minting it here was the bug: for the whole
# length of a build, a number was "taken" only in the sense that a lane
# INTENDED to take it, and two lanes intending the same number both pushed
# images before either reached this step. The winner's tag then named the
# loser's bytes, and the loser died here — after the damage.
#
# 422 USED TO BE A HARD ERROR, and that is exactly what made a resume
# impossible: the second run of a release whose fanout failed died here
# instead of continuing. A 422 whose ref already points at OUR sha is this
# same release, already receipted — idempotent success. A 422 pointing
# anywhere else is a genuine collision and still fails.
- name:Tag the release
# What remains is the assertion that nothing moved underneath us. A git tag
# is not mutable by accident, so this is expected to be quiet; it is here
# because the one thing worse than a moved tag is a moved tag that shipped.
- name:The claimed tag still names this commit
env:
GH_PAT:${{ secrets.GH_PAT }}
run:|
set -euo pipefail
TAG="v${{ needs.image.outputs.version }}"
CODE=$(curl -s -o /tmp/tag.json -w '%{http_code}' -X POST \
MODERNC="$(CGO_ENABLED=1 go list -tags "libsqlite3 sqlite_fts5 sqlite_math_functions" -deps ./cmd/... ./plugin/... 2>/dev/null | grep -c 'modernc.org/sqlite'||true)";\
["$MODERNC"="0"]||{echo"SQLITE-GATE FAIL: a per-app binary links modernc.org/sqlite ($MODERNC pkgs) under CGO=1 — double-registers \"sqlite\" with hanzoai/sqlite(mattn) and panics at init. Find it: CGO_ENABLED=1 go list -tags 'libsqlite3 sqlite_fts5 sqlite_math_functions' -deps ./plugin/<app> | grep modernc";exit 1;}
# RED gate — ENCRYPTION PROOF + the cek.go GOLDEN-VECTOR KAT, under the SAME CGO +
@@ -183,8 +209,8 @@ RUN --mount=type=cache,id=cloud-gomod-v4,target=/go/pkg/mod,sharing=locked \
# image). TestUnwrapGoldenFixture asserts a FROZEN pre-luxfi-swap 61-byte DEK
# sidecar still decrypts under the shipped luxfi/crypto-AEAD code — existing
# encrypted stores stay readable, or NO image.
RUN --mount=type=cache,id=cloud-gomod-v4,target=/go/pkg/mod,sharing=locked \
Description:"Lists the accepted (chain, token, treasury) triples, so a browser can\nrender \"send USDC here\" without the addresses being baked into its bundle.\n\nThis exists because the console previously gated its top-up UI on\nNEXT_PUBLIC_HANZO_HUSD_ADDRESS/_TREASURY — build-time constants. Enabling a rail\ntherefore meant rebuilding and redeploying the frontend, and with them unset the\nUI reported \"not available yet\" no matter what the server could actually accept.\nServing the set at runtime keeps ONE source of truth (the server's config) and\nlets a rail be switched on without shipping a bundle.\n\nEverything here is public on-chain data; no secret is exposed, and the set is\nempty on a deployment that accepts no crypto rail.",
Fields:map[string]string{
"railList.rails":"Rails is every (chain, token, treasury) triple this deployment accepts.",
"railView.chain":"Chain is the human chain name, e.g. \"Base\".",
"railView.chainId":"ChainID is the EIP-155 chain id the wallet must be on.",
"railView.decimals":"Decimals is the token's decimal places — 6 for USDC, 18 for an 18-decimal\ntoken. Cents are derived per-rail from it.",
"railView.id":"ID is the stable rail id to name when submitting a transfer, e.g. \"base-usdc\".",
"railView.symbol":"Symbol is the display symbol, e.g. \"USDC\".",
"railView.token":"Token is the ERC-20 contract address to transfer.",
"railView.treasury":"Treasury is the address on this chain to send funds to.",
Description:"Streams a stored photo. No credentials — see the file header.",
})
zip.Describe("GET /v1/csrf",zip.Doc{
Description:"IssueCSRFToken mints the anti-CSRF token a browser echoes as X-CSRF-Token on\nevery money write (mint/revoke a key, top up, onboard, and the billing/commerce\nwrite verbs). The token is bound to the caller's validated identity and expires,\nso one minted for one identity cannot authorize a write as another.\n\nIt is answered no-store, so it is never cached by a shared proxy. This is the\nsame-origin endpoint the embedded console reads — the Same-Origin Policy is what\nstops a cross-site page from reading the response and forging a write.",
@@ -61,19 +51,6 @@ func init() {
"apiKeyList.keys":"Keys is every key the caller holds, at most one per type.",
Description:"Credits the caller's org for a stablecoin transfer they already sent\nto the treasury. It reads the receipt from that rail's chain, confirms a mined,\nsuccessful ERC-20 Transfer to the rail's treasury, derives USD cents from the\non-chain value using the token's own decimals, records the credit, and returns\nthe amount plus the new balance.\n\nThe credited amount is the ON-CHAIN value, never a number the caller sends, and\nthe credit lands on the caller's own validated org — there is no way to name a\nthird-party subject. Nothing is credited that the chain did not confirm: a\nmissing, failed or non-matching transaction is refused, and a deployment with no\npayment rail enabled says so rather than inventing a credit.",
Fields:map[string]string{
"walletTopupReq.fromAddress":"FromAddress is the wallet the transfer was sent from. Optional; when given it\nmust match the transfer's on-chain sender.",
"walletTopupReq.rail":"Which accepted rail the transfer was sent on, e.g. \"base-usdc\". The client\nnames it rather than the server guessing from the tx: the same address can\nexist on several chains, so inferring would risk crediting against the wrong\ntreasury. It may be omitted only while exactly one rail is enabled.",
"walletTopupReq.txHash":"TxHash is the hash of the ERC-20 transfer that was already sent to the rail's\ntreasury. The receipt is read from that chain; nothing is credited that the\nchain did not confirm.",
"walletTopupResp.balance":"Balance is the org's new USD-ledger balance in cents. Best-effort: a read\nfailure reports 0, and the credit has already landed either way.",
"walletTopupResp.creditedCents":"CreditedCents is the USD credit recorded, derived from the ON-CHAIN value\nusing the token's own decimals — never a client-supplied number.",
"walletTopupResp.status":"Status is how commerce recorded the payment.",
"walletTopupResp.txHash":"TxHash is the transfer that was credited.",
Description:"Creates — or rotates — the caller's API key of the requested type and\nreturns it ONCE. A real IAM failure surfaces as 502, never a fabricated key.\n\nRotating is what creating means here: a user holds one key per type, so the\nendpoint is idempotent by (caller, type) and the superseded credential stops\nworking. Two live secrets for one user would make \"revoke my key\" a lie.",
Description:"Onboard creates the caller's organization. Two flows, keyed on whether the caller\nalready has a home org (mirrors app/onboard/route.ts):\n\n - FIRST-RUN (no owner): create + MOVE the user in as admin, so their next JWT\n carries the new owner and the cloud scopes everything to it.\n - ADDITIONAL (owner set): create the org but do NOT move the user — a move\n changes their IAM owner (stripping a SuperAdmin's status + orphaning their\n current org). They reach the new org via the OrgSwitcher, which re-scopes\n X-Org-Id without touching IAM membership. A personal-org request from someone\n who already has an org is meaningless → 409.",
Description:"Onboard creates the caller's organization. Two flows, keyed on whether the caller\nalready has a home org (mirrors app/onboard/route.ts):\n\n - FIRST-RUN (no home org): create + MOVE the user in as admin, so their next\n JWT carries the new owner and the cloud scopes everything to it. This is the\n path a fresh OAuth sign-up takes, from the sign-up application's org.\n - ADDITIONAL (owner set): create the org but do NOT move the user — a move\n changes their IAM owner (stripping a SuperAdmin's status + orphaning their\n current org). They reach the new org via the OrgSwitcher, which re-scopes\n X-Org-Id without touching IAM membership. A personal-org request from someone\n who already has an org is meaningless → 409.",
Fields:map[string]string{
"onboardReq.name":"Name is the organization's display name. Ignored when personal is true, which\nderives the name from the caller's own username instead.",
"onboardReq.personal":"Personal asks for the caller's own workspace: the name is derived from their\nusername and the slug auto-suffixes to stay unique. Meaningless — and refused\n— for a caller who already has an organization.",
"onboardResp.additional":"Additional is true when the caller already had an organization and this one\nwas created WITHOUT moving them into it — they reach it via the org switcher.",
"onboardResp.displayName":"DisplayName is the organization's human name.",
"onboardResp.org":"Org is the created organization's slug, which is what X-Org-Id carries.",
"onboardReq.name":"Name is the organization's display name. Ignored when personal is true, which\nderives the name from the caller's own username instead.",
"onboardReq.personal":"Personal asks for the caller's own workspace: the name is derived from their\nusername and the slug auto-suffixes to stay unique. Meaningless — and refused\n— for a caller who already has an organization.",
"onboardResp.accessKey":"AccessKey is the identifier of the org-scoped credential provisioning minted\nwith the organization. Present on a first run that actually minted one.",
"onboardResp.accessSecret":"AccessSecret is that credential's confidential half, returned ONCE — on the\nresponse that mints it and never again. IAM keeps only its argon2id digest\nand blanks the plaintext, so this is the single moment it exists in a form\nits owner can read; a replay of the same provision re-reveals nothing.",
"onboardResp.additional":"Additional is true when the caller already had an organization and this one\nwas created WITHOUT moving them into it — they reach it via the org switcher.",
"onboardResp.displayName":"DisplayName is the organization's human name.",
"onboardResp.org":"Org is the created organization's slug, which is what X-Org-Id carries.",
Description:"Mints credit for one org. It is the ONE admin mint surface, and it\ndoes NOT mint in-process: it forwards the request to commerce's already-mint-gated\nPOST /v1/billing/credits, authenticated by the service token and scoped to the\ntarget org, then writes one tamper-evident compliance record. Commerce stays the sole\ncredit ledger; this is a thin, audited relay so there is exactly one place credit is\ncreated.\n\nThe body is commerce's OWN CreateCreditGrant contract, forwarded whole — every field\nit carries reaches commerce. The only two this layer reads are the target org (`org`,\nor `user` as the org-pool alias), which selects the namespace commerce's EdgeAuth\ntrusts, and `idempotencyKey`, which makes a double-clicked grant credit once.\n\nA FAILED grant is audited too, with the request body attached: an attempted mint is\nexactly as interesting to a compliance auditor as a successful one.",
Description:"Onboards a hosted service, or edits one, so a new host comes under the\nlaunch gate WITHOUT a redeploy. Re-registering an existing service PRESERVES its live\nswitch — editing the hosts of a service that is already open must not silently close\nit again.",
// IsMode reports whether a request mode selects the answer engine. An empty or
@@ -91,9 +108,11 @@ const (
"If the sources conflict or are insufficient, say so plainly and answer from general knowledge while noting the uncertainty. Never fabricate facts or URLs."
researchSystem="You are Hanzo Deep Research. Synthesize a thorough, well-organized report answering the question from the numbered web sources. "+
"Use clear section headings, compare sources, and surface the strongest evidence. "+
"Cite inline as Markdown links [title](url) after each supported claim. "+
"Do NOT add a trailing References section or bare URLs. Note gaps or disagreements between sources. Never fabricate facts or URLs."
"Write a structured report with section headings, compare sources, and surface the strongest evidence. "+
"Cite at least three distinct sources per section. "+
"Place each [title](url) immediately after the claim it supports; never a bare URL, never a period after a link, "+
"never a trailing References or Sources section and no footnote markers. "+
"Note gaps or disagreements between sources. Never fabricate facts or URLs."
)
// feeCents resolves the per-answer price in cents for a mode, most specific
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
hanzo-dev
