GitHub sends `labeled` only for a label applied AFTER creation. Labels chosen in
the new-issue form arrive as part of `opened` and produce no `labeled` event at
all, so watching only `labeled` missed the most natural way to ask — file the
issue with the label on it — and missed it silently.
`opened` therefore reads the issue's label SET, while `labeled` keeps reading the
one label that was just applied. That asymmetry is the point rather than an
inconsistency: reading the set on `labeled` would restart the run every time
somebody added "p1" to an issue already carrying the trigger label.
The loop's last leg. An issue started a run; the run pushed a branch; this is what
makes a pull request appear for it, so a person reads a diff rather than a branch
name.
THE OBVIOUS BUILD IS A TABLE and the table is wrong. The door would remember
"agent/x is for issue #7" and the completion path would look it up. That is
correct on one replica and silently wrong on two — the webhook lands on one
process, the push on another, the lookup misses, and nothing fails to say so.
cloud runs replicas=1 today with no HPA, which is exactly the condition under
which such a bug ships and waits.
So nothing is remembered, because every fact a pull request needs is already
durable somewhere:
which branches get one the forge's ref policy already says a coding run
writes refs/heads/agent/<x> and nothing else, ever.
So the question is about the NAME.
what it is called the head commit's subject — the agent's own words.
which issue it closes the commit message, because the run's prompt asked
for it. GitHub links a PR to an issue from the PR
body, the body IS the commit message, and the commit
cites the issue. The link is the agent's work
product, not our bookkeeping.
where it merges to the repository's real default branch, read from
GitHub. "main" is a guess, and a repo whose trunk is
named otherwise would get a PR against nothing.
IT HANGS OFF THE MIRROR, NOT OFF THE PUSH. A pull request needs its head to
EXIST on GitHub, and what puts it there is the outbound mirror. Subscribing to
the same native push the mirror subscribes to would race it — EmitLifecycle fans
out concurrently — and a PR opened first is a 422 against a ref GitHub has never
seen. "The branch is on GitHub now" is a fact only the mirror holds, so the call
is made where that fact is.
IDEMPOTENCY IS GITHUB'S. A second call for the same head and base is a 422
saying a pull request already exists, which is the state we were asking for: it
is a success, and it answers with the number a human clicks. We keep no
"already opened" set to discover the same thing more slowly and less reliably.
apps/git already imports apps/integrations (mirror_out reaches the App for the
per-org token), so this is a call along an existing edge and not a new seam.
Measured, not assumed: git, tracker, agents and integrations are all 403 on
api.hanzo.ai, so they are one binary and the call is in-process. The zip
framework's mustBuild data race under -race is pre-existing — it reproduces
identically on the untouched tree, which has zero occurrences of this call.
Proven: the title is the commit subject, the body carries "Fixes #7", the base
is the repo's actual default branch and the head is the run's branch; an
existing pull request answers with its number instead of an error; a
non-agent branch, an empty agent/, a non-GitHub target and an unparseable
remote all create nothing.
An issue labeled for the agent, or a comment that mentions it, starts one coding
run against the repository the issue lives in, and the run answers on that issue.
It is a DOOR, not a second engine: coding.Start over the plane, exactly as the
Slack door reaches it. A door that assembled its own dispatcher would have its
own pool and its own rules, and a run started from GitHub would be invisible to
the app that shares its name.
AUTHENTICATION IS NOT AUTHORIZATION, and this is the defect review found. The
HMAC proves GitHub sent the bytes; the installation id proves which tenant owns
them. NEITHER says the person who typed the comment may write the repository
they named — and an installation access token is write-capable across the WHOLE
owner namespace. Anyone can comment on a public issue. So without a repo-level
check, one comment starts a run holding authority over every repository under
that owner.
Two questions, two authorities, both required:
SCOPE is owner/repo among the repositories this org's installation actually
grants? An App can be installed on selected repositories, so "the
owner is connected" is strictly weaker than "this repository is
covered", and it is the covered set that bounds the token.
PERSON does the sender hold push on THAT repository? Asked of GitHub, which
is the authority on its own repository's permissions — never inferred
from org membership, which is a different fact.
Scope is settled first, so an ungranted repository is refused without our even
asking who the person is: which repositories an installation covers is itself a
fact about the tenant.
THE CREDENTIAL IS NEVER IN THE BOX. The installation token is minted here, used
here for two calls cloud makes itself — read a permission, write a comment — and
dropped. Nothing hands it onward. apps/sandbox/push.go carries the same property
for the objects: cloud reads the sandbox's commits and pushes them, so there is
nothing in the pod to steal rather than a small window in which to steal it.
That file arrives here too, adapted to this tree's core-API shape (Land beside
Run/Read/Write, a thin handler over it), and with it the fix that made `Secret`
true only when a body actually carries a credential — the cleartext guard was
refusing every run that had no token to protect.
The trigger is spawned BEFORE the tracker mirror and does not depend on it, for
the same reason the push path fires its build trigger before mirroring: a sync
outage must not silently stop work. It is detached, because authorizing a sender
and answering on an issue are several GitHub round trips and GitHub allows ten
seconds; it reads everything off the live request first, because a fiber buffer
is reused the instant the handler returns.
A test caught one real bug in the writing: "me@hanzo.ai" read as a mention. The
@ needs a whole-word boundary on BOTH sides — the right one is a different
account (@hanzo-ci), the left one is an email address.
Proven: the reader case gets no run and no comment while the permission call is
demonstrably made; the ungranted repository is refused before the person is
asked about; a writer reaches the link invitation, which is the sentence only
reachable past both checks. Unsigned is 401 and mints nothing; a spoofed
X-Org-Id beside a valid signature moves no tenant. go test -race, green.
The runtime already grew `tool`, an optional repo and `desktop`; this is the
cloud half of that contract.
`tool` (dev|claude|codex|python|node) and `desktop` are carried, not
interpreted. Cloud ships a NAME; the runtime owns the name→argv table, so
adding a tool is one edit over there and none here. `desktop` selects an image
variant — a tag — and nothing on this side branches on it.
THE REPO IS OPTIONAL, AND THE CREDENTIAL LIVES INSIDE IT. Every git field is
`omitempty` and `credential` is a POINTER: a value type always marshals, so a
repo-less run would still ship a blank `credential` object and the runtime could
not tell an absent grant from an empty one. A caller that supplies a credential
with no repo is REFUSED rather than quietly trimmed — silently dropping a secret
hides the bug that minted it, and the runtime says the same thing at its own
boundary, so both ends agree.
`Secret` is now true only when the body actually carries a credential. It was
unconditional, which meant the cleartext guard — a rule written to protect a git
token — refused every research and bare-exec run that has no token to protect.
SANDBOX_URL names where a run goes. A sandbox is not the bot: coding, deep
research and bare exec all want a computer to run something in, and none of them
wants the service that runs Slack channels. BOT_GATEWAY_URL stays correct for
bot traffic and is accepted here for one release, then deleted.
The transport learns none of this. `Call.Base` is a DESTINATION, which is a
transport concern; the caller resolves the address because resolving it inside
transport.go would mean that file learning what a run is — the exact line its
header draws. requireSecure now checks the base the call will ACTUALLY use;
checking the default while the bytes went elsewhere was a guard on the wrong hop.
A branch the forge does not have advertises the zero hash. Passing that to
packOut worked only because git cat-file happens to fail on it — an accident
standing in for an intention. Live proof re-run after the change: b48c4db6.
A sandbox could commit but not push, because it holds no credential. The
deleted daemon solved that by writing a token to /tmp at 0600 — readable by
the very code it was sandboxing. Every variant of putting the credential in
the pod has that shape, and a credential helper only swaps the secret for the
capability.
So the push does not happen in the sandbox. Cloud reads the commits (a read,
needing no credential) and pushes them itself. There is nothing in the pod to
steal, which makes the property structural rather than a small window.
POST /v1/sandboxes/:id/push. The sandbox builds a packfile with git's own
plumbing; it leaves through the existing exec channel, base64'd and chunked
under the 1 MiB stdout cap, checksummed end to end because that buffer
truncates quietly. Cloud streams those exact bytes to the forge over
receive-pack, so the commits that land are the sandbox's own — author, message
and parents intact — rather than one re-synthesized from a file list.
The credential comes from KMS at orgs/<org>/forge/push and fails closed: no
shared-token fallback, nothing minted, https only. The fast-forward check is
ours to make at this level of the protocol — receive-pack does not enforce it
and git's client does — so without it every push would silently be a force.
Live: sandbox-live-push-1786022970 committed 35116874 and cloud landed it on
git.hanzo.ai/hanzo/sandbox-push-proof, branch sandbox-proof-1786022997, with
the pod verified to hold no copy of the token.
TEST_TAGS said `sqlite_fts5`. The release image builds AND tests with
`libsqlite3 sqlite_fts5 sqlite_math_functions` (Dockerfile:213), and the
Dockerfile's own comment says sqlite_math_functions "is not optional under cgo"
because hanzoai/base's search layer calls the math functions.
So a local `make test` linked a SQLite the shipped one is not, and apps/base,
apps/code and apps/commerce failed on `no such function: acos`. That was read as
"this box's SQLite is old" — including by me, all of today, in every regression
count I reported. It never was. The tag list here and the tag list in the
Dockerfile are two statements of one fact, and they disagreed.
Measured, same box, same commit:
default cgo FAIL x3
cgo + sqlite_math_functions + fts5 ok, ok, ok
CGO_ENABLED=0 (pure Go) ok, ok, ok
Both engines were fine. Only the tag list was wrong.
Full suite with the corrected tags: ONE failure, apps/meet/ui — a real defect
(the auth module ships twice, so the OIDC code is redeemed once per copy and the
second redemption destroys the session the first established). Its SPA source is
not in this repo; apps/meet/ui carries dist/ only, so the vite resolve.dedupe fix
belongs where the meet SPA is built. Left red on purpose: it is true.
The stale prose above the variable is corrected too — it claimed the image builds
with two tags while the image builds with three.
The code interpreter tells the model to persist artifacts in /mnt/data. In a
deployed `exec` sandbox that directory was whatever the image shipped —
root:root 0755 — and the pod runs as runAsUser 1000, so every such write failed
with EACCES. The one directory the tool exists to fill was the one directory it
could not write.
Measured in a live sandbox rather than inferred:
$ kubectl -n hanzo-sandboxes exec m-643df8b317a6052579035359 -- sh -c '...'
id: uid=1000(sandbox) gid=1000(sandbox) groups=1000(sandbox)
drwxr-xr-x 1 root root 40 /mnt/data
mkdir: Permission denied
Only `dev` sandboxes ever got a mount at their workdir, because only they have a
project PVC. `exec` fell through the `if m.Volume != ""` and got nothing. It
survived because nothing checks: a run that prints its answer looks perfectly
successful, and only a run that SAVES something notices — as a traceback the
model apologises for rather than as an error anyone sees.
emptyDir, not a PVC: a code-interpreter session is exactly as long-lived as its
pod, which is what emptyDir already means. It is also what makes the mount
writable — the kubelet chowns an emptyDir to the pod's fsGroup, and the
securityContext already sets fsGroup 1000 — so the fix is the mount, not a chown
in the image. Bounded by SANDBOX_WORKDIR_SIZE (2Gi) because a container's
ephemeral-storage limit does not cover emptyDir usage on every runtime.
/v1/ask is described as agentic research across all of our data. It had one
contributor, and that one could not work: the advisor ships as its own plugin
binary (plugin/ask/main.go mounts ask.Mount and nothing else), so the read it
made for its figures — an httptest replay against its OWN router — could only
ever reach /v1/ask. books runs in a different process. The replay 404'd, the
gather failed, and every money question in production was answered by the
"I can answer questions about your finances" fallback with an empty figures
array, while books sat healthy one socket away. Measured on api.hanzo.ai:
"what is my MRR and how long is my runway?" -> {"figures":[],"domain":""}.
So the seam was not merely empty, it was unreachable, and filling it with more
in-process replays would have produced more contributors that classify a
question and then answer nothing. Every domain is now a PLANE call, which is
the one thing that crosses a process boundary.
One shape for all of them: plane.FiguresOut, a domain's headline figures for
the caller's org, already formatted by the domain that owns the number. One op
each — books_figures, projects_figures, git_figures — because the ANSWER is
each app's own (only books knows what a dollar of revenue is) while the shape
is shared, so the advisor needs no per-domain branch and a new domain is a new
op and one line, never a router edit. apps/ask keeps ONE contributor type; the
three domains are values of it.
TENANCY. plane.FiguresIn is an empty struct. There is no org argument to
validate because there is no org argument: zip forwards the gateway's own
assertion off the in-flight request and each op reads cloud.Who(ctx).Org,
refusing anonymous rather than defaulting it. The advisor hands the peer
cloud.As(c, "") and not c.Context() — an UNTYPED handler is not given the
request on its context, so the peers would have seen "nobody is calling" and
correctly refused every question ever asked.
Verified in the production topology: four plugin binaries, four processes, four
sockets, real IAM-validated tokens. "how many repositories do I have and what
changed recently?" answers domain=git, sources=[git/figures], Repositories 3,
Code stored 412 B, most recently advisor-core — byte-identical to what the git
process reports for itself at /v1/git/usage. The same token switched to an org
with no repos answers 0, and a non-admin principal's forged X-Org-Id is ignored
entirely.
Skipped: o11y (its read needs ClickHouse and an allowlisted product slug — no
plane read op exists and I could not honestly verify one), knowledge (no plane
ops, needs a Qdrant this repo does not deploy) and index (org-scoped and ready,
but a query needs a UID and nothing indexes an org's code under a nameable one).
Two working domains beat five that classify and return nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzo-inc main was RED and had silently lost a mechanism. `go test ./apps/sandbox/`
failed two cases here and passed on the forge, which is the tell:
imageFor("dev") = "sandbox:dev-2026.6.7", want "sandbox:2026.6.7-dev"
a digest wins over any tag: got "sandbox:dev-2026.6.7", want "sandbox@sha256:2baf7ede"
b028ab0f rewrote imageFor from a base that predated dbcb9040, so it did not change
one line — it reinstated an older file over newer work and took two things out
with it:
the ORDER back to `<class>-<version>`, so it asks for `dev-2026.6.7` while
the registry holds `2026.6.7-dev`. Every version-pinned pull 404s.
digestFor DELETED. `SANDBOX_IMAGE_DIGEST_<CLASS>` stops being read at all.
The second one is the serious half and it is silent in the worst way. The live
deployment sets all three digests, and universe records why: they are the only
name that cannot be rewritten under a running fleet. A binary without digestFor
IGNORES them and falls through to SANDBOX_IMAGE_TAG=latest — a moving tag,
which is the exact class of name that a `crane copy` put stock node:22 onto in
the first place. The deployment would look correctly pinned and would not be.
Every signal stays green; the only tell is a `whoami`.
Its commit message says "apps/sandbox builds and its tests pass". On the tree it
was written against, that is true. On this one it is not, and that gap IS the
bug — the same shape as the defect it was fixing.
This restores apps/sandbox/runtime.go from forge/main, which CD reads, and
nothing else. One file, by path. The two mains now agree on this function.
That also settles the fallback: forge has `<class>-unset`, this had `<class>-latest`.
I had independently written `-latest` too and it is the weaker answer. `-latest`
is still a MOVING tag, so it fails OPEN into whatever was pushed last; `-unset`
is published by nothing and fails CLOSED. The argument for `-latest` rested on
the bare tags being unrepairable — but they were repaired: a pod requesting
`:dev-latest` today resolves to sha256:4e5deb07, the same digest universe pins,
and inside it whoami is `sandbox`, /etc/sandbox-version is 1.0.0 and the OS is
Ubuntu 26.04. Not node:22. For a component that runs untrusted code, a fallback
reached only by misconfiguration should refuse, not improvise.
TestImageForNeverComposesTheBareClassTag passes under BOTH answers, which is why
it was written as an invariant rather than a table row.
Merging inc2 onto forge surfaced four failures. All four were the gates working;
none was merge damage.
principal.ProjectFrom / WithProject — apps/crawl's scopeOf reached for
cloud.Request, the pinned escape hatch, to read two facts: org and project.
OrgFrom already existed; Project had no ctx counterpart, so the hatch was the
only way across the typed-op seam. It has one now, and crawl reads the context
like everything else.
WithProject parks UNCONDITIONALLY, which is the one asymmetry with WithOrg and
is deliberate: an org is an AUTHORITY, so an unvalidated request must park
nothing rather than an empty tenant a query would honour. A project is a
NARROWING — every consumer ANDs it with an org that already gates — so gating it
here would state the authority twice and let the two statements disagree.
agents/builtin_test — the test asserted the default assistant carries no tools;
the code now offers it the whole door. The CODE is right: an empty offer reads
to a model as "there is nothing here", and the assistant was reporting it could
not reach the cloud while the door served 88 tools one socket away. The test
sentence was the stale half, so it now says what is true.
floor tracker 10 -> 9 — GET and DELETE on /v1/tracker/projects/{key}/issues/{num}
stopped registering, and that is deliberate: tracker moved to reading the forge
(dd6cd6de, 7d42283f), so tracker.go registers the list route and no longer the
two by-number ones. A milestones op arrived, hence -1 net. Lowered here, next to
the reason, which is what the ratchet asks for when a deletion is real.
openapi.yaml regenerated: 1772 paths / 2442 operations, measured by the
generator rather than merged as text — the third time today git resolved the
generated document to one side while the routes merged to the union.
go build 0. Full suite 4 red: apps/base, apps/code, apps/commerce (the
environmental three — this box's SQLite has no acos/fts5) plus apps/o11y, which
passes standalone and is ok on main; it flakes under the parallel sweep.
Every sandbox in production ran docker.io/library/node:22. imageFor's default
returned the BARE class tag — <image>:dev / :exec / :desktop — and all three are
sha256:0557ac14, byte-identical to upstream node:22.
Forensics: a Job `sandbox-image-seed` ran `crane copy docker.io/library/node:22`
onto them at 20:08 UTC 2026-08-06, from a locally built +dirty crane rather than
CI's pinned 0.20.2. A second burst at 20:41 also clobbered exec-latest,
dev-latest and desktop-latest — those three were republished by the sha-8af2c8a
build at 21:39-21:42 and are correct again. The bare three are not, and never
will be: NO LANE EMITS A BARE CLASS TAG. hanzoai/ci and bot both publish
sha-<short>-amd64-<class>, <class>-latest and <version>-<class>. A tag nothing
can create is a tag nothing can repair.
The failure was SILENT, which is the expensive part: node:22 starts, reads EOF,
exits 0. A run that did nothing and reported success — the same shape as the
Slack bug that cost a day, where the callee logged 200 while the caller had
already hung up.
Default is now <class>-latest, a real published tag. It is not the strongest
form — SANDBOX_IMAGE_TAG_<CLASS> pins a version and a deployment that cares
should set it — but an unpinned tag that is CORRECT beats a pinned-looking one
that is a stock node image. The docstring's preference for a pinned version was
never satisfiable by a name carrying no version at all.
apps/sandbox builds and its tests pass, including the one that referenced the
bare tag — so nothing was pinning that behaviour.
Carries the sandbox image and network work onto this remote: the unset-tag
refusal (Red's -unset), the bare-tag invariant, and everything else already on
the forge. The two mains had diverged by one commit each way; the forge is the
one CD reads.
I arrived at this line independently and with a WEAKER answer: `<class>-latest`,
because that is a name hanzoai/ci actually publishes. Red got here first with
`-unset` and Red is right. `-latest` is still a MOVING tag, and a moving tag
someone can overwrite out of band is precisely what put stock node:22 under all
six of these names in the first place; resolving to it fails OPEN, into an image
that starts, reads EOF and exits 0 as root. `-unset` fails closed. Their line
stands unchanged.
What is added is the same rule as an invariant over every class and every tag
state, rather than the one `dev`-with-no-tag row that proves the fix. The bare
form is not one wrong answer among many — it is the ONE spelling in this
function that resolves to something in the registry which is not ours, and three
different roads reach it: an empty SANDBOX_IMAGE_TAG, an empty
SANDBOX_IMAGE_TAG_<CLASS>, or a later edit that reorders the concatenation and
drops a separator. A table covers the roads someone thought of.
bot's imageFor now asserts the identical invariant on its own side. Two
consumers, one rule, written twice because a Go service and a TypeScript one
share no code — only a registry.
The bare `exec`, `dev` and `desktop` tags all exist in the registry and all three
resolve to ONE digest: stock node:22. No toolchain, no agent, and User is unset,
so it runs as root. imageFor fell back to exactly that spelling whenever the tag
was empty — so one unset env var silently swapped a hardened sandbox for a root
shell, and every signal stayed green. The pod runs, the API answers, and the only
tell is `whoami`.
Nothing publishes `-unset`, so the pull now fails with a name that explains
itself. A loud stop beats a silent downgrade; the whole point of this subsystem
is that the thing running someone else's code is not root.
Found by RED, which noticed that fixing `{class}-latest` and leaving `{class}`
bare left the gun loaded rather than unloaded.
The verdict surface took a principal or nothing, so an SDK holding a project
key -- the credential the wire was designed around -- had no way in and read
'X-Org-Id required' for a header it cannot mint. A key now resolves its org
through the ONE IAM seam the event door already uses, fails CLOSED on an
unresolvable key rather than falling back to the host, and is refused at every
write: reading a verdict and changing what everyone reads are different powers.
Authorship keys on whether a validated identity named the tenant, not on the
actor email, which a principal need not carry.
Brings stt: an OpenAI-compatible /v1/audio/transcriptions provider, so
cloud can front the in-cluster speech service (whisper on CPU) the same
way it fronts every other model plane.
The Slack front door ran on a literal `enso` behind a BRIDGE_AGENT_MODEL knob,
justified in its own doc as "the auto-routing SKU that selects per query in the
gateway's own catalog". That is not what enso is.
enso is one fixed route entry: deepseek-v4-pro, 1M ctx, reasoning: medium
(zen-svc catalog-enso.yaml, and the live enso-catalog ConfigMap it is served
from). One rung, no ladder, no per-query selection, no escalation. Three code
comments in three repos and the Slack App Home menu all asserted otherwise, so
the front door's tier had never been chosen — it had been inherited from a
property nothing in the system has.
So I measured it instead, paired and interleaved against the live enso service so
upstream load drift cannot flatter either side. n=42 full turns, the real
assistant instructions, the real describe-then-call protocol:
enso p50 3489 ms p90 8000 ms 1.71 model round-trips/turn
enso-flash p50 3905 ms p90 17269 ms 1.71 model round-trips/turn
paired diff 1512 ms median in enso's favour, t=-3.29 — significant.
The tier is RIGHT and the stated reason was invented. Round count is identical;
the difference is generation rate. On a tool-shaped turn enso-flash emits at
11.7 tok/s against enso's 19.1 for the same ~75-token answer, so "flash" is
quicker only when the answer is short enough for terseness to beat rate — a
greeting, not a question about the fleet. "what did we deploy today?" ran
5,386 ms on enso and 15,122 ms on enso-flash.
I very nearly shipped the opposite. A single-call benchmark says flash wins
(p50 1648 vs 3250 ms, t=2.15) and it is the wrong benchmark: it measures the one
turn shape — a greeting — where terseness dominates, and the Slack assistant is
a tool-driving agent. Two runs of the full turn disagreed with each other before
n was large enough to separate the effect from the upstream's variance, which is
the real reason a turn is sometimes slow: p90 is 2-3x p50 for both tiers.
What changes:
- the tier and its evidence move to cloud.ChatModel, in the file that owns
model policy, beside DefaultModel and FallbackModel. It is deliberately NOT
DefaultModel: that one serves one-shot text, this one drives tools, and the
tiers do not rank the same on those jobs.
- BRIDGE_AGENT_MODEL is gone. No deployment ever set it, and a second place can
only disagree with the first.
- App Home stops selling `enso` as "Picks the right model for each message" and
stops calling enso-flash "Fastest, for quick questions" — it is not fastest on
the questions people ask. It now names the tiers and marks the default, read
from cloud.ChatModel rather than from the menu's own first row.
- builtin_test pinned "the default carries no tools" against code that had
already been given the whole door, so the package's tests were red.
The App Home pin still wins over the default, and is still limited to the three
SKUs the menu offers.
Carries one line of an in-flight fleet.Describe rename in onbehalf.go that landed
in the shared worktree mid-edit; reverting it would have dropped that work.
(cherry picked from commit fce13e455d7c15157d4c6f62b7184d34c917db13)
The meet client mints its LiveKit join token on a bearer-only host, where
cloud's session cookie never arrives, so POST /v1/meet/getToken has to carry
the Authorization header the session read already carries. Without it every
room join answers 401 and the lobby lists rooms nobody can enter.
The tracker client picks up the same-origin predicate, the token-resolved
org header, the hostname-derived brand registry and the tenant-keyed cache.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`COPY . .` was shipping 1087 MB. /bin/ was already excluded, but nothing that
mattered lived there:
hanzo 586 MB untracked
o11y 126 MB untracked
host 26 MB untracked
.claude ~92 MB gitignored — agent worktrees, i.e. second checkouts of this
same repository, cache-keyed into the layer so every agent
run invalidated it
All build litter; none of it tracked, so nothing in the build can want it.
Measured after: 152 MB.
NOT excluded, deliberately: `sandboxes` (50 MB), which IS tracked. COPY . . may
legitimately carry it, and a build output somebody committed is a question for
that commit — not something to drop from the image behind their back while
chasing a number.
The .claude/ line is the same defect the zipdoc gate hit, in a second place: an
agent worktree is a whole other checkout, and any tool that walks the tree
without excluding it reads our own copy as our own source.
One cloud binary answers every brand's API host, and its validator trusts EVERY
white-label issuer — trustedIssuers unions BrandIssuers, deliberately, because
api.hanzo.ai and api.lux.network are the same process. So a lux.id- or
zoo.ngo-issued token is genuinely valid on the hanzo deployment.
The forge, though, is resolved from the DEPLOYMENT's own domain (brand.Sibling
→ git.hanzo.ai), never from the principal's. So a token vouched by another
brand's IAM arrived with an attested org and an attested username, and both of
this surface's controls then did exactly what they were built to do: the org
scoped the query, and the username was Sudo'd against git.hanzo.ai. But "alice"
on this forge is a DIFFERENT HUMAN from lux.id's alice, and the forge answered
with that person's private issues.
Two individually-sound controls composed into a cross-brand private-repo read,
because neither of them asked WHO VOUCHED. The vouching brand must be this
deployment's own, so that is now checked — once, in the one resolver every read,
every write and the milestone rollup passes through, rather than six times on the
routes.
An absent vouching brand still passes: that is an hk-/sk- key minted by this
deployment's own IAM, which is by construction this brand, and principal.Brand
publishes "no brand" as "no second fact to compare" rather than as a brand.
Normalised on both sides, because a case difference must not decide a tenancy
question. Same pair, read the same way, as apps/tenant.
Tests: a lux-vouched principal is refused on all six routes and nothing reaches
the forge; the deployment's own brand passes, in any case; an unbranded
own-IAM principal passes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The assistant was asked what the weather was and had no path to the internet.
It was right: the fleet's MCP door carried 88 grouped tools and none of them
was websearch or crawl. Not because the capability was missing — apps/websearch
is a working keyless meta-search and apps/crawl a working fetch-and-extract,
both in-process, both serving over HTTP the whole time — but because ONLY TYPED
OPS PROJECT. A raw handler appends nothing to zip's op registry, and that
registry is the single value every projection reads: the route, the OpenAPI
operation, the SDK method, the CLI command and the MCP tool. A subsystem of raw
routes serves perfectly and is invisible to the agent.
websearch gains its NATIVE door. Its two existing routes are foreign-protocol
adapters — LibreChat's frozen searxng and firecrawl contracts — and neither can
be a typed op: one is registered with All and zip has no typed All, the other
answers 200 to a malformed body on purpose. Those are facts about the ADAPTERS,
and the package doc had read them as facts about web search itself. POST
/v1/websearch runs the SAME metaSearch over the SAME engines and answers the
SAME envelope, at an address the registry can hold.
crawl's POST /v1/crawl becomes a typed op with NOTHING moved on the wire. Its
two documented blockers were real and both were carried rather than dropped:
the 400 with `{"success":false,"error":"missing url"}` is stated by the ANSWER
through zip's StatusCoder, so the document publishes 400 with that schema; and
the 1 MiB bound and body-tolerance are facts about BYTES, which a typed op
never sees, so they are asked in middleware where the bytes still are.
TestCrawlWireSurvivedTyping is the measurement — same three inputs, same three
answers, byte for byte.
Both gates moved into the HANDLER, because a tools/call reaches a typed op with
no route and therefore no middleware: a gate that lived only in middleware would
be no gate at all for the two projections this exists to create.
fleet/reachable_test.go is the proof, and it stubs nothing: each subsystem
composed the way cloud.Serve composes a plugin child, on its own unix socket,
under the real composed door — so refuse() runs for real and an operation that
passes is one an agent can reach. post_v1_websearch, post_v1_crawl and
post_v1_exec all project. Their names are in surface_internal_test.go's
survivors table, which is the one place the rule is asserted; no second gate
site was added.
The work items on /v1/tracker were a SQLite table beside a github.com feeder,
while the estate files, labels and closes its issues on git.hanzo.ai. Two stores
under one prefix are two answers to what the state of a piece of work is, and
they disagreed the first time anyone touched the forge directly, which is every
day. The forge is now the store: every read is a read OF it, every write a write
TO it, and nothing here caches or mirrors a row.
A board is a repository, a column is a LABEL, and a card is an issue. Reading
the column off a label is what makes the board and the forge web UI the same
object seen twice — relabel in either and the card moves in both. So the
repository lifecycle is NOT on this surface: creating, renaming and deleting a
board are forge operations under forge permissions, and a second door onto them
here would be a weaker guard on the same object. Those three answer 405 naming
the forge, which is a different fact from 404.
Milestones are repo-scoped upstream and there is no org-level list, so the org
view is a server-side fan-out over the repositories the caller can see —
bounded, and failing whole rather than returning a partial rollup that reads as
complete.
TWO INDEPENDENT CONTROLS, because neither is trusted to be sufficient and this
forge really does host private orgs. The org comes from the validated principal
(principal.OrgFrom) and never from a path, query or body. Then every call is
made with Forgejo Sudo as the caller's own IAM username, which DROPS PRIVILEGE
to that user — measured against the live forge: the machine token reads
hanzo-private/patents (200), the same token sudoed as a non-member gets 404,
byte-identical to anonymous. So a bug in the first control cannot leak a private
repository on its own, and a write is attributed to the HUMAN rather than to a
shared bot.
One credential, held in KMS at orgs/hanzo/deploy/FORGE_TRACKER_TOKEN@prod —
never an env file, never a browser-side PAT. A separate secret from the universe
pin token on purpose: one credential per capability, so a compromise of the
tracker cannot deploy. Resolved lazily with a TTL so rotation is live without a
restart, invalidated when the forge rejects it, and fail-closed at every step —
an anonymous client would quietly serve public repos and read as "your board is
empty" rather than "this deployment is misconfigured". The forge host is
brand.Sibling of the deployment's own API host, so a white-labelled deployment
cannot read another brand's forge.
Also closes two tenancy defects found beside this work:
plane.AgentPRIn carried an Org the agent-PR seam read off the wire and passed
straight into the per-tenant store selector, so a caller on the plane could file
a work item onto ANOTHER tenant's board by naming it. Its sibling on the same
socket, IssueIn, has never had one. The field is gone, the handler reads
cloud.Who(ctx), the caller carries the org in the ENVELOPE (which is re-checked
by the same OrgOf rule as the HTTP boundary), and a reflection test now fails if
an org-shaped field returns to either input.
The audit trail's Home field means "a platform SuperAdmin acted inside another
tenant". Its predicate was home != effective, which WAS impersonation back when
a SuperAdmin was the only principal who could act outside their home org. Since
membership-based org switching, any ordinary member of two orgs trips it the
moment they work in their second one — telling auditors that routine work was an
admin impersonation, and burying the real events in volume. The predicate is now
the fact it always meant: home is the reserved admin org (authz.AdminOrg, the
issuer's constant).
Tests: 22 on the forge client (fail-closed with no actor, credential never in an
error, bounded fan-out, pagination that terminates against an endless forge, a
compile-time refusal of tenancy in the filter), 12 on the surface (cross-tenant
read, no-principal refusal, CSRF-refused writes never reaching the forge,
attribution of a move). The retired SQLite HTTP surface's tests go with it; the
two security properties they held — the CSRF gate and the per-IAM-project store
isolation, which still backs the plane doors — are re-pinned against what
survives. forge/live_test.go exercises the real forge, skipped unless
FORGE_LIVE_TOKEN is set, because a stub can only confirm we built what we
believed and not that what we believed is true.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TWO defects, both silent, both found by trying to start a real sandbox.
THE TAG ORDER WAS BACKWARDS. imageFor composed <class>-<version> and asked for
`dev-2026.6.7`; the registry holds `2026.6.7-dev`. So the default path 404'd on
an image sitting right there, and the only reason anything ever ran was a test
that overrode the image entirely. The publisher wins this argument: hanzoai/ci
appends a per-image tag-suffix to the version, so the whole fleet is
<version>-<suffix> and a consumer spelling it the other way is simply wrong.
A VERSION TAG IS NOT A PIN HERE. The sandbox image ships from hanzoai/bot under
BOT's package.json version, last bumped 2026-06-07. A rebuild today — commit
d1904514f4, "box: uv ships at the root" — republished `2026.6.7-dev` from source
two months newer. A tag that gets rewritten is not a pin, and one that LOOKS
pinned is worse than `latest`, which at least admits what it is.
So SANDBOX_IMAGE_DIGEST_<CLASS> is honoured ahead of any tag. `repo@sha256:…`
names bytes, and bytes do not change under a running fleet.
PER CLASS, not one variable. A single SANDBOX_IMAGE_DIGEST would have handed
every class the exec image while every log line still read correctly — the same
shape as the tag bug above. image_test.go pins that case specifically, along with
the order, because both of these are invisible until a pod fails to pull.
The SDK's leaf and the plane's root reach the same evaluator; the comment
said whose protocol it was rather than what the line does. The refusal said
'X-Org-Id required' to callers that can never send one — a project key does
not carry a tenant on this surface — so it now says what would satisfy it.
The image ends `USER sandbox` (uid 1000). A mounted volume — the emptyDir at the
exec class's workdir, or a project PVC — arrives owned root:root 0755, so uid
1000 takes EPERM on its first write. The pod is Running, the API answers, every
health signal is green, and the agent cannot save a file.
Reproduced before fixing, in both shapes:
uid=1000(node) gid=1000(node)
drwxr-xr-x 2 0 0 /mnt/data touch: Permission denied
drwxr-xr-x 3 0 0 /work touch: Permission denied
A Dockerfile `chown -R sandbox /work` cannot fix it — the mount happens after
the image layer and shadows it. fsGroup is the only mechanism that reaches a
volume: the kubelet chowns it to that GID and adds it as a supplemental group.
runAsUser/runAsGroup are stated rather than inherited from the image, because
the two have to agree and the one checkable from outside the image should be the
one that says so.
The image's own comments already asserted this existed — "the pod runs
runAsNonRoot with runAsUser 1000, so the uid is fixed by the securityContext."
Nothing fixed it. The live proof missed it because a stock node:22 runs as root,
so the one path exercised was the one that happened to survive. That is the
whole hazard of proving a mechanism with a substitute image.
The assistant reported it "can't query your projects directly" and named the
exact tools it would have used — hanzo_projects after hanzo_describe. It was
right, and the two halves of the system disagreed:
instructions "your tools are grouped hanzo_<subsystem>; call hanzo_describe"
door.go:93 if org == "" || len(want) == 0 { return nil }
want comes from callableTools(a), which iterates a.Tools — and the builtin
agent's Tools was empty, deliberately, from when the tool loop decided what to
offer. So the model was taught a protocol and handed nothing to practise it on,
while the door served 88 tools one socket away.
ToolsAll ("*") says "whatever the fleet serves", resolved per run. It has to be
STATED rather than implied: an agent that declares nothing still gets nothing,
because a user-defined agent's tool list is its authority. But the default
assistant cannot enumerate a surface discovered at runtime — the whole point of
grouping was that the set changes when a subsystem ships, so any list written
here would be stale by the next deploy.
The wildcard offers the door's tools AS GROUPED, not the ops flattened back out:
1,189 flat tools were 977 KB (~244k tokens) merely to list, and the same
operations grouped are 88 tools in 63 KB. Flattening here would hand back every
byte the grouping saved, and would contradict the prose the model is reading.
Also merges main (242 commits) into the security branch. plane.go's conflict was
two additive type blocks at one offset — both kept. The two generated files were
regenerated rather than hand-merged.
Red re-reviewed the chrome and found four client-side holes. All are fixed in
hanzoai/admin apps/tracker@577bb4c; this is the rebuilt bundle. No Go changed.
What is different in the bytes:
- ONE parser-based same-origin predicate gates the bearer. The old check
short-circuited on `startsWith('/')` without parsing, and seven root-relative
shapes — including ones carrying a raw TAB, LF or CR — fold to a foreign
authority under the parser fetch itself uses. Those shapes took the Bearer
and X-Org-Id off-origin. The base client attached the bearer with no check at
all.
- X-Org-Id is resolved against the token that carries it. The selection lives
in localStorage (permanent, shared by every user of a browser profile) and
the token in sessionStorage (dies with the tab); nothing reconciled them, so
the second person to use a machine sent the first person's org and got their
OWN rows back displayed under it. cloud already refused to honour it — this
stops it being sent, drawn, or kept.
- The IAM base comes from a brand registry (hanzo -> hanzo.id, lux -> lux.id,
zoo -> zoo.id) mirroring cloud's own, selected by brand rather than by an
arbitrary URL.
Audited on the shipped chunks: React only (0 hits for svelte/vue/solid/preact/
angular/htmx/alpine), 0 credential surface, 0 origin-derived IAM base.
Still not verified here: `go build` and `go test ./apps/tracker/...` — this box
has no Go toolchain. embed_test.go's assertions were checked by hand against the
new dist/ (base /tracker/, /v1/tracker baked in, every asset index.html names is
present, .sync-stamp still dot-prefixed).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
live_test.go proves a POD runs code. It calls r.exec, r.stop and r.purge
directly, so it never issues a request — delete every line of Routes() and it
still passes. What ships is not the runtime, it is the seven addresses in front
of it, and nothing was reading those.
So: the same proof, driven only through the routes.
CREATED over HTTP: id=m_53ce6be390c9eab1903222b7 class=exec status=running
WROTE FILE over HTTP: {"bytes":63,"path":"/mnt/data/answer.js"}
EDIT PERSISTS over HTTP: export const answer = 42; // written over .../fs
EDITED CODE RUNS over HTTP: ANSWER=42
FAILURE IS DATA over HTTP: exitCode=3
CROSS-ORG REFUSED over HTTP: GET as other-org -> 404
LIST CONTAINS IT over HTTP
DELETE /v1/sandboxes/m_53ce6be390c9eab1903222b7 -> 204
Three of those could not have been caught below the route. A non-zero exit is
DATA on POST /:id/exec — 200 with exitCode 3, not a 500 — and an agent that
cannot read a failing build is no use; that shape is a handler decision and
lives nowhere else. Org isolation is likewise a route fact: the unauthenticated
call is checked FIRST, so a later 201 cannot be explained away by an open door,
and a second org asking for the same id gets 404 rather than the row. The write
lands at /mnt/data because workdirFor answers the class, which is the same
confusion that made the git proof cd into a directory that was never there.
Guarded by SANDBOX_LIVE like its sibling, so it skips where there is no
cluster and runs where there is:
SANDBOX_LIVE=1 SANDBOX_NAMESPACE=hanzo-sandboxes go test ./apps/sandbox/ -run TestLiveHTTP -v
ledger.await gave a background debit 2 seconds to reach the ledger. Alone that
is ample; on the release gate the same process is running the whole fleet's
suite, and TestFeatures_IsPricedFromItsWindow failed there at "only 0 of 1
debits reached the ledger" while passing 20 of 20 runs in isolation.
That is a red money gate that proves nothing about the money — the metering it
exists to defend was never in question — and it is the shape of red that teaches
people to re-run a gate instead of reading it.
The question the helper asks is whether the debit arrives, which has a yes/no
answer that does not depend on the machine's load. Give it 30 seconds. The poll
is unchanged, so the happy path still returns in milliseconds and nothing gets
slower; only a genuine failure takes longer to declare.
Measured: apps/risk green over 6 consecutive runs under 12-way CPU contention,
which reproduced the failure before.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two concurrent transitions to `published` posted the item to its channels
twice. The fan-out was already leased; the erasure that defeated it was not.
Transition reads the document, then writes it back with the new status. That
write carries the WHOLE document — UpdateData replaces it, so external_ids
cannot be left out of the map without being deleted — and the snapshot it
writes is read before any fan-out. So: both callers read an empty skip-set, A
publishes and records its external_ids, B's stale snapshot writes that skip-set
back to empty, and B's fan-out (correctly leased, correctly re-reading) finds
nothing to skip and posts the item again. The lease inside Publish could not
see this, because the erasure happened outside it.
Widen the section to match the invariant it was defending: on the one edge that
distributes, Transition takes the item's publish lease across read, edge-check,
status write and fan-out. The loser then reads after the winner recorded — a
no-op edge (CanTransition is true for from==to), re-stamping the same status,
skipping every channel already on record. One post, both callers succeed.
Publish keeps its own acquisition for its own callers and delegates the fan-out
to publishHeld, which states the held lease as a precondition; the lease is not
reentrant, so Transition calls publishHeld directly. A contender that cannot win
the lease inside the wait window is refused with 409 rather than writing a
document it would corrupt.
Measured on apps/content: 2 of 30 runs failed before, 0 of 200 after.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The cleanup this file promises — "a leaked pod on a shared cluster is somebody
else's problem tomorrow" — was deferring purge, and purge deletes the VOLUME:
func (r *runtime) purge(ctx context.Context, m Sandbox) error {
if err := r.ready(); err != nil { return err }
if m.Volume == "" { return nil }
Neither live test declares a Volume, so purge returned at that second line every
time and deleted nothing. The pod was never anybody's job. Measured on
hanzo-k8s: four sandbox-live-* pods still Running, the oldest twelve minutes,
none carrying a deletionTimestamp — every pod both tests had ever started.
stop is the one that ends the pod, and it is separate from purge on purpose:
the volume holds the only copy of a checkout, so deleting it is opt-in. A test
that owns both wants both. Deferred in the order the product uses them.
Verified — pods gone after the run rather than left Running, and the proofs
still hold on the way out:
RUNS CODE: SANDBOX-RUNS-CODE v22.23.2
EDIT PERSISTS / EDITED CODE RUNS: ANSWER=42 / FAILURE IS DATA: exit=3
GIT PRESENT: git version 2.39.5
COMMIT MADE: 49523ec the agent committed this
ok github.com/hanzoai/cloud/apps/sandbox 79.993s
Pinning @v1.0.38 — which this file's own comments call for — made the forge
stop constructing a run entirely. No run row, nothing to inspect: the 'dead CI
is not red, it is absent' failure the same comments describe, reached from the
other direction. Absent is worse than red, so this goes back to @v1.
The lane remains broken and was before any of this: gate never gets a runner
while containment succeeds on the same run against an idle fleet, and no
release has minted since v1.801.490. Reverting restores a lane that at least
REPORTS its failure; it does not fix it, and the comment now says so instead of
implying the pin was the answer.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestLiveSandboxDoesGit has been RED since exec's workdir split off from dev's.
The script opened with a bare `cd /work` — the `workdir` constant — against a
sandbox declared `Class: "exec"`, and workdirFor("exec") is /mnt/data. Kubernetes
creates a pod's workingDir and nothing else, so /work was never there:
live_test.go:163: git flow exit=2 stderr="sh: 2: cd: can't cd to /work\n"
So the one test that proves an agent can COMMIT was failing for a directory, and
"COMMIT MADE" has not actually been observed since.
TestLiveSandboxRunsRealCode hid the same mistake instead of failing on it: it
writes with `mkdir -p /work/src`, which CREATES the directory it then proves the
edit survives in. The edit was real and the persistence was real, but both were
happening in a directory the product does not read — confine() resolves every
caller path under workdirFor(class), so nothing served through /v1/sandboxes
would ever have seen that file. A proof that passes in the wrong directory is
weaker than a proof that fails.
Both now ask workdirFor(m.Class), which is the same question runtime.go asks when
it sets workingDir and confine() asks when it resolves a path — one source for
the path instead of a constant that is right for one class and wrong for the
other. Against hanzo-k8s, node:22, ns hanzo-sandboxes:
RUNS CODE: SANDBOX-RUNS-CODE v22.23.2
EDIT PERSISTS: export const answer = 42; // edited by the agent
EDITED CODE RUNS: ANSWER=42
FAILURE IS DATA: exit=3 stderr=to-stderr
GIT PRESENT: git version 2.39.5
COMMIT MADE: a9423dc the agent committed this
FORGE REACHABLE: exit=0
ok github.com/hanzoai/cloud/apps/sandbox 64.902s
The default was registry.hanzo.ai/hanzoai/sandbox. That host is a DEPRECATED
ALIAS — one Traefik file router serves both names from svc/registry:5000, so it
resolves, which is exactly what makes a stale name survive: nothing breaks, it
just spreads. oci.hanzo.ai is the canonical one.
Measured while confirming this: a probe pod in hanzo-sandboxes sits in
ImagePullBackOff on registry.hanzo.ai/hanzoai/sandbox:exec. The alias is not the
reason — the image does not exist under either name yet — but the next person to
read that error should be sent to the right host.
Two comments carried the old name too, including the one explaining why a
sandbox uses its own ServiceAccount rather than `default` (DOKS re-attaches its
own registry pull secrets to every namespace's default account, so a sandbox
inherited a credential for someone else's registry and asked ours anonymously —
a 401 that reads like a bad password and was no password at all).
The mirror moved while the ci pin was being fixed. Merged, not forced: the
forge is canonical but that does not make GitHub's commits disposable.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every cloud release since v1.801.490 has died in `gate`, and the rule that
prevents it is written in this file directly above the line that broke it:
pin the immutable patch tag, not the rolling `v1` alias.
`gate` is the ONLY job here that is a reusable-workflow call rather than a
literal `runs-on`, and it is the only one that never gets a runner —
runner_id 0, zero steps, ~32 minutes, then failure — while `containment` on
the same run succeeds against an idle 10/10 fleet. Runs 925 and 926 (at
7c50638e) and 928 (the forge merge) all died identically, so this predates
that merge; it is not what the merge introduced.
The alias is not just stale (v1 -> f098b39e vs v1.0.38 -> dfac7dc4) — it is
unpinnable by design: sync-from-github refuses to move a tag that already
exists, so `v1` on the forge can never catch up. Only a NEW immutable tag
syncs, which is exactly why the rule says to name one.
If this does not restore the lane, the next thing to read is why the forge
declines to schedule a called workflow's jobs while scheduling sibling jobs
in the same run.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
podSpec named no ServiceAccount, so every sandbox pod ran as `default` — and
DOKS's registry integration re-attaches its own DigitalOcean pull secrets to
every namespace's `default` account whenever it reconciles. A sandbox therefore
inherited a credential for a registry that is not ours and died asking
registry.hanzo.ai for its image anonymously: a 401 that reads like a bad
password and was in fact no password at all.
`sandbox` is ours and DOKS does not manage it. It is bound to no Role and
carries exactly one thing, the pull secret; the line above it still refuses the
pod a token, so naming an account grants nothing that omitting one withheld.
A constant and not an env var: which accounts exist in the sandbox namespace is
decided by the manifest that creates the namespace, so a second knob here could
only ever disagree with it. Declared in universe infra/k8s/sandboxes/registry.yaml.
Rebuilt from hanzoai/admin apps/tracker@e3d5672. The bundle changes what the
page does about identity and chrome; nothing in this package's Go changed.
ONE DOOR. The SPA signed in against `<origin>/v1/iam` — whatever IAM the host
serving it embeds. That is a second identity authority by construction, and
where the embedded store is not the one holding the accounts it fails as a login
screen nobody can get through. It now signs in at hanzo.id, which is the issuer
this binary validates against (brand registry: hanzo -> https://hanzo.id, JWKS
https://hanzo.id/v1/iam/.well-known/jwks). The bundle carries no form and no
password field: it asks IAM to authenticate somebody and holds the bearer.
That is the point of the cutover rather than a detail of it. tracker.hanzo.ai is
Huly today and Huly runs its own login form, so replacing it is only worth doing
if it REDUCES the number of places identity can be established.
TWO CONTROLS. The chrome is the org switcher (left) and the user menu (right)
over the board — no nav rail, no env/version/clock/theme chips.
THE SWITCHER RE-SCOPES. The selection rides as X-Org-Id on every /v1/tracker
call. It is a request, not a claim: SanitizeIdentity already deletes that header
and re-mints it, honouring the value only when the token's signed `orgs` claim
contains it — so the switcher lists exactly that claim, and cannot offer a
selection the server would silently swap for the home org. No change was needed
on this side; the mechanism was already here and tested.
embed.go and the README said the SPA "sends no tenancy of its own", which is now
half true and would read as a promise the page no longer keeps. They say what it
asserts (nothing) and what it asks for (a selection the server validates).
Verified against this bundle in a browser: sign-in offers one button and zero
inputs and leaves for https://hanzo.id/v1/iam/oauth/authorize with S256 PKCE and
client_id=hanzo-cloud; the board renders with exactly the two controls; choosing
acme sends X-Org-Id: acme and returns acme's board. Bundle audit: React only —
zero Svelte/Vue/other — and zero credential surface.
NOT verified here: `go build` and `go test ./apps/tracker/...`. This box has no
Go toolchain. The Go sources are unchanged, and embed_test.go's assertions were
checked by hand against the new dist/ (base /tracker/, /v1/tracker baked in,
every asset index.html names is present, .sync-stamp still dot-prefixed).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
openapi.yaml is GENERATED. git merged it as text, so the routes merged to the
union (1764) while the document took one side (1759) — five paths served and
unpublished, which every SDK generator would have missed. TestTheServedDocument
IsTheArtifact caught it and named the fix; this is the fix.
fleet/mcp.go kept both halves: Serve publishes the door at an internal address,
signpost answers the framework default with a 308. d.Serve IS on.Post(path,
d.serve), so the abstraction subsumed the raw line rather than racing it.
Every sweep in mk/fleet.mk was a `for` in a shell recipe, and a shell loop does
one thing at a time: 121 independent links, one after another, with the Go
toolchain itself pinned to two compilers. From an empty cache that shape takes
586s. The same builds scheduled across the box take 300s, and 25s warm.
They are static-pattern targets now — one per app, so the set is CLOSED and
`build/nosuchapp` names the app instead of quietly matching. (They are also
explicit rules, which is what makes them work at all: an implicit `build/%:` is
never searched for a target listed in .PHONY, so the phony declaration these
obviously wanted turned every one of them into "nothing to be done".)
`make -k` replaces `set -e`, and that is the part that was actually dangerous.
The old loop stopped at the first failure and never ran the apps behind it,
which reports nothing, and nothing is indistinguishable from passing — it is how
an unbalanced brace in apps/commerce hid ~45 apps that silently never
regenerated. make continues past every failure and names each one itself.
THE EXEMPTIONS WERE EXEMPTIONS FROM DESCRIBING, APPLIED TO BUILDING. kafka needs
a live broker and zen is coresident, so neither can MOUNT alone — and `describe`
was the only sweep that ever touched an app, so neither was ever COMPILED. Both
could have stopped linking on main with every gate green. `binaries` carries no
exemptions, because there is no such thing as an app that cannot be compiled by
itself; `describe` builds those two and skips only the projection. All 121
manifest apps link alone, verified as an exact bijection against ./bin.
`check` CALLS describe instead of restating it. They were one sweep written
twice with different skip messages and different error text, which is what let
them disagree — the bug its own exemption comment already had to be written
about once.
`dist` is that same recipe with the platform spelled into the name:
dist/<app>-<os>-<arch> for every app and every platform, 242 binaries, which is
the triple manifest/release.go resolves a plugin by. CGO_ENABLED=0 there is
load-bearing — a plugin fetched over the network runs on a box we did not build.
Two bugs found writing it: `VAR=x cmd1 && cmd2` sets the env for cmd1 only, so
the floor cross-compiled and everything behind it built for the host; and
`go generate` inherited the cross-target and tried to exec an amd64 zipdoc on
arm64. A generator runs where make runs, always.
SIZING IS ONE RULE, J apps at once with P compilers each, and J*P is what the
box sees. J is bounded by MEMORY read from the cgroup before /proc/meminfo,
because the git-runner pod is 26Gi on 6 CPU while nproc inside it reports the
NODE's cores — sizing off nproc there asks for a dozen concurrent links in a
cgroup that holds eight. The heaviest link measures 1.67 GB resident, not the
6.23 GB the pod's config still cites; that number predates the light host.
mk/go.mk stops imposing -p=2 on SOLO builds, which never had the problem it was
protecting against: one app from an empty cache is 57.6s at -p=2 and 41.4s with
the box. The runner's own injected GOFLAGS still wins, via `?=`.
Two negative results are recorded rather than dropped, because both are the
first thing the next person reaches for:
* Prebuilding the shared floor is SLOWER. J cold builds each compile the
587-package root, so warming it first is the obvious fix; at J=10 P=2 it went
300s -> 339s (the root) -> 327s (all 4149 packages). One process on a
dependency-shaped graph idles the box longer than the duplication costs.
* More parallelism is slower past a point: 431s at J=20 P=2 against 300s at
J=10 P=2, the same work with twice the actions.
hanzo.yml names the binaries: lane it cannot yet declare, and both blockers,
instead of tripping over them: ci's run:/out: lane indexes per RECIPE rather
than per FILE (unreadable to manifest/release.go), and `bucket:` needs an
S3_ADMIN_* credential that is in KMS for no org — declaring it would publish
nothing and red every tag build, which is the state ci's own site: lane shipped
in.
Two defects behind "it is SUPER SLOW and it seems stupid". Measured in
production, a turn is:
slack ack 0-6 ms (well under Slack's 3s retry threshold)
plane overhead ~25 ms (53,009 vs 52,985)
model completion 9,955 / 35,893 / 52,985 ms <- all of it
So the plumbing is not slow; the model takes ten to fifty seconds, and until now
the person saw an EMPTY THREAD for the whole of it. That is indistinguishable
from a dead bot, and it is what the "does nothing" reports actually were.
setStatus is Slack's own affordance and the only one available: there is no SSE
to a Slack client, so the honest vocabulary is a status then a message, never a
token stream. A second placeholder message would be worse — it occupies the
thread with something the reader must skip. Best-effort by construction: a
3-second budget, every error swallowed, because a courtesy that runs BEFORE the
work must never delay the answer it announces. Skipped when there is no thread
rather than faked.
THE MODEL DID NOT KNOW ITS TOOLS HAD A PROTOCOL, and this is why it looked
stupid rather than merely slow. The surface was collapsed from 1,189 flat tools
(977 KB, ~244k tokens just to LIST) to 88 grouped hanzo_<subsystem> tools whose
only argument is an `op` enum of bare names — the schemas are fetched on demand
through hanzo_describe. That trade is a WIN only if the model is told how to
make the fetch. It was not: builtinAgentInstructions was three sentences that
never mentioned tools at all. The model saw 88 tools it could not interpret and
answered from memory, which is exactly the observed behaviour.
The instructions now state the protocol — grouped by subsystem, choose an op
from the enum, call hanzo_describe for a shape you do not know — and say plainly
that answering from memory is wrong here, because the question is about THIS
org's live cloud and no training data contains it.
Kept short on purpose: every sentence is read on every turn and spends context
the user's actual question needs.
go build ./... clean; apps/integrations passes.
(cherry picked from commit 0d4093193e)
Two defects behind "it is SUPER SLOW and it seems stupid". Measured in
production, a turn is:
slack ack 0-6 ms (well under Slack's 3s retry threshold)
plane overhead ~25 ms (53,009 vs 52,985)
model completion 9,955 / 35,893 / 52,985 ms <- all of it
So the plumbing is not slow; the model takes ten to fifty seconds, and until now
the person saw an EMPTY THREAD for the whole of it. That is indistinguishable
from a dead bot, and it is what the "does nothing" reports actually were.
setStatus is Slack's own affordance and the only one available: there is no SSE
to a Slack client, so the honest vocabulary is a status then a message, never a
token stream. A second placeholder message would be worse — it occupies the
thread with something the reader must skip. Best-effort by construction: a
3-second budget, every error swallowed, because a courtesy that runs BEFORE the
work must never delay the answer it announces. Skipped when there is no thread
rather than faked.
THE MODEL DID NOT KNOW ITS TOOLS HAD A PROTOCOL, and this is why it looked
stupid rather than merely slow. The surface was collapsed from 1,189 flat tools
(977 KB, ~244k tokens just to LIST) to 88 grouped hanzo_<subsystem> tools whose
only argument is an `op` enum of bare names — the schemas are fetched on demand
through hanzo_describe. That trade is a WIN only if the model is told how to
make the fetch. It was not: builtinAgentInstructions was three sentences that
never mentioned tools at all. The model saw 88 tools it could not interpret and
answered from memory, which is exactly the observed behaviour.
The instructions now state the protocol — grouped by subsystem, choose an op
from the enum, call hanzo_describe for a shape you do not know — and say plainly
that answering from memory is wrong here, because the question is about THIS
org's live cloud and no training data contains it.
Kept short on purpose: every sentence is read on every turn and spends context
the user's actual question needs.
go build ./... clean; apps/integrations passes.
The paywall could not be switched on. A brand-new org's wallet is $0, the
automatic starter grant was deleted (41b23f12), and the "$5 free credit" the
catalog advertises was never provisioned -- so enforcing SpendGate would have
402'd every new signup on request #1. This adds the missing rung.
WHERE IT SITS. One call site, in standing()'s proven-unpaid branch, past the
enforcement check: the last point at which an account that has never been
funded can still be told apart from one that will not pay. Both authorities
have already answered, so the rung inherits "no subscription and no credit"
as a finding rather than re-reading it.
WHY THIS IS NOT THE MECHANISM THAT WAS DELETED. The old grant ran as app-wide
middleware on first credential contact and minted into any wallet that did not
exist yet. This one:
- is not money -- it posts the plan's own advertised term under the shared
starter-credit tag, which billing/bucket classifies non-cash: spendable on
metered usage, never refundable, never paid out;
- is not unbounded -- the amount is credit.StarterCreditCents, the constant
the catalog's entry plan advertises, and fund() takes no amount at all, so
no request body reaches it;
- is not repeatable -- starterRef keys the deposit on the address it credits,
with no time, nonce or request id, so finance dedups it inside the same
transaction as the insert;
- is not unscreened -- every grant is a PRIVILEGED Decide at StageSignup, so
a flagged signup receives nothing and a scorer that is present and silent
withholds. That is the bound on "$5 x as many fake signups as can be made";
- has no switch of its own -- it is unreachable while the paywall is dark, so
the mint and the refusal it cures are the same flag. That is the structural
answer to "a disabled money-mint is one flag away from an enabled one".
Two further guards keep one grant to one customer: the shared signup org is
excluded (a login is not an account), and only the caller's home org is funded,
because a founder is a member of every org they create and position -- orgs[0],
minted as X-User-Owner -- is what marks the one that is theirs.
Funding is not an authorization decision: a grant that cannot be posted leaves
the gate to refuse in its own words, with the actionable 402 body.
No switch is flipped here. SwitchPaywallEnforced and enableSignUp stay as they
are; this only makes the rung exist so the flip is a pricing decision.
Every mechanism above is mutation-proven: removing the rung, the ref dedup, the
screen, the Privileged bit, the lifetime-usage leg, the catalog amount, the
dark-gate guard, the home-org guard or the signup-org exclusion each turns a
test red.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The paywall could not be switched on. A brand-new org's wallet is $0, the
automatic starter grant was deleted (41b23f12), and the "$5 free credit" the
catalog advertises was never provisioned -- so enforcing SpendGate would have
402'd every new signup on request #1. This adds the missing rung.
WHERE IT SITS. One call site, in standing()'s proven-unpaid branch, past the
enforcement check: the last point at which an account that has never been
funded can still be told apart from one that will not pay. Both authorities
have already answered, so the rung inherits "no subscription and no credit"
as a finding rather than re-reading it.
WHY THIS IS NOT THE MECHANISM THAT WAS DELETED. The old grant ran as app-wide
middleware on first credential contact and minted into any wallet that did not
exist yet. This one:
- is not money -- it posts the plan's own advertised term under the shared
starter-credit tag, which billing/bucket classifies non-cash: spendable on
metered usage, never refundable, never paid out;
- is not unbounded -- the amount is credit.StarterCreditCents, the constant
the catalog's entry plan advertises, and fund() takes no amount at all, so
no request body reaches it;
- is not repeatable -- starterRef keys the deposit on the address it credits,
with no time, nonce or request id, so finance dedups it inside the same
transaction as the insert;
- is not unscreened -- every grant is a PRIVILEGED Decide at StageSignup, so
a flagged signup receives nothing and a scorer that is present and silent
withholds. That is the bound on "$5 x as many fake signups as can be made";
- has no switch of its own -- it is unreachable while the paywall is dark, so
the mint and the refusal it cures are the same flag. That is the structural
answer to "a disabled money-mint is one flag away from an enabled one".
Two further guards keep one grant to one customer: the shared signup org is
excluded (a login is not an account), and only the caller's home org is funded,
because a founder is a member of every org they create and position -- orgs[0],
minted as X-User-Owner -- is what marks the one that is theirs.
Funding is not an authorization decision: a grant that cannot be posted leaves
the gate to refuse in its own words, with the actionable 402 body.
No switch is flipped here. SwitchPaywallEnforced and enableSignUp stay as they
are; this only makes the rung exist so the flip is a pricing decision.
Every mechanism above is mutation-proven: removing the rung, the ref dedup, the
screen, the Privileged bit, the lifetime-usage leg, the catalog amount, the
dark-gate guard, the home-org guard or the signup-org exclusion each turns a
test red.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
podSpec named no ServiceAccount, so every sandbox pod ran as `default` — and
DOKS's registry integration re-attaches its own DigitalOcean pull secrets to
every namespace's `default` account whenever it reconciles. A sandbox therefore
inherited a credential for a registry that is not ours and died asking
registry.hanzo.ai for its image anonymously: a 401 that reads like a bad
password and was in fact no password at all.
`sandbox` is ours and DOKS does not manage it. It is bound to no Role and
carries exactly one thing, the pull secret; the line above it still refuses the
pod a token, so naming an account grants nothing that omitting one withheld.
A constant and not an env var: which accounts exist in the sandbox namespace is
decided by the manifest that creates the namespace, so a second knob here could
only ever disagree with it. Declared in universe infra/k8s/sandboxes/registry.yaml.
The drift gate stopped on commerce again, this time on the four reads mount.go
registered so the billing app's tabs would stop rendering empty: GET
/v1/billing/transactions, /credit-balance, /accounts and /accounts/{id}/members.
They were routed and described nowhere, and an operation that says nothing about
itself publishes an operationId and no sentence — an SDK method that cannot
explain itself and an MCP tool a model cannot pick.
They are raw by nature — the handlers live in the commerce module, so there is no
doc comment here for zipdoc to lift — so the prose is declared with
openapi.Describe beside the route table, which is where this file already keeps
the other seventy-five.
The prose is written from the handlers and from the chain mount.go gives them,
and it leads with the fact a reader is most likely to get wrong: these handlers
filter on a user or userId parameter, and PinBillingSubject OVERWRITES that key
with the caller's own account.Payer subject before the handler runs. So the
parameter is not the caller's to choose — naming another subject returns your own
rows, not theirs. Saying otherwise, or saying nothing, would leave a generated
client's author believing they had found a way to read another tenant's ledger.
accounts/{id}/members has no subject key to pin and guards itself instead, by
comparing the path segment against the resolved org and answering 403; that is
stated too, because it is the one member of the family whose refusal comes from
somewhere else. Both refusals are 401 unauthenticated rather than 403, because a
browser re-authenticates on the first and merely reports the second.
Regenerated from source: 1759 -> 1768 paths, 2430 -> 2440 operations, billing 39
-> 43. Every number rises; nothing shrank, so the floor moves up rather than
being lowered.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Follow-up to 7e34af0a, closing the two things it left open.
The terminal handler answered with `application/json; charset=utf-8` — this
file's usual spelling — while zip's mounted route answers `application/json`.
An agent could therefore tell WHICH adapter carried its answer, which is the
second door reappearing as a header. Unified to zip's media type, and
TestDoorlessPluginAnswersItsOwnDoor now compares the two paths' Content-Type
directly, so a future divergence fails rather than merely looking untidy.
TestCatchAllNeverShadowsTheDoor pins what was NOT the cause. Registration order
was the obvious suspect and it is innocent: zap-proto/fiber's insertRouteSorted
ranks a longer static literal ahead of the greedy `/*`, so a control route
registered late still sorts ahead of the console catch-all. Nor was it the
zip v1.25.1 -> v1.27.0 bump — generation.go is byte-identical across the two.
The cause was only ever installMCP declining to mount a route for an app with an
empty edge registry. Both dead ends are recorded in the test and in
manifest/mcp.go so the next reader does not re-run the investigation.
manifest/mcp.go's claim is corrected at its source. It said the host's framework
path "is claimed by nobody, which is exactly why the front door has to name it".
The host claims it now, from fleet.Mount, and a plugin answers there with its own
door — so the comment describes the mechanism that exists rather than the one
that was replaced.
v1.50.16 (already live) stopped the rail from handing out addresses nothing
could credit. These four carry the machinery that will eventually let it be
lifted, and all of it ships COLD:
- v1.50.18 the EVM deposit watcher. Exactly-once is a property of the KEY —
sha256(chain:txHash:logIndex) against a backend that upserts — so re-scans,
crash retries and N replicas all yield one credit with no leader election.
Disabled unless CRYPTO_DEPOSIT_* assets are configured, which they are not.
- v1.50.19 GET /v1/billing/crypto/options answers from the watcher's assets
instead of the MPC processor's mintable chains, so the picker cannot name an
asset nothing is watching.
- v1.50.20 GenerateAddress returns {Address, ID} and the intent records the
custody handle, so a credited deposit is one we can also sweep.
The gate (cryptoDepositsCanBeCredited) is still false and is a constant, not
config — nothing in this bump can take money.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Red re-reviewed the eight-door work and found the ref policy bypassed by the
code added to close its own lowest-severity finding. L2 was "shallow and signed
pushes fail to parse" — an availability nit. The push-certificate parser written
to fix it disagrees with real git about where a certificate's commands are, and
the disagreement is total.
git concatenates every cert pkt-line payload into one buffer and takes the
commands between the FIRST "\n\n" and the signature offset
(builtin/receive-pack.c, queue_commands_from_cert). This parser treats one
pkt-line as one logical line. Embed a "\n\n" inside a header payload and the two
part ways: we parse ZERO commands, git parses one and applies it.
Zero commands WAS the exploit. checkRefPolicy over an empty slice iterates
nothing and returns nil, because an empty command list satisfies every policy —
including a grant confined to one agent ref. The raw bytes are then forwarded to
git receive-pack, which executes the command we never saw.
Verified by Red against git 2.43: a 262-byte certificate whose `pusher` payload
carries "\n\n<old> <new> refs/heads/main\n" makes a grant scoped to
refs/heads/agent/<session> write refs/heads/main — which fires cloud.OnGitPush
and deploys a model-authored, unreviewed commit. No signature and no nonce are
needed; the forge sets no receive.certNonceSeed. Negative control: the same
trunk write as a PLAIN push under the same grant is correctly refused, so the
policy works and the hole is specifically the framing.
TWO GUARDS, either sufficient, so neither is load-bearing alone:
1. parseRefCommandsCaps refuses any pkt-line payload carrying an embedded
newline. This makes the parser's own one-line assumption TRUE rather than
assumed. Narrower and safer than teaching it git's concatenation: parity with
a second implementation must be re-proved every time either side changes,
whereas a payload with no embedded newline can only be read one way BY BOTH.
Nothing legitimate is lost — git's send-pack emits one line per pkt-line.
2. checkRefPolicy refuses a grant-bearing push that names no ref. A grant exists
to write one named ref, so naming none means the frame did not parse the way
we think it did. For a principal an empty push stays a harmless no-op.
Tests use Red's exact 262-byte exploit and FAIL without the guards — verified by
reverting them and re-running. The negative control (an ordinary single-line
push still parses and yields its one command) is what keeps the fix from being
worse than the bug.
Red's other findings are NOT closed here and are follow-ups, all lower:
writer 9 (corePush -> initBare sets HEAD to an agent branch on a NEW repo,
dead-ending checkHeadRef, principal-only), a check-then-set TOCTOU in corePush
(go-git SetReference is not CAS; the wire path is safe because git does CAS ref
locking), and zapface minting a socket for any credential string.
apps/git shows the same 3 pre-existing failures as baseline; go build ./... clean.
TestNoSecondMCPDoorInSource matched any string literal ending in /mcp, which made
`"github.com/zap-proto/mcp"` read as a second door. It is the opposite: webui's
terminal handler imports the PROTOCOL precisely so it can hand a frame to zip's
existing door rather than write an envelope of its own.
The pattern now requires a leading slash, which is what actually distinguishes
the two — a door is a route path, rooted and ending at /mcp; an import path is
neither. The gate still fails on a real second door, and apps/tasks' raw net/http
mux handler is still the reason it reads SOURCE rather than the document.
Every per-app plugin binary answered POST /mcp with 308 to /v1/mcp, and then 404
there. Measured on bin/kms and bin/tasks before this change; the door a subsystem
owns was unreachable over HTTP.
The console's terminal handler sent it. Its reasoning was that a plugin serving
its own door at the framework default matches a real route and never reaches a
terminal handler — so anything arriving there had to be a caller who guessed the
default on a host that moved it. That is false for a whole class of app: zip
mounts the /mcp route only when there is something to expose (installMCP), and a
plugin whose typed ops live on the internal plane has an EMPTY edge registry.
kms is the honest example — its four secret ops are on cloud.Plane() precisely so
no route runs from the edge to a secret — so kms has no route at its own door,
fell through to the console, and was told its door lived at an address only a
host serves.
The cost is not cosmetic. The fleet composes tools/list by asking every child at
FrameworkMCPPath and reading any non-2xx as an outage (fleet/fleet.go ask), so
such a child drops out of the composed list AND is reported down — for the crime
of having no tools. An empty list is the honest answer and it is a 200.
Two changes, each in the place that holds the fact:
The console SERVES the door instead of redirecting it. The door is not the route:
it is zip.App.MCP, a frame in and a frame out, which exists whether or not
anything was mounted over it. Serving zip's own door is not a second surface;
re-implementing one, or redirecting to a door this process does not have, is.
The signpost moves to fleet.Mount, which is the one place that KNOWS the door
moved, because it is the call that registers the new address — and it registers
the signpost only when the two differ. A host still tells a caller who guessed
/mcp where /v1/mcp is; a plugin, which moved nothing, no longer claims it did.
Verified against freshly built binaries, not by reading code:
bin/kms POST /mcp initialize -> 200, protocolVersion 2025-06-18, serverInfo kms
POST /mcp tools/list -> 200 {"tools":[]} (was 308 -> 404)
bin/blueprint POST /mcp tools/list -> 200, 2 tools, both carrying lifted prose
POST /mcp tools/call -> 200, get_v1_blueprint_health returned live data
git.hanzo.ai is canonical for every repo; GitHub is a mirror. These two had
diverged and BOTH sides held real work:
forge only (8): the platform deploy/git-lane series — b17f9e6a..26cfa8e5
hanzo-inc only (1): 827f3e66 plane: a call expires on the CALLER's deadline
Production was running 827f3e66 — a commit that existed ONLY on GitHub and not
on the canonical tree. That is the failure mode the one-remote rule exists to
prevent: the deployed sha was unreachable from the repo everything else reads.
Merged rather than force-pushed in either direction, so no commit is lost from
either side.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three conflicts, all mechanical: forge renamed Mount's router to zapp, moved
fleet.go's openapi.Describe calls out (so that import goes), and switched
runnerBuild to a plain ctx. Kept forge's shape in each and re-applied the
delivery wiring, the org attribution and the namespace import on top. Renamed
this package's test helper keysOf to mapKeys — forge added a keysOf of its own
with a different signature.
Verified by DIFF, not by commit count: this estate has a merge that kept a fix's
commit and dropped its diff, so every security-critical seam was re-checked in
the working tree — seal-by-default, the audit's caller-declared public set, the
tenant- reservation, owner() on both boards, checkFence, the rendered-patch
guard, tenantNamespace's single derivation, per-org build attribution, and the
deleted resource/logs handlers.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A typed op mints five things from one registration. Three defects here let an
operation reach all five carrying no sentence, and one hand-written file
described a surface nobody generated.
POST /v1/exec published a summary and no description. The handler had a doc
comment; the package had no zipdoc directive, so nothing lifted it — and the op
registered on the cloud.Router parameter, which zipdoc cannot follow to a prefix,
so it refuses to lift even with the directive. openapi.Complete accepts EITHER a
summary or a description, so every gate stayed green over the hole. The
registration moves to the *zip.App (the apps/meet and apps/blueprint pattern);
routes move, the credential middleware stays on the scoped router where the
prefix guard applies to it. Its 15 published fields carry prose now too.
docs/automations-openapi.yaml is deleted. 25 KB of hand-authored OpenAPI over 17
operations of /v1/automations, referenced by nothing and compared by no gate — so
it had drifted the way a second copy always does: it claimed two operations the
fleet does not serve (GET /v1/automations/health, POST /v1/automations/mcp) and
omitted three it does (connectors/{id}/run, flows/{id}/versions,
hooks/{source}/{event}). The document already describes that surface.
The field tranche closes 343 published properties across seven apps — authors
39->0, label 37->0, channels 34->0, prompts 29->0, leaderboard 40->0, campaign
42->6, affiliates 123->10 — and campaign's two result types stop being defined
types over another struct, which published all 30 of their properties bare; the
struct is declared under its published name and the domain name is an alias, so
there is still one shape.
The 16 that remain are not app defects and are recorded as such in LLM.md: zip
keys a promoted field's prose under the type that DECLARES it and looks it up
under the type that PROMOTES it, so no comment written here can reach them.
LLM.md records what the running deployment measures, taken by making the request:
the served document is the committed one at the revision the header names (1735
paths / 2474 ops, identical sets), /v1/commands answers 2448 commands under a
working ETag, POST /v1/mcp lists 88 tools and tools/call returns lifted prose —
and those tools reach only 1189 of 2422 operations, with 134 declared refusals
accounting for a tenth of the gap.
Red proved the classifier missed 8 of 8 real credential shapes, each for its own
reason: PGPASSWORD (libpq's own variable, one token to any splitter), *_PW (`pw`
absent), a symbol-rich password (the check returned false the moment it saw a
symbol, so a STRONGER password was MORE likely to be published), KUBECONFIG
(cluster-admin, base64 material, no PEM armour), a base32 MFA seed. Every fix
would have been another special case in a denylist over an unbounded set.
And the three gates were ONE gate: leakedSecret called the same mustSeal the
split called, so a miss passed both and reached git. An independent-looking
backstop that shares its single point of failure is worse than none, because it
is counted as defence.
THE FIX IS THE POLARITY, NOT A BETTER CLASSIFIER. This lane's output is a commit
in a repository replicated to every clone and impossible to unpublish, so what
matters is not how good the guess is but which way it fails. Every env value is
now a KMS reference UNLESS the caller explicitly marks it public. A miss means an
operator cannot read back a config value — a support ticket. The old miss meant a
password in git history — an incident with no rollback. It needs no list, so
there is no shape left to overlook, and it matches what the console already
sends. It also closes the over-seal complaint from the other side: GIT_COMMIT,
IMAGE_DIGEST and TENANT_ID are marked public and stay readable.
The audit now shares NO code with the decision it checks. It parses the RENDERED
bytes and asks a fact — "is this key in the set the caller marked public?" — not
a judgement. The public set is carried on the spec straight from the request and
is NOT derived from spec.Env: an audit that asks "is this key in Env?" of a
document rendered FROM Env answers yes by construction, which is the same
tautology in new clothing. A value reaching the render by any path the caller
did not authorise is refused, whatever it looks like.
V6 — the templatePatch guard scanned for the substring "project", which the very
engine that runs it defeats: `{{ "pro" }}ject:` and "\x70roject:" both set the
key and both evade a scan. It renders the patch with the generator's own funcs
and data and reads the RESULT as YAML. The real fleet patch (dig/syncPolicy)
still passes; anything resolving to a project is refused.
The operator lane keeps shape classification — it writes a database column it
can rewrite — but the named misses are closed there too, since they cost
nothing: substring matching for the names with no separator to tokenise,
`pw` added, and the symbol short-circuit inverted so a symbol-bearing,
space-free string of password length is password-SHAPED rather than exempt.
V2/V4 — evidence, and the containment, recorded at the emission site. The leading
slash is RIGHT: eleven live values files in universe use `secretsPath: /<path>`
and are healthy, and the chart schema requires it; the operator lane's no-slash
form is the outlier, flagged and not changed because it is live. Still
unverified: a CUSTOMER org's projectSlug resolving, since every live example is
platform-tier. Nothing from this lane can reach a pod yet — a declaration lands
on a branch the generator does not read, and modeCommit is refused by checkFence
until universe carries the companion rule. Both gates open deliberately, by a
human, and the first merge is where this resolves.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A key-NAME regex running in a BROWSER decided which env values were sealed. Two
things are wrong with that and the second is the one that matters: the key is
not the secret, the VALUE is — GCP_SA_JSON names nothing credential-ish and
holds a private key, while STRIPE_SK, SK_LIVE, GH_PAT, PGPASS, SMTP_PASS, HMAC
and TLS_CERT are ordinary names that pattern missed — and a control the caller
can edit is not a control. secretshape.go decides on the SERVER now: issuer
prefixes and PEM armour, connection strings carrying a password, and a narrow
entropy tier for tokens with no recognisable prefix, OR'd with a server-side key
test so DB_PASS=hunter2 is caught too. Both signals, because they fail in
opposite directions. The client flag stays as UX and may only ADD secrecy.
Not default-seal-everything: PORT, NODE_ENV and LOG_LEVEL would become
write-only values an operator cannot read back, and every config line would take
a KMS round trip on the deploy path.
★ THE DECLARE LANE WAS WORSE THAN THE OPERATOR LANE. sealSecretEnv protects a
database column; a values file is committed to universe, replicated to every
clone, and cannot be unpublished — a credential written into one is cleartext in
git history forever. The split now happens BEFORE anything is rendered: secrets
are sealed into KMS at the coordinate the operator lane already uses, and the
file carries a `kmsSecrets` reference plus env valueFrom.secretKeyRef. That is
the chart's own rule, in its own words: "Secrets are REFERENCES, never values —
that is what makes this values file safe to publish." Fail-closed harder here
than in the operator lane: no KMS, no deploy. And a last-line guard re-asks the
question of the BYTES about to be committed, so the decision is proven to have
held all the way to the write rather than assumed.
V1 — reservation extended to the namespaces the cluster actually runs (adnexus,
bootnode, team-go, preview, registry, pars-*) plus `*-system` as a SUFFIX, so the
next operator installed is not claimable in the window before someone extends a
list. Deliberately STATIC rather than derived from the live namespace set: a
derivation is the stronger rule but fails OPEN exactly when the apiserver is
unreachable, which is the moment it is most needed. Wrong only by omission,
never by outage.
V2 — fenceOf saw only spec.template. templatePatch is rendered as TEXT and
merged OVER it, so a patch setting spec.project overrides the fence unseen;
evaluating the merge faithfully means reimplementing the generator, so a patch
that mentions `project` is refused instead. And `env`/`expandenv` are removed
from the function map: they read the PROCESS environment, so the same expression
would render from cloud's environment here and the controller's there — one
expression, two answers, and the deciding one is not ours.
tenantNamespace derives ONCE. It sanitized an input that tenant(s, c) had
already sanitized, and Sanitize is not idempotent — re-folding its own
<fold>-<hash> output appends a second hash. So for every org whose name is not
already a clean label, platform wrote App CRs into tenant-<double> while
apps/deploy and apps/provisioning both scanned tenant-<single>: fails closed,
nothing crosses a boundary, but the org's board is silently empty and its CRs
orphaned. It was the outlier of three copies; the other two already took the
slug. Non-label input now resolves to the inert "unknown" rather than rendering a
malformed namespace, so a caller that skips tenant(s, c) fails closed too.
Deleted the unrouted, unscoped appResource/appTree/appLogs. They resolved a
namespace with no org filter and no redaction, and the typed client already
ships tree/resource/logs methods — wiring them would have been a cross-tenant
unredacted read. resource.go and logs.go go entirely; tree.go keeps buildTree,
which dashboard and detail actually route.
Follow-ups flagged, not slipped in: a SanitizedOrg type in hanzoai/namespace
(26 call sites across three packages, a cross-repo release) is what would make
double application a COMPILE error instead of a guard test; and the legacy
tenant-* namespaces still want migrating onto the bare-org layout.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
R1 (HIGH, red) — THE RENAME'S OWN SHADOW. Dropping the `tenant-` prefix from
this API made `tenant-` an ordinary name, but the cluster's live fences are
still spelled with it: universe project-tenants.yaml declares AppProjects
tenant-hanzo|lux|zoo|zen|maxpower, each admitting namespace tenant-<org>. A
clean label, so Sanitize passes it through untouched — an IAM org named
`tenant-maxpower` would have claimed the REAL maxpower tenant's namespace under
maxpower's own fence and read its CD rows. Reserved as a FAMILY, so the next
tenant onboarded under the legacy layout is not claimable in the window before
someone extends a list. Red's PoC (7 tests) is kept and green.
That fix exposed a conflation in my own predicate, and splitting it is the real
change here. `reserved` was answering two questions at once:
reserved(dir) — may an org CLAIM this directory? The WRITE question. Wider
than platformOwned by exactly the tenant- family: those
directories are not the platform's, but they are already
somebody's.
owner(ns) — which org does this namespace BELONG to, "" for the
platform's own? The READ question. It decodes BOTH layouts,
because both are live.
Conflated, reserving tenant- would have blanked every legacy tenant's own board.
Two names, two questions, and the boards share the second one.
R3 (MEDIUM, LIVE) — /v1/platform/fleet asked nsOrg, which maps the brand
namespaces onto org "hanzo", so the ORG ADMIN of the brand org was handed the
platform tier — iam, kms, gateway — while the delivery board refused the
identical caller. A per-org isAdmin is never platform-privileged (HIP-0519);
this file's own deploy route already applied that reasoning to RESTARTING one of
these services, and observing them is the same class of act, only quieter, which
is why it survived. Both boards ask owner() now. Two tests asserted the old
behaviour in so many words ("an OrgAdmin of the platform org sees its own org's
whole board"); that expectation WAS the bug and they now assert the refusal.
R2 (MEDIUM) — checkFence refused one known-bad substring, so every OTHER unsafe
template passed: an unconditional `project: hanzo-platform`, a file with no
project, an empty file, garbage. A denylist of one is not a check. It now
asserts the POSITIVE — parses the ApplicationSet, evaluates its project template
for this org with the same engine and options the generator uses (text/template
+ sprig, missingkey=error), and compares to declareProject. Anything it cannot
answer is refused, including a template that reaches past the path for the fence
(the "thing being fenced chooses its fence" defect the ApplicationSet itself
rejects). Red's six templates are the refusal table.
R4 (LOW) — DECIDED: the per-org build ceiling stays SOFT, and the reasoning is
recorded where the check is. It is check-then-act; the only atom available is
the Job name, already spent on idempotency, and a counter object would be a
second source of truth about how many builds are running. The overrun is bounded
by requests in flight and confined to the caller's OWN org. The cluster bound
belongs where bounds are enforced atomically — a ResourceQuota on the build
namespace, applied at admission, which no race can widen.
Follow-up flagged, not done here: retire the legacy tenant-* namespaces and
AppProjects onto the bare-org layout. That is a migration, not a rename — it
moves live workloads — and the reservation holds until it lands.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
CTO naming rule: no prefix on an org. The values directory, the destination
namespace, the AppProject and the image path are all `<org>` —
charts/app/values/<org>/<app>.yaml, namespace <org>, project <org>,
ghcr.io/hanzoai/<org>/<app>. One value, four roles, nothing added to it.
The `tenant-` prefix was a naming convention doing a policy's job, and removing
it is only safe once the policy exists, because namespace.Sanitize is the
IDENTITY on a clean label: without a separator, customer directories and the
cluster's own namespaces share one name space, and an IAM org named
`kube-system` resolves to the real `kube-system`. RESERVATION replaces it —
one predicate, asked on read and on write, over the platform's namespace
family: the brands and their environments, the control and delivery planes,
kubernetes' own, and `admin`. A reserved directory is SuperAdmin-only even when
it is the caller's own org.
`tier` is gone with the prefix. An org is its name, so placement is not a field:
`org` is an ACT-AS, defaulting to the caller's own, and naming another requires
SuperAdmin. Both refusals refuse rather than downgrade, so an escape attempt is
never indistinguishable from a normal request.
The fence's real rule lives in universe's ApplicationSet, so it is VERIFIED, not
documented: checkFence reads the live template out of the clone the write
already makes and refuses to put a declaration on main while that template would
fence it wider than this API reports. It clears itself when universe lands the
reservation form — no flag, nothing to remember — and a branch write is exempt
because nothing is generated from a branch. A comment saying "land the companion
change first" is not a control, and neither is a test that fails on a
developer's machine and skips in CI.
RED FINDINGS, all fixed with red's PoC kept and green:
F1 (HIGH, cross-org read) — the write path derived a directory with
namespace.Sanitize(org) while the read path confined with the RAW owner claim.
Sanitize on ONE side of an authorization compare is a collision waiting to be
named: org "Acme" owns "acme-<hash>", so it never matched its own rows, and any
org whose raw name IS that literal string matched them instead — offline
-computable, since the slugger is public code. Both sides are canonicalised now,
in cd.go owns AND in fleet.go scopeNamespaces, which carried the identical
compare and is LIVE on /v1/platform/fleet. Sanitize is injective, so this is
collision-free and not merely symmetric. cdApps had no tests; it now has a table
over every shape of name Sanitize treats differently.
F3 (build DoS) — a declare build was charged to the constant "platform", so one
org looping deploys exhausted a shared ceiling of 3 and locked the fleet out of
building, with no attribution in the Job labels to see it by. launchDirectBuild
takes the org it is charged to; the ceiling is per-org, like /v1/runner's.
F4 — the default host interpolated the RAW owner claim, so any org without a
clean name got "web.Acme.hanzo.app", not a hostname at all. It is built from the
canonical org, the same value that is the directory.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every client retries. A branch write based on main committed the same tree at a
different timestamp on the second call, so a different sha, so a non-fast-forward
push and a raw git hint the caller could not act on. A branch write now BASES ON
ITS OWN BRANCH when that ref exists — the push is a fast-forward, and when the
declaration is already exactly this the whole call is a clean no-op reporting the
same ref and review URL.
The env comparison was the same defect one layer up: an existing declaration
refused ANY env, so a retry of a create-with-env could never succeed. It now
compares environments as sets of name=value and refuses only a genuine change.
The test that pins this sleeps a second between the two calls, and that sleep is
the point. Without it both commits land in the same second, git mints the
identical sha, the push is "everything up-to-date" and the bug is invisible —
which is exactly how the first version of this test passed while the bug was live.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A commit to main proves the image pullable, and the image a build launched by
the same call would produce does not exist until the Job finishes — so the
combination could only ever fail, after a privileged BuildKit Job had already
been spent on it. Refused at validation instead, naming the two-step flow the
caller wants: deploy (which returns the build's tag), then commit that tag once
the build is green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/platform/apps is the delivery surface for the ONE deploy plane. An app is a
values file in hanzoai/universe under charts/app/values/<namespace>/<name>.yaml;
the fleet ApplicationSet renders charts/app against it and cd.hanzo.ai reconciles
the result. Declaring the file is the whole of deploying the app, so the surface
is three moves: build the repository through the existing BuildKit lane, write
the declaration that names the image, and report the Application reconciling it.
Two things a caller may never choose, because both are fences.
The DIRECTORY. The generator derives the Application name, the destination
namespace, the Helm release name and — load-bearing — the AppProject from the
file's own path: tenant-<org> admits ONE namespace, no cluster scope and six
kinds, while hanzo-platform admits namespace * and ClusterRole/ClusterRoleBinding.
A caller that could name its directory could name its fence, so the directory is
derived from the validated owner claim and is not a request field. tier=platform
is the one lever over placement and it is SuperAdmin only.
The IMAGE REPOSITORY. A declaration is what the cluster pulls, so it is derived
per tenant on the same injective path the build lane already uses. There is no
image field to try.
A branch is not a deploy. The generator reads main, so the default mode pushes
deploy/<ns>/<name>/<tag> and deploys nothing; merging the review is the
deliberate act. mode=commit writes main and proves the image pullable first,
because a declaration naming an image the registry cannot serve is an
ImagePullBackOff with no rollback path.
The git seam is pin.go's, not a second one: the same shallow clone, the same KMS
-held token carried as an http.extraHeader rather than in argv, the same
fast-forward-only push retried by re-reading the tip. This composes with the pin
rather than duplicating it — declare ADDS a service deliberately, which is
exactly what resolvePinFile refuses to do, and the pin then moves its tag. An
update here moves that one scalar too and refuses anything that would rewrite a
hand-maintained declaration.
/v1/platform/cd reads the Applications in hanzo-cd. A cluster with no CD answers
an empty plane; a plane that cannot be READ answers 503 and says why, because
those are opposite facts. /v1/platform/ci answers 501 and names what is missing
rather than fabricating an empty run list — this deployment has no forge API
client. Static sites and bucket listing are NOT added: /v1/platform/sites and
/v1/s3/buckets already serve them.
Proved against the real chart, not a fixture: the generated declaration renders
through hanzoai/universe charts/app with its values.schema.json enforced, and a
key the chart does not declare is refused by that schema. A dry run against a
clone holding the real 105-file inventory wrote a scratch branch and left main
byte-identical.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
refpolicy.go claimed receive-pack was "the point every path that changes a ref
passes through and no client can decline to". That was false, and it was not a
detail — it was the entire argument. It guarded ONE of eight ref writers into the
same bare repository, and the other seven took the credential the run already
held.
The cheapest was not the git wire protocol at all. One JSON POST to
/v1/git/repos/:name/push lands a FAST-FORWARD CHILD on the branch a human just
approved: worse than the force-push the wire door refused, because append-only
trips no "the branch was rewritten" signal anywhere. The reviewer approved A and
merges A+B. /repos/:name/mirror was worse still — `+refs/*:refs/*` with --prune
force-overwrites every ref from a source the caller names and DELETES any ref
that source omits. SSH receive-pack piped straight into git with no policy in
front of it, and any org member may enrol a key. Verified by test: before this,
a mirror from an attacker-chosen upstream deleted a reviewed agent branch and
created one of its own.
Underneath all seven was the real problem. The credential was an org-wide `agent`
key sealed in KMS — an ordinary IAM sk-, which IAM resolves to a USER, from which
cloud mints a full org principal, which cloud.Member admits everywhere. The one
process executing untrusted model output held something that opened
GET /v1/kms/secrets (every other secret the org has, including whatever posts to
its Slack) and every org-scoped API in the platform. Custody had not shrunk to a
safe place; it had shrunk to the worst place, holding a skeleton key.
So the credential stops being an identity. A run now asks the forge for a GRANT
(apps/git/grant.go): a bounded permission to drive the pack protocol against ONE
repository, creating ONE ref, until it expires. It authenticates nobody. It is
not a JWT and carries none of APIKeyPrefixes, so validatedPrincipal returns nil,
no X-User-Id is minted, principal.Validated is false, and every cloud.Guard and
every tenantOf refuses it BY DEFAULT — nothing had to be told to say no. The one
exception is resolvePackRepo, so the set of doors a grant opens is the set of
callers of that function, and a principal always wins because the grant is
consulted only where there is none. git owns refs, so git decides who may write
one; the orchestrator is no longer a credential custodian, and there is no
org-wide agent secret left in the world to seal or to leak. The constants that
named one are deleted, not deprecated: a constant naming a secret is an
instruction to seal one.
The policy then moved to all eight writers, enumerated in refpolicy.go beside the
rule so a ninth is a change to the list and not just a new function. Five state
their intent as the same refCommand value the wire door parses and call the same
function. Two cannot be judged command-by-command and are refused structurally
instead: the mirror takes a negative refspec so the machine namespace is outside
its refmap and therefore outside --prune, and HEAD — which no refspec reaches —
goes past checkHeadRef, including the first-push case where a client-less push
would otherwise make a run's branch the repository default, which is what the
deploy reactors gate on. One (tag fetch) was already structurally safe and now
says so.
Also, from the same review:
- Base was TrimSpace and nothing else while Repo and Org were shape-checked. It
reaches a `git clone -b <base>` argv on a CUSTOMER'S machine, where a leading
dash is not a branch but a flag, and --upload-pack= / --config=core.fsmonitor=
are each arbitrary execution there. BaseRE is git's own branch shape; the
load-bearing part is that a branch is alnum-led. Project is shaped too.
- The PR head is pinned to the cloud-issued BranchFor(sessionID). Adopting the
sandbox's self-reported branch let a compromised one answer `main`, which
passed VerifyRef (which only asks whether a ref EXISTS) and came out as
CreatePR{Head: "main"} — a pull request headed at the trunk, filed by us, for
a run that never had permission to write there. Same on the routed path,
where the reporter is a customer's machine.
- A refusal naming many refs was wrapped in ONE side-band packet. A pkt-line
length is four hex digits, so past ~65516 bytes it renders five and the client
desynchronises — the report the control exists to deliver became the
"RPC failed / unexpected disconnect" it exists to avoid. Now chunked, with
each line bounded independently.
- A push from a --depth clone (`shallow <oid>` lines) and `git push --signed`
(a push certificate wrapping the commands) both failed to parse and answered
400. Fail-closed, but closed on ordinary clients doing ordinary things, which
is how a control gets switched off. Both parse now, and a certificate's
commands are still judged — parsing one is not a bypass.
- A run's budget was unbounded, so a caller could hold one of its org's two pool
slots for a day and ask the forge to delegate a push for just as long. Capped,
and the grant's lifetime is now derived from it.
Proved with the real client against the real server, one test per door:
refwriters_wire_test.go drives the real git CLI over the real SSH listener and
over smart-HTTP, and refpolicy_framing_test.go replays wire bytes captured from
git 2.43. A grant clones its repository and pushes its one ref, and is refused —
with git's own report-status, in words the pusher can read — for the trunk, for
another run's branch, for a non-agent ref, for deleting the trunk, and for
appending to its own branch. The legitimate paths are asserted too: an agent
branch still creates, the builder's client-less push to main still lands, and an
ordinary SSH push to main still lands.
The sandbox remains INERT: BOT_GATEWAY_URL and HANZO_CODING_SANDBOX_IMAGE stay
unset until Red has re-reviewed this.
Every @hanzo turn in Slack answered "the agent hit an error handling that". With
the bridge's new non-silent branch the reason finally surfaced:
zip: call agents_run_on_behalf at /var/lib/cloud/run/agents.sock:
zaphttp: read response: i/o timeout
zap-proto/http@v0.3.1 sets readTimeout: 30 * time.Second (client.go:72) on every
dialled transport. An agent turn runs a real model completion and enso spends
40s+ on one — production measured 41,153ms, answered 200 by BOTH `ai` and
`agents`. The work succeeded. The caller had already hung up, and the reply was
written to a socket nobody was reading.
That is the worst shape a timeout can have: the expensive work is done and
billed, the callee logs success, and only the caller reports failure — so every
log you would naturally check says the system is healthy. It is why this
survived a day of looking at a healthy `ai`.
The context budget could not save it. The bridge bounds a turn at 110s on a
detached context; that governs the CALL and never reaches the transport's own
SetReadDeadline, which is wall-clock on the connection and shorter. Two
deadlines for one operation, and the smaller one — the one nobody chose — won.
The fix is a registration, not a patch. zip resolves a scheme through
RegisterTransport (transport.go:109) and its default zap Dial is one line, so
re-register the same scheme with the same dialler plus the knob zap-proto/http
exports for exactly this (SetReadTimeout, client.go:81). The stock Serve is
restated verbatim because re-registering replaces BOTH halves — omitting it
would stop every plugin listening.
15 minutes, matching the host's own plugin-start budget, so there is ONE answer
to "how long may an in-flight plane call take". It is a CEILING, never a floor:
every caller still bounds itself, and a caller that gives itself ten seconds
still gets ten. What changes is that it can no longer be cut off BELOW its own
budget by a default it never saw.
Tests pin the property rather than the wall clock (a unit test cannot hold a
socket for 40s): the ceiling must exceed the longest real caller budget,
re-registration must be idempotent, and networkOf must match zip's rule — it is
copied because zip keeps it unexported, and a drifted copy would hand a unix
path to a tcp dialler.
The two private lines had diverged 191/18 off a merge-base six hours old.
Production builds from inc2; every push went to forge; neither knew.
The 14 conflicting hunks, and why each resolved the way it did:
- bridgeReply loses its ctx parameter (inc2). Both lines fixed the same
production bug — the webhook's ctx made zip drop the stated org, so the
balance gate answered "no org on the call". forge passed a detached ctx in;
inc2 deleted the parameter so the webhook's cannot be passed at all, and
states the tenant with cloud.For. Unrepresentable beats discouraged. Its
body now calls forge's bridgeIdentity, so the link lookup exists once.
- commerce prefixes are the union: forge's /v1/cart and the /v1/billing/topup
stem, inc2's accounts, credit-balance and transactions. topup/token is
dropped — the stem owns its subtree. A prefix missing here never reaches
commerce; it falls to ai's bare /v1 and answers ai's 404.
- zipdoc-check keeps inc2's target (forge repeated the loop twice) with
forge's body: per-package, because whole-module load extracts differently
than the generator it polices, and skipping dot-dirs, because an agent
worktree is a second checkout and the walk read 203 packages where there
are 104.
- plane.go keeps both new sections — sandbox ops and the coding-run seams are
orthogonal, no shared names.
- plane_debit_harness_test.go takes forge's, which delegates to
internal/planetest instead of holding a second copy. Same API. The copy is
what hid a bind failure: a unix path is 108 bytes and t.TempDir() spells
the test's name into it.
- metering keeps Actor and the token counts; plane.Usage has the fields, and
a debit without them can be re-read but not re-derived.
- o11y v1.5.62 over v1.5.61 — upgrade, never downgrade.
go build ./... and go vet clean. Full suite: 3 red (base, code, commerce),
identical to both parents — this box's SQLite lacks acos and fts5.
Red returned do-not-ship on two CRITICALs that chain to unauthenticated
cross-tenant code execution. Both are proven by apps/exec/auth_test.go, and all
four assertions FAIL when the old behaviour is put back — mutation-checked, not
asserted.
1. THE TENANT CAME OFF A HEADER. callCtx preferred cloud.Who(ctx).Org, which is
zip.CallerOf, which reads the X-Org-Id REQUEST HEADER (zip caller.go:377) —
and for a request with no validated bearer, SanitizeIdentity deliberately
RESTORES the client's own header (middleware_identity.go:455). So
`X-Org-Id: victim-corp` made storeFor open the victim's SQLite file: the run
executed in their store and /v1/files + /v1/download read their artifacts
back out. principal.OrgFrom is the org a VALIDATED principal resolved to and
nothing else; every other app resolves through it and this one did not.
plane.go's own note — "an org in the argument is an org the caller chose" —
is the rule it was breaking.
2. THE CREDENTIAL CHECK WAS A LOWERCASE PREFIX LIST OVER c.Path(). fiber routes
case-insensitively; cloud.RoutePath exists in this repo for exactly that and
two other gates already use it. POST /V1/EXEC matched the route and missed
the list: no key at all ran code, a wrong key read another session's bytes,
and CODE_EXEC_API_KEY UNSET — the documented fail-closed 503 — still ran code.
Same root, so one fix: authorization stops being inferred from the SPELLING of a
request. The middleware normalizes with RoutePath AND parks two facts on the
context — principal.WithOrg (inherited by the typed op, typed.go:82) and an
unexported `admitted` marker. tenantOf is the ONE tenant decision and refuses a
context carrying neither. The prefix list may still drift; drift is now
fail-CLOSED — a missed path is a 403 on a route that should have worked, never a
route that works without a credential.
AND THE DOOR NO LIST COULD HAVE COVERED. A typed op is also an MCP tool and an
op-plane op, and tools/call invokes it DIRECTLY (zip typed.go:474) — no route, no
middleware. `POST /mcp name=post_v1_exec` with no key ran code. Typing /v1/exec
for its SDK value is what opened that door; the handler-side check closes it,
because those doors park no marker and carry no principal.
3. [HIGH] AN EXEC SANDBOX HAD NO CEILING. Single-attach bounds dev/desktop via
their project; an exec sandbox has none, and the code tool sends no
session_id, so every call mints a fresh pod on a 15-minute lease — 40 calls,
40 pods, each 250m/512Mi/2Gi. The reaper is the FLOOR, not the ceiling: it
ends leases that are over, bounding the steady state and never the burst.
maxLiveExec=16 is written in node capacity (8Gi, 4 cores) and refuses with
429, because the caller's correct response is to wait. Counted with a real
COUNT(*), not len(List) — List is LIMIT 200, which stops counting exactly
where refusing starts to matter.
4. [HIGH] TWO CLIENT-SHAPE BUGS, both silent. hanzo.chat primes an attachment as
{id, session_id, name} (Files/Code/process.js) while @hanzochat/agents spells
it storage_session_id (tools.d.ts) — reading only the second skipped every
attached file AND the "not available" note, so a user's CSV was invisible with
no error. CodeFile.Session() reads both, and a ref with neither is reported.
Artifacts were COLLECTED recursively (find) and LISTED top-level (ls -1A), so
a nested artifact appeared in the reply and was missing from /v1/files/{sid},
which the client reads as expired. One find answers both now.
Also: apps/functions.go's doc no longer cites the deleted CODE_EXEC_UPSTREAM.
Tests: 22 in apps/exec (14 + 5 auth + 3 client-shape), 6 in apps/sandbox. Full
suite 3 red — base, code, commerce — all pre-existing and environmental (this
box's SQLite lacks fts5 and acos). Zero new failures. make check green.
NOT FIXED, and it is a decision rather than a defect: the real chat client sends
no credential at all (EnvVar.CODE_API_KEY undefined, handleTools.js sends no auth
header, the chat pod has no LIBRECHAT_CODE_API_KEY), so the guard 401s every
legitimate request. Fixing the bypasses does not make the feature work; settling
what the client presents does, and the house rule says Hanzo IAM rather than a
second shared secret.
Every image built from main has failed, after a green gate, at one line:
#28 ERROR: process "/bin/sh -c CGO_ENABLED=0 go build -ldflags=\"$GO_LDFLAGS\"
-o /boxd ./cmd/boxd" did not complete successfully: exit code: 1
cmd/boxd was deleted with the design that needed it — a sandbox is a POD, and
commands reach it over the Kubernetes exec subresource, so there is no daemon
inside to talk to and nothing to ship into it. The Dockerfile stage that built it
outlived the source, and `go build` on a directory that is not there is not a
warning. So the lane could TEST a commit and could not BUILD one, which is the
whole reason a sandbox executor that is merged and proven has never been served:
no image has published since v1.801.476.
This deletes the stage and the COPY that carried its output. The consumer they
existed for — hanzo/bot's Dockerfile.box, `COPY --from=cloud:<pin> /boxd` — has
to move to a pod nothing needs installing into; leaving a stage that cannot
compile does not keep that consumer working, it only stops everything else from
shipping.
Verified against this tree, with the Dockerfile's own tags: ./cmd/cloud,
./plugin/smoke and ./plugin/sandboxes all build, all 121 manifest apps have a
plugin/<app> directory (the loop hard-fails otherwise), and no non-comment line
mentions boxd.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Start required a non-empty org and nothing more. The org is then interpolated
into the KMS reference for the agent credential (credRef) and into the git
namespace, so an org carrying a separator or a `..` segment would address
another tenant's secret.
It arrives already validated from the gateway or the plane. This is the second
lock, placed on the side that would actually be harmed if the first ever failed
— the same rule the git subsystem states for itself (git.go orgRE), spelled
where the concatenation happens.
apps/exec was a reverse proxy to code-exec.hanzo.svc.cluster.local:8000, and
that Service has had ZERO endpoints for 33 days — /v1/exec answered 503 in
production the whole time. It was not failing to reach the executor; there was
no executor. apps/functions read the same CODE_EXEC_UPSTREAM and was failing
against the same absence.
cloud already runs the one compute primitive. A LibreChat session IS a sandbox
("a code-exec call = a sandbox with a session lease", apps/sandbox's package
doc), so session_id is the sandbox id, upload/download/list are Write/Read/Read
on that sandbox, and exec holds no store, no session table and no lifetime of
its own. The lease ends on the reaper.
THE BLOCKER, AND THE DECOMPLECTION. Every sandbox operation existed ONLY as
`func X(s *Service, c *zip.Ctx) error` — the domain braided into the transport,
so nothing else in the process could use a sandbox. apps/sandbox/api.go is the
domain as a VALUE: Lease, Get, List, Run, Read, Write, End. The HTTP handlers
become adapters (bind, call, JSON) and apps/sandbox/plane.go is a SECOND adapter
over the same functions. Errors are zip errors in the core, so "not yours" is
404 once, decided where the fact is known.
(Functions taking *Service, not methods: `Service = cloud.Service[state]` is an
alias for a generic type in package cloud, and Go cannot define a method on a
non-local type. apps/git's files.go is the same shape for the same reason.)
cloud.Ask AND NOT A GO IMPORT, and that is correctness, not style. Apps ship as
separate plugin binaries (plugin/<app>/main.go "links only its own subsystem"),
so importing apps/sandbox would give exec a SECOND sandbox service — its own
OrgStore on the same per-org SQLite files, its own reaper racing the real one.
plane.Ask already collapses to an in-process dispatch where the fleet fuses two
apps (zip.Serving/zip.Here), so the ONE call is a function call when they are
co-resident and a socket hop when they are not.
THE CONTRACT IS MEASURED, from ~/work/hanzo/chat, not remembered. Download is
TWO segments — /download/{session_id}/{fileId} (crud.js) — not /download/{id}.
Upload answers {message:"success", session_id, files:[{fileId, filename}]} and
the client throws unless `message` is that literal. /files/{sid} answers a BARE
JSON ARRAY of {name, lastModified} whose name is that same two-segment
identifier. And the code tool tells the model to persist artifacts in /mnt/data,
so sandbox gains workdirFor(class): /mnt/data for exec, /work for dev. Listing
/work would have reported no files after every successful run.
/v1/exec/programmatic answers 501 rather than being routed into the interpreter.
It is a DIFFERENT protocol — a run suspended on each tool call and resumed from
a continuation token — and answering it with a Result its parser cannot read is
a wrong answer where a refusal is an actionable one.
TENANCY, and the trap. cloud.For states a caller only on a context with NO
request behind it: zip.CallerOf reads the request's headers first, deliberately,
so no handler can assert an org a caller did not arrive with. A typed op's ctx
HAS a request behind it, so cloud.For(typedCtx, org) silently states nothing and
the peer answers 403. exec.callCtx is the one rule — pass a ctx that already
carries a principal through unchanged (hanzo.chat forwards the user's IAM
bearer), otherwise detach to Background, state the deployment's brand org, and
put the request's cancellation back with context.AfterFunc.
apps/functions moves to the same door, because task 3 (deleting the orphan
Service) would otherwise turn its 502 into a DNS failure. It calls exec.Run —
one operation, one home — and ENDS the lease, because a function invoke is over
when it answers.
THE FLOOR IS LOWERED HERE, next to its reason. exec published 40 operations
because a proxy cannot describe a contract it does not own; it now implements
one, so it publishes 5 — POST /v1/exec is a TYPED op (CodeRun -> CodeResult),
which is the schema, the MCP tool, the CLI command and the SDK method a proxy
could never carry. The four that stay untyped have real reasons: multipart in,
bytes out, a bare array, and a protocol we do not serve. Numbers taken from the
ratchet's own report, not computed:
paths 1762 -> 1759 (-3)
operations 2465 -> 2430 (-35)
download 10 -> 1 · exec 10 -> 2 · files 10 -> 1 · upload 10 -> 1
Deleted: newProxy, defaultUpstream, CODE_EXEC_UPSTREAM, and the ledger of 40
untyped operations. CODE_EXEC_API_KEY stays — it is the credential the chat
presents, and the guard is now middleware because a typed op takes no handler
chain.
Tests: apps/exec 14 (a sandbox peer double on the REAL plane, so an op renamed
or a field moved fails here and not in production), apps/functions green
including TestInvoke_AllowsAndDebitsCallerOrg, which was RED before this change
— metering moved debits onto the plane and the test still had only an HTTP
double.
Measured against HEAD: 12 packages fail before and after; zero new; one fixed.
LLM.md also carries a concurrent agent's insights.hanzo.ai section, present in
the tree and committed rather than dropped.
A coding run could not complete, for four independent reasons, any one of which
was fatal:
1. The run was spawned on a bare context.Background(). Every seam it touches —
session, clone URL, ref verify, PR row, and the balance gate behind them —
authorizes on the CALLER's org and never on an argument, so each answered
`authorize: no org on the call`. The routed target lookup was worse: it ran
on the webhook's REQUEST context, where cloud.For is a silent no-op. This is
the same shape that killed every chat turn until bridgeRunContext fixed it,
and start.go's runContext is the other half of that fix.
2. There was no /v1/coding. The app had no way to start a run at all.
3. The chat surface assembled its own Dispatcher, so Slack and the app would
have been two engines with two pools and no shared address for a run.
4. (unchanged here, stated in the handoff) the sandbox hop and its image are
deployment configuration, and both currently refuse.
The engine now lives in one process — agents, which already holds the session
store, the durable engine and the routed mailbox — and the START travels instead,
exactly as AgentsRunOnBehalf made one brain reachable from every chat platform.
POST /v1/coding and the coding_start plane op are two doors onto the same
coding.Start: same pool, same validation, same credential custody, same detached
tenant-stated context. The chat adapter is an adapter again — parse, authorize,
dispatch, reply — and no longer holds the org's git credential at all, which the
engine now reads from KMS at the one moment it dispatches.
The forge gets the rule that holds when the rest has failed. A run executes
untrusted model output against a real checkout, and the model also READS the
repo, so a README can carry an instruction. Assume the run is hostile and holds
the credential: refs/heads/agent/* is CREATE-ONLY, and the default branch cannot
be deleted by a push. That refuses the bait-and-switch — open a clean PR, let a
human read it, force-push the payload before the merge — structurally, at the one
point every push must pass through, rather than by asking reviewers to be
vigilant. It refuses with git's own report-status, not an HTTP 403, because a
control that can only say "RPC failed" is a control someone switches off.
Proven with the real git CLI against the real server: an agent branch pushes and
re-clones; --force over it, deleting it, and deleting main all come back
`! [remote rejected]` with the reason; feature-branch force-push and trunk
updates are untouched.
The run id was minted at the end of executeRun, beside the row it fills in.
That reads naturally and made the run unobservable: every span the run
produced — the step, each tool dispatch, each model call — had already ended
and exported by the time the run had a name, so none of them could carry it,
and neither could the per-token debits the metering decorator makes one round
at a time. The id existed only on the record of a thing that was already over.
Mint it before the work instead. One value is now on the span, on the row and
on the money, which is the whole of "drill into this run":
- every span (agent.run / agent.step / agent.tool / the gen_ai client span)
carries hanzo.agent.run_id, so attribution never depends on walking a
parent chain that sampling or a truncated batch may have broken;
- the run row carries the trace id, which is the key the run history and the
span store had no version of — two accounts of one event that could not be
joined in either direction;
- types.ChatRequest.RunID rides to metering.Usage.RequestID, the field whose
stated job is exactly this, so a run's per-token cost is a SUM over ledger
rows rather than unanswerable. It is the correlation id and NOT Ref: a tool
loop settles once per round, and pinning the idempotency key to the run
would dedup every round after the first into the first one's debit.
WHO, not just which tenant. org answered "whose ledger" and nothing answered
"which person" — an actor reached the debit and the session row but was never
written to the run, so a scheduled or on-behalf run had no answer at all.
Runs now record it and the spans carry hanzo.user.
A tool dispatch is readable as a dispatch: the span names the tool, its call
id, the owning subsystem, the round it happened in, and its outcome — set on
every exit, including the happy one, because a status written only on failure
cannot tell "succeeded" from "never finished". So "it called six tools and
failed on the fourth" is a fact you can read instead of infer.
toolSubsystem derives the owner from the operation name rather than looking it
up: the fleet door spells ops <method>_v1_<subsystem>_<rest>, so it is a fact
the value already states, and it answers the same in the fused binary and in a
single-app plugin process — where cloud.SubsystemOf reads a boot-time mount
index that knows only that plugin's own routes and would answer "" for every
sibling's tool.
GET /v1/agents/runs is the org-wide feed the per-agent history could not be:
an operator asking what a tenant's agent plane is doing does not start out
knowing an agent ref, and listing the agents to page each one's history is N+1
round trips to rebuild an ordering the org index already has. The org comes
from the caller's identity; there is deliberately no org field to forge.
scanRun is one function because runCols was named once to stop the projection
drifting and both readers were spelling the column order out by hand anyway.
The tests run a real turn — real handler, real tool loop, real OpenAI-wire
client — and assert on spans that actually reached an exporter, because a span
that is created and never exported is the failure this is about. The provider
is installed once per process and only the sink swaps: OTel's global delegates
ONCE, so a tracer handle taken at package init binds to the first provider and
keeps it, and a test that installed its own and shut it down on cleanup left
every later span-asserting test in the binary seeing nothing.
The usage debit moved off HTTP onto the internal plane (ac09a86d), for a good
reason: metering.Usage.Ref is `json:"-"` so no request body can set the ledger's
idempotency key, and json.Marshal therefore dropped it — every split-deploy debit
reached the ledger anonymous and a sealed retry was charged twice.
Nine packages never moved with it. Each had hand-rolled a usage counter against
POST /v1/billing/usage, so each went on counting an endpoint nothing calls: 20
tests across apps/{agents,cloudflare,content,functions,ml,projects,provisioning,
risk,storage} reported "0 debits" for work that had in fact been billed. That is
the release lane's break, and it predates the sandbox work — bisected to
ac09a86d, which is a metering commit, not a sandbox one. Before it, the same
tests pass.
THE CROSSING WAS DROPPING FIELDS OF ITS OWN. plane.Usage had no Actor and no
token counts, and metering.Usage carries both — so the crossing lost WHO acted
and WHAT WORK the amount priced, and the receiver rebuilt a ledger row without
them. Exactly the failure the Ref fix was written for, one field over. That is
not a test problem and is fixed on the wire: plane.Usage gains Actor,
PromptTokens, CompletionTokens and TotalTokens, the sender sends them, and
commerce's record handler writes them. apps/agents already asserted an actor is
present; it now passes because the actor arrives, not because the assertion left.
ONE PEER, NOT TEN. internal/planetest is commerce's half of the money plane on a
real socket, following internal/iamtest: one fixture for one wire, because the
copies are what drifted. It also ends a bug all the copies shared — they bound
the socket under t.TempDir(), which spells the TEST'S NAME into the path, and a
unix address is a fixed 108-byte field. TestMeterUsage_OneActIsChargedOnceIn-
EitherTopology/split_deploy produced a 106-character address; the bind failed
inside a goroutine where nothing read the error, and the symptom was a debit
that never arrived. Whether it failed at all depended on how long $TMPDIR was,
which is to say on whose machine it ran. The shared peer uses a short directory
and refuses an address it cannot bind instead of failing silently.
Verified on the gates CI runs, with its tags and skips:
go vet -tags "sqlite_fts5 sqlite_math_functions" ./... clean
make -f mk/fleet.mk check openapi.yaml unchanged — 1762 paths
go test -tags "..." ./... the 9 packages green, root green
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Wire the embedded sentry runtime to the ONE binary-wide key resolver: a pk- key
posted to /v1/sentry resolves to its org through the SAME seam /v1/event uses
(sharedKeys), and the DSN's project is auto-provisioned under that org on first
ingest. So sentry.hanzo.ai accepts the exact key that already feeds analytics +
insights — no per-project DSN secret, dark no more.
- auth_apikey.go: export ResolvePublishableKeyOrg (the pk- -> org seam for
sibling packages that resolve a key out-of-band of the Authorization header).
- apps/o11y/embed.go: implsentry.SetIngestKeyResolver at the o11y mount.
- apps/o11y/o11y.go: migrate the removed o11y.SetHandler(gh) -> SetRuntime(Whole(gh))
(o11y v1.5.61's mount API), the drop-in "one door" replacement.
- go.mod: github.com/hanzoai/o11y v1.5.58 -> v1.5.61 (carries the sentry change).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every part of this already existed and nothing called it. ExpiresAt was written
on every create, LastUsedAt was stamped on every call, Store.Expired was
implemented, stop was implemented — and a sandbox, once created, ran forever.
Measured rather than inferred: a proof pod from a finished test was still
Running 64 minutes later and had to be deleted by hand.
That is one bug wearing two faces. A sandbox that never sleeps is a sandbox that
bills forever, so this is simultaneously the idle-sleep feature and the honest
meter. They are not separate components and building them separately would give
the fleet two clocks that disagree.
TWO TRIGGERS, ONE ACTION:
ExpiresAt the LEASE — the caller said how long they wanted it.
LastUsedAt ATTENTION — a sandbox nobody has touched for an hour is abandoned
even with hours of lease left, and holding a pod for a session
someone walked away from is the entire cost this exists to stop.
Both end the pod and both KEEP THE VOLUME, because purge is a separate opt-in
and ending a lease must never destroy what the tenant made. So the checkout and
the caches survive and the next call for that project gets a fresh pod against
the same disk — "resume is cheap" with no suspended state to define and no
resume route to serve.
A `status: suspended` was the obvious other design and it is worse: Status is
pending|running|error, nothing resumes, and a row in a fourth state that no
route can leave is a dead row wearing a live sandbox's clothes.
Idle is computed here rather than queried, because "untouched for an hour" is a
POLICY and the store holds facts. The constant that defines it lives beside the
code that applies it.
Started from Mount, not from a route and not from a caller. A reaper that
something has to remember to call is the bug it was written to fix.
An agent that can edit but cannot commit has done nothing durable, so the second
half needed its own proof. Against hanzo-k8s, 62s, stock node:22:
GIT PRESENT: git version 2.39.5
COMMIT MADE: 7fb283f the agent committed this
FORGE REACHABLE: exit=0
git needs no routes of its own — it is a command, and exec runs commands. The
predecessor had a /git/clone and a /git/push endpoint and a credential file
written to /tmp at 0600, which the code it was sandboxing could poll for during
the push window. None of that survives, because none of it was git.
Identity is set locally rather than globally: a sandbox is per-tenant, and a
global identity is one more thing to reset between leases.
The ls-remote is deliberately NOT a push. It exits 0, which proves the network
reaches the forge — and proves nothing about a credential. Those are different
gaps and conflating them sends someone to fix the wrong one. Pushing as a tenant
still needs a per-org token the sandbox does not yet have.
Stage 3 calls itself the load-bearing check and it could not pass. It read
o11y_traces.o11y_index_v3 and o11y_logs.logs_v2; both databases are gone, so
the query errored, `mins` came back empty, and the script printed "stale --
nothing is draining into the store" and TELEMETRY CHAIN: BROKEN on every run.
A check that always fails reports nothing. Worse, it spends the alarm it exists
to raise: the one run where telemetry really has stopped looks exactly like the
hundred before it.
Traces and logs are event.span and event.log now, and both date themselves with
the same `time` column, so the per-probe expression -- and the special case that
overrode it for traces -- has nothing left to vary and is gone. Measured while
writing this: both signals 0 minutes stale, 3.3M spans and 131.4M logs.
Every @hanzo turn answered "the agent hit an error handling that". The tenant
fix landed and was correct — the run reached agents with org=hanzo and passed
the balance gate — and then the model call died one hop lower:
Post "http://ai/v1/chat/completions": EOF
deps.AI dialed zip.SocketPath("ai") and spoke ordinary HTTP to it. That address
is wrong twice, and each half is independently fatal.
The WIRE is not HTTP. The socket is served by zaphttp.Server — ZAP, a framed
binary protocol. A cleartext request is not slow there, it is unintelligible:
the peer reads a malformed frame and closes. Measured on a healthy pod, every
request over that socket EOF'd — GET /v1/health, GET /v1/models, a bogus path,
POST /v1/chat/completions — and agents.sock and commerce.sock did the same, so
this was never about `ai`.
The SURFACE is not the app's. What binds there is the app's PLANE, the typed-op
door plane.Ask uses; the app's own routes are on a listener it knows nothing
about. Framed correctly as ZAP, ai.sock still answers 404 for /v1/models while
ai's own listener answers 200.
So `ai` never logged the request, because the request never arrived, and the
run came back error-status with a nil error — which is the branch that logged
nothing at all. Hence a day of looking at a healthy `ai`.
The address that does serve it is the fleet ROUTER's own listener: it owns the
route table that sends /v1/* to `ai`, and it owns starting a cold app. Entered
on 127.0.0.1 it never leaves the pod — no DNS, no Service hop, and no trip out
through Cloudflare and back to the pod's own public address, which is what the
configured base URL does. Only the port is read from CLOUD_LISTEN; the host half
a process binds is not an address a client may dial. Measured there:
/v1/models 200, bogus 404, /v1/chat/completions 401 for the Bearer the M2M
client already mints.
There is deliberately no second mechanism left. A raw route reached
process-to-process is not something this fleet offers; ops are. The custom
transport is deleted rather than repaired.
The non-"ok" branch in the bridge now says so, with the run id. It is the branch
a broken inference path lands in and it was the silent one.
Every metric the console reads through PromQL failed on the live pod with
"Database o11y_metrics does not exist" -- 96 times in 30 minutes, and each one
renders as an empty chart rather than an error, so service health, HTTP traffic,
ingest, memory, plane throughput and alert delivery all read as silence. The
data was never missing: event.metric holds 145.7M rows and was 3 seconds behind
real time throughout. Only the address was wrong.
o11y v1.5.62 points its PromQL client at event.series/event.metric via the
telemetrymetrics constants, which is where cloud's own gauge reader
(apps/o11y/metricsgauge.go) has been reading all along.
The bump also crosses o11y's runtime seam: SetHandler(http.Handler) became
SetRuntime(Runtime), which resolves a handler PER ADDRESS so a declared op that
the runtime does not serve is a nil rather than a 404 indistinguishable from a
typo. Both of cloud's installs are o11y.Whole -- the embedded runtime is one
router that matches the request's own path, and the proxy fallback has one door
with the far side selecting the route. Neither has anything to resolve per
address, and Whole is that shape stated honestly.
I moved this to hanzo-sandboxes on the reasoning that everything running
submitted code belongs in the one namespace whose policy denies it the cluster.
That reasoning is right and the change was wrong: there is no code-exec Service
in hanzo-sandboxes, so it renamed a 502 into a DNS failure. A peer caught it and
said so plainly — target ns is EMPTY, do not ship.
Moving the NAME before moving the SERVICE is the same mistake as a policy whose
selector matches no pod, and this file would have been the fourth this week.
The Service moves first; then the address follows it.
Neither namespace makes this path work today — the Service in hanzo has had no
endpoints for 33 days. The fix is not a better address, it is /v1/sandboxes,
where a caller gets a pod rather than a proxy hop to a workload nobody deployed.
sha-1fa6f964cb4d mounted GET /v1/billing/{transactions,credit-balance,
accounts} and shipped them — the route strings are provably in that
image's commerce binary — and all three still answered 404 in production.
An address has two halves. mount.go states what commerce will answer;
manifest.Apps states what the host may hand it, and the commerce row is
an explicit leaf allowlist, not a /v1/billing subtree ("Nobody claims the
bare /v1/billing REMAINDER"). A leaf absent there is never delivered: it
falls to the "/v1" remainder on ai's row and answers ai's bare 404. From
outside that is byte-identical to a route that was never mounted, which
is why the first fix verified correct at every layer anyone thought to
check — source, image bytes, deployed digest, running version — and
changed nothing a caller saw.
credit-balance is stated separately from credits on purpose: they are
sibling prefixes, so matching one does not match the other, and next to
an existing "credits" entry the missing one reads as already covered.
accounts covers its /:id/members child.
The test asserts the ROUTER's half specifically, because nothing else
can. apps/commerce's route tests cannot see this table, and a byte-level
check on the built image cannot either — the string was in the binary the
entire time it was returning 404.
The merge collapsed two manifest rows into one and left a name that matched
nothing. The app was called `sandbox`, it served `/v1/sandboxes`, and its
operations were tagged `sandboxes` because the tag comes from the path — one
thing under three spellings, and the Makefile named a fourth.
It is `sandboxes` now, which is what its path says and what every sibling
collection already does: bots, agents, tasks, books. The package stays
apps/sandbox and pkgOf carries that, which is the case pkgOf exists for.
The floor's `sandbox: 26` was the BOX POOL — a warm pool of pods, a daemon
inside each one, a shared pool-wide key, and a bind handshake so a box could
tell which box it was. All of it is deleted, and the row goes with it rather
than being pinned at 0: the product does not exist, and a floor entry for a
product that does not exist is a claim that something is missing.
That deletion is right and it is not mine to mourn. runtime.go says why in its
own words — the boundary is the pod's runtimeClassName, one field, and
"everything the predecessor built to approximate that boundary in Go is deleted
rather than kept for defence in depth, because a second half-boundary is a
second thing to keep true." Every problem the pool had, I spent tonight fixing:
the key that opened every box, the row whose address the CNI had reassigned, the
identity a warm pod could not learn. None of them exist without the daemon.
There is no key, and a pod is addressed by NAME through the apiserver, which
cannot go stale.
Four operations published an operationId and nothing else. These routes bind
through zip.Ctx rather than being typed ops, so zipdoc has no comment to lift
and openapi.Describe is the seam — the other three already used it. They say
what they do now, including the two facts a caller cannot guess: a 404 rather
than a 403 for another org's id, because a 403 confirms the id exists; and that
the volume SURVIVES a delete unless purge=1, which is the one part that cannot
be undone.
1758 paths, and the weave is green.
apps/lsp ran gopls, tsserver and the rest INSIDE a fleet pod: server.go,
workspace.go and langs.go checked repositories out, fetched their dependencies
and executed a third-party toolchain over untrusted bytes in the same process
that holds a principal, a ledger and a KMS-injected environment. That is the
wrong shape however carefully it is written, and hanzoai/lsp exists to be the
right one — a jailed daemon with gVisor, no egress but a module proxy, and no
git credential of any kind.
So this side becomes a PROXY over that daemon, and keeps the three things the
daemon must never hold:
TENANT the org is the validated principal's, never a body field, and it
is the daemon's isolation key.
REPOSITORY the revision and the tree come from git's own object plane, for
the caller's own org. The daemon cannot fetch a repository — a
credential that could reach every repository is exactly what must
not sit next to an unjailed compiler — so a cold revision is
/ask → 409 {"need":"tree"} → /root → ask again, ONCE.
LEDGER the gate runs at the prepare price before any work, the debit
after the answer. "prepare" and "query" are the two Models, and
spend.go's meteredApps entry still holds.
The surface moves from /v1/lsp to /v1/code/lsp, and from one door with a
`method` field to five typed ops — hover, locate, symbols, diagnostics,
complete. code and lsp are two reads of ONE repository, not two products: code
is the static index, lsp the live server that follows a symbol out of the
repository and into a dependency. One home means one place to look for it, in
the document and in the MCP tool list alike, and an agent picks a tool by its
name rather than by a union behind one. Nested static prefixes resolve by
specificity, so /v1/code/lsp beats /v1/code and both beat ai's bare /v1 — the
same relation storage's /v1/s3/buckets already has to provisioning's /v1/s3.
git gains ONE op, git_rev: the commit a ref names. It is separate from git_files
because the two questions cost differently — resolving is a ref lookup and runs
on every position query; reading is a walk of the whole tree and runs only when
the daemon says it holds no root. Folded together, a hover would drag a monorepo
across a socket. One resolve (coreRev) now backs both.
plane/agents was generated before agents_run_on_behalf existed; running the
generator carries it in.
The published surface GREW: 1762 → 1766 paths, 2480 → 2484 operations. The
floor's "lsp" product is gone because its five operations are tagged "code" now,
which is the move, and code rises 7 → 12.
The box vocabulary is gone. It named the same thing three ways — a box in the
Go, a machine in the types, a sandbox in the API — so a policy could select
hanzo.ai/box-class while the code set hanzo.ai/sandbox-class and both looked
right in isolation. That is not a style preference: it is how the containment
shipped selecting nothing.
apps/sandboxes -> apps/sandbox package sandbox, not sandboxes
Machine -> Sandbox the type is the noun the API serves
machines.go -> sandbox.go
MACHINE_* -> SANDBOX_*
/v1/sandboxes stays plural because a collection is plural. The PACKAGE is
singular because it is one thing.
DELETED, not deprecated: cmd/boxd (~2000 lines), the old apps/sandbox pool and
proxy, apps/sandbox/wire. The daemon existed to serve a filesystem and a shell
from inside the pod over HTTP; the Kubernetes exec subresource already does
that, so boxd was a second implementation of a thing the cluster ships. Deleting
it also deletes what it forced: a shared service key living in an environment
the submitted code could read out of /proc, a bind protocol so a recycled pod IP
could not serve another tenant's checkout, and an HTTP server that had to be
built into every sandbox image.
wire went with it, except for one string. It held the types boxd and the proxy
both marshalled, which is a real reason for a package — but the last survivor is
the /v1/exec path shared by apps/exec and apps/functions, and a package that
exists to hold one constant is not a package. It moves to exec.Path, in the app
that serves it. The reason it must have exactly one home is unchanged and worth
keeping: those two consumers disagreed about it in production, one asking for
/v1/exec and the other building upstream+"/exec", and because the executor did
not exist neither was ever wrong out loud.
Routes registered. Create, List, Get and Delete were exported functions no
request could reach — a caller could exec in a sandbox it had no way to create.
The comment said they were "shared with the compute surface", which was true
while this served /v1/machines (visor's, where a second registration is a
conflict) and false the moment it moved. Handlers present, routes absent,
nothing saying so: the same shape as the policy that selected no pod and the
installer that installed nothing.
Re-lands 2f9ffc2d, which 0d7011e2 reverted for want of evidence. The evidence
now exists, and it is a run whose gate had already passed.
Run 895 (41a0e5a3, event push, branch main):
gate success 09:07:35Z -> 09:46:45Z
cicd success 09:07:35Z -> 09:46:45Z
containment success 09:07:33Z -> 09:09:10Z
image cancelled never started -> 10:25:26Z
rollout cancelled never started -> 10:25:26Z
reach cancelled never started -> 10:25:26Z
fanout cancelled never started -> 10:25:26Z
receipt cancelled never started -> 10:25:26Z
The run's own conclusion is `cancelled`. On a push, `${{ github.event_name ==
'pull_request' }}` is supposed to be false and nothing should have cancelled
anything. The forge does not evaluate it to a boolean; a non-empty string is
truthy, so every push cancels the run before it.
The revert argued these runs "were superseded while still QUEUED, not killed
mid-gate, so cancel-in-progress was never the thing acting on them". That holds
for the runs it measured — 889 through 892 never received a runner — and cannot
hold for 895: a run that never started cannot contain three jobs that completed
successfully. Both failure modes are real. Only this one is a line in this file,
and it is the one that throws away work the gate already finished.
Runs 891-909, every one a push to main: 16 cancelled, 3 failure, 0 success. The
single run that earned a release lost it 39 minutes after going green, which is
the whole of "the gate is green and nothing ships" — and why a cutover that has
been on main all day is still 404 in production.
A literal false is read correctly and restores what the comment above already
described. It does not queue without bound: one run holds the group and the
rest collapse into a single pending run, so pushes during a release are
coalesced, not accumulated. The cost is that a stale PR run is no longer
cancelled by its own next push, because the setting can no longer ask what event
it is — cheap next to a release lane that cannot finish.
The other half of the diagnosis under the revert stands and is not addressed
here: ten runners of hanzo-build-linux-amd64 serve five orgs, and a run that
waits longer for a runner than the interval between pushes still starts late.
That is fleet sizing, and it belongs in universe, not in this file.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
MeterUsage documents a debit as exactly-once on the act's ref, and a surface that
already holds the act's server-assigned name sets it precisely so the work and its
charge are one thing under one name — apps/company mints a formation ref and hands
that same value to the debit.
The promise held only while the ledger was co-resident. In the shipped split
topology the debit leaves through meterPeer, which built its plane.Usage from
Project and Service alone: the ref never boarded. The receiver mints a fresh name
when an arrival carries none (apps/finance RecordUsage), so nothing could dedup and
a re-driven formation debit charged the customer twice for one company.
So the name crosses with the act, and the act is named ABOVE the topology branch.
Sealing below it was what let the two paths key the same act differently; sealing
above means neither can. Seal is idempotent, so a caller that already holds the act's
own name keeps it, and the co-resident path — where the metering client seals what it
is given — is unchanged by having been sealed one step earlier.
Two calls are still two acts: an unnamed usage is sealed with a fresh server-minted
name, so identical inferences bill twice and the fix cannot fold distinct work into
one charge.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The same act can hold BOTH usage keys at once — one entry under the empty program
and one under the wallet — whenever its ref was nameable by the client across the
change that scoped a usage ref to its wallet. An org that renewed the same domain
once before and once after writes `domain:renew:foo.com` into both program
namespaces.
Re-keying the legacy row onto the wallet then collides on
UNIQUE(kind, program, ref), and a plain UPDATE makes that abort the statement: the
migration fails, so Open fails, so storeFor fails, so the prepaid gate has no ledger
to read and fails closed — correctly — on EVERY paid request that org makes, for
good, on a store no retry can open. The repair runs at first open after a deploy, so
it was armed for every org not yet opened.
OR IGNORE skips the colliding row instead. That is the honest outcome rather than a
concession: the wallet row IS that act's entry, so the probe that asks whether the
act is already paid for finds it, and the legacy row's empty program can never fund a
second debit. It is also curative — a store that already aborted opens on the next
attempt, which is what un-bricks an org that has already hit this.
The repair also moves BELOW the DDL re-apply. On a legacy amount_cents file the
migration rebuilds treasury_postings without its indexes, and the repair reads each
candidate entry's postings twice as correlated subqueries — so running it first
scanned the whole postings table once per usage row, on the one open where the file
is largest and a customer is waiting on it.
And the kind is now the opener's to name. This is a generic journal; which of its
kinds are scoped by wallet rather than by the book is a fact only the app that writes
them holds, so finance passes its own KindUsage and the house book — whose accrual,
seed and payout kinds are all book-scoped — passes none. A literal here could be
renamed in finance and silently become a no-op that bills one act twice; the finance
tests now fail if the kind stops arriving.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Measured in-cluster on cloud's exact shape, same node pool, same buildkit,
fresh pod, run in both orders:
gzip / mode=max zstd / mode=min
exporting layers 194.6 245.5 256.1 s 39.2 58.1 53.3 s
cache export 212.1 s 2.0 s
image size 1,646,830,883 B 1,626,409,300 B
Derived on the real build: -219s cache export, -136s layer export = ~354s off
17m09s. Both scale WITH the image, so a smaller image compounds them.
WHY mode=max was wrong. The build stage holds the same 4.2 GB the final image
does, so mode=max gzips it a SECOND time and pushes a second 1.6 GB blob. What
it bought: two independent builds imported the cache and got exactly the same 8
CACHED steps — apk add / adduser / symlink / WORKDIR in each stage. Every
expensive step missed both times, and must: COPY . . sits in front of them, and
the compiles write into --mount=type=cache, which is worker-local and never
travels in an exported cache. The four build-stage records mode=min gives up
cost 4.6s cold. It was paying 221s per build to save 4.6s on the next one.
WHY 175s for one export. The plugin layer is 1,627,346,308 of 1,646,830,883
bytes — 98.8% of the image — and 4.15 GB at 175s is 23.7 MB/s: single-core
gzip, with seven of eight CPUs idle.
Three flags are load-bearing, not decoration:
- oci-mediatypes=true is REQUIRED; Docker schema2 has no media type that can
name a zstd layer.
- force-compression=true is REQUIRED. Without it buildkit publishes an OCI
manifest whose layers are still tar+gzip, reusing blobs it already has —
silent, and in the "still works, just slow" direction.
- the cache's compression must MATCH the image's, or mode=min re-compresses
what the image export just wrote and hands the saving back.
AND THE ONE THAT WOULD HAVE BITTEN. imagePullable in pin.go negotiates the
manifest by Accept, and its list omitted application/vnd.oci.image.manifest.v1
+json. Measured against ghcr with a real zstd image, with a bogus-tag control:
zstd image, old header -> 404 zstd image, new header -> 200
gzip image, old header -> 200 nonexistent tag, both -> 404
ghcr honours Accept strictly, so shipping zstd without this makes EVERY release
pin fail, insisting the registry does not have an image that is sitting there.
That list was coupled to the builder's compression setting without saying so;
the coupling is now written down.
SEQUENCING: this pin.go fix must be DEPLOYED before the build flags flip.
imagePullable runs in the deployed cloud binary and gates both v* and sha- pins.
NO LAYER SPLITTING. Every plugin embeds the per-build version string
(-X ...Version=${VERSION}, confirmed in 5/5 shipped binaries), so all 119 change
bytes on every commit. There is no stable subset to split off, and splitting
would add manifest entries while saving zero export work. Revisit only if the
version stamp moves out of the per-plugin binaries.
NOT VERIFIED: no full real cloud build with these flags yet — the numbers are a
same-shape synthetic whose gzip baseline reproduced production within 11%. zstd
is proven INSIDE this cluster (ghcr stores it; containerd 1.7.28 pulled and RAN
it on a node that had never seen the bytes). Anything pulling from outside — an
old Docker, an external mirror — is untested.
Four apps kept their own copy of the projection's method set and asserted it.
Dropping TRACE and OPTIONS from the document therefore surfaced as eight
failures that looked unrelated to it — bots and dns and o11y naming routes
"which this subsystem no longer serves", exec insisting on 56 operations where
40 are published.
Nothing about the WIRE changed. All() still binds every method and the executor
still answers them; exec's own subtest proves exactly that and is why it stayed.
What changed is what we OFFER, and the ledgers were describing the offer from a
second copy of the truth.
exec reads openapi.Methods() now, the same fix websearch took an hour ago and
for the same reason: one set, one place, both halves move together. The others
had literal entries, which are simply gone.
The counts move with it and say why in the prose beside them, so the next reader
gets the reason rather than a number that used to be right:
bots 10 -> 8 six product methods on the relay, plus two typed ops
exec 56 -> 40 8 paths x 5 published methods
SEPARATELY, and not caused by any of that: TestJWKSHasOneDerivation walks the
tree from ".", and git puts WORKTREES inside the working tree. It failed naming
.claude/worktrees/agent-…/apps/base/pool.go — a file from another commit of this
same repo, which this commit is not responsible for and cannot fix. It skips
.claude and .worktrees now. The gate could never fire in CI, which checks out
clean, so it only ever cost whoever had a worktree open — which, here, is
everyone running agents.
Verified against d2c44bab rather than asserted: the metering timeouts and the
frozen-app-order failure are pre-existing, and three of them could not even RUN
at that baseline because the duplicate Describe panicked their package at init.
Fixing that panic did not break them. It made them visible.
MeterUsage documents a debit as exactly-once on the act's ref, and a surface that
already holds the act's server-assigned name sets it precisely so the work and its
charge are one thing under one name — apps/company mints a formation ref and hands
that same value to the debit.
The promise held only while the ledger was co-resident. In the shipped split
topology the debit leaves through meterPeer, which built its plane.Usage from
Project and Service alone: the ref never boarded. The receiver mints a fresh name
when an arrival carries none (apps/finance RecordUsage), so nothing could dedup and
a re-driven formation debit charged the customer twice for one company.
So the name crosses with the act, and the act is named ABOVE the topology branch.
Sealing below it was what let the two paths key the same act differently; sealing
above means neither can. Seal is idempotent, so a caller that already holds the act's
own name keeps it, and the co-resident path — where the metering client seals what it
is given — is unchanged by having been sealed one step earlier.
Two calls are still two acts: an unnamed usage is sealed with a fresh server-minted
name, so identical inferences bill twice and the fix cannot fold distinct work into
one charge.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The same act can hold BOTH usage keys at once — one entry under the empty program
and one under the wallet — whenever its ref was nameable by the client across the
change that scoped a usage ref to its wallet. An org that renewed the same domain
once before and once after writes `domain:renew:foo.com` into both program
namespaces.
Re-keying the legacy row onto the wallet then collides on
UNIQUE(kind, program, ref), and a plain UPDATE makes that abort the statement: the
migration fails, so Open fails, so storeFor fails, so the prepaid gate has no ledger
to read and fails closed — correctly — on EVERY paid request that org makes, for
good, on a store no retry can open. The repair runs at first open after a deploy, so
it was armed for every org not yet opened.
OR IGNORE skips the colliding row instead. That is the honest outcome rather than a
concession: the wallet row IS that act's entry, so the probe that asks whether the
act is already paid for finds it, and the legacy row's empty program can never fund a
second debit. It is also curative — a store that already aborted opens on the next
attempt, which is what un-bricks an org that has already hit this.
The repair also moves BELOW the DDL re-apply. On a legacy amount_cents file the
migration rebuilds treasury_postings without its indexes, and the repair reads each
candidate entry's postings twice as correlated subqueries — so running it first
scanned the whole postings table once per usage row, on the one open where the file
is largest and a customer is waiting on it.
And the kind is now the opener's to name. This is a generic journal; which of its
kinds are scoped by wallet rather than by the book is a fact only the app that writes
them holds, so finance passes its own KindUsage and the house book — whose accrual,
seed and payout kinds are all book-scoped — passes none. A literal here could be
renamed in finance and silently become a no-op that bills one act twice; the finance
tests now fail if the kind stops arriving.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/zen was linked, copied into the image and pulled on every deploy to be executed
never. cmd/cloud's mount() returns at `if a.Coresident` BEFORE it can resolve a
path or spawn a child; zen's behaviour ships inside /ai, which links apps/zen and
mounts the Claim ahead of ai's catch-all.
The build list is still derived from manifest/apps.go: `names` (119) still guards
the plugin/<app> bijection, `spawned` (118) earns a binary. Flip
Coresident:false and the binary returns next build.
Checked rather than assumed: the other a.Plugin() call site (locate()) reads only
.Addr; with /zen absent resolve() falls to pluginIn() and fetch() returns nil,nil
because CLOUD_PLUGINS is unset — no stat error, no network. Nothing in helm,
compose, smoke or the workflows names a /zen binary.
MEASURED: 4225 -> 4060 MB uncompressed (-164.7 MB), 1.65 -> ~1.60 GB compressed.
While measuring, two things worth recording because they contradict the obvious
guesses:
- Sqlite/CGO is not the cost, it is a 3.29 MB SAVING per binary. CGO=1 with
libsqlite3 links the small mattn shim against system libsqlcipher; CGO=0
links pure-Go modernc. Dropping the tags for "non-storage" plugins would ADD
388 MB. The Dockerfile's uniform-tags decision is right.
- 100% of the delta is the duplicated library. plugin/dns carries 741 lines of
app source and is 32 BYTES larger than an empty app. 119 x 20.23 MB =
2,407 MB of 4,095 (58.8%) is the same request tier byte-for-byte, 119 times.
There is no Mount-style registry indirection to break here — manifest is 344
packages of stdlib+zip importing zero apps, and every plugin main imports only
cloud + its own app. The fat deps are honest (geth<-treasury, esbuild<-
connectorruntime, k8s<-ai/controllers). A perfect diet of the shared floor is
bounded at <=519 MB (12.7%), because the host is a real serving binary at
15.86 MB.
The big prize is a multi-call binary: 4,095 MB -> 336 MB (-92%), one file plus
118 symlinks, every app still its own process. Built and linked in 12.3s; NOT
shipped because argv[0] dispatch cannot be verified end-to-end without a
container, it needs gen-app-cmds to derive a fused main from 119 hand-authored
mains, and it reverses a documented decision in manifest/release.go. That is a
designed change with a test plan, not a slip-in.
Also rejected with numbers: -trimpath (-10 MB fleet, 0.25%, cold-busts the build
cache), reflect method-pruning (three independent triggers; closing two buys
+128 B), embeds (<=63 MB, 1.5%).
`go generate -run zipdoc ./...` was 355.9s of a 17-minute image build — 35% —
spent regenerating 99 zipdoc_gen.go files that are already committed. The image
now trusts them.
This is strictly stronger, not a trade. The build-time pass would happily build
a CORRECT image from a STALE commit and leave main wrong, silently. That is not
hypothetical: plane/plane.go documents RunOnBehalfIn.Model and HEAD's
apps/agents/zipdoc_gen.go carries not one word of it. Main was stale the whole
time that RUN was "doing its job" every build. Freshness is a property of the
COMMIT, so it is now enforced where commits are.
Two gates existed and neither could fire:
- make test has carried a zipdoc -check loop for a long time, but CI never
runs make test — hanzo.yml's test: block runs raw go steps. It has never
executed in CI, not once.
- app-contract → surface-check regenerates these files as a SIDE EFFECT
(mk/plugin.mk: describe: build, build: generate) and then scopes its
porcelain check to openapi.yaml / openapi/floor.json / plugin/. A stale lift
was silently repaired in the runner's tree and never reported.
So the new zipdoc-current step runs BEFORE app-contract. After it the tree is
already regenerated and the check asserts nothing.
The two inline -check loops in test and test-fast collapse into one zipdoc-check
target all three callers use — the same discipline app-contract already follows,
and exactly the drift that let the Makefile's own check end up policing nothing.
The old remediation rendered `././...` for d=.; the new one names a command that
works from a bare checkout.
The gate regenerates and diffs rather than asking -check for a second opinion: a
gate must never be able to disagree with the tool it polices. It uses
git status --porcelain, not git diff, so a NEW package's untracked zipdoc_gen.go
cannot slip through.
Also commits the live staleness in apps/agents/zipdoc_gen.go.
Measured: 40.4s warm on an 8-core box, 355.9s cold on the BuildKit runner.
go build ./... clean; TestSpecCarriesProse passes and is byte-unmodified — it
now proves the prose in the file that actually SHIPS is live.
Same shape as the bots entry: a ledger describing a surface that moved, and a
freeze that a new app walked past. No behaviour changes.
ceff43ac drew the line that the document publishes what was DECLARED rather
than whatever the router happened to bind under All(), which retired the
OPTIONS and TRACE entries every wildcard used to carry. Three ledgers still
named them, so each described operations its own document no longer has:
- apps/dns: five relay operations at the greedy wildcard, not seven.
- apps/o11y: the /v1/sentry catch-all, same two.
- apps/exec: derived rather than listed, so the fix is one line —
servedMethods is the five methods zip has a typed registrar for, and the
cross falls from 8x7=56 to 8x5=40. methodReason went with them; it existed
only to explain OPTIONS and TRACE.
Worth stating because the two facts disagree: OPTIONS and TRACE are still
SERVED. All() registers every method and the executor answers them, which
the proxy subtest asserts and which still passes. They are no longer
PUBLISHED. The ledger counts the document, so they leave it.
manifest: apps/sandbox joined manifest.Apps between exec and websearch without
joining the frozen sequence, so the freeze guarded 120 of 121 apps. Frozen at
the position it already mounts at, which is what the test compares.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Transactions, Credits, Team and Settings on billing.hanzo.ai were empty
because GET /v1/billing/{transactions,credit-balance,accounts} and
accounts/:id/members answered 404. They read as a stale deploy and were
not one: commerce declares all four on its api.Route() `user` group
(api/billing/handlers.go), but the co-resident embed registers on the
HOST's router and never compiles that table, so a commerce route reaches
production only if Mount names it. The handlers shipped in the pinned
module all along — v1.50.11 has every one of the four symbols — while no
binary in the fleet served them, which is why grepping the library found
them wired and the tabs stayed blank.
Chain is GetTier's, and each link earns its place on a money read:
IAMTokenRequired resolves the org but FALLS THROUGH when there is no
validated principal, so PinBillingSubject is both the gate and the IDOR
control — it fail-closes that fall-through as 401 (a browser
re-authenticates on 401 and only reports 403) and overwrites every
subject key {user,userId,customerId} with the caller's own account.Payer
subject. That is load-bearing, not decoration: ListTransactions filters
on ?user and GetCreditBalance on ?userId, both unpinned client values, so
an unmounted-but-naive mount would have returned every subject's rows in
the org namespace. Because the pin SETS the key rather than validating
it, the subject is exactly the one account.Payer debits — a read cannot
disagree with the wallet it describes. TokenRequired follows the pin so
the trusted S2S reader still resolves an org; IAMTokenRequired admits
only IAM principals and would leave GetOrganization nil, which panics.
The test asserts 401-not-404. Both are "no data" to a browser, but 404
means the gate never ran and 401 means it ran and refused — only the
second proves the route reached its middleware, and a library-side grep
cannot tell them apart.
The isolation boundary is the pod's RUNTIME — one field, SANDBOX_RUNTIME_CLASS,
holding gvisor or kata-fc or kata-clh or nothing. That is the whole of the
runtime decision: no fork in the code, no second implementation, and a swap is a
deployment change rather than a release. Empty means the node's default runtime,
which is honest rather than a hole — runsc has to be installed on the nodes
first, and installing it restarts containerd under 204 running pods, so that is
maintenance and not something a release does on its way past.
No daemon inside the pod. Commands go through the Kubernetes exec subresource,
so cloud talks to the apiserver rather than to a pod IP. That deletes three
problems the predecessor had to solve: a shared service key sitting in an
environment the submitted code could read out of /proc, a bind protocol to stop
a recycled pod IP serving another tenant's checkout, and an in-pod HTTP server
that had to be built into every image.
PROVEN LIVE, not asserted — go test ./apps/sandboxes -run TestLive against
hanzo-k8s, 27.7s, image node:22:
RUNS CODE: SANDBOX-RUNS-CODE v22.23.2
EDIT PERSISTS: export const answer = 42; // edited by the agent
EDITED CODE RUNS: ANSWER=42
FAILURE IS DATA: exit=3 stderr=to-stderr
The file is written through one exec and read back through a SEPARATE one, so
what is proven is a filesystem and not a buffer. The last line is the one an
agent depends on most: a command that exits 3 is a successful call carrying a
failed program, because a caller that cannot tell "your tests failed" from "the
sandbox is broken" will retry the wrong one forever.
The image is an env var, and that is why this could be proven before our own
image exists. A stock node:22 exercises the identical create/exec/edit path.
Named sandboxes, not machines: /v1/machines already exists and is visor's —
whole GPU and VPS instances, sized, quoted and billed. Two products cannot share
a noun. The namespace and pod labels follow the same word, which matters more
than it looks: universe's containment policy selects hanzo.ai/sandbox-class in
hanzo-sandboxes, and a policy naming a label the code does not set is the
enforce-nothing failure this fleet has shipped three times this week.
Four gates went red because two new apps arrived without the entries every app
owes, and because a change to what the document publishes was not carried into
the ledger that counts it. None is a behaviour change; each is the declaration
the gate exists to demand.
- spend.go: plugin/lsp declares Price: cloud.Metered, so lsp joins
meteredApps. Without the entry the surface is free the moment SpendGate
enforcement is switched on, which is precisely what
TestMeteredSurfacesRequireStanding refuses to let drift. apps/lsp/meter.go
already owns the debit — a cold checkout+index is billed, a warm query is
not.
- product_gate_test.go: apps/lsp and apps/sandbox mount no product and held
no pin. Neither has an upstream repo to mount, so both are `unextracted`,
the bucket that says the functionality lives only here and owes its
extraction at migration.
- typed_request_gate_test.go: apps/lsp/lsp.go takes cloud.Request for the
money gate in front of a query, which needs strictly more of the validated
principal than the org — the payer and the validated project sub-scope,
neither of which may become an In field without letting a caller name who
it bills. One call site, guarded by onHTTP so it fails closed off the HTTP
path.
- apps/bots/typed_wire_test.go: OPTIONS and TRACE on the relay wildcard left
plugin/bots/openapi.json in ceff43ac, which drew the line that the document
publishes what was DECLARED rather than whatever the router happened to
bind under All(). The ledger still named them, so it described a surface
that no longer exists and the partition summed to 10 against a served 8.
Five relay entries now, not seven.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/deploy/deploy.go:350 calls registerEngineRoutes, and the file that defined
it went away in 77f187f7 — a commit whose message is a commerce dependency bump
about crypto deposit addresses and which says nothing about apps/deploy. It took
engine_mount.go (172 lines) with it and renamed engine.go to source_git.go,
which reads like a dirty worktree committed alongside the bump rather than a
decision: the same file and the same call site are both still present on the
GitHub lineage the author works from.
The tree has not compiled since. `go build ./apps/deploy/...` fails on an
undefined symbol, so go-unit reports "[build failed]" for apps/deploy and
plugin/deploy, the gate cannot pass, and no run can reach the image car — which
is why main has published nothing while commits kept landing on top of it.
Restored verbatim from 77f187f7^. The engine stays gated by
DEPLOY_ENGINE_ENABLED (default off), so this brings back the wiring and no
behaviour.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
github.com/hanzoai/ai v1.832.30 -> v1.832.31, which removes the second debit
on the casibase chat answer path. There, recordCasibaseChatUsage and
AddTransactionForMessage both reached the usage recorder cloud installs, so
one completion charged the customer twice.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GPU is metered like any other resource through the machine launch, whose
balance gate and per-hour meter both live upstream in Visor. Two references to
the bespoke prepay charge outlived its deletion.
apps/billing/balance.go named gpu_charge.go as a second consumer comparing this
balance against a GPU's price. The point the paragraph makes — that `available`
is spent against rather than displayed — rests on its remaining example, the ai
balance gate that reads the field over the S2S HTTP path, so the clause goes and
the sentence keeps its subject. Comment-only: the FloorMinor rounding it
documents, and every caller that compares rather than debits, are untouched.
.hanzo/workflows/cicd.yml still spelled the drift incident with the two concrete
addresses that the same deletion generalized in cmd/reach/main.go and
openapi/fleet.go. It now describes that incident the way its siblings do — one
build serving a renamed route under its new name while still publishing the old
one — so the release train's own preamble stops naming routes the fleet no
longer serves.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GPU is metered like any other resource through the machine launch, whose
balance gate and per-hour meter both live upstream in Visor. Two references to
the bespoke prepay charge outlived its deletion.
apps/billing/balance.go named gpu_charge.go as a second consumer comparing this
balance against a GPU's price. The point the paragraph makes — that `available`
is spent against rather than displayed — rests on its remaining example, the ai
balance gate that reads the field over the S2S HTTP path, so the clause goes and
the sentence keeps its subject. Comment-only: the FloorMinor rounding it
documents, and every caller that compares rather than debits, are untouched.
.hanzo/workflows/cicd.yml still spelled the drift incident with the two concrete
addresses that the same deletion generalized in cmd/reach/main.go and
openapi/fleet.go. It now describes that incident the way its siblings do — one
build serving a renamed route under its new name while still publishing the old
one — so the release train's own preamble stops naming routes the fleet no
longer serves.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The sink commits one row per message, and insertSettings asks the store to
batch it (async_insert) while wait_for_async_insert holds the call open until
the block is durable. With a single puller those two deadlock: the one
in-flight row IS the whole batch, so every insert pays a full materialized-view
cascade to land a single fact, and the next message cannot start until it
finishes. Throughput collapses to 1/insert-latency — measured at ~24 rows/min
against a ~2.5s cascade — no matter how much is queued behind it.
The warehouse was ~3.6 hours behind live traffic and losing ground, so every
dashboard, heat map and rollup read empty: they filter on `time`, and `time`
had not advanced since the backlog formed. Nothing was lost — the door commits
to JetStream before it answers, and the stream holds 72h — but the window is
finite.
A JetStream pull consumer load-balances across every client bound to it, so
binding the durable N times is concurrency without a second cursor. Each
message is still delivered once and still acked only after ITS OWN row lands;
redelivery, MaxDeliver and the abandoned-fact advisory are untouched. That
gives async_insert real blocks to coalesce and lets the store overlap the
cascades, so the ceiling becomes the store's rather than the round-trip's.
Also drops insights.raw_sessions_v3_mv in the warehouse (done out-of-band):
854ms and 9.2MiB of every insert spent projecting a table with no readers.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nothing credits a crypto deposit. The intent never leaves Pending, no watcher
observes the deposit address, and GenerateAddress discards the wallet_id, so
money sent to a minted address is received and not credited — and not readily
recoverable either. The rail now refuses before any keygen.
Card, bank transfer and wire are unaffected.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two things, one seam.
boxd assembled its own request tier out of net/http: a ServeMux, an
http.Server, a hand-rolled guard wrapping a Handler, and three helpers
(writeJSON, writeErr, and a bind that re-applied a body ceiling per call) that
zip has had all along as c.JSON, zip.Errorf and c.Bind. It is a standalone
binary but it is not a different kind of program, and there is one web framework
here. It is zip now: one middleware seam, one error shape, one place a route is
declared. The METHOD moved onto the route where it belongs — three handlers
opened with the same six-line method switch precisely because a mux matches a
path and leaves the verb to whoever wrote the body, which is how PathFsDelete
came to share a handler with the fs root.
BodyLimit is stated rather than defaulted: zip's default is 4 MiB and a box
takes source, so leaving it implicit would have cut the ceiling 8x while every
test that writes a small file went on passing.
Two things the conversion had to get right rather than translate:
- A streamed body is written from inside c.SendStreamWriter, not to a captured
ResponseWriter. agentRun's terminal-frame contract is unchanged — every
return still ends the stream where it always did.
- A file handed to c.SendStream is read AFTER the handler returns, so it must
not be closed on the way out. It was, and the LibreChat artifact test caught
it as "file already closed" on a body nobody had written yet. Ownership
transfers to the stream; the one path that never reaches the stream closes
it itself.
zip has no multipart accessor, so the upload path parses the body zip already
read with the stdlib rather than reaching through c.Fiber() and putting the
framework we are hiding back into a handler. That gap is worth closing in zip —
this is the second consumer that needs it.
SECOND: the box now learns its identity, which is what makes X-Box-Id mean
anything. The header was being sent and compared against BOX_ID — an env var,
read at startup, set nowhere and unsettable in principle: boxes come from a WARM
POOL, the pods are already running when a tenant appears, and a claim is a label
patch on a live pod. There is no moment at which env could carry a per-box id,
so the guard compared against "" and skipped, forever.
Cloud binds the box at claim instead, over POST /v1/box/bind, before the address
is handed to anyone. A failed bind fails the claim and releases the pod rather
than returning an address that would 409 at its own owner. Binding twice to the
same id is fine (a retry must be able to finish); binding to a different one is
refused, because release() deletes a pod rather than recycling it and there is
no legitimate second tenant.
An UNBOUND box now refuses a named call too — that is the recycled-address case
exactly: a pod that died without release() leaves a row pointing at an IP the
CNI has reassigned, and the replacement is warm and bound to nobody, so serving
it would hand a stranger's request to a fresh checkout.
The two tenancy tests count box-bound requests, and the claim now legitimately
makes one. They assert a DELTA rather than an absolute, and the baseline is
itself asserted to be exactly 1 — so the setup cannot quietly absorb a leak.
Two halves of the same seam, entangled in the same files.
THE DOOR THE AGENT USES. An agent's tools now come from the fleet's own MCP
door, published on the plane socket every child already dials
(mcp.Serve(door, manifest.MCPPath) in serveWake). Not a second gather: refuse()
is applied inside gather(), where the routing table is written, so any second
projection would be a second policy site and the first time someone forgot it
the agent would see CreateServiceAccountKey while Slack did not. An agent is now
just another MCP client of the one door and cannot see a wider surface than an
external client -- there is no wider surface.
ONE TOOL PER SUBSYSTEM. tools/list published 1,189 flat tools in 977 KB --
roughly 244k tokens to merely enumerate. Clients truncate; Slack keeps 128. So
1,061 operations were unreachable no matter how well ordered. rank() fixed the
order and could not fix a hard cap.
Now: hanzo_<app> with inputSchema {op: enum[...names...], input: object}, plus
hanzo_describe returning one operation's own descriptor bytes. Names in the
enum, schemas on demand -- the schemas were the 977 KB. The grouping key is
gather()'s existing owner map, so there is no second lookup and no second
source. The envelope is a DECODING, not a route: call() unwraps {op,input} into
the (name, message) a direct call carries and takes the same hop with the same
headers, so a flat tools/call by op name still works byte-for-byte and Slack's
128 cached names keep functioning.
MEASURED against the fleet's own corpus (plugin/*/openapi.json, replayed through
real zip children over real sockets), baseline a real curl of api.hanzo.ai:
BEFORE 1189 tools 977636 bytes 822 B/op
AFTER 116 tools 106847 bytes 47 B/op
10.2x fewer tools, 9.1x fewer bytes, for 1.9x MORE operations (17x per op)
Like-for-like the baseline's own 1,189 ops would cost ~56 KB, ~14k tokens
instead of ~244k.
hanzo_describe is published FIRST, not last: it is what makes every other tool
usable, and a client that truncates and drops it holds 116 enums it cannot read.
HEADROOM: 116 of 128, against a 119-app manifest. This fits today and will not
fit forever. When it stops, group by productStems bucket (17) -- do not add a
second projection. Recorded in LLM.md.
SECURITY. TestARefusedOpIsInvisibleUncallableAndUndescribable: the child
genuinely serves CreateServiceAccountKey (asserted against MCPTools(), so
refusing it proves something), and the name appears nowhere in the tools/list
bytes, hanzo_console{op:...} returns -32602, and hanzo_describe{op:...} returns
-32602. Both messages are checked to stay non-oracular. All three paths read the
one set gather() produces. fleet/surface.go has ZERO diff -- refuse() is still
the only gate, unchanged.
describe re-asks gather() on every request rather than caching: a remembered
descriptor could describe an op the rule has since refused.
30 of 117 subsystems answered 308 to the door's tools/list and were reported as
outages. They were not down. Three links, each verified:
1. zip skips installing the MCP door when an app has no typed ops
(zip mcp.go:99). The deciding term is len(a.Registry())==0 on the SERVED app.
kms/billing/tasks/platform/index look like they have ops but register them on
cloud.Plane() -- a DIFFERENT *zip.App -- so their served app has none.
Census: all 30 dark apps have 0 typed ops on the served app; all 87 working
ones have >=1.
2. webui.Mount runs in every plugin process, not just the host (serve.go:413),
and registers the console catch-all.
3. So POST /mcp fell through to webui/mcp.go's signpost, which 308s to
manifest.MCPPath -- an address that is a 404 inside a child.
The signpost's own comment stated the assumption that made it safe: "a plugin
that serves its own door at FrameworkMCPPath matches a real route and never
reaches here." Link 1 made that false.
Following the redirect would mask it -- the hop lands on 404 in the child. It is
the front door's topology leaking into a process where it is untrue.
Fix is one line in cloud.App(), the constructor every Hanzo program reaches:
MCP.Source is now always non-nil (the declared Plugin.Door, else an empty
per-caller half), so hasCaller() holds and zip installs its own door. Still
exactly one MCP implementation, and zip returns the pre-rendered bytes verbatim
when the per-caller half is empty, so the memcpy stays.
MEASURED, and this is the honest part: 29 of 30 recover their door (kafka needs
a live broker to mount) and they contribute ZERO new tools. Every one has 0
typed ops -- which is why they were dark. This turns 30 phantom outages into 30
honest empty doors and makes hanzo.ai/unavailable mean what it says. It does not
conjure tools that were never registered.
The tool famine is a SEPARATE defect: MCP tools come only from typed ops, and
/v1/chat/completions and /v1/embeddings are raw routes. apps/ai declares exactly
one typed op, which is exactly the "one tool matching chat" the live door shows.
fleet/childdoor_test.go reproduces the 308 against a real apps/exec child and
asserts bytes, not status. Its sibling proves a genuinely-down app is still
reported, so this cannot be "fixed" by making every child look alive.
Every @hanzo turn failed in production with "authorize: no org on the call",
and the previous fix did not work because it stated the tenant in a place zip
deliberately ignores.
commerce takes the org from the CALLER's identity and never from an argument
(balance_rpc.go:36), so no caller can name the books it charges. The org has to
ride the caller. But zip reads a STATED caller only where there is NO request
behind the context (caller.go:352-356) — otherwise CallerOf reads the request's
own headers. That rule exists so that stating an identity can never override an
authenticated one.
The prior attempt called cloud.For inside the agents op, which is reached OVER
THE PLANE — a real request. The statement was silently discarded and the gate
still refused. It read exactly like the working background callers elsewhere in
the tree; the difference is invisible without the precedence rule.
So the org is now stated by the DISPATCHER, on a detached context, before the
hop: bridgeRunContext(org) = WithTimeout(cloud.For(context.Background(), org)).
Caller.headers renders it onto the wire (caller.go:302) and it rides onward for
free. This matches the existing convention in this package —
TestTheDetachedImportStatesItsTenant covers the git-import path the same way.
Detaching was independently required: the turn runs in bridgeSpawn's goroutine
after the webhook has already answered Slack 200, so on the request context the
model call was being cancelled the instant we replied.
bridgeReply loses its ctx parameter. An unused context argument here is an
invitation to pass the webhook's, which is precisely the bug; removing it makes
that unavailable rather than merely discouraged. The no-op in the agents op is
replaced by a comment stating why it must NOT be done there.
Also: the install URL asked Slack for 9 of the 13 bot scopes the app manifest
declares. Slack grants exactly what the consent URL requests, so anyone
installing through it got a token with no `commands` — /hanzo would fail at
first use, long after the install looked successful. Adds channels:read,
commands, team:read.
go build ./... clean. apps/integrations passes whole. apps/agents shows the
same 7 pre-existing metering-plane failures as baseline, no new ones.
@hanzo got past "agent not found" and then failed with
bridge: agent run err="authorize: no org on the call"
A run BILLS. The balance gate is a plane call to commerce, and commerce takes
the org from the CALLER's identity, never from an argument -- deliberately, so
no caller can name the books it charges (apps/commerce/balance_rpc.go:36, and
meter_rpc.go says it outright: "The org is the CALLER'S and can never be named
in the input"). A turn dispatched over the plane has no inbound request to carry
that identity, so the gate refused and every Slack message died after the agent
had already resolved.
Fixed with cloud.For, which is the function that exists for exactly this: it
states the tenant a BACKGROUND call acts for -- one with no request to forward
-- and it CANNOT launder an identity, because zip prefers a gateway assertion
over it whenever a request exists (plane/ask.go:236-242).
The org is trustworthy for the same reason every other Slack path trusts it:
the bridge resolved it from the Slack-verified team_id through the install-to-org
map, never from a payload field.
Tests: a plane run acts for a named tenant, and an empty org states nothing
rather than becoming a blank tenant that would bill an account named "".
Measured on a 23-minute image build, the four steps that cost it:
#27 422s the plugin loop
#24 396s go generate -run zipdoc ./...
#41 244s exporting cache to registry (mode=max)
#39 210s exporting + pushing layers
#27 was ~60 `go build` invocations run one at a time by a shell `for` loop, on
an 8-core runner. Each is a separate process, so Go's own intra-package
parallelism does nothing for the set: seven of eight cores idled for seven
minutes. They are independent binaries with no ordering between them, which is
the definition of embarrassingly parallel.
Now `xargs -P $(nproc)`. The existence check keeps its own pass so a missing
plugin/<name> still fails with the message that names the fix, before any
compile starts and while the output is still readable.
A failing build exits 255, which is xargs' stop-everything code — the whole
step fails fast rather than compiling 59 more plugins to report one error at
the end. It also prints WHICH plugin failed, because parallel output is
interleaved and the go error alone no longer says.
GO_LDFLAGS survives the subshell: it is ENV (Dockerfile:259), so it is in the
environment xargs' sh inherits, not a shell variable that would silently vanish
and leave every binary stamped "unknown". The existing strings|grep check on
the artifact still proves that on the bytes rather than on the flag string.
apps/integrations carried two openapi.Describe calls for GET
/v1/integrations/slack/install, written by different hands into different init()
funcs. Describe panics on a duplicate — correctly, because two descriptions of
one operation means one of them renders and nobody can tell which — and that
panic fires at init, so it took down every app that links integrations: ads,
automations, campaign, catalogsync, channels, cloudflare, company, content,
destinations, git, guide, integrations and sync all failed to describe.
The earlier one survives. It was already the superset: it has the attribution
constraint (Slack refuses a slack.com URL in that field, so the click has to
route through an address of ours to be counted) AND the tenant point (public, no
principal, binds no org, because minting an org for an anonymous click is the one
thing that would break isolation). The later one had a single fact the first did
not — 503 where the app is unconfigured, rather than a consent URL with an empty
client_id that Slack renders as its own dead end — so that sentence moved across
before the duplicate went.
The floor drops for the merge's own deletion too: /v1/billing/gpu/charge and
/v1/billing/gpu/eligibility are gone because GPU is metered like any other
resource now, and the bespoke prepay path with it. Checked rather than assumed —
a -1 that is not a multiple of two is not a TRACE/OPTIONS removal, and an
unexplained shrink is exactly what the ratchet is there to make someone look at.
1762 paths, 2480 operations, 185 products. Every one carries an operationId and a
summary; 51 still want a long description and 49 of those are hanzoai/ai, whose
prose belongs on its controllers, in that repo.
Two ways the same confusion showed up, and one build that could not run.
TRACE and OPTIONS were published — 33 each — because All() binds every method
at a path and the describe loops read that route table as a product surface.
CONNECT and HEAD were already excluded on exactly that principle. TRACE echoes
the request back; it is the Cross-Site Tracing verb and should not be reachable
on a public API at all, so advertising it in the contract is worse than merely
routing it, because the contract is what the SDKs, the CLI and the MCP tool list
are built from. OPTIONS is CORS preflight: a browser sends it, a person never
does, and `hanzo meet options` is not a command anyone wants. Between them, 66
operations — and the generated CLI already carried none of them, so this is the
document agreeing with the projection that was right first.
websearch held its own copy of that method list. It said seven, including the
two, and the day the document stopped publishing them the copy went on
describing operations that no longer existed — which openapi.Methods() exists to
prevent and which the describe gate caught by name. It reads the projection's own
set now.
The floor drops by exactly 66, in this commit, next to the reason.
Separately: every app Makefile claims "Generated by plugin/gen-app-cmds" and none
of them were. The generator wrote plugin/<app>/main.go and left a human to
remember the second file, so `sandbox` arrived with a main and no Makefile, so
`make describe` had no rule for it, so it published no subset, so `make openapi`
failed on an app that was otherwise finished. The generator writes both now, from
the same manifest row, taking the package from each app's own composition root
rather than a lookup table. It reproduced all 117 existing Makefiles with zero
APPS values changed, which is the evidence the rule is the one already in use.
External modules fall out structurally — no package directory here, no Makefile —
so mk/fleet.mk's EXTERNAL list is not copied.
The Home tab rendered Slack's own "this is still a work in progress"
placeholder -- what Slack shows for any app that enables home_tab_enabled and
never publishes a view. The choice was never Home vs no Home; it was our page
vs Slack's apology.
The page now carries what a person opening it actually needs:
Model enso (auto) / enso-flash / enso-ultra, THEIR choice pre-selected
Mode chat only, or chat + code
Connected who the turns run as, and the org they bill
Started DM, @mention, /hanzo, and a coding run
Both controls write the SAME userLink the chat path reads -- no second settings
store, no third source of truth for which model answers. The turn then carries
that model per-request (plane.RunOnBehalfIn.Model), because it is a preference
of the PERSON asking, not a property of the agent: two people in one workspace
can prefer different models of the same assistant.
Interactivity arrives at the SAME signed URL as events, form-encoded rather
than JSON, so it is sniffed and handled before routing -- that is where the two
encodings actually diverge, and no header separates them.
Three refusals, all deliberate:
- an UNLINKED user is shown how to connect and given NO controls; a model
chosen by an identity we cannot resolve is a control that does nothing
- a value not on our own menu is DROPPED, not stored: the options came from
us, so anything else is a stale client or a forged payload choosing what
this org pays for
- the per-turn model overrides only the BUILT-IN agent; a person's Slack
preference must not silently re-point an agent their org configured
Isolation is unchanged: the org comes only from the install-to-org map for the
Slack-verified team_id. The payload names which setting and which user, never
the tenant.
11 tests green. apps/integrations and plane suites pass; go build ./... clean.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/lsp — one door, because there is one value: a language server
rooted at a workspace, asked about a position. code (static index) and
lsp (live server) are two reads of the ONE checkout.
{repo, rev?, path, line, character, method} → hover | definition |
references | typeDefinition | implementation | documentSymbol |
completion | diagnostics
Positions are the LSP's and pass through untouched: 0-based line, 0-based
UTF-16 character. Re-basing them would corrupt every multi-byte line, and
the callers already speak LSP.
Isolation is structural, not checked. The org comes from the validated
principal and is BOTH the pool key and the OWNER SEGMENT of the git URL;
a caller supplies only a repo slug. There is no input from which one
tenant can name another tenant's repository.
Scripts-off by default. A dependency fetch that runs dependency-authored
code is RCE triggered by whatever the caller asked us to check out, and it
buys nothing — servers resolve from source. So: npm ci --ignore-scripts,
cargo fetch (not build), go mod download; the python fetch builds sdists
and therefore does not run. rust-analyzer is additionally told not to run
build.rs or expand proc macros, because otherwise the server does at load
time exactly what the fetch was chosen to avoid. One predicate, one place.
The deployed worker must still be sandboxed — see fetchable's comment.
Language table ported verbatim from hanzo-tools-lsp (same binaries, same
argv, same root markers); install_cmd deliberately dropped — a worker that
can npm install -g at request time is one an attacker can make write to
its own filesystem.
server.go is the testable core: Content-Length JSON-RPC with a SINGLE
reader goroutine demultiplexing responses, server→client requests and
notifications. The python tool reads inline from each call, which drops
every frame that is not the awaited response — which is why it cannot
report diagnostics, and why an unanswered client/registerCapability
deadlocks it. 24 tests drive a fake server over in-process pipes; no
toolchain, no network.
Two bugs the tests found, both real in production:
- Close() wrote a polite shutdown unconditionally, so a server that had
stopped reading its stdin blocked it forever — holding a pool slot and,
at Shutdown, the whole binary.
- rel() compared a symlink-resolved root against an unresolved target, so
any data dir with a symlink component (/var on a Mac, a mounted volume)
made every location "outside" the checkout and handed the caller the
worker's absolute path instead of a repo-relative one.
Metered, not Free: cost is the COLD start (checkout + fetch + first index),
not the query. Warm point queries are recorded and free, so the pricing
does not teach callers to re-key their workspace. Gate before the work.
apps/code has NO checkout to reuse — it indexes files POSTed to it and says
so. apps/deploy has the only working-tree checkout and is not importable
(it would link k8s into this binary). This is therefore a second one,
following deploy's invocation and hardened env exactly; the fix is hoisting
it into the root cloud package, not made here.
Mount order: apps.Wire() and its integers are gone. manifest.Apps slice
position IS the order, so lsp's row sits after code's, with order_test's
frozen sequence updated in the same commit.
apps/commerce would not run a single test. openapi.Describe PANICS on a
duplicate (openapi/register.go) and this package registers from an init(), so
eight re-described billing routes took the whole package down before any test
started:
wire, crypto/options, crypto/deposit, crypto/deposit/:id,
methods (GET and POST), methods/:id, portal/methods
Each already had a description at lines ~222-315; a later block at 444-552
described them again. Removing that block leaves 91 unique Describes and no
duplicates. The unique webhooks/:provider entry that followed it is untouched.
WHY THIS MATTERED MORE THAN A PANIC. Tests that fail to RUN are worse than tests
that fail, and this hid several: with the package dead at init, the billing
subject-pin guard added minutes earlier could never go red — a guard whose whole
job is to stay red until a route pins its subject. Removing the duplicates makes
it execute (14 subject-taking routes inspected).
It also uncovered six pre-existing failures the panic had been masking. All six
are the same environmental class, not defects: 42 occurrences of "no RAM-backed
scratch for the pure-Go SQLCipher codec (need tmpfs at /dev/shm or
HANZO_SQLITE_RAMFS_DIR)". macOS has neither, so the encrypted per-org store
refuses to open rather than decrypt to persistent storage. They run in CI.
The surviving descriptions are the earlier ones. The removed set was in places
more detailed — if that wording is wanted, replace the earlier text rather than
adding a second registration.
Two call sites had picked the wrong one of the three Minor variants, and both
failed silently in a way that looked like something else.
BACKFILL (blocked the finance cutover). apps/admin/finance/backfill.go read a
commerce balance with Minor(), which REFUSES anything finer than a cent. The
ledger keeps eighteen decimals and per-token charges are routinely finer than a
cent, so every org that has spent anything carries a sub-cent tail — and the
backfill returned "read commerce balance: amount … is finer than its minor unit"
and migrated ZERO. Fail-safe, and completely stuck. It now FLOORS, which is the
direction a migration must take: the amount moved must never exceed the amount
held, or the migration mints the difference. The dust stays behind and settles.
MONEY BOARD (display). apps/admin/moneyboard.go read the treasury reserve the
same way, so the same tail rendered as SrcOf("treasury", err) — "could not reach
the treasury" — for a treasury that was reachable and correct. It now ROUNDS.
Nothing is spent from that figure, and refusing to show it is worse than a half
cent.
The rule the two sites got wrong is now pinned in plane/money_test.go against the
real production balance, which distinguishes all three:
149913.078983985999994361 -> Minor() ERR Floor 14991307 Round 14991308
Minor() a DEBIT exact or refuse — rounding money you TAKE is theft
one way and a gift the other
FloorMinor() a COMPARISON never overstate — admitting spend a balance cannot
or MIGRATION cover leaves a negative balance nobody authorised
RoundMinor() a DISPLAY nearest — nothing is spent from it
RoundMinor also gains the coverage it never had: nearest in both signs, and it
still errors on overflow and on a malformed amount rather than wrapping or
rounding to zero.
Found by an adversarial review of the money path (its M1/M2), which asked the
question I had not: where should I have floored and did not.
The built-in default agent I added minutes ago took cloud.FallbackModel, which
is "best". That constant's own doc says what it is for:
"keeps a bot's reply landing when the flash tier is saturated; the
interactive chat path never uses it"
A Slack turn IS the interactive chat path, so I had wired the DEGRADED tier as
the default brain — I reached for the nearest available model constant instead
of asking what the chat model should be.
It is now enso, Hanzo's own auto-routing SKU that selects per query in the
gateway's catalog, with BRIDGE_AGENT_MODEL to override per deployment. Studio
names the same default for the same reason (STUDIO_CHAT_MODEL or "enso").
Forwarded verbatim and never validated here: the catalog resolves it, and a
check in cloud could only disagree with the thing that decides.
Tests pin both halves — the default is enso and is never "best", and the
override wins.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The forge already carried the front-door body-limit fix (internal/edge). These
seven were only on GitHub, so the next pipeline release would have shipped
without them:
84e26665 commerce: the 128 silent operations say what they do
702ecbf7 preflight admits the four headers the console actually sends
bd4115d3 o11y: the product scope is the fleet, sub-cent debit is not an outage
8f1f50e0 CORS: allowed origins derive from verified site hosts
3bfdeee0 slack: the agent turn crosses the plugin boundary over ZAP/UDS
c78fbe72 a console the customer owns can call the API it was forked from
Resolutions:
- apps/billing/usage_coresident.go — FloorMinor, not RoundMinor. Both fix the
same 502, but balance.go:126 already floors a row from the same ledger, and
two roundings on one ledger is how the next discrepancy starts.
- apps/framework — RoleAssignment/roleRef. Without the rename the weave refuses
the whole document: iam publishes a Role ENTITY and framework published a
(user, role) GRANT under the same name, and a generated SDK binds whichever it
read last. revokeRole still named the pre-rename type; it takes *roleRef now,
which is the URL-path form it actually addresses.
- manifest/apps.go — UNION, not a side. github adds /v1/event.js (the hosted
analytics tag, previously falling to ai's bare /v1 remainder, which a browser
reads as a broken script tag); the forge has /v1/replay, which github lacks.
Taking either side alone drops a live route. The superset check I wrote for
the last merge is what caught it — it asserted, and it fired.
- Generated artifacts (openapi.yaml, plugin/*/openapi.json, */zipdoc_gen.go) are
taken from github, not regenerated here: `make describe` on this tree still
fails for 14 apps, and github's are the set that was verified green at 1735
paths. A half-regenerated document is worse than a whole borrowed one.
go build ./... exits 0. TestTheServedDocumentIsTheArtifact and the door
body-limit tests both pass.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
I introduced this tonight. To make the S2S tier reader resolve an org I mounted
the route RequestContext -> IAMTokenRequired -> TokenRequired and stopped there.
TokenRequired AUTHENTICATES and does not PIN, which is the exact hazard the
comment on /v1/billing/methods already spells out: any authenticated browser can
then name another subject.
GetTier reads ?user= verbatim and answers with that subject's wallet —
prepaidAvailable, creditsRemaining, effectiveAvailable. Every one of the other
subject-taking billing routes carries accountclient.PinBillingSubject(); this one
did not.
It is CROSS-CUSTOMER, not merely cross-subject. Every self-serve signup lands in
the SAME org (account.SignupOrg = "hanzo") with a per-person subject, so the org
namespace was closed while the subject was open. Measured live against the running
pod before the fix — one caller, four wallets:
?user=hanzo/z prepaidAvailable 10966
?user=hanzo prepaidAvailable 6375
?user=hanzo/admin prepaidAvailable 10000
?user=hanzo/dev prepaidAvailable 0
The pin does not undo the fix it sits on top of: PinBillingSubject admits a
verified service-token caller that names its own org and leaves its query
UNTOUCHED (apps/account/billing_coresident.go), which is exactly what ai's rate
limiter and apps/metering send. TokenRequired stays after it so that caller still
resolves an org.
The guard is a CLASS guard because a one-route fix would not have found this one.
It walks the AST of the mount table and fails any /v1/billing route that takes a
caller-named subject without a pin.
Two things the guard had to survive to be worth committing:
- The first version used a regexp and SILENTLY SKIPPED the route it was written
for. The non-greedy chain match ended at the first `\n\t)` and the scan resumed
past several registrations, so it reported "14 routes checked", passed, and
would have passed with the leak present. It now asserts it can SEE
/v1/billing/tier, so a walk that stops reaching the table fails loudly instead
of reading as green.
- The alerts group is allowlisted only after checking it: billingSubject derives
from the RESOLVED ORG (org.Name, spend_alerts.go:62) and every read is
org.Namespaced, so there is no caller-named subject to pin. An unverified entry
in that list turns this guard into a rubber stamp for the leak it exists to catch.
Verified both ways: removing the pin fails the guard naming mount.go:603.
Found by an adversarial review of the money path, which ranked it the one thing
worth fixing before launch.
Closes the last two surfaces where a client could name the usage idempotency ref
(the ai answer surface and the domain register/renew/transfer refs) so Usage.Seal()
mints them, and backfills the program column on pre-existing finance.usage rows so a
ref straddling the deploy debits once. R7 asserts no plane op carries metering.Usage;
R8 routes the split-deploy debit over the plane, not the tag-stripped HTTP wire.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The span-signal slice of the analytics fan-out gets the sibling the error slice
already has. AddSpanSink mirrors AddErrorSink and filters on routeOf -- the
plane's ONE routing rule -- so the lens can never carry a fact the plane filed as
something else: an event with a span BODY that routes as an act stays an act.
apps/o11y writes the row, and it writes a ROW rather than calling a module
because that is what the read side is. llmobs.Module has no ingest surface; its
observations/traces/sessions/users are querier reads of event.span filtered by
`gen_ai.system EXISTS AND gen_ai.hanzo.org_id = <caller's org>`
(impllmobs.genAIFilter). So the only way a span becomes an observation is to BE
such a row, and the rows go through insertSpans, which owes event.span and the
event.trace partial the trace list resolves against -- a span written without its
partial is a span every trace read answers empty over.
Only gen_ai spans project. The marker is the reader's own, so the lens admits
exactly the spans the LLM views return; an ordinary span would be write
amplification no read can answer with, and a second copy of a span the ZAP door
already owns. An attribute that renders empty is dropped rather than stored
blank, because a key stored empty satisfies EXISTS while naming no provider.
The tenant is the SLUG, stamped last and unconditionally. gen_ai.hanzo.org_id is
the only org discriminator the span views have, and the handler binds it from the
validated X-Org-Id -- so the o11y org UUID the Sentry lens derives would be a row
every LLM view returns zero of. A wire-supplied org is overwritten, never read.
The span slice carries the scrubbed property copy the fact row stores, which is
where this seam differs from the destinations slice: that consumer FORWARDS and
must hash raw match keys, this one STORES, and scrubText's contract is that a
token in a property is redacted before storage. A projection must not store more
than the plane stores.
CLOUD_LLM_LENS turns it off; default on, the CLOUD_SENTRY_LENS posture. It
installs from mountPlaneIngest rather than mountRuntime, which is the same rule
against a different dependency -- a lens installs where the thing it writes
through becomes available, and mountPlaneIngest sets the plane sink after
mountRuntime has already run. ShutdownO11y detaches it first, beside the error
lens, so no in-flight ingest dispatches into a closing connection.
Mutation: widen the filter to admit any event carrying a span body and
TestFanOutSpansIsThePlanesRoute goes red with
want the 2 span-routed events, got 3: [... {Name:click Kind:track SpanID:x}]
-- the act the plane filed as an act, arriving on the span projection under its
own name and kind. Restored, green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A buyer who picked SOL reached the confirm screen and met 'the custody service
is not accepting new addresses'. It could never have worked: a Solana address is
an Ed25519 key and the fleet runs only the secp256k1 ceremony, so mpcd never
emits sol_address. The asset picker is rendered from the processor's currency
list, so listing it is what put the button there.
Every other chain mints — ethereum, bitcoin, polygon and lux all verified
returning addresses through our own MPC.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
THREE things, all measured against production tonight.
1. AGENT NOT FOUND. With the plugin-boundary fix deployed the turn finally
reached agents, which answered `agents: agent not found`. The bridges ask
for the conventional ref ("hanzo") and Store.Resolve is a plain row lookup
with NO seeding anywhere — so an org that connected Slack and did nothing
else had no agent, and @hanzo could never work out of the box in ANY
workspace. The conventional ref now resolves to a BUILT-IN default: not
persisted (a row would fork the definition per org and strand already-seeded
orgs on a later change), and a row the org DOES create still wins because
Resolve is tried first. An unknown ref stays a miss — silently substituting
the chat agent would make a typo in `code: repo` run the wrong thing and
look like it worked.
2. TOOL CALLING. executeRun did one completion and returned the text; Agent
.Tools was stored, updated and displayed but never read by a run, so the
agent could not reach anything. It now runs a bounded tool loop and takes
the actor, so every tool call is attributable to the org and user that
caused it.
3. TOOL SURFACE. fleet/mcp.go + fleet/surface.go curate and gate what a client
may see: the live server projected 1,323 internal ops with zero annotations,
and Slack saved the first 128 ALPHABETICALLY — a window containing zero
product tools and 36 credential/auth ops including CreateServiceAccountKey.
Cross-agent collisions found and fixed while integrating: tools.go's truncate
collided with an existing test helper (renamed truncateToolResult, which says
what it bounds), and executeRun gained an actor parameter that two test callers
had not been updated for.
go build ./... clean. fleet and plane suites green; apps/agents' one failure,
TestTargetOpsProjectEverywhere, is the known pre-existing op-id drift.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
413 lines of harness and tenancy tests referenced stateWarm, classDesktop and
pool.nodeTaint from an implementation that was replaced, so the package compiled
and its whole suite did not — zero coverage on the one component whose job is
keeping one tenant out of another's box.
The substitutions are both outside the implementation: client-go's fake dynamic
client for the pool, and an http.RoundTripper swapped into the package-level
boxClient for the box. forward() is therefore exercised byte for byte — headers,
query handling and all — against the production route table rather than a
paraphrase of it. Route tables are shared vars, so a route that moves breaks the
tests instead of leaving them testing a path nobody serves.
Six tests are skipped rather than deleted, each naming the mechanism it needs.
That is the point of the port: a skip that names a missing mechanism is a
finding, and deleting it is how the mechanism stays missing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/exec pointed at code-exec.hanzo.svc while apps/sandbox schedules boxes into
hanzo-boxes, so the two halves of one executor sat in different namespaces under
different containment — and the half in `hanzo` was the half sitting beside the
datastores. They run the same binary on the same submitted code; only the
lifetime differs, which is not a reason for two blast radii.
Also corrects two comments that described the containment as absent. It was:
the policy selected `app: code-exec` in namespace `hanzo` and matched no pod.
universe now declares one policy over hanzo.ai/box-class in hanzo-boxes, so
saying so is accurate rather than aspirational — which is the property those
comments were written to protect in the first place.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three things were true at once and each hid the next. cmd/boxd was written and
untracked, so no release could contain it. Cloud's Dockerfile built /cloud and
/smoke and nothing else, so committing boxd would still not have put it in an
image. And apps/sandbox had a Mount with no plugin/sandbox beside it, so the
package compiled, linked into nothing, and answered 404 in production while the
code to answer sat in the tree.
That is why `COPY --from=ghcr.io/hanzoai/cloud:<pin> /boxd` in hanzo/bot's
Dockerfile.box was red by construction — grep -c boxd Dockerfile was 0 — and why
every consumer already pointing at the executor (hanzo.app's ProjectFs,
apps/exec's upstream, apps/functions' invoke) had nothing behind it. The
33-day-old code-exec Service with <none> endpoints is the same fact seen from
the cluster.
boxd builds HERE rather than in hanzo/bot because its types ARE
apps/sandbox/wire's types: the scheduler and the daemon agree because they
compile against one declaration, not because two repos were kept in sync by
hand. It is copied into the FINAL layer, not just the build stage — a binary
that exists only in the builder is not in the published image, and the box image
reads the published one.
boxd is also the half that runs untrusted code, so it refuses to start in the
configuration where that code can steal the pool key: submitted work running as
boxd's own uid can read CODE_EXEC_API_KEY out of /proc/<pid>/environ, and that
key is shared across the pool, so the credential a box hands its own workload is
the credential that opens every other tenant's box. Reproduced end to end. Two
cheaper defenses do not work and pgroup_unix.go records why measurements, not
reasoning: scrubbing the child's env misses that the child reads boxd's, and
os.Unsetenv misses that the kernel serves /proc/environ from the exec-time stack
block rather than the live environment.
.gitignore takes boxd in both spellings. It is the one binary people build from
inside its own directory, since it runs standalone on a laptop, and that file
already carries what committing binaries cost this repo once.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Closes the last two surfaces where a client could name the usage idempotency ref
(the ai answer surface and the domain register/renew/transfer refs) so Usage.Seal()
mints them, and backfills the program column on pre-existing finance.usage rows so a
ref straddling the deploy debits once. R7 asserts no plane op carries metering.Usage;
R8 routes the split-deploy debit over the plane, not the tag-stripped HTTP wire.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`make describe` refuses to regenerate openapi.yaml while any operation is bare,
so the artifact could not be rebuilt AT ALL and had drifted: on clean main the
host served 1698 paths against the golden's 1696 and
cmd/cloud.TestTheServedDocumentIsTheArtifact was red.
128 of the bare operations were commerce's, and commerce registers its routes
from an embedded module, so there is no doc comment here for zipdoc to lift —
openapi.Describe beside the route is the seam. Nine are written out: the wire
top-up rail, the crypto custody rail, the saved-card family, and the tenant's
payment-rail toggle. The other 119 are seventeen merchant kinds behind ONE
generic REST scaffold, so the mechanics are written once and composed with the
kind. Seventeen hand-copied paragraphs describing one generator is the drift
DescribeRest already exists to prevent one level down.
Every sentence is written from the handlers, and what earns space is what a
caller gets wrong:
- PUT is a true REPLACEMENT — the body is decoded onto a FRESH entity, so a
field the body omits is written back as its zero value;
- POST /<kind>/{id} with NO override is a PARTIAL UPDATE, never a create;
- a wallet read renders the account's ENCRYPTED key blob and its salt, so
whoever may read one can attack it offline down to the owner's passphrase —
which is the reason the kind is admin-gated;
- a webhook's delivery consults neither `enabled` nor `live`, so enabled=false
does not stop delivery; deleting the row is what does;
- a discount is enabled by DEFAULT, so a bare create makes a live discount;
- the per-kind permission table covers 5 of the 17 kinds. On the other 12 the
scaffold logs that it is skipping the check and ALLOWS, so the route gate is
the whole authorization story. Each kind now says which it is.
THREE defects were hiding BEHIND that refusal, because describe-apps stops at
the first app that fails and commerce sorts early. Each is repaired, not
recorded:
- integrations: GET /v1/integrations/slack/install was bare. The handler
already carries the prose; it is a raw route, so zipdoc cannot lift it.
- the weave refused two schema names that meant two things. `Role` was iam's
role ENTITY and framework's (user, role) GRANT — the grant is now
RoleAssignment, converted at the handler boundary with the engine type
untouched. `Application` was iam's OAuth client and crm's startup-program
submission — the latter is now ProgramApplication.
- manifest: /v1/event.js, the hosted analytics tag, is served by analytics and
was routed to ai's bare "/v1". A prefix owns SEGMENTS, so "/v1/event" never
covered it, and a browser reads that 404 as a broken script tag rather than
as a routing mistake. Claimed on the analytics row.
All three predate this change and are provable on clean main: regenerating the
stale plugin/iam/openapi.json alone makes the weave fail the same way.
The regenerated zipdoc_gen.go files are the same class of staleness — prose that
was in the Go source and had never reached the artifact.
make describe exits 0. openapi.yaml carries 1735 paths, up from 1696, served
byte-identical to the golden. go build ./... is clean. The pre-existing red
tests (apps/commerce TestBalanceCents/TestInProcessClient, apps/framework's
three, apps/projects TestForkCreatesProjectFromTemplate) fail identically before
and after.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestADisposalIsIdempotent failed the release gate on a 409 — "a materialisation
of this dataset is running" — for a materialisation that had already finished.
The two facts land at different instants. plane.run publishes the version with
p.record and gives the tenant's scan slot back from a DEFERRED p.release, so
between the status going terminal and the slot clearing describe answers
status: ready, running: true. settled returned on the status alone, inside that
window, and dispose refuses a held slot — so the next call 409'd on a job that
was over.
The CI log says exactly this: the FAIL is printed BEFORE the 'dataset
materialised' line for the same dataset, and that line is logged AFTER
p.record — so the version was published, the test read it, disposed, and was
refused by a slot whose owner had not yet returned.
It is microseconds wide, which is why it passes on an idle machine and failed on
a loaded runner. running is already on the wire for this (typed.go view), so
settled now waits on the state the refusal is made from instead of a proxy for
it. No production behaviour changes: a client polling describe was always told
running: true, and always had to wait for it.
30 consecutive runs green under CPU contention.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every fact the plane files under the error signal fans out, detached and
fail-soft, to an error sink apps/o11y installs when its embedded runtime is
up: analytics.ErrorEvent -> the Sentry wire -> o11y's normalize+fingerprint
-> Modules.Sentry.Ingest, landing in o11y_sentry_events + the o11y_issues
lifecycle under the org's canonical project (get-or-create, cached; the org
UUID is the read side's own UUIDv5 formula, pinned by TestDeriveOrgUUID).
So an error a product beacons to /v1/event surfaces on sentry.hanzo.ai
beside the errors a Sentry SDK posts to /v1/event/{project}/envelope.
Re-expressed on the trunk where it moved past the branch:
- AddErrorSink returns a remover, the same plural-sink discipline as
AddSink; o11y detaches it first in ShutdownO11y.
- The filter is routeOf — the plane's ONE routing rule — so the lens can
never carry a fact the plane filed as something else; the branch's
bespoke isErrorEvent detector goes away.
- The envelope's first-class release/environment/service/site/trace
qualifiers win over the legacy property spellings; service_name tags
the wire's service when it named one, else the emitting surface.
- The branch's pk_ mint-door flow test is superseded: IAM mints pk- at
POST /v1/keys and capture_keyorg_test covers the resolver seam.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
This reverts 2f9ffc2d. The diagnosis under it was wrong and the change bought
nothing, so it should not sit in the release lane implying otherwise.
What the evidence actually shows: runs 889, 890, 891 and 892 all report
started_at = 1970-01-01 — they never received a runner at all. They were
superseded while still QUEUED, not killed mid-gate, so cancel-in-progress was
never the thing acting on them and a literal false could not have saved them.
Superseding a queued run with a newer commit is moreover the RIGHT behaviour for
a release lane; the newest commit is the one worth shipping.
The real constraint is capacity against push rate: the ten runners of
hanzo-build-linux-amd64 serve five orgs and were 10/10 busy throughout, while
main took a push every few minutes. A run that waits longer for a runner than
the interval between pushes can never start. That is a fleet-sizing and
scheduling question, not a line in this file.
Reverted rather than reworded because the setting is only honest as the
expression that states the intent; a comment describing a mechanism I could not
reproduce is a worse landmine than the bug it claimed to fix.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
X-Actor-Id, X-Act-As-Project, X-Act-As-Org and X-CSRF-Token are stamped on every
signed-in console call (client.ts baseHeaders + applyCsrfToInit) and none of them
were in corsAllowHeaders. A header absent from that list fails PREFLIGHT: the
browser reports an opaque "TypeError: Failed to fetch" and the request never
reaches a server log, so there is nothing to find on our side. Measured from a
signed-in console.hanzo.ai page: adding any ONE of the four blocked the call,
the identical request without it returned 200.
Naming a header only lets the browser SEND it. Each stays exactly as trustworthy
as before — SanitizeIdentity still strips and re-mints client-supplied identity,
so an intent is validated, never believed.
Taken as a one-hunk cherry-pick rather than by merging PR #385: that branch is
443 files and 49,500 insertions off an ancient base, and merging it would drop
~17 commerce prefixes from manifest/apps.go and revert usage_coresident.go's
FloorMinor back to RoundMinor.
NOTE: this constant's own doc says it mirrors hanzoai/gateway routes.go
corsPreflightMiddleware so the browser contract is byte-identical through either
path. The gateway needs the same four or the mirror is broken.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
o11y probed visor.hanzo.svc:19000/health. visor serves its health as a typed op
at /v1/health, and everything outside /v1/ falls through to a static passthrough
that answers 200 with index.html. So the probe was green whatever the service was
doing -- it measured the file server, not visor.
This is the same bug found twice tonight in two repos. visor's OWN liveness and
readiness probes requested /api/health, also not a route, also served the SPA at
200, so a pod with a dead database stayed in the Service. One bug twice is a
pattern, and a pattern earns a gate rather than a second one-line fix.
The gate is narrow on purpose. It cannot know which path a given service serves
-- that is the service's business -- so it refuses only what is provably wrong:
a probe URL with no path, and a bare /health on a host known to answer under
/v1/. Anything it cannot prove wrong it permits, because a gate that guesses is a
gate someone disables.
Mutation: revert the probe to /health and it goes red with
visor: probes "/health" on visor.hanzo.svc, which serves under /v1/. Outside
/v1/ the request falls to the static passthrough and answers 200 with HTML, so
this probe is green whatever the service is doing.
Restored, green. The first run of that mutation reported "clean=1" and I nearly
read it as a pass -- the package was not compiling, because the test named the
target slice `probes` and it is `fleetTargets`. A package that fails to BUILD
reports zero test names, which is exactly why go vet answers "does this build"
and a test-name diff does not.
concurrency.cancel-in-progress was ${{ github.event_name == 'pull_request' }},
which on a push is supposed to be false. The forge does not evaluate it to a
boolean — it cancelled every push run regardless, so the setting written to keep
releases from being killed midway was itself killing them.
Measured on main this morning: runs 883-890 all CANCELLED, each by the next push
arriving seconds later, and 27 consecutive runs produced no release. The gate was
not merely red; even once green it could not have finished, because a repo pushed
more often than its gate takes to run never reaches the end of one. That is the
whole reason a feature merged yesterday is still 404 in production.
A literal false is read correctly and restores the behaviour the comment already
described: ONE release at a time, queued, never cancelled. The image is pushed
early and the tag and pin come last, so a killed run leaves an orphan image — the
ten between v1.801.335 and v1.801.350 are what that looks like.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The five branch commits join the trunk. Their event plane — the fact, the
stream, the writer, the /v1/event door — already lives here in evolved form
(one event.fact table discriminated by signal; the door table in event.go;
/v1/insights/e retired), so content conflicts resolve to the trunk
throughout, and the branch's two proofs land re-expressed against it:
- manifest/shadow_test.go: a prefix belongs to exactly ONE app. The router
merges identical patterns silently (mutation-checked: a duplicated
prefix passes TestAppsMountWithoutConflict), so the table is where the
invariant must hold. The branch's two recorded collisions are both
fixed on this side (storage's deeper /v1/s3 leaves; zen's Gates), so
the gate lands with no exception list.
- apps/analytics/signal_live_test.go: one batch, four signals, four rows —
act/error/log/span each land exactly one row under its own signal, with
the columns their reads sort and group by, sharing one trace.
The branch's fleet-wide zipdoc/openapi regeneration, its o11y receiver
ports, and its telemetry rungs are superseded by the trunk's plane sink,
typed ops and generated surfaces, and resolve to the trunk.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The image job skipped the smoke on `resumed`, and `resumed` was read from the
registry. So a run that PUSHED and then FAILED the smoke came back, found its own
bytes under its own version, declared itself resumed, and skipped the build AND
the proof — handing rollout an image nothing had ever booted, with every job
green. One flag was answering two questions and only one of them had an authority.
An artifact existing proves it was built. It never proves it was tested.
`built` stays the registry's answer and decides the build alone. `smoked` is a
separate fact with a separate authority: refs/smoked/sha256-<digest> at the
commit, created by the same compare-and-swap the version claim already uses — one
mechanism for recording a fact, used twice. Kept in git rather than beside the
image because the registry is the thing being attested and ns hanzo-build alone
had five ways to write it; an attestation stored inside what it attests can be
written by anyone who can write that.
Keyed by DIGEST, never by version. A receipt naming a version proves a version
was smoked and says nothing about which bytes that name resolves to now.
The smoke boots repo@digest instead of repo:tag for the same reason, and rollout
now refuses to ship when the digest pin.sh wrote is not the digest this run
smoked — a values file whose tag moved and whose digest did not deploys the old
image and reports success.
Measured on the real conditions, before vs after, at the state that caused it
(bytes exist, never smoked): before SKIPPED the smoke, after RUNS it. The two
agree on every other state.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The merge that brought the github lineage onto the forge one took the SOURCE of
several ops — agents' session count, affiliates' royaltyFailures among them —
without the artifacts derived from it, so the drift gate read the tree as a
surface that had changed and a document that had not.
Nothing here is hand-written: zipdoc_gen.go for seven apps, affiliates' subset
and the weave, all regenerated from the source already on main. A generated file
committed stale is the same defect as one never generated, and it fails the gate
in the same place for both.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two lineages each justified the same exemption, and the merge kept both entries.
A duplicate key in a map literal is a COMPILE error, so the root package's tests
did not run at all — go-unit reported "[build failed]" for the one package whose
job is to hold this gate, which is the failure mode the gate exists to prevent.
Kept the entry that states the mechanism (screen.op / seen, what comes off the
decoded In versus the request, and that a call with no request resolves no payer
and is screened as that state rather than exempted from it), and folded in the
one fact only the other carried: the request is parked by the app-wide Bridge,
which is why the same resolver works on the /mcp leg and the browser's alike.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/billing/topup was mounted on commerce and named nowhere in this table, so the
fleet published the path and handed it to ai's bare "/v1" remainder — the route
of last resort. The prepaid balance gate on that row would have made topping up
require the balance the top-up exists to create: the exact trap the /v1/cart
paragraph above describes, this time on the door that funds the account.
It was invisible until the operation was described, because router_test.go's
oracle compares PUBLISHED paths against what the router hands each app, and an
undocumented route publishes nothing to compare. The document and the table are
each other's check; a route missing from one hides a defect in the other.
The stem /v1/billing/topup replaces the leaf /v1/billing/topup/token rather than
joining it: commerce serves both top-up doors, a prefix owns its whole subtree,
and the subtree has no other claimant.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
@hanzo answered every message with "the agent hit an error handling that".
Cause, from the log: `cloud: app is not deployed here: agents`.
bridgeReply called agents.RunOnBehalf DIRECTLY, and that function gates on the
agents package's `mounted` global. A PLUGIN IS A PROCESS: a package global is
per-process, so `mounted` is nil on the bridge's side unless agents happens to
be in the same binary. The in-process seam was the ONLY door, which made
co-residency an undeclared requirement — and integrations and agents are
separate plugins in every real deployment, so every chat bridge failed
identically, for Slack, Discord, Teams and Telegram alike.
The turn now travels as a typed plane op over ZAP/UDS:
plane.AgentsRunOnBehalf + RunOnBehalfIn/RunOnBehalfOut (plane/plane.go)
POST /agents/run-on-behalf (apps/agents/onbehalf_rpc.go)
plane.Ask from the bridge (apps/integrations/bridge.go)
Both doors run the SAME runOnBehalf, so org isolation, linked-subject
attribution and billing are identical whichever way the call arrived —
sessions_rpc.go's "two doors, one teardown" shape, applied to the run.
The bridge no longer imports apps/agents at all: reaching into another plugin's
package WAS the coupling, and the dropped import is the proof it is gone. The
plane door lives in its own file so onbehalf.go keeps its promise to know
nothing of zip.Ctx or the wire. Status travels explicitly rather than being
inferred from a non-empty Output — "ran and had nothing to say" and "failed"
are different answers and a bridge must not post the second as the first. An
empty subject is refused, never defaulted to the org: a turn that lost its
caller must not bill the tenant for an unattributable act.
ALSO: a DM is answered INLINE. Threading every DM reply buried a one-line
answer behind a "1 reply" click; a channel still threads because the reply
shares the room, and a DM the user deliberately threaded is honoured.
Tests: DM inline, explicit DM thread honoured, channel mention still threads;
the existing route test now STATES the inline intent rather than the old
behaviour. apps/integrations and plane suites green. apps/agents'
TestTargetOpsProjectEverywhere fails identically on unmodified main (op-id
naming drift) — verified by stashing tracked AND untracked.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/billing/topup and /v1/integrations/slack/install were both registered on the
router and described nowhere, so the drift gate stopped: an operation that says
nothing about itself publishes an operationId and no sentence, which is an SDK
method that cannot explain itself and an MCP tool a model cannot pick. Neither
route was reachable from any generated client.
Both are raw by nature, so both state themselves with openapi.Describe rather
than a lifted doc comment: topup is charged and redirected, install answers a
302. The prose is written from the handlers — the gate each one gets, the tenant
scope, and the one rule a caller would otherwise get wrong: for topup that
paymentMethodId is NOT covered by the billing-subject pin and a card owned by
another subject answers 404 rather than 403, because 403 would be an ownership
oracle over other people's cards; for install that it is anonymous ON PURPOSE
and the org is resolved at the provider callback like every other install.
Regenerated: openapi.yaml 1751 -> 1753 paths, 2509 -> 2511 operations, and the
per-app subsets beside them.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The forge is what builds cloud (the build job clones git.hanzo.ai/hanzoai/cloud;
hanzo-inc/cloud has no build workflow at all, only Dependabot). So work landing
on GitHub main was shipping nowhere. 133 forge commits vs 52 GitHub commits off
merge-base 1720eb20, 15 conflicts.
Resolutions, each decided rather than defaulted:
- Dockerfile: ARG CONSOLE_IMAGE deleted. Deleting it IS the change — the console
is a published site now, not an embedded image.
- apps/billing/gpu_charge.go: delete kept. The forge removed the bespoke prepay
charge path on purpose ("GPU is metered like any resource"); GitHub modified a
file whose reason for existing was gone.
- go.mod: higher version wins both ways (commerce v1.50.12, plans v1.4.14).
Forward, never backward.
- manifest/apps.go: forge kept, after PROVING it a superset — GitHub contributed
no prefix the forge lacks, the forge adds /v1 and /v1/cart.
- apps/commerce/mount.go: GitHub's comment, because it CORRECTS the forge's. The
forge said byte-identical patterns "merge silently, first wins"; measured
against the pinned zip they do not — the composer refuses the program and names
every conflicting pair. Plus the forge's POST /v1/billing/topup route, which
GitHub lacks.
- apps/affiliates: the forge's typed accrualsOut, given the field GitHub's
untyped map had and the typed shape did not. royaltyErrs was already counted
and then dropped on the floor — the same silence the typed leg exists to end.
- slack_events.go: the forge's fuller comment plus GitHub's app_home_opened arm.
- plane/plane.go: both sides had independently added an identical
ProjectsOwnership constant. Kept the one grouped with ProjectsResolveKey.
- go.sum regenerated by tidy, never hand-merged.
Two defects the merge exposed and this fixes: duplicate imports in commerce
mount.go, and affiliates calling s.Log.Warn where its two siblings in the same
loop call o.s.Log.Warn.
Carries the front-door body-limit fix (internal/edge): cmd/cloud built its app
with no BodyLimit, so the door enforced fasthttp's 4 MiB while the pod's
GATEWAY_BODY_LIMIT read 100 MiB. Measured on api.hanzo.ai: 4,194,304 bytes
passed, 4,194,305 answered 400.
go build ./... exits 0.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The bespoke prepay endpoints existed on both sides of this seam: cloud served
/v1/billing/gpu/{charge,eligibility}, and commerce served the handlers behind
them. Cloud's went in the previous commit; v1.50.13 is the commerce release that
deletes ChargeGPU and GPUChargeEligibility, so the module this binary links no
longer carries the path at all.
Nothing here referenced those two handlers — cloud names commerce's billing
handlers one by one and never called RegisterHandlers, which is why the route
was already gone from this binary. The bump is what makes that true of the
module graph as well, rather than leaving a deleted door still compiled in.
The ledger rule is untouched: billing/bucket still classifies a gpu-tagged
withdrawal as prepaid and never a grant, for every writer.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two agents built apps/sandbox concurrently in the same working tree and each
overwrote the other's files. What survived was halves: proxy.go from one,
client.go from the other, both declaring boxPath with different signatures, and
the package did not build at all.
Resolved to ONE implementation. sandbox.go wires forward() from proxy.go and
nothing referenced client.go's httpBox transport, so client.go was the orphan
and is deleted rather than kept as a second way to reach a box.
proxy.go is now zip-native. It had been written against Fiber's context —
c.QueryString, c.BodyStream, c.Get, c.Set, c.Response — none of which exist on
zip.Ctx. zip offers two escape hatches that would have made it compile,
AdaptNetHTTP and c.Fiber(), and both are declined on purpose: either one makes a
second way to write a handler in a codebase whose whole premise is that there is
one. The real surface does the job — c.Query, c.Body, c.Header, c.SetHeader,
c.Status, c.SendStream.
The query forwarding is the part that changed shape rather than syntax. Copying a
raw query string blindly would make this an opaque tunnel that cannot tell a real
parameter from a smuggled one. wire.QueryParams now names the five the box
surface actually reads (path, depth, q, regex, limit), declared once where both
ends can see it, and the proxy forwards exactly those.
Response streaming is preserved and matters: SendStream rather than a buffered
read, because a `pnpm install` streams for minutes and the caller should watch it
arrive.
Also s.State.stores.Get -> .For; cloud.OrgStore has no Get.
NOT DONE, and it should not look done: harness_test.go and tenancy_test.go were
written against the OTHER implementation and reference stateWarm, classDesktop
and pool.nodeTaint, none of which exist here. The package BUILDS but `go test`
does not compile. Those two files need porting to this implementation — they are
kept rather than deleted because tenancy isolation is exactly the property worth
testing, and deleting the test would be the worst way to make the suite green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three console cards were dark for three unrelated reasons. Two of them are here.
METRICS — /v1/billing/usage answered 502 "billing upstream unreachable" with
nothing upstream involved. usage_coresident.go asked plane.Money.Minor() for
cents, and Minor() REFUSES an amount finer than a cent rather than round behind
the caller. A per-token AI charge is routinely finer than a cent, so one such row
failed the whole page; usage() tests err before coResident, so the refusal
surfaced as a dead upstream. This is the 2026-08-03 balance bug exactly —
balance.go and apps/ai were converted to FloorMinor then and this row was missed.
Measured on the live fleet, same org, same ledger, same process:
GET /v1/billing/balance -> 200 (FloorMinor)
GET /v1/billing/usage -> 502 (Minor)
Only the call differed. The row now also carries the EXACT amount through, so the
envelope's `decimal` is populated on this branch too — fixing the 502 without it
would have traded a visible error for a silently understated bill.
FLEET — knownServices, a hand-kept list of 26 k8s workloads, was the only door
into resolveService. manifest.Apps lists 119 routed apps. The overlap was TWELVE.
The other 107 answered honest-empty on status, product metrics and scoped logs —
not because their telemetry was missing (TracingMiddleware has been stamping
http.route on every one of their request spans into event.span all along) but
because this package had never heard of them. The set is now DERIVED from the
manifest the host already builds its router from, so a new plugin gets
metrics/logs/status the moment it has a row, which it must have to be routable at
all. 117 of 119 now scope. The two that do not are the two that cannot:
zen is Coresident — it routes no prefix of its own.
ai claims the API ROOT — it routes everything nobody else claimed.
Neither is a product boundary, for mirror-image reasons.
Routes come from the manifest rather than "/v1/"+name because for 48 apps that
convention names a subtree nobody serves (plan serves /v1/plans, storage serves
/v1/s3/buckets, account serves /v1/orgs and five more). Scoping RED to a path with
no spans returns zero, which reads as a healthy idle service — an answer, and
wrong.
Longest prefix wins, as the router already decided it. manifest.OwnerOf warns that
"anything deciding policy from a bare HasPrefix scan will attribute those paths to
the wrong app", and a RED query decides policy: admin owns /v1/admin and seven
apps live under it. Eleven apps need this; for the other hundred the exclusion set
is empty and costs nothing. Measured against six hours of live spans: a bounded
app with seven exclusions is 4.3s against a ~5s baseline, i.e. free — while
expressing ai's fallback as "/v1 minus 251 nested prefixes" costs 15.5s to compute
a number that would have counted KMS's 161,705 requests as inference. That is the
second reason the fallback has no route scope.
Addresses still come from probes.go: an app the fleet does not probe has no URL
and is not probed, which stays the honest outcome.
No new routes and no new Owns* flag. A health-shaped auto-registration loop would
have declared /v1/<name>/{metrics,logs,status} on top of 15 addresses 13 apps
already serve for unrelated business reasons, and zip refuses a twice-declared
address by crash-looping the binary.
Tests fail when they find nothing: the fleet gates guard their iteration source
with a magnitude floor, assert both directions, pin the two exemptions at exactly
one each, and refuse the vacuous run. Verified by reverting each fix and watching
them fail — the sub-cent test with the exact production error, the fleet gate with
106 unresolved apps, the attribution gate with ai swallowing /v1.
Also drops a comment citing query.go as the pin to follow. That file is gone, and
its removal is what broke the console's Logs card.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Visor typed its agent-binding routes (hanzoai/visor 575fc60), which drops the
casibase envelope with them: the answer IS the value, the status IS the outcome.
Nothing here could have kept working — an AgentBinding carries its own `status`
field, so the envelope reader takes "Pending" for an envelope status, decides
the upstream failed, and answers 502. A read of an unbound machine and every
unbind would have gone the same way, on a 204 with no body to decode.
So the client says which wire it is reading, per call site:
cl.call the casibase {status,msg,data} envelope at HTTP 200 (23 sites)
cl.op a typed op: the value, 204 for void, 404 for a miss (9 sites)
Not a fallback and neither sniffs — which wire an upstream op speaks is a
property of that op, and the two cannot be told apart by looking. They share
`do`, so there is still one request builder, one identity rule, one status map.
`call` shrinks to zero as visor finishes and goes with the last noun.
The addresses converge rather than translate. This package already published
/v1/machines/agents and /v1/machines/:id/agent to its own callers while spelling
vm's side three other ways; vm answers on ours now, so the translation is gone
instead of moved. The owning org travels once, as ?owner — vm derives the
binding's org from the same resolved principal, so the body field repeating it
is gone too.
An upstream 404 is a FACT about the machine (it runs no bot), not a fault:
notFound() recognises it so getAgent answers in its own words and messageBot
keeps its 400 — "no bound agent to message" tells a caller what to do, where a
relayed 404 says only that some lookup missed.
The fake vm in bots_http_test.go now speaks both wires, written to match what
visor's own controllers/agent_wire_test.go asserts against the real handlers —
a fake agreeing only with this client would prove the two agree with each other
and nothing about the service.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/v1/billing/gpu/charge and /v1/billing/gpu/eligibility were a second,
product-specific way to move an org's money. They duplicated the general
metered path and nothing called them: no first-party caller exists for either
address, for gpuCharge/gpuEligibility, or for the commerce handlers behind them.
A machine is launched through /v1/machines, which fronts the compute provider's
resell endpoint. That path already owns both halves this one hardcoded: it
authorizes the first launch hour against the org's balance BEFORE provisioning
and refuses when the funds are not there, then records the usage keyed on the
machine id the provider minted, and an hourly sweep meters it from there. One
meter bills every resource, so a GPU needs no door of its own.
Deleted with it: proxyGPUCharge and the commerceProxy.post it was the only
caller of, pinSubjectBody, and the manifest prefixes, retired-name mappings and
published operations that named either address. The floor drops by the two
operations that left, so the reduction is reviewed next to its reason.
The ledger rule the endpoint was built to protect is untouched and lives where
it always did: a gpu-tagged withdrawal draws prepaid, never a credit grant,
because billing/bucket classifies it that way for every writer.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/integrations/slack/install landed as a route but not as a DECLARATION,
and TestSurfaceIsRegistered compares the live router against the union of
typedOps and rawRoutes: 42 live, 41 declared. The gate did exactly its job —
a route nobody recorded a decision about is the thing it exists to catch.
It belongs in rawRoutes rather than typedOps for the reason already written at
the top of that map: a 302 is not a JSON body, so it cannot be a typed op. It
sits with the other redirect legs it is a sibling of.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzo.yml states the invariant as "main push -> build -> smoke -> tag -> pin ->
prove live", and every job in cicd.yml upholds its own link. Nothing upheld the
chain itself: each job can only see the release it is running, so an image
published by a lane that never enters the workflow is invisible to all of them.
That is how 82 versions came to be published with no tag in any repo. Seven of
them are recent — 478, 479, 480, 481, 482, 483, 484 — and 480 served
api.hanzo.ai while reporting {"revision":"unknown"} on its own health endpoint.
485 was published untagged while this was being written, which is the point: the
hole is open now, not in the log.
The lane is not a broken gate, it is a bypassed one. 484 and 485 came from
Jobs applied straight to ns hanzo-build (build-cloud-slackinstall,
build-cloud-slackfix) whose buildctl line reads
--opt=context=https://github.com/hanzo-inc/cloud.git#refs/heads/main
--opt=build-arg:VERSION=v1.801.485
--output=type=image,name=ghcr.io/hanzoai/cloud:v1.801.485,push=true
No build-arg:REVISION, so the Dockerfile's `unknown` default wins and the binary
can never name its commit. No version claim, so the number is taken rather than
owned. No tag, because nothing in that command mints one. A context of
refs/heads/main is not even a fixed input — it resolves at fetch time, so the
image is unreproducible as well as untraceable.
So the check is lane-agnostic on purpose. It does not ask who built an image or
whether some workflow succeeded; it asks the registry what is published and the
remotes what is tagged, and any published version that is neither tagged nor
recorded is red. A gate phrased in terms of a lane can only catch that lane, and
the lane that caused this was the one nobody thought to instrument.
It runs at the end of receipt rather than in rollout: a pre-existing orphan must
not be able to block a deploy, because a gate that wedges production on someone
else's old mess is switched off within a day. It is on the main path rather than
a cron for the same reason — a scheduled reconciliation can stop running and
look identical to a clean fleet.
Tags are read from BOTH git.hanzo.ai and github.com/hanzoai/cloud and the union
counts. cloud is canonical on the forge, but cicd.yml claims its version through
the GitHub refs API, so receipts genuinely live in two namespaces; asking only
one would paint every release red from the other. Two of the recorded entries
(319, 340) look tagged from a checkout and are not — the tag exists in one local
clone and was never pushed. Reading `git tag` instead of the remotes is how they
stayed invisible, and it is why the record is derived from the remotes.
None of the 82 were tagged retroactively. Fifteen carry a real commit in
org.opencontainers.image.revision and several name a commit on main, so they
could have been. Here a v* tag is a RECEIPT, minted only after build and smoke
pass; minting one now would assert that a pipeline proved something it never
ran, trading a gap everyone can see for a false claim nobody can detect later.
The image -> commit link those fifteen do have is recorded in column two, which
costs nothing and claims nothing. The other 67 report revision=unknown and no
mapping is guessed for them.
The tests are the mutation: same registry listing every time, and only the tag
list moves. Untagged is red with the refusal quoted verbatim, tagged is green,
recorded is green, and an unrecorded orphan beside a recorded one is still red.
An unreadable source exits 2 rather than 0 or 1, because "could not look" and
"nothing found" demand opposite handling and collapsing them is how a verifier
comes to pass by accident.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Upstream /v1/k8s/nodes is now a typed zip op, so it answers {"nodes":[...]} with
no {status,msg,data} around it. Both consumers move together: listK8sNodes and
managedMachines' third source.
client.call is split at the seam that actually exists. do issues the request —
URL, forwarded identity, credential, body limit, transport errors, one path for
everything. call reads an ENVELOPE (visor's untyped controller routes). op reads
a typed op's Out. Two readings because visor really serves two shapes right now;
call and envelope shrink as routes are typed and go with the last one.
The two readings must not be merged, and that is the trap this closes: decoding
an envelope into visorNodes does not fail — the keys are simply unknown — so
Nodes stays nil and an operator running eight clusters is told, with a 200, that
they have no worker nodes. The op always writes the key, so nil means "this Visor
does not serve this op": listK8sNodes answers 502 and the fleet fold drops that
one source and logs why, leaving the registry and live-droplet sources intact.
TestK8sNodesRefusesTheOldEnvelope and TestMachinesDropDOKSNodesOnSkew pin both
halves against a fake that speaks the pre-typed wire. Without the check the first
serves `200 {"nodes":[]}`.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cmd/cloud built its app as zip.New(zip.Config{AppName: "cloud", MCP: ...}) with
no ReadBufferSize and no BodyLimit. It is the FRONT DOOR -- every public request
reaches it before the program behind it -- so it terminated HTTP on fasthttp's
defaults: 4 KiB headers, 4 MiB bodies. cloud.App() set both correctly for the
program, and none of it could be reached past the door.
Measured on api.hanzo.ai: GATEWAY_BODY_LIMIT=104857600 in the running pod's
environment, and 4,194,304 bytes passed while 4,194,305 answered 400. That is
4 MiB exactly. In-cluster, straight to cloud.hanzo.svc:8000, the same 400 --
so ingress and gateway were never involved.
The consequence nobody saw: the 16 MiB default had ALSO never taken effect. Its
docstring says it exists so a 1M-token prompt (~4.3 MB of JSON) can reach the
1M-context models; at 4 MiB those models were unreachable, and fasthttp's wire
error is the opaque 400 "Error when parsing request", which reads like a
malformed payload rather than a size cap. Someone wrote that comment believing
it was fixed.
app.go already says an app is obtained one way and a program never builds a
zip.App itself. The door does, deliberately -- it links zip, manifest and webui
and nothing else, and importing the cloud root to reach a number would re-fuse
the monolith it replaced. So the numbers move to internal/edge, a leaf both can
import. One home per number; a second home is how this broke.
TestTheDoorAcceptsABodyLargerThanTheFrameworkDefault sends a real 4 MiB+1 KiB
body through a real door and fails on 400. It reads the wire, not the config,
because reading the config back is what the old code would also have passed:
the value was computed correctly and never reached the transport. Mutation-
tested -- hardcoding the framework defaults back turns both tests red.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
projectIsForeign answered false for a foreign project-id on every surface that
installed it, so the guard that keeps one org from asserting another's project
was a no-op fleet-wide. The registry it consults lives in the `projects` plugin
and is never mounted in the process where the edge middleware runs, so the
lookup could not succeed anywhere it mattered — measured false against a project
id belonging to another org.
The question "does this project belong to this org" cannot be answered inside
the identity boundary alone, so it becomes a plane op and is asked across the
boundary: plane.ProjectsOwnership, with the same fail-closed posture the rest of
the edge has. A caller that asserts a project it does not own is refused rather
than admitted on a lookup that silently returned false.
Cherry-picked onto the ship branch with its plane op, which is the 33 lines the
guard needs to reach the registry. Both were on the mirror main and not on the
integration main this image builds from, so production carries the hole today.
It ships with the tracker/meet cutover rather than behind it: a live cross-tenant
isolation gap outranks a UI host migration, and one image closes both.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Disabling a model made it invisible to the one caller who must still see it.
The read path delegates existence to the embedded @hanzo/pricing bundle, which
is a stale snapshot of first-party prices. The authority for what exists and
whether it is enabled is the OVERLAY, and commerce backs that. Those two
disagree in exactly one case — a model an admin has disabled — because the
bundle then stops serving it, and its 404 returned before the gate below could
decide. VisibleCatalog already keeps disabled entries for admins
(`if !isAdmin && !visible { continue }`); it simply never ran.
So a 404 from the bundle no longer ends the question: when the overlay knows the
id and the caller is an admin, the admin gets the overlay's answer, annotated
with the same _overlay state the list surface returns.
Measured before changing anything, because the shape invited two wrong guesses.
The op prints, for the admin request: bundle status=404 isAdmin=true. The
identity bridge was fine and the gate was fine — only their ORDER against the
bundle was wrong.
Not a regression this change introduced: apps/pricing was green at fe9bacb0 and
red after, and nothing in this cutover touches it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three defects, each of which fails silently rather than loudly, and all three
found by another agent reading hanzo.chat's client source instead of our own
docs. Verified against ~/work/hanzo/chat before changing anything.
1. POST /v1/upload omitted message:"success". crud.js:108 is
`if (result.message !== 'success') throw` — so EVERY upload failed, with
"Error uploading file: undefined" and nothing on our side to say why.
2. Upload elements were {name,id}. crud.js:112 builds the identifier as
`${result.session_id}/${result.files[0].fileId}`, and the client's own JSDoc
types the element as {fileId, filename}. Our shape produced
"<sid>/undefined": the upload looked fine and the file was unreachable.
3. GET /v1/files/{sid} returned {session_id, files}. Two readers assume a bare
array — ProgrammaticToolCalling guards with Array.isArray and returns [],
and process.js:294 calls response.data.find(...) directly. An object does not
error there; it makes every session read as permanently empty, which is much
harder to notice than a crash.
The three endpoints need three DIFFERENT shapes, which is why one struct could
not serve them: exec returns files:[{name}] (unchanged — newFiles is correct and
"fixing" it to match upload would break it), upload returns {fileId, filename},
and files/{sid} returns bare [{name:"<sid>/<fileId>", lastModified}] because
process.js matches on that prefix and reads that field.
librechat_contract_test.go pins all four shapes with the consumer line number
that reads each one, so a later tidy-up has to argue with hanzo.chat rather than
with a naming preference. Keep it. Credit for the findings goes to the agent
that read the client; only the file was lost to a concurrent-write collision.
cmd/boxd builds and its suite passes, including the four new conformance cases.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TWO defects, both measured in production.
1. EVERY Slack DM answered "Sorry — I couldn't reach your Hanzo account just
now." getUserLink already had the right branch:
if errors.Is(err, kms.ErrSecretNotFound) { return userLink{}, false, nil }
but the live error was the STRING `kms kms_get: kms.get: store: secret not
found`. A plugin is a PROCESS: an error crossing that wire is re-created from
its text, so the sentinel does not survive and errors.Is is false on the far
side even though the store said exactly that. An unlinked user therefore took
the BROKEN branch instead of "not linked yet", and the link prompt that
teaches them how to connect was unreachable — the feature could never be used.
Repaired in kmsGet, the ONE door onto the store, so the sentinel is whole for
every caller above it. Text matching is kept to the store's own phrase: a
broken store must never be laundered into a silent "unlinked".
2. The App Home tab rendered Slack's own "this is still a work in progress"
placeholder, which it shows for any app that enables home_tab_enabled and
never publishes a view. The choice was never Home vs no Home — it was our page
vs Slack's apology. app_home_opened now publishes a view answering what this
can do and what to type. A failed publish is logged and swallowed: a cosmetic
surface must not fail an event Slack would retry into an identical render.
The install route is DECLARED in ops_projection_test's rawRoutes rather than
bumping a count: that test asserts the surface is exactly what someone decided
on, and it caught the addition, which is it working.
Full apps/integrations suite green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every control plane for agentic execution exists; the thing they control does
not. This settles where it lives and what it looks like on the wire.
Four consumers, not three: apps/exec (proxy with no upstream),
apps/functions/invoke.go (same unset env), apps/coding (bot-gateway
/v1/coding-tasks, fail-closed on a docker daemon no pod has), and hanzo.app's
ProjectFs. hanzo.chat's execute_code resolves end to end onto zero endpoints
today.
Decisions: control plane is apps/sandbox in cloud (three of four consumers are
already in this binary; the standalone twin of that same pattern was deleted
last week for being a duplicate). Image is three layered tags in hanzoai/bot,
where the recipe already exists digest-pinned and non-root and has never been
built because bot has no CI at all. boxd is Go in cmd/boxd so its wire types
are one declaration shared with apps/sandbox and drift is not expressible.
Records a live defect proven with real output: apps/exec preserves the request
path (/v1/exec) while apps/functions appends /exec, so no value of
CODE_EXEC_UPSTREAM satisfies both consumers.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Retiring zen4 cost eight red gate runs on one line, because a test that
writes a catalog id down is making a claim about the snapshot and snapshots
retire models. The single-segment id started being read from the mounted
catalog; the SLASHED one did not, and the same literal was spelled a second
time in the enablement flow. The bundle already carries opus-4.7, 4.8 and 5,
so 4.6 ages out the way zen4 did — between two survivors, invisibly.
Both ids are now derived by SHAPE through one reader. A provider-qualified id
is what the greedy wildcard exists for and a single-segment one is what the
ordinary route serves; which id has that shape is the catalog's business.
The read is the PUBLIC view, which also makes the assertions mean more than
they did: the model is baseline-visible before it is disabled, so the
customer's 404 is the visibility gate refusing, not the lookup missing. Those
two answers reach the wire identically, which is why the failure pointed at
the admin gate instead of at the snapshot.
The precondition the route-shape tests rest on — the catalog serves both
shapes — is now stated by a test of its own instead of surfacing as a 404 far
from its cause.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Eight gate runs red on one line: the admin HTTP test hardcoded zen4 (and
zen5), and pricing v1.4.10's regenerated snapshot retired the zen4
generation — so the single-model section 404'd an id that no longer exists
and every push since the catalog bump failed the gate. A literal model id
in a test is a claim about the snapshot, and snapshots retire models; the
single-segment id is now READ from /v1/pricing/models on the mounted
subsystem, so the sections prove the ROUTE SHAPES (greedy wildcard vs
single segment, the visibility gate, the over-deep 400) against whatever
the catalog actually carries.
The allowed browser origins were a static list of domains we own, so the
shipped feature — fork hanzoai/console, deploy it on your own domain — could
not work: no list of our domains can name a customer's domain.
An origin is now admitted from either of two sources. DECLARED is the platform
allowlist an operator writes. PROVEN is a host whose site_hosts row is
verified, which is the DNS-01 proof that subsystem already takes: the claimant
publishes a 128-bit token as TXT at _hanzo-challenge.<host>, fqdn.Verify is
fail-closed, and ResolveHost's status='verified' filter is the existing
hostname-hijack boundary. CORS asks that boundary rather than growing a second
one, so a pending claim on a name grants nothing here either.
The Origin header is chosen by the caller, so the cheap total rules run first
and the store read runs last: exactly scheme://host reconstructed and compared,
https only, no port, and fqdn.Valid — which is also what keeps bare project
slugs out, since site_hosts holds every project's slug as a row that is always
verified and a bare label is not an FQDN. Answers are cached in BOTH
directions and the cache is bounded, because the attacker picks the key.
Two defects fixed on the way. Vary: Origin was assigned, not appended, so it
fought middleware_markdown's Vary: Accept and the loser's protection silently
vanished; and it was set only when the origin was allowed, though the answer
that carries no ACAO depends on Origin just as much. It is now appended on
every origin-dependent answer, including the denial.
And there was a second CORS authority. hanzoai/ai is mounted in-process and
inserts a filter ahead of every /v1 route that 403s any origin outside 21 apex
domains compiled into the module. The preflight short-circuits here and never
reaches it, so a customer console would have passed the preflight and failed
the call — allowed preflight, denied request. Its positive half is already
dead in production (setCorsHeaders returns early on X-Forwarded-Host, which
the ingress always sets), so only the refusal is live; clearing Origin on the
hop into it, for origins this edge already admitted, takes its own origin==""
no-op path. Denied origins are passed through untouched and still 403.
WebSocket upgrades keep their Origin: dev_bridge's CheckOrigin admits an empty
one for CLI clients, so clearing it there would fail open, and a socket's
origin check is a different mechanism from CORS.
The per-IP cap moves ahead of CORS. Resolving an unknown origin is a plane hop
in production, and a caller rotating a fresh hostname per request would defeat
the answer cache and turn each inbound request into an internal one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three parts, all required before Slack will OFFER @hanzo as an agent:
1. assistant:write bot scope (slack.go)
2. assistant_thread_started / _context_changed events (slack_events.go)
3. the Agents & AI Apps toggle in the app config (not code)
Without all three the app answers @mentions and DMs fine but never appears in
Slack's "Add Agents" picker, which is why only Claude was listed.
Both events merely ack, which the default arm already did, so the wire
behaviour is unchanged. They are named explicitly because a reader asking
'does this app support agents' must find the answer here; a silent default
cannot say yes.
GET /v1/integrations/slack/install -> 302 to Slack's consent URL. Slack REFUSES
a slack.com URL in the Direct install URL field and requires one of OURS that
302s to slack.com -- because the field is an ATTRIBUTION hook: routing the
Marketplace click through our address is what lets an install be counted, and
forcing the 302 stops the counter becoming a detour that never reaches consent.
A redirector and nothing more. Same destination every time, built by the SAME
slackAuthorize the console's Connect button uses. It carries no state and binds
no org: the org resolves where it always did, at the generic
/v1/integrations/:provider/callback. Minting a principal for an anonymous click
is what the isolation bar forbids.
Applied here because the BuildKit lane builds github.com/hanzo-inc/cloud#main,
which had NONE of these -- it is 49 commits ahead of the forge while the forge
is 124 ahead of it. Consolidating those is separate work; this is the change.
go build clean; go test -run Slack ok.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The fleet publishes /v1/commerce/{collection,product,wallet,webhook,...} — 17
resource families commerce genuinely serves — and routed every one of them to
`ai`, which owns the bare "/v1" prefix. commerce's manifest row enumerates 30
specific prefixes and none of them reached these, so a prefix deeper than the
sibling that currently wins was simply missing.
They were DARK: published in the document, therefore offered by every generated
SDK, the MCP tool list and the spec-derived CLI, and reaching a different app on
the wire. That is the exact defect the router oracle exists to catch, and it
could not see them until they had prose — an app that cannot project its own
document produces no paths to check, so this gap sat behind the describe failure
rather than behind a passing test.
Naming the 17 prefixes is the fix the oracle asks for, and it is the narrower one:
a bare "/v1/commerce" would capture the whole subtree including addresses other
apps may later claim, while a prefix per family owns exactly what commerce serves.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/integrations/slack/install -> 302 to Slack's consent URL.
Slack REFUSES a slack.com URL in the Direct install URL field and requires one
of OURS that answers 302 to slack.com. That is not a hoop: the field is an
ATTRIBUTION hook. Routing the Marketplace / Add-to-Slack click through our own
address is what lets an install be COUNTED, and forcing the 302 is what stops
the counter becoming a detour that never reaches consent. Same destination
every time.
A redirector and nothing more. The consent URL is built by the SAME
slackAuthorize the console's Connect button uses, so there is one source of
truth for what we ask a workspace to grant however the click arrived.
It carries NO state and binds NO org, which is correct rather than a gap: the
org is resolved where it always was — the generic /v1/integrations/:provider/
callback, from the signed state a console connect minted or OrgForExternalID
for a workspace already connected. An install beginning here finishes there
under the same rules as every other install, and that callback already returns
the browser to the console on a labeled failure rather than a JSON dead end.
Minting a principal here would be inventing an org for an anonymous click,
which the isolation bar forbids.
Registered as a LITERAL before the /:provider wildcards (fiber resolves by
registration order) and Terminal like its siblings — the person clicking
Install in Slack's directory has no Hanzo session, so it must not 403.
Unconfigured answers an honest 503 rather than a consent URL with an empty
client_id, which Slack renders as its own error page with no way back.
Tests assert the contract Slack validates: 302 to slack.com/oauth/v2/authorize
carrying client_id, assistant:write and our callback as redirect_uri.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Same fix as billing's lane: the KMS read returns a flat {env, name,
value}, both pin steps indexed .secret.value, so the self-pin reported
UNIVERSE_PIN_TOKEN missing even once it existed. Tolerant read; the
automation identity (forge user 'deploy', write on hanzo/universe, token
sealed at deploy/UNIVERSE_PIN_TOKEN env prod) is live.
The account had four vaulted Square cards and $149k of balance, and no
customer-facing door could charge any card on file: subscribe/card demanded
a fresh nonce per attempt (vaulting one more identical row each time), and
POST /v1/billing/topup — the saved-card top-up commerce already ships, the
same chargeAndCredit core the auto-recharge cron uses — had no co-resident
route, so it fell to the account bridge and died in self-dispatch.
commerce v1.50.12 (this bump) makes subscribe/card accept paymentMethodId,
stamps brand/last4/expiry + fingerprint on vaulted rows, dedupes a re-saved
card into its existing row, heals legacy rows on list, and adds the
ownership guard Topup's body field always needed. The mount here routes
POST /v1/billing/topup with topup/token's exact chain — risk screen
included, both credit the spendable wallet.
TestPlaneWireCarriesNoFieldNames observed the bytes crossing the plane's socket
and proved they were ZAP rather than JSON -- field names absent, values present.
That was right, and zip v1.27.0 made it unobservable: Ask now asks zip.Serving
whether the peer is this process and hands the call to the op's own invoke seam,
so there are no bytes. The test read "no body was observed" and went red on a
change that was correct.
The encoding assertion did not need rescuing, it needed deleting: internal/zapenc
already pins it at the encoder, in TestNothingIsJSON and TestBytesAreAZAPMessage.
Cloud was keeping a second copy of a fact zip owns, at a level that could only
observe it by accident of the transport.
What is true HERE, and was not true before v1.27.0, is that the wire is gone. So
the middleware is still the instrument and an EMPTY observation is now the
result: if a body ever appears, cloud has started serialising a value, handing it
to the kernel, reading it back and parsing a fresh copy, to reach a function
pointer that was in memory the whole time.
Mutation: disable the zip.Serving short-circuit in plane/ask.go and the same call
puts 39 bytes on the socket --
"ZAP\x00\x01...\x00acmeusd"
-- which is both the failure and a demonstration that the frame is still ZAP
carrying values with no field names when a wire is genuinely involved.
zip v1.27 gave plane.Ask a co-residence short-circuit: zip.Serving(app) finds a
peer this process is already serving and zip.Here runs the op through the same
opByName and invoke the socket plane uses, with nothing encoded. That is right,
and it made this file's mutation test assert the wrong thing — it unlinked a
socket the call had stopped using and called the success a failure.
The two halves are now pinned separately, because they are two facts:
co-resident zip.Serving finds iam, the call runs in-process, and unlinking
the socket changes nothing because nothing was going to touch it
separate the socket is the whole dependency: unlink it with the peer alive
and the call fails, rebind and it succeeds
"Separate" is arranged the way zip.Serving actually decides it — the lookup keys
on SocketPath(name), so a caller resolving iam.sock in a different runtime
directory misses the local registration and dials for real. Same arrangement
planewire_probe_test.go uses, and the frames it captures are what says this path
is a wire at all.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The waitlist gate resolved approval with an HTTP GET to /v1/iam/get-account, and
because HTTP gave it no way to say WHO was asking, it said so by forwarding the
caller's own Cookie and Authorization header — cloud taking a user's raw
credential and presenting it to another service so that service would answer
about that user. It worked. It is the wrong shape twice: the credential is
handled where it need not be, and the answer is only as good as a bearer the
gateway had already validated once.
plane.IAMApproval carries the validated principal instead. iam reads the subject
with cloud.Who, resolves it with GetUserBySubject, and answers the recorded
approvalStatus — raw, judged by nobody. Whether "pending" gates a person stays
in admission next to the gate, so the identity store and the gate can never
disagree about who is on a waitlist.
EnforceConfig.IAMBase goes with it, and it was already dead: nothing in the tree
set it, so the resolver it built had an empty base and the lookup it installed
returned ok=false forever — a gate that had been failing open by configuration
accident rather than by decision. A peer is addressed by name, so there is no
base to supply and no way for one to be absent.
The availability rule is unchanged and now pinned against a real socket rather
than a JSON table: iam SAID pending → gated; iam SAID nothing → approved; iam
could not be ASKED → approved, and that verdict is NOT cached, so a recovered
iam re-gates on the very next request rather than a ttl later.
TestApprovals_NoCredentialCrossesTheWire proves the point on the bytes, not from
source. A recording relay sits at the socket the caller dials; the caller's
secret appears nowhere in the frames and the gateway-asserted subject does. The
structural half is that a plane handler has no header accessor at all, so a
forwarded credential would have nothing to read it.
TestApprovalStatusFromAccount is deleted rather than ported: it parsed IAM's
{status,data,properties} envelope in both its top-level and data-wrapped shapes,
and that envelope had no reader left.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The canonical project read was an HTTP GET to /v1/iam/projects. IAM is a peer —
the pod boots ~25 single-app processes and iam is one of them — so the only
thing the hop needed was a way to say which tenant was asking, and HTTP had
none. Platform built one: a per-org "<org>-platform-kms" IAM application, minted
on first need, sealed into KMS, cached for a minute, and admitted by IAM's
authorize as read-only/own-org-only.
The plane carries the tenant. cloud.For puts the org on the call, the callee
reads it with cloud.Who, and it is not a field a caller can set — so the whole
apparatus was standing in for something the transport now does. A minted secret,
a KMS seal, a cred cache and a grant, all deleted, and the narrowest grant is
the one that was never issued.
The workaround went with it. IAM frames its single-project read as a POST, which
the machine grant rightly refused, so Get was derived from List and the reaper
logged "projects: get hanzo/index: status 403" every cycle — a client tripping
the wall it came through. One op, one verb, nothing to route around.
iam declares the op because iam owns the store (apps/iam/projects_rpc.go), next
to the roster read that is there for the same reason. The projection is five
fields; the identity record stays home.
Proven on the wire, not by reading it: planewire_probe_test.go puts a recording
relay at the socket the caller dials and asserts the capture is ZAP —
ZAP magic, Content-Type: application/zap, addressed by name at
/.well-known/zip/op/iam_projects, X-Org-Id riding the call, and the reply
reading acmewebWebthe site2026-08-01T00:00:00Z
five values with ZERO field names, which is what zapenc's positional layout
means and what a JSON fallback could not look like. Plus the mutation: unlink
the socket with the peer alive and the call fails, rebind and it succeeds.
Co-residence is unchanged and still wins — no IAM_URL means this binary IS the
IAM and iamProjects reads the store directly. A Go call beats any transport.
ServePlane freezes the plane app, so every test here drops it on cleanup;
without that a later Mount panics on a frozen registry, which is what the two
new files sorting before pushbuild_e2e_test.go surfaced.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zz_probe_seam_test.go was a throwaway: it printed what zip.Op and cloud.Request
look like at each of the four doors so the design could be built on measurements
rather than on a reading of the framework. It answered its question -- op.Path is
the same value four times, the request is an envelope on three of them and absent
on the fourth -- and toll_test.go asserts that answer properly, in
TestOperationIsTheSameValueAtEveryDoor, where a regression fails a build instead
of printing a line into a log nobody reads.
It was swept into the tree by an add-all and never meant to land. Deleted rather
than kept "in case", because a test that asserts nothing is a test that goes stale
without anybody noticing it has.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Toll stands down for a request BillingGate has already answered, so an operation
reached over REST is charged once rather than twice. The first version worked out
whether that had happened: if the request's own path names a declared surface then
the edge must have answered for it. That is true, and its truth rests on a line in
a composition root two files away.
Unmount BillingGate and the inference still says yes. The toll still stands down,
nothing charges anything, and both gates are present in the source the whole time.
An invariant spread across two lines nobody reads together is an invariant nobody
is keeping -- which is the same shape as the bug this seam exists to close, where
the gate was real, the coverage was real, and the hole was real, all at once.
So the claim is made rather than deduced. BillingGate parks it at the point it
commits to the charge -- after every decision NOT to charge, because a request it
waved through is one the op seam must still weigh, and over MCP and the plane the
path it read was an envelope and not the operation at all. Toll reads the claim.
Unmount the edge gate and the claim stops being made, which is exactly what makes
the op seam take over.
TestTollTakesOverWhenTheEdgeIsGone is that property, and it is the test the
inference version would have passed. Composed without BillingGate, REST must still
be charged exactly once -- by the toll -- and refused at zero.
Four mutations, four distinct failures:
gate removed MCP + ZAP "balance settled at 25c, want 0c", CLI serves free
stand-down removed REST "books hold 1 deposits and 2 debits, want 1 and 1"
price the REQUEST MCP + ZAP unbilled again -- the original bug, reproduced
claim never made REST double-charges
flow and auto said in prose that cloud sets no zip authorizer and that no
transport but HTTP reaches a money gate anywhere in the fleet. Both were true when
written and neither is now, and an MCP tools/call was never the payer-less case
they grouped it with -- it arrives as POST /mcp and carries a request like any
other. Corrected where it is written, not left for the next reader to disbelieve.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Slack's "Add Agents" picker lists only apps that declare the Agents & AI Apps
surface. Ours declared none of it, so the workspace was offered Claude and
nothing else — not a bug in the bot, which answers @mentions and DMs fine, but
the app was never OFFERED as an agent.
The declaration is three parts and ALL are required:
1. the assistant:write bot scope <- added here (slack.go)
2. assistant_thread_started +
assistant_thread_context_changed events <- added here (slack_events.go)
3. the Agents & AI Apps toggle in the app
config at api.slack.com <- NOT code; owner action
Both events merely ack, which the default arm already did — so the wire
behaviour is unchanged. They are named explicitly because a reader asking
"does this app support agents" must be able to find the answer, and a silent
default cannot say yes; it also gives setSuggestedPrompts an obvious home later.
No reply is needed to open the thread: the user's first message arrives as a
normal message event with channel_type=="im", which the existing arm already
routes to the agent, so conversation works as soon as the app is listed.
Zero runtime risk on existing installs: Slack never grants new scopes to an
existing token, so this changes only the consent URL for a re-install.
Based on forge/main (70a67035) — github.com/hanzoai/cloud main is 4504 commits
behind and does not carry apps/integrations at all.
go build ./apps/integrations/ ok; go test -run Slack ok.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The next agent needs two facts this file did not hold. First, that most of the
paths apps/commerce/transport carries are not registered in this binary at all —
commerce's api.Route() bundle is behind //go:build cloud — so the rollup and the
transaction list dispatch to a 404 and the balance re-enters cloud's own customer
handler and 401s, while a split deploy leaves the base URL empty and every reader
answers a silent zero. Not a degraded path: one that has never worked in either
shape.
Second, the line between what converted and what did not, and WHY. books, usage
and payout (with referrals, affiliates and authors) ask the ledger by name, because
the ledger holds their answer. metering's tier and cap, admin's subscriptions and
costs, storefront's store reads and billing's remaining proxies do not, because
their answer lives in a hanzoai/commerce handler whose logic is in unexported
helpers — so those want the payments.go pattern, a value-taking core exported from
the module, and NOT a second implementation in cloud. Writing a spend verdict twice
is how a cap and a rate limit come to disagree about which requests they bind.
Until those land, maxDepth, the goroutine-id parsing and CLOUD_COMMERCE_HTTP_URL
stay. The file says so, so nobody reads a half-finished conversion as a finished
one.
I changed ListEntries to take the `test` selector every other read on that ledger
already took, and toll_test.go — which landed while I was rebasing — calls it.
The root package stopped COMPILING, and I read the suite by comparing failing
test NAMES against a baseline, which reported no regression: a package that fails
to build reports zero test names, so the comparison could not see it. The name
diff said clean while the package was not compiling at all.
go vet ./... compiles every test file and is what actually answers "does the tree
build". It is the check that belonged in front of the name comparison, and it is
how this was found.
coResidentUsage resolves ListUsage by INTERFACE ASSERTION and treats a miss as
"this ledger cannot list usage" — a fall-through, not an error. So a fake left on
the old signature does not fail to compile; it stops satisfying the interface,
the capability disappears, and the customer's usage view answers from nowhere.
Three tests said ok=false and one said the envelope was not JSON, which is what
that looks like from the outside.
Worth stating because the shape recurs: an optional capability resolved by
assertion turns a signature change into a silent behaviour change. The tests
caught it here. Nothing in the type system would have.
Two service-token GETs — /v1/billing/usage/rollup and /v1/billing/transactions —
sent through the commerce transport, which co-resident dispatches back into this
binary's own router BY PATH. Neither route is registered here (commerce's
api.Route() bundle is behind //go:build cloud), so both were 404s. Split into
per-app binaries the base URL was empty, the reader called itself "not
configured", and the whole spend block degraded to honest zeros. A customer's
usage page showed a blank month while their wallet was being debited all along.
Under that, isSpend matched "withdraw" — a word the ledger has never written; it
writes finance.usage. So even a row that arrived would not have counted, and the
category breakdown and the series would have summed to zero on live data. The
kind crosses as the ledger's own spelling and is parsed once by the one
recognizer, which makes that disagreement a compile error.
rollupWire is gone and rollup replaces it, with two fields where there were four.
There is ONE balance because the ledger has one number: what it calls available
IS the settled balance, and two names for one value is how a reader comes to
subtract one from the other. There is no overage because overage needs a plan
ALLOWANCE and none exists here — this is a prepaid wallet, there is nothing to
exceed, and the field it replaces was read out of a JSON body that never came.
The scope tests moved with the read. They proved isolation by asserting an
X-Org-Id header and a ?user parameter the reader itself set — which proves the
reader agrees with itself. The org rides the CALL now, so they assert what the
LEDGER acted for, and a forged ?user=victim&org=other is not a thing the request
can express.
referrals, affiliates and authors all qualify and accrue on ONE figure: what has
this org spent. They read it through payout.Client, which was an http.Client
aimed at GET /v1/billing/usage/rollup and sent through the commerce transport.
Co-resident, that transport dispatches back into this binary's own router BY
PATH, and /v1/billing/usage/rollup is registered nowhere here — commerce's
api.Route() bundle is behind //go:build cloud and is never compiled in. The read
was a 404 wearing an upstream failure's clothes. Split into per-app binaries it
failed differently and worse: the base URL is empty in every process but
commerce's, so Configured() was false and SpendCents returned a silent ZERO. A
program that pays a commission on spend, told the spend was zero, pays nothing
and reports nothing wrong. Referrals qualified nobody. Affiliates accrued
nothing. Authors were paid nothing.
It asks the ledger by name now. The client holds no address and no credential,
because reaching a peer takes neither — Configured() and ErrUnconfigured named
things that do not exist for a peer, and both are gone. What replaces them is
ErrNoLedger, which is the ROUTER'S word and the only absence a program may act
on; every other failure is an outage and is returned as one. The user argument is gone from
the read: the figure was always the org's, and the rollup's user parameter was
the org's own name being sent back to it.
The tests were green through all of it, because each stood up an httptest.Server
for the wire under test and proved the mock answers — which nobody doubted. They
drive a real per-org encrypted ledger and a real plane socket now, and the
no-mint proof got stronger for the move: the money-IN ops are registered live
beside the read, each failing the test if invoked. A caller cannot reach those by
half-matching a path. It has to name the op.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The depth cap it replaces was never a fix. maxDepth = 8, counted in a map keyed
on a goroutine id parsed out of the header of runtime.Stack, existed because a
co-resident caller entered the ROUTER: the transport dispatched an http.Request
into the whole shared fiber app, every edge middleware ran again, and a
middleware that itself reads commerce read again on a cache it had not yet
filled. A counter measures that recursion; it does not remove it.
A caller that enters the OP cannot recurse into a middleware, because it never
runs one. So the test does not count how deep the nesting goes. It unlinks the
socket the plane serves on, proves nothing can dial it, and shows the nested
read still answering — which no call carried by a transport could do.
The mutation is the other half: point the runtime dir at a directory this
process serves nothing in, so zip.Serving stops answering for commerce, and the
same call FAILS. Point it back and it answers again. Without that, the first
test is a coincidence with good manners.
Measured under strace, whole binary, connect() traced: 6 op handlers ran and
FOUR connect() syscalls occurred, all AF_UNIX to commerce.sock and all four the
test harness's own — three readiness probes waiting for the listener, one after
the unlink to prove the wire was really gone. None of the six calls opened
anything.
Two defects, one cause. books ingested commerce's money through an S2S
GET /v1/billing/transactions sent over the commerce transport, and that
transport dispatches co-resident requests back into this binary's own router by
PATH. /v1/billing/transactions is registered nowhere here — commerce's api.Route()
bundle is behind //go:build cloud and is never compiled in — so the read was a
404 wearing an upstream failure's clothes. Split into per-app binaries it was
worse: the base URL is empty in every process but commerce's, the reader called
itself 'not configured', and the general ledger ingested nothing at all.
Underneath that, the classifier matched 'deposit' and 'withdraw' — spellings the
ledger has never written; it writes finance.deposit and finance.usage. So even a
row that arrived would have classified as nothing and posted nothing. The tests
passed because the fake source spoke the same invented vocabulary. The kind now
crosses as the ledger's own and is parsed once with finance.ParseKind, which
makes that disagreement a compile error.
A refund is not a third kind. The ledger writes two, and a return of unspent
funds is a NEGATIVE deposit — the reversal rule already books it, against the
clearing account the cash arrived through. The refund arm and its Bank leg are
deleted with the word that reached them.
FinanceTxns takes an input now: which books, and how many rows. Sandbox money
and real money live in physically separate files, so the selector has to travel
— a reader that posts test rows into real revenue restates the company's income
and nothing downstream can tell. ListUsage and ListEntries take the same
selector every other read on that ledger already took; they arrived later and
had forgotten it, so a sandbox caller was silently answered from real money.
Six subsystems needed one number — what has this org consumed — and none of
them could get it. Each asked commerce for GET /v1/billing/usage/rollup over
the commerce transport, which co-resident dispatches back into this binary's
router BY PATH. That route is registered nowhere here: commerce's api.Route()
bundle is behind //go:build cloud and never compiled in. So every one of those
reads was a 404 wearing an upstream failure's clothes. Referrals qualified
nobody, affiliates accrued nothing, authors were paid nothing, the usage page
showed a blank month, and the admin fleet read broke — while the figure sat in
a SQLite file in whichever process mounts commerce.
finance_spend answers it there, from the ledger's OWN windowed sum: the same
source the rolling spend cap reads, so a program that qualifies an org on its
spend and the gate that stops that org spending cannot disagree about the
amount. One balance field, because the ledger has one number.
plane.Ask now runs an op that is registered in THIS process directly, through
zip.Serving + zip.Here (zip v1.26.1). A fused binary registers every app's
internal ops on one plane app, so commerce asking commerce for a balance was a
function call wearing an address; dialing our own socket for it encoded a value
we already held and parsed it back into a copy. The decision is made once, in
the one dispatcher — a caller names the op and never learns where it ran.
The three fleet targets were surface-check, describe-apps and openapi-weave:
compound words, and worse, words from the implementation's vocabulary rather
than from what you get. "weave" and "subsets" describe how the thing is built.
"openapi" and "describe" describe the thing.
surface-check -> check regenerate FROM SOURCE, fail on any diff
describe-apps -> describe every app describes itself, and runs zipdoc first
openapi-weave -> openapi compose every app's document into openapi.yaml
The root Makefile's own `describe` target is gone: it chained the generate, the
loop and the compose, which made two definitions of one word. mk/fleet.mk's
describe now absorbs the `go generate -run zipdoc ./...` step, so there is one.
`openapi` deliberately does NOT depend on describe -- it reads the committed
documents and stays a three-second command; `check` is the twelve-minute one
because regenerating is the whole of what makes it a gate.
55 references across 25 files, including hanzo.yml and .hanzo/workflows/cicd.yml,
so the gate CI runs is the gate a person runs by hand.
Also here: zip v1.27.0, which is where the toll's op.invoke seam and the
co-resident Here call come from. The working tree carried
`replace github.com/zap-proto/zip => /home/z/work/zap/zip` while that was
unpublished -- a path that exists on exactly one machine, and go.mod is not the
place to record which one.
Money was gated by HTTP middleware, and HTTP middleware reads the path the
TRANSPORT carried. For a plain REST call that path IS the operation, so the gate
looked right and tested green. It is not the operation anywhere else. One typed
op registered once is seen by the edge as four different things depending on the
door it arrived through, and PriceOf("/mcp") is Undeclared while
Billable("/.well-known/...") is false -- so an op reached over MCP or over the
call plane was free. Not by anyone's decision: the gate was asking about the
wrong value. Free never errors, so nothing said so.
That was survivable while everything spoke HTTP. It stops being survivable the
moment internal hops move to ZAP over UDS, because each converted call site is a
hop that leaves the only surface where the gate exists. The leak would have grown
with the migration rather than with traffic.
zip funnels every projection of a typed handler -- REST, MCP tools/call, the
by-name call plane, CLI LocalInvoke -- through one dispatcher, op.invoke, and
offers one hook inside it. Its argument is a zip.Op: {Method, Path, OperationID},
the operation as a value, identical on all four paths. So DefaultPrice and
Billable are asked there, about op.Method and op.Path, and answer the same thing
however the call arrived. The edge gates stay, because an untyped handler has no
op to ask about, and the toll stands down when the request's own path already
names a declared surface -- one operation, one answer.
Proven on four doors with both controls: an unpriced op moves nothing, and a read
of a priced surface moves nothing, each re-funded first so it cannot pass by
being broke. Neutering Toll fails MCP and ZAP at "balance settled at 25c, want
0c"; restoring it passes.
toll_wire_test.go closes the gap that proof leaves. The four-door test builds its
own app and installs the hook itself, so deleting both lines from serve.go leaves
every subtest passing -- measured, not supposed. Two claims, one gate: the toll
charges correctly, and the server actually mounts it. The second is now pinned,
and goes red naming which of the two lines went missing. A gate nothing mounts is
worth what no gate is worth, and worse, because the passing suite reads as
assurance.
pricing: the embedded catalog was a snapshot from 2026-03-14. /v1/pricing/models
was therefore selling 10 zen4 rows and 26 zen3 rows — both generations retired —
while the five zen5 rows that DO exist carried no price at all. 38 phantom rows
gone; the real SKUs now carry the numbers the gateway serves, derived from zen's
own cost x margin rather than hand-kept.
That regeneration only became correct today: the zen SERVICE was pinned at v1.4.1
while the library was at v1.4.11, so `npm run sync` reads ZEN_URL and would have
copied the same DigitalOcean-era prices straight back in. A sync is only as true
as the service it reads.
console: the playground's six starters suggested `zen-omni` or `zen-coder`, and
the gateway serves neither. Nothing threw — applying a starter falls back to the
selected model when the suggestion is absent — so the card advertised a model
that could never be the one that ran.
I installed this seam and wrote nothing that would notice its removal. ai's
builtin registry DECLARES web_search unconditionally and holds no backend; this
host supplies one. Delete that line and nothing fails to compile, no route 404s,
no test goes red — the tool just answers "not available in this deployment" to
every agent, forever. That is the same shape as every other wired-but-never-
consulted defect in this tree, and I had just re-created it.
installWebSearch takes the searcher as a PARAMETER rather than calling
websearch.Search directly, so the adapter — the truncation and the field mapping
— runs without a network round trip. Production passes the real one; the tests
pass their own.
What the tests hold:
- the seam is CLOSED after install (the deletion case)
- Content maps to Snippet. websearch.Result names the body Content and the tool
contract names it Snippet; lose that rename and every agent gets empty
snippets, which a model reads as "the web had nothing to say about this"
rather than as a bug
- the limit is honoured, and 0/negative mean UNSET rather than "truncate to
nothing"
- an empty upstream is an empty SLICE, not an error — the distinction the whole
tool layer rests on: an error means the capability is unavailable and the
model should say so, an empty list is a genuine finding
TestMountInstallsTheWebSearchSeam is a source assertion and says so. Mount needs
a real zip.App and cloud.Deps (it runs ai's Bootstrap), so no test in this
package constructs one; it pins the call site against deletion while the
behavioural tests pin the adapter. Naming the limit beats implying coverage that
is not there.
Three negative controls run, each failing only its own test: dropping the call
from Mount, blanking the Snippet mapping, and ignoring the limit.
webui/console.go's //go:embed all:dist and webui/dist/ are gone, as is the
CONSOLE_IMAGE Dockerfile stage and the Makefile webui target. cmd/cloud loads
the ACTIVE release of the hanzo-console site into RAM at boot and swaps it on an
atomic.Pointer, resolving it through sites.CurrentResolver() — the same path cd
uses, so there is one answer to 'which console is live'.
Publishing drops from an image build plus a Recreate rollout (a measured 2m15s
api.hanzo.ai outage to ship a CSS fix) to a sub-second release activate that
restarts nothing. Rollback likewise: two independent levers now, one for the
binary and one for the console.
Dockerfile conflict resolved toward this change — ARG CONSOLE_IMAGE is deleted
because deleting it is the point. Kept main's still-true warning that no host
routes to the standalone console Deployment, since rolling it still reaches
nobody.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The stub this removes was invisible for as long as it existed. It compiled. It
had test coverage -- two tests asserted the stub's error and passed. It logged
"deps.IAM -> ZAP RPC" at boot. Every angle said the wire was up except a running
fleet, and nothing in the repo looked from that angle.
So the gate is lexical and blunt, because the defect announces itself in the
words it uses: "not yet wired", "zapc-gen", "TODO" anywhere in clients/ is a
transport somebody is trusting and nobody has built. doc.go is exempt -- it
explains the deleted shape on purpose, and history is how the next reader learns
why the shape is what it is.
The second gate closes the visible half. A peer is addressed by NAME
(plane.Ask -> zip.SocketPath), so there is no endpoint for a deployment to
supply; the eight CLOUD_<X>_ZAP_ADDR knobs were what let a deployment believe it
had chosen a transport. It matches the SHAPE rather than a list of subsystem
names, because a list has to be kept in step and the ninth entry added is the one
that slips through. Note the digits in the suffix match: O11Y is exactly why a
[A-Z_]+ pattern would have missed one, and did, when I first swept for these.
Both were mutation-tested rather than merely observed to pass. Reintroducing the
stub in clients/relapse.go and a CLOUD_IAM_ZAP_ADDR read in config.go fails both
with the reason attached; removing them passes. A gate that has only ever been
green is a gate nobody has checked.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The mint has refused a payment whose two organisations are two organisations since
the split that fixed the receipt read. It refused from the wrong side of the money.
screen.record runs the ledger boundary AFTER next(c) — after commerce's card core has
taken the card — so the one caller the rule exists for, a SuperAdmin acting inside a
customer's org, got the worst of both halves: the charge cleared on the CUSTOMER's
merchant account, the credit had no address to land at, and the door answered 500 over
real money that was now permanently uncreditable. Worse, chargeable again: the retry
the answer invited replays into the same refusal, and a fresh idempotency key takes a
fresh card authorisation, every fifteen minutes for as long as somebody keeps trying.
Both names are already resolved by seen(), off a request that has moved nothing, so
the question is asked in screen.decide — one place, both doors, every projection of
the typed op — before the handler runs. It answers 409 rather than the screen's own
403 because it is not a risk verdict (nothing was scored, the scorer is not asked) and
an operator reading the door's refusals should never have to guess which of the two it
was, and the sentence names POST /v1/admin/grants: a platform operator meaning to fund
a customer is not doing anything wrong, they are at the wrong door. The predicate is
one method, payment.diverged, because the boundary keeps its own reading of the same
rule as a backstop — the pre-charge refusal is a property of the two doors the screen
is composed onto, not of the ledger, and a mint reached any other way still refuses.
Callers that pay for themselves are untouched: principal answers one string for both
names for an ordinary member, a service token and a SuperAdmin at home.
MigrateOrg keeps the exactly-once it documents over an input that MOVES. Its ref is
fixed per org and its amount is whatever commerce held when somebody ran it, so the
honest second run — a retried operator call over a balance the customer has spent from
since — collided with the deposit's conflict rule and came back an error. That rule is
right for a SETTLEMENT, where a ref is one payment and a differing amount is a
different payment wearing a taken key; a backfill ref names an ORG's one cutover and
nothing else may write it. So a conflict on its own ref is read back as "already
carried": the original entry id, no money moved, the same answer on every subsequent
run. depositByRef is untouched.
And a refusal now says WHETHER A RETRY CAN CLEAR IT. "The customer's own retry is the
recovery path" was true of a lost write and false of every refusal that is a fact about
the settlement itself — a charge that settled in another currency or for nothing, a
settlement the door could not identify, a reference already posted for another payment,
the two names that cannot be one name. Telling a customer to retry one of those is
worse than telling them nothing: the same key replays into the same refusal and a fresh
key charges the card again. Each branch answers through uncredited or stranded; the
RECONCILE line carries a terminal field to alert on and names what does settle it (the
admin grant credits the balance, the processor refunds the charge), and the customer is
told not to retry.
finance's deposit conflict is exported as ErrRefTaken, the twin of the usage plane's
ErrRefReused and for its reason: it is the one deposit failure no retry can clear, so a
caller that renders it as a transient fault sends a customer round a loop that cannot
end. The credit door reads it to classify its refusal; the backfill reads its own ref
back through it.
Note a response-shape change already in flight from the deposit-ref work: the ledger's
idempotency lookup returns the whole entry, so POST /v1/admin/treasury/seed's
seedData.Entry (apps/treasury/treasury.go) now carries `postings` on the REPLAY branch
too, where the created branch always had them. Additive, and the two branches agree now.
Tests are mutation-proven. The pre-charge refusal is asserted on a COUNTED charge
handler, because a status code cannot tell a refusal from a refusal that happened after
the money moved: leave the rule only at the boundary and the masquerade row runs the
handler once and answers 500; invert it and both org == ledger rows lose their 200 and
their money. The receipt-read discrimination moves to a direct settle call, which is
where that read still exists now no door reaches it. A structural check pins every
fin.Deposit call site in the package with the reason it may mint, so the next one is
added deliberately.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The mint has refused a payment whose two organisations are two organisations since
the split that fixed the receipt read. It refused from the wrong side of the money.
screen.record runs the ledger boundary AFTER next(c) — after commerce's card core has
taken the card — so the one caller the rule exists for, a SuperAdmin acting inside a
customer's org, got the worst of both halves: the charge cleared on the CUSTOMER's
merchant account, the credit had no address to land at, and the door answered 500 over
real money that was now permanently uncreditable. Worse, chargeable again: the retry
the answer invited replays into the same refusal, and a fresh idempotency key takes a
fresh card authorisation, every fifteen minutes for as long as somebody keeps trying.
Both names are already resolved by seen(), off a request that has moved nothing, so
the question is asked in screen.decide — one place, both doors, every projection of
the typed op — before the handler runs. It answers 409 rather than the screen's own
403 because it is not a risk verdict (nothing was scored, the scorer is not asked) and
an operator reading the door's refusals should never have to guess which of the two it
was, and the sentence names POST /v1/admin/grants: a platform operator meaning to fund
a customer is not doing anything wrong, they are at the wrong door. The predicate is
one method, payment.diverged, because the boundary keeps its own reading of the same
rule as a backstop — the pre-charge refusal is a property of the two doors the screen
is composed onto, not of the ledger, and a mint reached any other way still refuses.
Callers that pay for themselves are untouched: principal answers one string for both
names for an ordinary member, a service token and a SuperAdmin at home.
MigrateOrg keeps the exactly-once it documents over an input that MOVES. Its ref is
fixed per org and its amount is whatever commerce held when somebody ran it, so the
honest second run — a retried operator call over a balance the customer has spent from
since — collided with the deposit's conflict rule and came back an error. That rule is
right for a SETTLEMENT, where a ref is one payment and a differing amount is a
different payment wearing a taken key; a backfill ref names an ORG's one cutover and
nothing else may write it. So a conflict on its own ref is read back as "already
carried": the original entry id, no money moved, the same answer on every subsequent
run. depositByRef is untouched.
And a refusal now says WHETHER A RETRY CAN CLEAR IT. "The customer's own retry is the
recovery path" was true of a lost write and false of every refusal that is a fact about
the settlement itself — a charge that settled in another currency or for nothing, a
settlement the door could not identify, a reference already posted for another payment,
the two names that cannot be one name. Telling a customer to retry one of those is
worse than telling them nothing: the same key replays into the same refusal and a fresh
key charges the card again. Each branch answers through uncredited or stranded; the
RECONCILE line carries a terminal field to alert on and names what does settle it (the
admin grant credits the balance, the processor refunds the charge), and the customer is
told not to retry.
finance's deposit conflict is exported as ErrRefTaken, the twin of the usage plane's
ErrRefReused and for its reason: it is the one deposit failure no retry can clear, so a
caller that renders it as a transient fault sends a customer round a loop that cannot
end. The credit door reads it to classify its refusal; the backfill reads its own ref
back through it.
Note a response-shape change already in flight from the deposit-ref work: the ledger's
idempotency lookup returns the whole entry, so POST /v1/admin/treasury/seed's
seedData.Entry (apps/treasury/treasury.go) now carries `postings` on the REPLAY branch
too, where the created branch always had them. Additive, and the two branches agree now.
Tests are mutation-proven. The pre-charge refusal is asserted on a COUNTED charge
handler, because a status code cannot tell a refusal from a refusal that happened after
the money moved: leave the rule only at the boundary and the masquerade row runs the
handler once and answers 500; invert it and both org == ledger rows lose their 200 and
their money. The receipt-read discrimination moves to a direct settle call, which is
where that read still exists now no door reaches it. A structural check pins every
fin.Deposit call site in the package with the reason it may mint, so the next one is
added deliberately.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
clients/rpc.go declared ZAP RPC for nine subsystems -- iam, base, commerce,
ai, o11y, vfs, mq, payments, vault -- and every one of its thirty-odd methods
returned the same thing:
cloud: ZAP RPC client for iam@iam.hanzo.svc:9653 not yet wired
(zapc-gen pending) -- VerifyJWT
It was live wiring, not dead code. BuildDeps selected it whenever
CLOUD_<X>_ZAP_ADDR was set, and logged "deps.IAM -> ZAP RPC" when it did. So
the single signal an operator had said the wire was up, while nothing crossed
it. Setting CLOUD_VFS_ZAP_ADDR replaced a working S3-backed blob client with
one that failed every Put and Get. A missing client is diagnosed in seconds;
one that reports itself configured and then fails elsewhere is diagnosed in an
incident.
pickKMSClient already made this argument and already acted on it -- it dropped
CLOUD_KMS_ZAP_ADDR for the plane, with a comment saying a stub that looks
configured is worse than none. The argument was never specific to KMS. This
applies it to the other eight.
The peer plane is the transport, and it is the only one: plane.Ask over the
peer's own socket, addressed by NAME, with plane/gen emitting a typed client
per app. It takes no endpoint, which is why removing eight config fields and
eight env knobs removes nothing real -- no manifest in universe sets any of
them, so no deployment loses a path it had.
What each dependency does now is what it did: co-resident when mounted here,
and the disabled stub -- which says "enable the subsystem" -- when not. Payments
and vault stay non-nil because commerce nil-checks them, but they stop claiming
a cardholder-data transport that never carried a byte; nothing in the tree
invokes CreateIntent, ConfirmIntent, GetIntentStatus or Charge at all.
The two tests that covered this asserted the fake. One pinned that a configured
address yields a client which is NOT disabled -- true, and true only because it
failed in a transport-specific way instead. Both now assert the honest
property: a subsystem that is not here reports itself DISABLED, and no address
can say otherwise.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The comment above this mount said byte-identical patterns "merge silently,
first wins". Measured, they do not. zip refuses the whole program and names
every conflicting pair with file:line and the group path it came through:
zip: this program does not compose, so it has no projection:
zip: GET /v1/commerce/collection: declared by "/collection" at
rest/rest.go:133 (via root -> /v1/commerce -> /collection) and by
"/collection" at rest/rest.go:133 (...)
The correction matters because the false version makes an addition look free.
Commerce's api.Route already registers order, customergroup, promotion, region,
taxregion, pricelist, claim, role and the rest; the admin needs them at
/v1/commerce, so someone will reach for this leaf. Adding a kind here while
api.Route still registers it does not merge — it takes the STANDALONE down at
boot, in a different binary from the one that changed. One address, one
declaration.
WHERE the refusal lands differs by zip and both are before traffic: this tree
resolves v1.25.1, which accepts the second Route() call and refuses at
COMPOSITION; commerce standalone pins v1.24.2, which refuses inside Route().
That difference is not academic — the first version of this test only
registered, read "accepted", and would have shipped the claim that the rule was
gone. It now asks for the composition, and it is proven by construction: remove
the second Route() call and it fails.
Also drops the stale "two equal-specificity params with different names" framing
from the sibling guard's message, which described a narrower collision than the
one that actually fires.
console.hanzo.ai is this binary and stays this binary — same host, same
origin, /v1 still answered here, session cookie still first-party. What
moves is the SOURCE OF THE BYTES.
The console was //go:embed'd from webui/dist, overwritten at image build
from a pinned console-embed. That welded the frontend's lifecycle to the
backend's: a CSS fix cost a ~22-minute cloud build plus a Recreate
single-replica rollout — a measured 2m15s outage of api.hanzo.ai to ship
a stylesheet. It also needed a whole discipline to stay honest (never
:latest — v1.801.215 built 12 minutes early and silently shipped the
previous console, green — and never re-point a cut tag), a discipline
that existed only because the console's identity lived in a Dockerfile.
It is a site release now, through the site-release subsystem already in
this repo: the ACTIVE release of the hanzo-console site, read from the
same S3 the site edge reads, resolved through the same registry
cd.hanzo.ai resolves through (sites.CurrentResolver — one resolution
path, not a second). Publish is under a second, rollback faster, and
neither builds nor restarts anything.
webui keeps the whole handler — per-Host white-label <title>,
static-export route shells, precompressed negotiation,
cache policy, the API-namespace 404 — and takes the FS as
an argument. Still a leaf. The cached index.html is gone:
against a live release a cached shell names the previous
build's chunk hashes, so it is read per request now, via
the one HTML-shell path the route shells already used.
webui/release loads the release whole into RAM (68 files / 9.3 MB) and
swaps it atomically when the active pointer moves. The
steady state is one resolver call per poll; bytes are
re-read only when the release actually changed.
Required at the front door, reported in a child. cmd/cloud starts
projects (idempotent; bounded, so it refuses rather than hangs) and
fails with the reason if the release will not load. cloud.Listen cannot:
a child would have to ask projects, only the host can start projects,
and the broker every child needs boots first — so a child logs the
reason and answers 503 naming it. Never a blank shell.
Also: _next/static and assets/ are immutable by PATH at the site edge.
The basename rule needs a separator before the hash run, which Next does
not emit, so 5 of 44 _next/static objects were served max-age=3600 and
re-fetched hourly forever. The prefix stays narrow — _next/ also holds
_next/data and a build-id dir, and this path serves arbitrary tenant
sites, where a year of immutability on a service worker outlives every
deploy.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The 119 derived sentences were true about tenancy and silent about everything
else, and the seventeen families they cover are not uniform on the axis that
matters to a caller. Read off mount.go and commerce's api/resources:
- transfer, wallet and webhook are ADMIN-gated (TokenRequired(permission.Admin)).
A plain member's token is refused 403 there, on reads as much as writes.
- collection, discount, product, variant, saleschannel and stocklocation sit
behind paywall.Require: no active subscription, trial or redeemed invite is
402 subscription_required, and an unreadable billing store fails closed 503.
- collection, product, return, subscriber and variant additionally carry a
DefaultPermissions table that CheckPermissions enforces per method (403).
The other twelve have no table, and the miss logs and ALLOWS — so claiming a
scope check on those would be prose the server does not honour.
One sentence repeated seventeen times said none of it, which is worse than the
bare operationId it replaced: it told a reader that a plain token may write a
wallet and that the only refusal on a product is 404. A description nothing
downstream can check is believed. So the table now carries the gate — the only
thing that actually varies — and the sentence is generated from it.
Three corrections to the shape prose while reading the handlers:
POST on the item address is the METHOD-OVERRIDE door, not "an address that
accepts every method". With no override it runs r.Patch; `_method` (form or
query) or X-HTTP-Method-Override set to PUT, PATCH or DELETE runs THAT one, so a
POST here can DELETE the record. An invalid override is ignored, not 405'd —
IsValidMethodOverride only admits those three and the method stays POST.
LIST returns the Pagination envelope {page, display, count, models, facets}, not
a bare array, and it fails closed to an EMPTY PAGE with 200 when no org
namespace resolves — so empty `models` does not mean "no records exist".
PUT decodes onto an empty record carrying only the existing key, which is why
omitted fields are reset; PATCH decodes over the loaded record. Both 404 on an
unknown id. DELETE writes a copy aside under an internal deleted key first, so
the row stops answering but is not erased.
Also: a family that outgrows the table comes OUT of it rather than gaining a
Describe as well — Describe panics on a duplicate key, so doing both aborts the
binary at init. The old comment said the explicit one "wins", which would have
sent the next reader into that panic.
Regenerated: openapi.yaml and plugin/commerce/openapi.json. 119 operations, none
bare; regenerating from source reproduces these bytes exactly.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every chat and vision SKU (zen5-mini, zen5-flash, zen5, zen5-coder, zen5-pro,
zen-vl, zen-guard) now routes through openrouter. DigitalOcean cannot be billed
right now, and a provider we cannot pay is not a route.
It is cheaper on every rung, because openrouter publishes each upstream's own
rate: zen5 quotes 2.28/7.26 per MTok where it quoted 4.176/13.2. Margin is
untouched at 3x — the saving reaches the caller.
OPENROUTER_API_KEY must be in the environment for this image. zen's resolver
reads env first and KMS only as a fallback, so a missing key is not a degraded
route, it is a 401 on every chat. The universe values carry it in the same
commit that pins this image.
Four SKUs stay on DigitalOcean and nothing can move them: openrouter is a
chat-completions aggregator with no /embeddings, /images/generations,
/videos/generations or /rerank, so zen-embedding, zen-image, zen-video and
zen-rerank have no rung to move to.
A usage debit was idempotent on the caller's X-Request-Id. That header is the
CLIENT's: zip's RequestID middleware keeps an incoming one verbatim and the
gateway CORS-allows it from a browser, so the payer chose the key the ledger
deduped on. Pin one value and every call after the first replayed into the
first one's entry — the work ran, the wallet did not move, and the spend cap,
which sums that ledger, never rose. It reached finance from every metered
surface: the edge BillingGate, the net/http meter, ~25 ResourceMeter callers,
and zen, whose own field comment called the header "idempotency".
The key also carried no subject. (kind, program, ref) held program empty for
usage, so ONE namespace per org held every subject's refs and the first subject
to take a ref owned it for everyone: two people in one org sharing a ref meant
the second one's call debited nobody.
Both halves are the same mistake — the key named neither who paid nor which act
— and both are fixed where the key is formed rather than at the 27 places that
supply attribution:
- The scope is the WALLET. usageProgram is walletAcct, so the account a ref is
unique within IS the account the posting debits; the key and the money can
no longer name two wallets.
- The name is the SERVER's. metering.Usage.Ref carries it, Usage.Seal mints
it, and Record seals what it is given — one seam, so every caller is fixed
without touching one of them. Seal is idempotent, so a caller holding a
server-assigned act id (a message row's id, an x402 settlement id, a domain
registration ref) sets Ref instead and its own retry stays exactly-once.
- RequestID stays, as what it always was: correlation, on the log line and the
attribution wire, never the money.
- A ref names ONE charge. A hit whose amount differs is ErrRefReused, not the
first entry — the rule the deposit already states, so the GPU charge, the
one surface whose key a caller may name, answers 409 instead of handing back
work nobody paid for.
Exactly-once is preserved and proven both ways: three attempts at one sealed act
debit once; two acts identical in every field debit twice.
Mutation-proven: key the ledger on the header again and 20 pinned calls bill 1¢
instead of 20¢; drop the subject from the key and the second subject's wallet
never moves; make the mint non-stable and one act charges three times; drop the
amount check and a $5.00 charge under a $0.30 name is answered "ok".
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A usage debit was idempotent on the caller's X-Request-Id. That header is the
CLIENT's: zip's RequestID middleware keeps an incoming one verbatim and the
gateway CORS-allows it from a browser, so the payer chose the key the ledger
deduped on. Pin one value and every call after the first replayed into the
first one's entry — the work ran, the wallet did not move, and the spend cap,
which sums that ledger, never rose. It reached finance from every metered
surface: the edge BillingGate, the net/http meter, ~25 ResourceMeter callers,
and zen, whose own field comment called the header "idempotency".
The key also carried no subject. (kind, program, ref) held program empty for
usage, so ONE namespace per org held every subject's refs and the first subject
to take a ref owned it for everyone: two people in one org sharing a ref meant
the second one's call debited nobody.
Both halves are the same mistake — the key named neither who paid nor which act
— and both are fixed where the key is formed rather than at the 27 places that
supply attribution:
- The scope is the WALLET. usageProgram is walletAcct, so the account a ref is
unique within IS the account the posting debits; the key and the money can
no longer name two wallets.
- The name is the SERVER's. metering.Usage.Ref carries it, Usage.Seal mints
it, and Record seals what it is given — one seam, so every caller is fixed
without touching one of them. Seal is idempotent, so a caller holding a
server-assigned act id (a message row's id, an x402 settlement id, a domain
registration ref) sets Ref instead and its own retry stays exactly-once.
- RequestID stays, as what it always was: correlation, on the log line and the
attribution wire, never the money.
- A ref names ONE charge. A hit whose amount differs is ErrRefReused, not the
first entry — the rule the deposit already states, so the GPU charge, the
one surface whose key a caller may name, answers 409 instead of handing back
work nobody paid for.
Exactly-once is preserved and proven both ways: three attempts at one sealed act
debit once; two acts identical in every field debit twice.
Mutation-proven: key the ledger on the header again and 20 pinned calls bill 1¢
instead of 20¢; drop the subject from the key and the second subject's wallet
never moves; make the mint non-stable and one act charges three times; drop the
amount check and a $5.00 charge under a $0.30 name is answered "ok".
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Mounting the bundle was not enough and the last release proved it. The light
host routes by manifest.Apps: commerce's row names each leaf it serves, `ai`
holds the bare "/v1", and longest-prefix wins. A leaf nobody names goes to ai's
catch-all, so commerce sat there serving routes the host never handed it.
Production said so precisely — /v1/commerce/tenant (named) answered 200 from
commerce in the same second /v1/commerce/product (unnamed) answered 404 from ai.
So name them, which is the rule this file already states: nobody claims the bare
/v1/commerce remainder, every leaf is named deeper. Seventeen leaves, read off
the live router rather than guessed. Each is gated in Mount — token, admin token
and paywall — so this changes who is ROUTED, not who is allowed.
The guard is the real repair. Its first version asserted the bundle BINDS
/v1/commerce/product; it did, and production still 404'd, because binding says
nothing about the host's routing table. It now asserts every bound leaf is named
on commerce's manifest row, and it is proven by reverting the row: 17 failures,
one per leaf, then green.
That closes a gap the existing oracle cannot see. manifest/router_test.go
compares routing against plugin/commerce/openapi.json — a GENERATED file — so a
bundle whose spec has not been regenerated is invisible to it and it passes
vacuously, which is exactly what it did here. Reading the live router needs no
regeneration step, so an upstream kind fails the moment it is linked.
Separately and NOT fixed here: `describe` for commerce is already red without
this change. It refuses 128 operations that "say nothing about themselves", 9 of
which are pre-existing (/v1/billing/methods, /v1/billing/wire,
/v1/billing/crypto/*, /_/commerce/providers/{name}) and untouched by this commit.
So plugin/commerce/openapi.json is stale at 73 paths and cannot be regenerated
until those operations carry doc comments. Routing does not depend on it; the
published spec and the generated SDKs do.
console-embed 8.5.58 is the launch build: the sidebar down to chat, the builder,
models, keys, usage, billing and settings with everything else behind the beta
flag; a playground that offers only what the gateway routes and holds frontier
prices behind a plan; the org's own logo in the mark row; search across the whole
catalog. This pin named 8.5.48, so none of it was reachable.
The routing fact is the one worth writing down: hanzo-domains.yaml sends
console.hanzo.ai to `service: cloud`, and nothing routes to `service: console`
at all. The SPA a user loads is the one //go:embed bakes in here, so a console
release ships only when this line moves. console:v8.5.59 was built, pinned and
rolled to Ready today and served no one.
TWO THINGS STOOD BETWEEN THE FLEET DOCUMENT AND SOURCE, both revealed only once
licensing stopped failing first — surface-check stops at the first app that cannot
project, so everything behind it was invisible rather than passing.
commerce publishes 17 generated CRUD families over the same store, 7 operations
each: 119 sentences differing in one word. They are DERIVED now, for the reason
openapi.DescribeSPA derives the two SPA addresses — prose written 119 times drifts
119 ways, and the next family the module adds would arrive undescribed and stop the
gate again. Adding one is a line in `resources`. The vocabulary stays plain on
purpose: these are generic store resources, so each sentence states the shape (what
the address is, what the method does, what the id means) and stops. A resource that
earns its own prose keeps an explicit Describe above — Describe panics on a
duplicate, so the hand-written sentence and the loop cannot both claim an address.
The weave then refused for a different reason, and it was right to: schema
"Artifact" meant a RELEASE artifact in licensing (download_url, cosign_signature)
and a RESEARCH artifact in research (git_sha, run_id, retention_class). Schema names
are global in the woven document, so every generated SDK would have bound whichever
it read last — one name, two shapes, silently. research's is the one this repo owns,
so it becomes ResearchArtifact; licensing's stays, being the external module's.
Regenerated: openapi.yaml, the affected subsets and the floor.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two money bugs on the deposit path, one introduced and one waiting.
A card top-up names TWO organisations and the split that fixed the receipt read
left the MINT choosing between them. The charge clears on the EFFECTIVE org's
merchant account and commerce writes its receipt there; the credit is addressed
to principal.WalletOf. For every caller but a masquerading SuperAdmin those are
one string. For that one they are two, and the deposit took the second: a
customer's card funded a platform admin's spendable balance and the door
answered 200. Neither address is right — crediting the ledger moves a customer's
money into the admin's books, crediting the charged org has a masqueraded session
top up the org it is only inspecting — so the mint refuses a payment whose two
names are not one name, with the door's own 500 and a RECONCILE line naming both.
A SuperAdmin funding a customer uses the admin grant, which is a credit that
states whose it is. The refusal sits AFTER the receipt read, so the masquerade is
still the one caller that can tell payment.org from payment.ledger and the read
this replaces cannot quietly go back to the payer's namespace. Callers that pay
for themselves are untouched: org == ledger, so the guard cannot fire.
A deposit's idempotency key is (kind, program, ref) and carries neither subject
nor amount, so two DIFFERENT payments naming one ref in one org's books collide.
Both the in-transaction replay branch and the recovery read answered the second
of them with the FIRST one's entry id: alice's $1 posted, bob's $500 was told
SUCCESS, bob's wallet stayed at zero and nothing said a payment had been dropped.
A ref hit is a REPLAY only when it is the same money to the same wallet; anything
else is refused, because crediting anyway breaks the exactly-once the ref exists
to give and answering with the other payment's entry is the swallow. Nothing
reaches it today — the settlement ref is a Square payment id — but the webhook,
the backfill and the credit RPC all pick their own. Both callers now read the ref
through one function, so they cannot answer the question two ways.
The subject lives in the entry's legs, so the ledger's idempotency lookup returns
the whole entry the way its journal listing does rather than a header the caller
must complete; one postings read serves both, over a connection or inside a
transaction.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two money bugs on the deposit path, one introduced and one waiting.
A card top-up names TWO organisations and the split that fixed the receipt read
left the MINT choosing between them. The charge clears on the EFFECTIVE org's
merchant account and commerce writes its receipt there; the credit is addressed
to principal.WalletOf. For every caller but a masquerading SuperAdmin those are
one string. For that one they are two, and the deposit took the second: a
customer's card funded a platform admin's spendable balance and the door
answered 200. Neither address is right — crediting the ledger moves a customer's
money into the admin's books, crediting the charged org has a masqueraded session
top up the org it is only inspecting — so the mint refuses a payment whose two
names are not one name, with the door's own 500 and a RECONCILE line naming both.
A SuperAdmin funding a customer uses the admin grant, which is a credit that
states whose it is. The refusal sits AFTER the receipt read, so the masquerade is
still the one caller that can tell payment.org from payment.ledger and the read
this replaces cannot quietly go back to the payer's namespace. Callers that pay
for themselves are untouched: org == ledger, so the guard cannot fire.
A deposit's idempotency key is (kind, program, ref) and carries neither subject
nor amount, so two DIFFERENT payments naming one ref in one org's books collide.
Both the in-transaction replay branch and the recovery read answered the second
of them with the FIRST one's entry id: alice's $1 posted, bob's $500 was told
SUCCESS, bob's wallet stayed at zero and nothing said a payment had been dropped.
A ref hit is a REPLAY only when it is the same money to the same wallet; anything
else is refused, because crediting anyway breaks the exactly-once the ref exists
to give and answering with the other payment's entry is the swallow. Nothing
reaches it today — the settlement ref is a Square payment id — but the webhook,
the backfill and the credit RPC all pick their own. Both callers now read the ref
through one function, so they cannot answer the question two ways.
The subject lives in the entry's legs, so the ledger's idempotency lookup returns
the whole entry the way its journal listing does rather than a header the caller
must complete; one postings read serves both, over a connection or inside a
transaction.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
console-embed 8.5.58 is the launch build: the sidebar down to chat, the builder,
models, keys, usage, billing and settings with everything else behind the beta
flag; a playground that offers only what the gateway routes and holds frontier
prices behind a plan; the org's own logo in the mark row; search across the whole
catalog. This pin named 8.5.55, so none of it was reachable.
The routing fact is the one worth writing down: hanzo-domains.yaml sends
console.hanzo.ai to `service: cloud`, and nothing routes to `service: console`
at all. The SPA a user loads is the one //go:embed bakes in here, so a console
release ships only when this line moves. console:v8.5.59 was built, pinned and
rolled to Ready today and served no one.
hanzoai/licensing used to register ONE untyped wildcard — app.All("/v1/licensing/*")
over its own net/http mux — and an untyped route cannot carry prose, so its seven
method descriptions were declared here, in cloud, keyed to that pattern.
At v0.1.10 it types them instead: app.Group("/v1/licensing") with zip.Post(g,
"/issue", ...) and siblings. A typed op carries its sentence in the handler's own
doc comment and zipdoc lifts it, so the seven declarations in this file now name no
operation at all. That is worse than missing prose: they render nowhere while
reading, in the source, exactly as though they had landed.
Deleted with the route they described. The gate that caught it is the same one that
catches the opposite case — an app that cannot project its own document — which is
why it is worth having pointed at both directions.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nothing else in this package calls commerceresources.Route, so the two ways it
can fail were both invisible. Registration into a shared router panics on two
equal-specificity params with different names — a BOOT panic that passes every
build check. And the routes simply going missing is what 404'd every admin data
view in production.
Reads the route TABLE rather than driving a request: a request exercises
commerce's datastore and tenant resolution, and a panic in there reads as "the
route is missing" when the route is fine.
Trailing slashes are trimmed because the bundle binds /v1/commerce/product/ and
every client calls /v1/commerce/product. Those are one route only because
fiber's StrictRouting defaults false and zip never sets it, so that is pinned
too — defensively: zip.Config exposes no such field, so unlike the absence
assertion (proven to fire on a bogus kind) it cannot be falsified today.
api.hanzo.ai/v1/commerce is THE commerce endpoint, and commerce is a PLUGIN of
this binary — there is no commerce backend pod. So the merchant surface is
reachable through this binary or it is not reachable at all. It was not: every
admin data view 404'd in production while sign-in, catalog and billing answered
200. Honest, and total.
Mounts commerce's api/resources LEAF, not its api.Route. Route also binds an
index route, a permissive CORS policy and a wildcard OPTIONS onto whatever
router it is handed — right for a process that owns its tree, wrong in this
shared one, where byte-identical patterns merge silently and two
equal-specificity params with different names panic at registration. Importing
that package would also drag checkout, subscriptions and thirdparty/netlify,
which this binary deliberately does not carry.
commerce v1.50.7 -> v1.50.11 for the leaf.
Package tests unchanged: 40 failures before and 40 after, the identical set —
all pre-existing ledger/payments/settlement cases on this line.
Five checks were measuring something other than what ships.
captable pinned two MCP tool names in their pre-v1.26 spelling. zip v1.26.0
settled ONE rule — an id derives from the absolute path the occurrence answers at
— so the tools are patch_v1_captable_stakeholders_by_id and
post_v1_captable_rounds_by_id_close. bab535c2 taught the bots and agents
projections that rule and missed this file. The segments still name the same two
ops unambiguously and what the test proves is unchanged: the op is addressable
through its In alone, over a transport with no URL.
pricing declared /v1/pricing/datastore and /v1/pricing/services and proved
neither, so the byte-identity golden had quietly stopped covering two sections.
They are covered now. The scanner also skips fiber's "/" route, which is
middleware territory — every app.Use rides it as a handler chain, by design — and
never was a section address.
projects pinned the template repository at hanzo-templates/synapse. It is not
there: 4203c7f2 found 45 of 66 gallery entries pointing at repositories that do
not exist, verified every source against the GitHub API, and repointed the static
sites to hanzo-apps/template-<slug>, which is that org's convention. Only 21
templates (expo-*, flutter*, swiftui*, desktop-*) ever moved to the
hanzo-templates layout this line was written against — the catalog is mixed by
design, and the variant case beside it still pins hanzo-templates/prism-react.
The assertion's intent is untouched: a fork seeds the REPOSITORY, not the gallery
page that is an HTML 404, and the provider is github.
git's ZAP round trip installed its identity shim through app.Fiber().Use before
Mount. At zip v1.26 Fiber() materialises a DRAFT router and every later
registration invalidates it, so the next call builds a new *fiber.App and the shim
was written onto a throwaway. The handlers saw no X-Org-Id, principal.Org failed,
and createRepo answered a 403 that zapface rewrites to a bare UNAUTHORIZED naming
none of it. It is a zip component now, installed by the same compose the server
runs (f6c9605b, same class).
And apps/commerce/risk.go joins allowedRequestUses with its reason. The fraud
screen in front of the typed mint op reads the payer, the door actually reached
and the jurisdiction signals a credit decision is made on — an identity gate
reading strictly more than the org, which is one of the sanctioned reasons. The
amount comes off the decoded In and the settlement off the returned receipt, both
deliberately not read from the wire, so the request is consulted for exactly the
facts no projection can carry on a type.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
POST /v1/content/generate published an operationId and nothing else — no request
schema, no response schema, no MCP tool input, no CLI flags, an untyped `any` in
every generated SDK — for the ONE agentic call in the content loop. It stayed a
raw handler on a stated belief: that a studio-render billing denial answers the
platform's nested {"error":{"code","message"}} at 402/503, and a typed op can
answer only its Out schema or zip's flat {status,code,error}.
That belief is out of date. cloud.Denied carries the money wire's own status and
body as an ERROR — the one refusal channel a typed op has — and cloud.DenyEnvelope
writes those bytes back verbatim; apps/projects, apps/dataset and apps/risk
already pair them. So the op is typed and the 402 body is byte-for-byte what it
was, which TestGenerate402IsTheMoneyWireBody asserts by shape: exactly one key,
`error`, nested {code,message}, code insufficient_balance. Off the HTTP path
(MCP, CLI) deniedErr.Unwrap keeps the status and the sentence.
Two details the conversion turns on. 402 is DECLARED so a generated client knows
the refusal is a shape it can read; it is never the status this op RETURNS, since
statusFor takes Statuses[0] when the Out states none, so a success is still 201
and the 402 reaches the wire from the envelope. And every GenerateInput field is
`url:"-"`, because zip's binder fills an In field from the QUERY as well as the
body: without it the route silently starts accepting ?doctype= and ?source_media=,
inputs the raw handler's c.Bind never took — and a source_media reachable from a
query string is an SSRF surface reachable from a link. A wire widening no status
code shows, pinned by TestGenerateStillReadsTheBodyAndOnlyTheBody.
The prose moves from the openapi.Describe init onto the handler, which is where
zipdoc lifts it from for a typed op, so the description is declared once rather
than beside the wire fact that no longer holds. The regenerated document is the
point of the change: /v1/content/generate now carries a GenerateInput requestBody
and GenerateResult on 201.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three surfaces installed a gate on a group and then registered the routes it was
meant to cover somewhere else, which zip v1.26 does not forgive: a Use is scoped
to the group INSTANCE it was installed on and its children, and is never matched
by prefix across the app. Measured, not reasoned: a probe registering the same
prefix twice shows the second group's routes with none of the first's middleware,
and a route on the App with none of any group's.
books rebuilt `app.Group("/v1/books")` in scan.go and bank_api.go, so six reads —
bank/transactions, inbox, vendors, rules, transactions, bank/unreconciled — and
the vendor/rule upserts answered per-org money WITHOUT Cache-Control: no-store,
while /gl and /questions next door carried it. Each file now installs noStore on
the group it builds. It keeps building that group from a literal prefix because
cmd/zipdoc files a doc comment by the prefix it can READ in the file: passing the
group down as a parameter drops every leaf below out of the document and the MCP
tool list. The five RAW handlers stay on the App with their whole paths for the
same generator reason in reverse — a raw handler on a readable prefix is one
zipdoc describes FOR it, and the only doc comment at that call site is
cloud.Handle's "binds a Service-scoped handler to a route", which would ship as
what /v1/books/scan does.
company and validators declared their collection root on the App, outside the
group carrying the cap and the identity gate. An oversized body at /v1/company
answered 400 from the decode instead of 413, and an anonymous POST to
/v1/validators with a malformed body answered 400 where the surface has always
answered 403 — the refusal has to precede the decode, and it was not running at
all. Neither root can hang off its own group with an empty leaf, which joins to
"/v1/company/" and moves the published address. So the gate rides the ROUTER:
With() wraps the leaf it registers and installs nothing at "/v1". That distinction
is load-bearing — a Use there gates every other subsystem's /v1 routes, and
cloud's own ownership gate refuses it in as many words. .Group("/v1") is then only
an address, and one zipdoc can read, so both ops keep their prose. Same shape
apps/sync already uses.
Pinned by TestReadsAnswerTheirFrozenEnvelope, TestVendorAndRuleUpsertsEchoThe-
NormalizedRow, TestBodyCapCoversJSONNotTheDeck and TestIdentityRefusalStill-
PrecedesTheBody.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Commerce is a plugin of cloud, so its merchant surface is reachable through this
binary or it is not reachable at all. It was not. There is no commerce backend
pod; commerce-api.hanzo.ai routes here; and this embed carried no resource
bundle — so every admin data view 404'd in production while sign-in, catalog and
billing all answered 200. Honest, and total: /v1/product AND /v1/commerce/product
both 404'd, so no client-side prefix change could have reached anything.
Products, variants, collections, discounts, sales channels, stock locations,
subscribers and webhooks now answer at api.hanzo.ai/v1/commerce/<kind> — the
endpoint every Hanzo surface is told to use.
It mounts the resources LEAF (commerce v1.50.11, api/resources), not api.Route.
Route also binds an index route, a permissive CORS policy and a wildcard OPTIONS
onto whatever router it is handed: right for a process that owns its tree, wrong
in this shared one, where byte-identical patterns merge silently and two
equal-specificity params with different names panic at registration. Importing
that package would also drag checkout, subscriptions and thirdparty/netlify —
weight this binary deliberately does not carry, and the real reason the bundle
was skipped rather than a decision anyone made.
The leaf's dependency graph contains netlify zero times; api's still contains it
twice. That boundary is the whole mechanism.
productEvents is nil here: the storefront publish loop belongs to the
standalone's event bus, and the CRUD does not depend on it.
Package tests unchanged: 23 failures before this commit and 23 after, the same
set — all pre-existing ledger/payments/settlement cases on forge/main.
A card top-up names TWO organisations and the settlement held one value for both.
commerce writes the receipt under the EFFECTIVE org — the org the door is acting in,
resolved by iammiddleware for the browser route and by payingOrg for the typed op —
while the balance it funds is principal.WalletOf's, which for a platform SuperAdmin is
its own books. Those are the same string for every caller except a SuperAdmin
masquerading into a customer's org, and for that one the receipt read looked in the
admin's books for a row commerce had written in the customer's: not found, so the door
refused a charge that had already cleared. Permanently — the retry replays the same
receipt into the same absent namespace, and a fresh idempotency key charges the card
again. The payment now carries both names as principal names them, `org` for the data
namespace and `ledger` for the billing key: the receipt is read from org, the deposit
lands on (ledger, subject), and the screen and the accrual key on ledger as before.
A settled charge the credit refuses is still taught to the risk model. Teaching is
telemetry about a payment that HAPPENED, so gating it on the credit dropped real
settlements — a charge in a currency this ledger does not hold, a receipt it could not
read — out of the payer's velocity, and a caller who could provoke the refusal could
pay all day and accrue nothing. Both doors now run one value (screen.record) that holds
the order: the money first, because only it can refuse the door, then the model, always.
A deposit that fails on money ALREADY in the books answers with the money. finance
dedups a replay inside its insert transaction; it cannot cover a transaction that never
reached its own read — the write lost to a concurrent poster of the same Ref, or the
request context that died between the charge and the post. Both leave the ref credited
and the caller holding an error for it, which at a credit door is a 500 on a card that
cleared. A failed deposit now asks the only question it has left, on a context detached
from the caller's because the caller's is what may have just died.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
(cherry picked from commit de1e5c4341)
A card top-up names TWO organisations and the settlement held one value for both.
commerce writes the receipt under the EFFECTIVE org — the org the door is acting in,
resolved by iammiddleware for the browser route and by payingOrg for the typed op —
while the balance it funds is principal.WalletOf's, which for a platform SuperAdmin is
its own books. Those are the same string for every caller except a SuperAdmin
masquerading into a customer's org, and for that one the receipt read looked in the
admin's books for a row commerce had written in the customer's: not found, so the door
refused a charge that had already cleared. Permanently — the retry replays the same
receipt into the same absent namespace, and a fresh idempotency key charges the card
again. The payment now carries both names as principal names them, `org` for the data
namespace and `ledger` for the billing key: the receipt is read from org, the deposit
lands on (ledger, subject), and the screen and the accrual key on ledger as before.
A settled charge the credit refuses is still taught to the risk model. Teaching is
telemetry about a payment that HAPPENED, so gating it on the credit dropped real
settlements — a charge in a currency this ledger does not hold, a receipt it could not
read — out of the payer's velocity, and a caller who could provoke the refusal could
pay all day and accrue nothing. Both doors now run one value (screen.record) that holds
the order: the money first, because only it can refuse the door, then the model, always.
A deposit that fails on money ALREADY in the books answers with the money. finance
dedups a replay inside its insert transaction; it cannot cover a transaction that never
reached its own read — the write lost to a concurrent poster of the same Ref, or the
request context that died between the charge and the post. Both leave the ref credited
and the caller holding an error for it, which at a credit door is a 500 on a card that
cleared. A failed deposit now asks the only question it has left, on a context detached
from the caller's because the caller's is what may have just died.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
studio, bot and platform leave appProducts. No catalog version licenses them, and
nothing enforces them either — RequireProduct has zero production callers and the
three sites that would call it say so outright ("DEFERRED — DO NOT ENABLE YET ...
flip on once the catalog licenses it"). Listing them bought no strictness: it made
CheckEntitlement answer Active:false for every org on every tier, enterprise
included, so the projection reported all three LOCKED to every customer behind a
purchase that does not exist. When they launch they come back ATOMICALLY — this
list and the granting plans in one change, never half-wired.
A NOT-GATED PRODUCT IS NOT A LOCKED ONE, and that distinction is why the keys stay.
@hanzogui/shell maps over these names unconditionally, so dropping them from the
projection would break the console rather than correct it. shellApps now says what
must be REPORTED and appProducts says what is licence-CHECKED — the two stopped
being one list the moment cloud stopped claiming products nothing grants — and an
ungated app answers true, which is the truth, instead of a lock nobody can lift.
engine stays in the catalog and out of the console, deliberately. It is licensed by
a DIFFERENT authority: apps/plan/licence.go stamps it into a signed licence
("licensing.product:"+id) that a customer's own engine verifies offline. No console
surface reads it and none should, so the paid-for-nothing test is scoped to the
console's own vocabulary and names the two authorities where a reader will meet
them. engine-rocm resolved itself: the catalog dropped it at v1.4.14.
The test helper derives from shellApps instead of spelling the surface twice — the
count lived in a literal list AND a length check, two places that never mentioned
each other.
FOLLOW-UP, filed not fixed: @hanzogui/shell still advertises studio/bot/world/
platform to the browser as tier "pro", client-side, against a different endpoint.
That third vocabulary is the only place the product intent is written down, and
reconciling the console with the server is its own piece of work.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The console paywall asked the licence authority about "world", and no catalog
version has ever licensed that id. World access was conveyed by a tier-level
`bundles` field pointing at separate world-free/pro/team/enterprise PLANS;
@hanzo/plans carried that at v1.4.4 and deleted both the field and those plans at
v1.4.11, with nothing put in its place.
Asking about a product nothing grants is not a near miss — CheckEntitlement
resolves cleanly and answers Active:false for every org on every tier, enterprise
included, so GET /v1/entitlements reported world LOCKED to every customer
unconditionally. A lock no purchase can lift is worse than no claim at all, which
is what made this a contract failure rather than a missing feature.
apps/world already resolves its limits to the free floor when the plan does not
resolve (entitlement.go ResolveWorldLimits). That is now the whole answer rather
than a degraded one.
TestEntitlements_WorldTiers asserted world-pro/free/enterprise resolve through
the Go/goja path and cannot pass against a catalog that no longer sells them. The
seam it covered is worth keeping, so it becomes TestEntitlements_RetiredTierIsAnError:
a live tier still resolves and carries entitlements, and a retired id ERRORS
rather than answering an empty map — the distinction that matters, because an
empty map reads as "this tier grants nothing" and would silently strip a paying
subscriber of everything they bought.
Not touched: studio, bot and platform are asked about and granted by nothing
either, but unlike world they have never had a mechanism at all — that is a
product decision about whether they are sold, not a stale reference to remove.
engine-rocm resolved itself upstream: the catalog dropped it at v1.4.14.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`on:` declared workflow_dispatch twice - bare at the top, and again under the
comment explaining the resume entry point. YAML forbids a duplicate mapping
key, and the forge reads this trigger by DECODING the block: act/model
WorkflowDispatchConfig() decodes `on:` into map[string]yaml.Node and returns
nil the moment that fails. DispatchActionWorkflow reports nil as
`has no workflow_dispatch event trigger`, so every dispatch of this workflow
answered HTTP 422 - including the RESUME entry point the comment describes and
the call a sync lane makes after a fast-forward, which is the one door left
when a machine push has eaten the event.
Pushes still triggered runs throughout; only the dispatch door was shut.
Keep the documented occurrence, drop the bare one. The trigger set is
unchanged: push(main, tags v*), pull_request, workflow_dispatch.
engine-rocm was a SKU on the Enterprise plan, and nothing else. rocm is a cargo
feature in hanzo-cli/Cargo.toml beside cuda, metal, mkl and accelerate -- peers,
all built from one source tree -- so an AMD owner on Max was refused a binary an
NVIDIA owner on Max could download. That prices the plan by the customer's
hardware brand. Removed in plans v1.4.14, where the test that asserted the SKU
existed now asserts no plan sells an accelerator at all; licensing v0.1.10 says
the same thing where the distinction lives, on Entitlement.ProductID and
Release.Product. Nothing enforced it: no release carries Product "engine-rocm".
.gitignore line 62 was the bare word `tools`. A gitignore pattern with no slash
matches at ANY depth, so it covered apps/tools -- 25 tracked files, the /v1/tools
MCP catalogue -- and plugin/tools. Adding a NEW file to either was refused, and a
refusal to add is how work disappears with no diff to notice it. It sat directly
above a block that anchors every other root binary (/gateway /account /authz
/smoke /gen-app-cmds) and explains why. It is now /tools, in that block, proven
both ways: a new file under apps/tools is visible, a root ./tools binary is not.
Also here: iam v1.34.21 (13 more typed ops, 98 untyped -> 85), commerce's cart,
and the document regenerated from all of it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three defects, each of which alone made the meter charge nobody.
The edge gate billed reads. DefaultPrice priced by path and ignored method, so
declaring a surface at 25c charged 25c for GET /v1/<surface>/list. That is why
0 of 119 surfaces had ever declared a positive price: 95 chose Free, 24 chose
Metered, and DefaultPrice returned 0 for every path in production. Consumes(method)
in price.go is now the one rule, read by both DefaultPrice and spend.go's
Billable, so the charge question and the standing question cannot drift.
Two surfaces promised a meter and kept none. auto and flow declared Metered --
which makes the platform require standing and makes the edge charge nothing, on
the promise that a downstream meter owns it -- and had no meter. Every durable
run and every workflow run was free, invisibly. Both now gate before the work
and debit after. meteredWithoutAMeter is empty: 21 of 24 hold their own meter,
3 meter through the AI wrapper, 0 charge nothing.
The third was the launch blocker. The in-handler meter addressed money by bare
org; the customer's balance page, the edge gate and the paywall all address it
by principal.WalletOf. In a tenant org those coincide. In the shared signup org,
where a self-serve stranger lands, they do not -- so a stranger's top-up went to
hanzo/stranger while 21 products checked and debited hanzo, the platform's own
pool. Proven by mutation: revert the address and a customer who has just paid
gets 402 insufficient_balance. Fourth recurrence of a class apps/principal/wallet.go
already documents three instances of.
prepaid_e2e_test.go runs it over a real socket: real RSA key, real JWKS the
validator fetches, real RS256 bearer, real listener, real net/http client, real
double-entry ledger. Top up 100c, four 25c calls to 75/50/25/0, fifth refused
and the handler does not run; one balanced ledger entry per movement; an
unpriced op and a GET on the priced surface both move nothing (re-funded first,
so they cannot pass by being broke); another org's valid token cannot spend this
org's balance. Run against both addressing modes, because the address is half of
what is being proved.
Chargeable: 24 of 119 surfaces, and all 24 now actually charge. On the correct
payer address: 3 (auto, flow, tools). Migration is atomic per surface -- a
half-migrated surface gates one wallet and debits another, worse than
consistently wrong -- so the remaining 21 move one at a time: principal.Ledger(c)
-> principal.Payer(c) at both the Gate and the Meter in that surface, together.
booksOf makes it inert; a caller still passing a bare org gets the identical
value byte for byte.
Not closed: the money gates are HTTP-only. zip has a per-op authorizer at
op.invoke covering REST, MCP tools/call, CLI LocalInvoke and the ZAP call plane,
and cloud sets none, so a priced op reached over MCP or the plane bypasses
billing entirely.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The usage page 502'd on an honest ledger: per-token debits are
routinely finer than a cent (0.00589 USD is a live row), plane
Minor()'s exactness guard refuses those BY DESIGN — a rounded debit is
a ledger that drifts — and the co-resident usage read was making a
display go through a debit's gate. Sustained 'billing upstream
unreachable' on GET /v1/billing/usage, ~8/min, org hanzo, right now.
RoundMinor is the explicit display rounding that guard demands, made
once beside FloorMinor so every summary makes the same one:
half-away-from-zero to the currency's minor unit, for figures someone
reads — never a balance a gate spends against (FloorMinor: down is the
only safe direction) and never a debit (exact or refused, unchanged).
The test pins the 502's own row and that Minor() still refuses it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The first fully green gate pass this pipeline ever produced skipped its
image: the run was an API rerun, act_runner serves github.ref as null
on those, and the image guard read null != refs/heads/main as 'not
main'. The guard now accepts the rerun's null-ref shape (the workflow
only triggers from main pushes, tags and dispatch, and PRs are still
refused by event_name), and workflow_dispatch exists so a green head
can be re-driven without an empty commit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A customer's card was charged and the balance that decides whether an inference
request is served never moved. commerce's card core credits its OWN transaction
store; every spend gate reads apps/finance — build.go's balanceReader,
apps/metering's fetchAvailable, GET /v1/billing/balance are all fin.Balance — and
the two are different files on disk. The only door that reaches finance,
POST /v1/billing/credit, is not mounted. So a paid top-up funded nothing.
Both credit doors already compose one value onto their handler, which is where all
four projections of the typed op and the browser's raw route converge on "the
charge cleared". That is where the deposit is issued now, before the risk record
and, unlike it, synchronously: a credit that cannot be posted refuses the door with
a 500 and a RECONCILE line rather than answering success to a customer whose money
went nowhere. The customer's own retry is then the recovery — the money core
replays the receipt and the deposit completes the half that was missing.
The address is the payer's wallet, (payerOrg, principal.Subject), which is
byte-for-byte what principal.WalletOf hands the spend gate and apps/billing hands
the balance route. That also closes the split payments.go recorded: commerce's
typed door credits the org POOL, while for a member of the shared signup org — or
a credential carrying a signed person: billing_account claim — the wallet the gate
reads is the person's, so an agent's payment funded a balance nobody could spend.
The key is the settlement's own reference (firstRef: the processor's, else the
receipt), the same key the risk record is filed under, so a replay, a webhook and
both doors credit one payment once. The amount, the currency and the books come
off the RECEIPT, never off the request: commerce replays the first receipt for a
repeated idempotency key whatever amount the repeat carried, so sizing a credit
from the request would let a settled $5 charge be replayed as $5,000. A sandbox
charge credits the sandbox books, because the receipt states which bucket cleared
and finance keeps them in separate files.
The deposit goes through finance's own Deposit rather than commerce's injected
creditledger adapter for one reason: CreditInput has no test field, so that seam
cannot say which books to write.
Tests: a settled top-up at either door funds the wallet the gate reads, the
customer's /v1/billing/balance reports it, and a metered debit spends it; three
posts of one settlement credit once; one payment answered at both doors credits
once; a $42 settlement replayed under a $5,000 request credits $42; a sandbox
charge leaves the live books at zero; and every way the credit can fail refuses
the door instead of answering 200. Seven mutations — dropping the key, the credit
at either door, the test flag, keying on the receipt, sizing from the request, and
swallowing the refusal — are each caught by a named test.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
969d755c added a cloud.Request call site in apps/commerce/risk.go without an
allowlist entry, so TestRequestEscapeHatchIsPinned failed on that commit ALONE
(proven on a clean tree). That test is car 0 of the release train and every
later car `needs:` it, so main has been unshippable since it landed.
The escape hatch is justified and the reason was already written at the call
site; this records it where the gate reads it. The credit screen serves BOTH the
browser's REST call and an agent's tools/call, and reads the PAYER, the billing
ADDRESS and the JURISDICTION off the request it is being served over — parked by
the app-wide Bridge, which runs for /mcp and the op path alike.
None of the three can be an In field, for exactly the reason apps/dataset is
already allowed here: a caller that could name its own payer would charge
another org. And on the MCP leg no HTTP response exists yet when this returns,
so a response reader would watch an agent's payment settle and learn nothing.
The screen fails closed when the request is absent.
The gate now passes. Nothing else changed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
No overlap — 969d755c is apps/commerce and its generated openapi
artifacts; this branch is the plane seams, the JWKS derivation and the
two silent zeros. Clean merge, no conflicts.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
mountedStore answers ErrNoPeer, and three sibling functions answered the
same fact — agents is not in this process — with a bare string a caller
cannot test for. RunOnBehalf, namespaceFor and eachStore now say it the
one way, so "not here" is routable wherever it is raised rather than only
where it was noticed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two seams returned a zero with no error, and both are seams whose zero is
read as an answer.
agents.StopSessions answered (0, nil) when the package was not in the
process. agents ships as its own binary and the revoke lives in link, so
that was every deployment: a credential revoke replied 200 with
{"sessionsStopped":0} having torn down nothing, while the sessions kept
running under the revoked account. It now errors with ErrNoPeer, and link
takes the plane leg — two doors onto the ONE StopSessions, so the actor
scoping that bounds a revoke to its own user's sessions holds across the
boundary. The org is the caller's plane identity and never an argument;
the actor is built on the answering side from the org the plane proved.
authors.AccrueForOrg had no error channel at all. Same split, same
consequence, on the money path: the royalty leg of every affiliate sweep
latched nothing and reported a completed accrual of zero. It returns
(int, error) now, absence is ErrNoPeer, and a lookup failure is returned
rather than logged-and-zeroed — a sweep that could not read the authors is
not a sweep that found none. The sweep reports royaltyFailures beside the
count.
An empty match stays (0, nil) in both: fail-closed is a real answer, and
turning it into a fault would break a revoke that legitimately matches
nothing.
projects.LiveSites, the third of the three, was already fixed on main —
Ready() plus the plane leg in apps/catalog serving(), with corpus()
returning the error so a failed read cannot prune the catalog.
plane/gen was stale: the ops the seam work added were declared but their
typed clients had never been regenerated, so plane/git, plane/platform
and plane/projects were missing them and plane/sync and plane/tracker did
not exist. Regenerated.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
CLOUD_JWKS_URL exists because the public issuer host is fronted by
Cloudflare, which 403s a server-side loopback — iamurl.go's own comment
root-caused that on 2026-07-04. Production pins the in-cluster address
against CLOUD_IAM_ISSUER=https://hanzo.id for exactly that reason.
jwksURLFor honoured the pin and was unexported, so the two planes that
could not reach it built the URL themselves: durable.go's gated ZAP
listener and apps/base's per-app pool each concatenated the suffix onto
the ISSUER. Both fetched signing keys from the host that refuses them
while the edge validator used the working one — one fleet verifying one
set of keys at the front door and a different set behind it.
It is exported as JWKSURLFor now and all three callers go through it.
TestJWKSHasOneDerivation pins the behaviour, and because the defect was
not calling it wrongly but not calling it at all, it also walks the
source: one file may spell the path as a URL, and naming durable.go is
what the check does when the inline rebuild comes back.
Also lands the rest of the deps-policy delta over 06c672b: the Deps.Brand
doc named "osage" and "any customer brand", neither of which the registry
has (brand.For folds an unknown id to hanzo silently); apps/platform's
`var apexOf = brand.Apex` alias is deleted with its call sites reading
brand.Apex directly, and apps/sites' unrelated apexOf — the published-site
ZONE, a different question with the same name one package away — is
siteZone.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The screen was ROUTER middleware. zip records a typed op ONCE and four projections read
that one record — the REST route, the MCP tool, the by-name call plane and the CLI — and
all four dispatch to the op's HANDLER (registeredOp.invoke). Router middleware is composed
around the fiber handler the REST route is served through and around nothing else. So
POST /v1/payments was screened and `takePayment`, which is published in tools/list, was
not: an agent calling the tool reached commerce's one card money move with no decision in
front of it, and the settlement it produced taught the model nothing. The control was
present on the browser's door and absent on the agent's, and every HTTP test stayed green
because HTTP is the one plane it worked on.
The screen is composed onto the HANDLER of both doors now — screen.op for the typed op,
screen.route for the raw top-up — so it sits inside the value every projection invokes.
Only the composition point moved. The decision, the payer rule and the settlement key are
the landed ones, resolved once per payment into one `payment` value that the screen judges
and the record teaches, so the two halves cannot name different subjects.
Composing on the handler also stops the screen reading the wire, which is what lets it
work off the REST path at all. The amount is the DECODED input's, because over MCP the
request body is a JSON-RPC envelope with the payment inside `arguments` — a body reader
states no value on exactly the door an agent calls, leaving the sharpest axis a credit
door has blind there while it reads fine on the browser's. The settlement is the RETURNED
receipt, because over MCP the op's answer is wrapped in a tools/call result and no HTTP
response exists when the handler returns. The payer, the address and the jurisdiction
still come off the request, which the app-wide Bridge parks for /mcp and for the op plane
exactly as it does for a REST route, so all three resolve on every plane with a connection
behind it; a call with no request at all resolves no payer, is screened as that state, and
is refused by the handler's own gate rather than exempted from the screen.
The structural check reads WHERE THE VALUE WENT rather than which spellings appear. Every
credit door's handler expression must be built from an identifier that holds the screen,
and nothing anywhere may compose the screen as router middleware. The check it replaces
was satisfied by the exact registration that carried this bug.
Shadow is unchanged. No organisation is armed and defaultRegime is untouched; the screen
reaching two more planes widens the record, not the enforcement.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Seven remote seams returned a zero value when their owning app was not in
the process: OnGitPush answered nil, so every push built nothing and
reported success; OnServiceRelease answered nil, so every release patched
no CR; projectIsForeign answered false, so the cross-org project guard was
off across the fleet. Absence now travels as ErrNoPeer over the existing
Ask plane rather than as a zero that reads like an answer.
Four genuinely-local seams stay local, each with its reason written down.
TestAllListed parses the package for Register* and fails the build when a
new one is added without declaring itself remote or local, so the class
cannot come back.
plane/plane.go conflicted because both sides appended types after Site and
each left its last struct open around a shared closing brace. The two sets
are disjoint — LiveSite* is the site directory, the branch's are the git,
sync, tracker, platform and projects call shapes — so both are kept and
LiveSite is closed on its own.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
commerce holds ONE card money move and this binary opens two addresses onto it.
POST /v1/billing/topup/token was screened; POST /v1/payments ran the same
billing.TakePayment — the same bounds, the same idempotency guard, the same
mintauth-authorized deposit — with nothing in front of it. So the risk gate was
not a bound on the mint, it was a bound on one entrance, and the other entrance
is the one published as an MCP tool for an agent to call.
Both doors now hold the SAME screen, resolved once at the composition root and
handed to each registration. The gate becomes a zip.Middleware because a typed op
has no handler chain to sit in — zip composes middleware around it at
registration — so one gate reaches both forms; screenChain hands the raw route
its chain as `next`. The decision, the payer rule and the settlement key are the
landed ones, not copies.
Because both doors resolve the payer through payerOrg + principal.Subject and key
their settlement on the gateway's own payment id, a burst split across the two
accrues on ONE subject and one payment reached through both converges to one
observation. settlementOf learns the typed door's spelling of the ledger receipt
(`id`, where the browser answers `transactionId`) so a settlement whose processor
states no reference is still keyable there, and the settlement check reads the
whole 2xx band because the typed op declares 201.
The read op is not screened: it mints nothing, and a scorer that is present and
mute must not be able to withhold a customer's own receipt.
Shadow is unchanged. No organisation is armed and defaultRegime is untouched;
widening the gate widens the record, not the enforcement.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
TestRiskGate_ANotDeployedScorerDoesNotCloseTheCreditDoor and
TestScoreOverPlane_AnUndeployedPeerIsAbsentRatherThanAnOutage failed on every
Mac and passed in CI.
A unix socket address is capped near a hundred bytes — 104 on darwin — and
t.TempDir embeds the calling test's own name. The second of those names is 61
characters, so the sockaddr went over the cap and every dial answered
`connect: invalid argument`.
That is the part worth writing down: the failure looked exactly like the thing
under test. Both tests assert that an UNDEPLOYED peer is absent rather than an
outage, and an over-long address produces a dial error indistinguishable from a
peer that is not listening. A test whose environment failure mimics its own
subject cannot be read at a glance, and this one was read as a real regression.
shortRuntimeDir is the same fix, for the same reason, as servePeerLedger in
ledger_peer_test.go — an anonymous short-named dir keeps the address inside the
cap on every platform. Named and commented so the next test that needs a working
socket uses it instead of rediscovering the cap.
unusableRuntimeDir is untouched: it builds a deliberately over-long address to
assert the refusal path, which is the one case that WANTS to exceed the cap.
apps/commerce goes from 5 local failures to 3. The remaining three are
TestPeerLedger_*, which need a RAM-backed scratch (`/dev/shm` or
HANZO_SQLITE_RAMFS_DIR) for the pure-Go SQLCipher codec; macOS provides neither,
so they refuse to run rather than decrypt to persistent storage. Environmental,
not a defect, and not addressed here.
Verified both ways: reverting shortRuntimeDir fails exactly those two tests
again.
The org column on a risk_decided row is the bare slug, and an org name
is unique within an issuer rather than across issuers.
One warehouse serves every brand's deployments — there is no brand in
the datastore address, the database name, or anything else that reaches
it; two brands are two deployments of one artifact differing in a flag.
So two brands' identically named organisations are one value in that
column, in this table and in every other table on that plane.
The column stays what the plane's column is: filing risk's rows under a
qualified key would put them in a partition none of the organisation's
own lenses read, which is the whole point of stating decisions there.
What this app can do is refuse to add to the collision. The subject
column already cannot collide, because the digest covers the qualified
tenant, and the brand now travels as a server-minted attribute — so a
row states which issuer's organisation it describes and a read can bind
it. Qualifying rather than assuming costs one bounded attribute and is
correct whether or not a second brand ever shares the warehouse.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
severest and fuse both document that an action outside cloud's action
vocabulary ranks below allow and can never become the answer, and both
relied on the same predicate to hold it.
Asked as "not empty and not allow", that predicate answered YES for such
a string. A determination carrying one was therefore treated as a
finding, could lead the composition — it is only ever ranked against
other findings — and would then stand, contradicting the two comments
that promise it cannot.
It is the severity that answers now, because the vocabulary is a ranking
and the question is about rank: has this rule reached something stricter
than proceeding. Nothing in this package mints an unrecognised action
today; the guarantee is worth having from the predicate rather than from
the fact that nobody has broken it yet.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
"A score records nothing" is the invariant that keeps this plane from
measuring itself, and it held for exactly one hop.
Every decision is also stated onto the shared event plane — one
risk_decided row per decide, under product 'risk', with the decided
subject as its distinct id. The person rollup folded every signal='act'
fact with no product filter at all, so those rows came back as ACTIVITY:
a subject screened often enough became unusual for having been screened,
and the model learned from its own output one hop later.
The rollup that folds a plane now names the product it will not fold,
spelled from the same constant the emitter files rows under so the two
cannot drift to different words for one product. Only the person rollup
needs it — the session rollup requires a session id this app never
states, and the fault rollup reads a table it never writes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The rings' own claim is that they only move forward, and the guard that
made it true was measured against the wrong window.
velocity folds an event older than a window's span to that window's
leading edge — the right call for a compliance aggregate that must not
drop a record, and a detector evasion in reverse for a live rule, where
it means the event is counted as having just happened. placeable is that
guard, and it refused an event the WIDEST span the rings keep could not
hold. Thirty days is the one span nothing reaches: an event ninety
minutes old cleared it comfortably, and was then folded to now in the
one-hour ring — which is the ring the pace count and the pace accrual
are both read from. The burst window could be filled with last month's
history.
Placement is now measured against the narrowest window the rings keep,
which is the window the rules read. It is taken from the windows the
store is actually built with rather than written down as an hour, so a
deployment that changed them moves this with them instead of leaving a
rule reading a window nobody keeps, and a test holds the published value
against a real ring set's own answer.
What it costs is stated rather than hidden. velocity writes all four
windows in one call, so an event more than an hour behind the edge no
longer enters the wider rings either and the model loses a late arrival.
It still learns from the event itself. A wider aggregate missing one
late event is a slightly stale baseline; the narrowest one gaining it is
a rule freezing a payment for something that happened last month.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The reading a pace bound acts on was a statement about whenever the
rings were last taught something, not about now.
A velocity window sums the buckets in [lead-N, lead], and `lead` is the
newest event the rings received. So a subject that did sixty funded
things inside one hour last Tuesday read sixty events in "the last hour"
for as long as the residency lived. Nothing cleared it: the op that
reads the window records nothing, so the anchor never advanced and the
window never slid off. One busy hour, and every later payment by that
organisation was frozen by a burst that had finished a week earlier.
The reading is now believed only while the aggregates have been taught
something inside the span they claim to describe. Past that the two
aggregate halves fall silent for that axis and the event's own stated
facts are the whole rule — the same reading a subject with no history
gets, and the honest one, since a window with nothing in it is what
"has done nothing lately" means.
The edge is the tenant's and not the key's, and that is said out loud
rather than hidden: the engine's observation carries no per-key leading
edge, so the finest recency available here is "has this organisation
been taught anything inside the span". It clears the case that matters —
a self-serve organisation is its own payer, so its edge is its payer's
edge — and leaves one open, where a busy organisation's traffic keeps
the edge fresh for a quiet subject.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The credit door's pace and fan-out rules were structurally dead — not
mis-tuned, but unable to reach a correct answer for any input.
The door stated an amount, a currency, a country and an address, and no
aggregation axis at all. onFan reads a device and a counterparty, so it
could not fire for any top-up ever sent; onPace read one axis where it
reads three. A rule half nothing states an axis for is indistinguishable
from a rule half that found nothing, and the second reads as clean.
And nothing taught the model anything. A decide records nothing by
design, the published learn door is an organisation calling itself, and
at a self-serve credit door the organisation IS the payer — so the two
halves read an empty history for exactly the subject they exist for.
Five payments of eleven thousand dollars looked like five first
payments, every one under every stated bound by construction.
So a settlement now teaches, over a plane op of its own:
The server states it. The value is what settled, the moment is a
server clock, the subject is the one the credit landed on, and nothing
is stated unless the handler answered that money moved. A request that
fails, is declined or is refused upstream teaches nothing.
It is keyed on the settlement. Settlement is at-least-once, so the key
is the processor's own reference for the charge — the one identifier
the synchronous door and a replayed webhook share — falling back to
the ledger receipt where the processor states none. Velocity that
double-counted a retry would freeze a customer for paying once.
It cannot be pre-empted. Dedupe means the first writer of an id wins,
so the settlement id lands in a namespace the public learn door now
refuses outright. The same guard covers folded buckets, which could
already be claimed.
It cannot fail the payment. The money has moved by then, so the call
is detached, bounded, dropped at a ceiling and panic-guarded.
The door also states the address our own edge resolved as its
counterparty, which is what makes the fan-out reachable there. It states
no device, and that is declared at boot rather than left to read as a
rule that found nothing: no fingerprint reaches this binary from a
top-up, and a user-agent stated as one would summon a person for every
customer. Nothing is invented.
And the payer is now a subject kind of its own. An account's learned
history is its metered inference spend, so judging a top-up as an
account scored a payment against a distribution of money spent OUT —
and the windowed value bounds, which are a payments appetite, accrued on
that same key. A customer with a large inference bill was examined for
it, and no restatement of the number could fix it: one appetite over two
populations is two appetites.
Finally, a negative value is refused at the one door every bound is
applied at. The aggregates accrue a sum and the bounds are read off it,
so a negative amount does not describe a small event — it subtracts from
the finding, and one taught event cancels an hour of real payments. A
recorded row the door would now refuse is skipped and said rather than
abandoning the whole replay, which would blind a tenant for as long as
the row is retained.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Everything the risk plane writes down is its own. The observation record is
what a tenant TAUGHT its model; the policy record is what it STATED; both are
per-tenant files on this app's own shelf, reachable only through this app's own
ops. /risk/decide wrote neither — it is pure by contract — so a verdict existed
for the length of one plane call and then existed nowhere. Nothing joined any of
it to the traffic the organisation was already measuring, so the questions an
operator actually has could not be asked at all: did the gate start refusing the
hour the signup flow changed, is the shadow regime alerting on the campaign's
own traffic, which stage carries the refusals.
So each decision is ALSO stated on the plane the organisation already reads —
event.fact, filled by POST /v1/event, read by /v1/insights, grouped by every
product lens. One row per decide, in the same table as the pageview that led to
it, which is what makes "risk decisions beside product analytics" a join rather
than a project.
THE PLANE IS ASKED, NOT LINKED. analytics owns that table and the pod forks one
process per app, so its write core is in another pid — the shape this seam has
now met twice (cloud.SetRiskScorer, cloud.SetObsErrorIngest) and the answer is
the one both landed on: a plane op on the owning app's socket. analytics gains
event_capture, the WRITE side of the door obs_error_post already claims a slice
of, reaching the SAME write core every HTTP door reaches — one admission, one
normalizer, one storage projection. No new transport, no hop through the fleet's
front door, no second gate.
THE TENANT IS THE CALLER'S AT BOTH ENDS. The emitted row is filed for the tenant
the verdict was REACHED under, and the door mints its org from the plane
principal; the contract carries no field that could name one, so a peer can only
ever write into its own organisation's partition. An unidentified caller writes
nothing rather than defaulting to a tenant.
IT CANNOT FAIL THE DECISION. The gate that asks this scorer is the credit door,
so the emit happens after the verdict is computed, on a detached goroutine, with
its own budget, behind a ceiling that DROPS rather than queues. A queue defers
the loss instead of bounding it and makes this process's memory a function of
how fast the fleet is being screened; dropping says the true thing, which is
that a telemetry row is expendable and a decision is not. An absent peer, a
refusing one and a wedged one are the same fact here and none of them is the
verdict's.
WHAT TRAVELS IS BOUNDED, IN BOTH SENSES. The attributes are the stage, the
subject's kind, the action, the cause, the refusal, the model shape, the policy
version, the posture, and — on a scored answer only — the score and the cut it
was held against. Every value is server-minted, from a closed set the op already
refuses outside of, or narrowed before it travels: the country to alpha-2, and
the amount to which stated bound it was past rather than to itself. The subject
is a tenant-and-kind-scoped digest, so decisions about one subject still group
and the shared plane does not become a second copy of what the gate stated.
SHADOW IS EMITTED AND SAYS SO, which is the whole point of a shadow regime: what
the gate WOULD have done, on real traffic, changing nothing. No organisation is
armed and the default posture is untouched.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The rule beside the model reads the event in front of it: where the payer
acted from, and how much moves. Both are bounds on ONE payment, so the
way past them is arithmetic anyone can do — five payments of eleven
thousand instead of one of fifty-five. Every piece is under the bound by
construction, which is what splitting it is for, so no point-in-time test
can ever see it. The same blindness in the other direction: twenty
accounts sharing one device are each unremarkable taken alone, and the
pattern exists only in what they share.
Two more halves now answer beside the first, and both read the
organisation's OWN aggregates rather than the event. PACE is what one of
the event's identifiers — its subject, its counterparty pair, its device
— had already done inside the aggregates' narrowest window. FAN-OUT is
how many distinct subjects one device or one counterparty already ties
together, counted from that organisation's own record.
The value bounds are not restated. What one payment may not move, an hour
of payments may not move either: the accrued value is judged against the
same examining and freeze thresholds a single event is, so there is one
statement of appetite and two readings of it. Only the count needed a
bound of its own, and the freeze is a conjunction exactly as geography's
is — a busy hour alone is not a determination that anything is wrong, and
a shared identifier never freezes anything at all, because a household,
an office and a farm are the same shape from a count.
The window is TAKEN and never named. A rule that read a misspelled window
name would read zero, which is indistinguishable from a quiet subject, so
the burst window is the narrowest one the aggregates actually keep and a
ring set that keeps none is refused rather than answered with a zero.
Both bounds are stated where the geography rule's are, and the count is
held above what a fold alone puts in that window: at or under it, the
rule would fire on this organisation's own history the moment a residency
rebuilt.
The reading is the tenant's own, on both halves. The rings belong to one
resident and the record query carries the qualified tenant as its leading
predicate, so two brands' identically named organisations — which share
one shelf file — cannot see each other's. A reading that cannot be taken
is an error and never an empty one, because empty means "this subject has
done nothing", which is the answer that would silently allow.
Reading the aggregates is a bounded range scan of the record's own
covering index, measured against a record filled to its ceiling at 6.3 ms
— four per cent of the door's budget, and the reason there is no second
index to widen the record's published per-tenant bound.
Shadow is preserved whole. The findings are computed and recorded, the
outcome is left exactly where the model left it, and the cause says what
would have happened. No organisation is armed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
apps/commerce installs the risk scorer; apps/gateway arms its fleet-wide
abuse gate on cloud.RiskScorerInstalled() and refuses mode=live while that
reads false. As composed, plugin/commerce links commerce alone and
plugin/gateway links gateway alone, so the gateway process sees no scorer
and the gate cannot be armed. That refusal is the fail-safe direction and it
is currently unconditional — which is what makes it fragile: nothing
anywhere fails when it stops being true.
One import into either root would make that predicate answer true in the
process that arms the gate, and the gate would be armed on a co-residency
accident rather than on the question arming asks — whether the risk plane
can answer for the FLEET, which no in-process global settles.
So the invariant is checked structurally, over every `package main` under
plugin/ and cmd/, following first-party imports transitively because linking
is transitive. Anti-vacuity assertions require the scan to actually reach
each app, so a renamed package or a moved root fails the gate instead of
silently passing it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The operator's listing won on MEMBERSHIP alone, so one stated without an
`as_of` went in force whole — and reference.Jurisdictions then refused every
country on it, correctly, because "not listed" from a listing of unknown
currency is not a fact. onGeography swallows that refusal by design, so the
ACTION tier became unreachable, the freeze the rule exists for silently
vanished, and what remained — review at or past the freeze value — looks
exactly like a rule that is working. Nothing logged it: the error that
mattered was consumed per country.
A date is now part of what makes an operator listing usable, checked where
the listing is chosen, and an unusable one loses to the dated compiled
default instead of disarming the half of the rule it was stated to arm.
Unset and malformed are unchanged — both already resolved to the empty
listing and fell to that same dated default.
Choosing is split from reading the environment: resolve is a rule over a
stated listing and can be exercised against every shape one can take, where
a value resolved once at first use cannot be. The reason travels out on
listing.Gap rather than being logged from inside the resolution, because a
control that switched itself off has to be visible from OUTSIDE the process
— Mount says it at startup and /v1/risk/health keeps saying it, beside the
listing's date and age.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Arming and tuning arrive at one address in one body, and only one of them is
self-service. PUT /v1/risk/policy was gated by the billing check and the
tenant guard, neither of which asks about authority, so any member of an
organisation could state {"live":true} and take its model out of shadow —
the decision that lets the model freeze that organisation's customers'
payments, taken by whoever held a token.
Setting `live` now requires cloud.Admin: an admin OF THAT ORG, with
SuperAdmin as the stated superset rather than the requirement, because the
organisation is arming ITSELF and requiring platform sudo would make every
customer's governance decision ours to take. Stating the appetite and the
sample stays self-service.
The check reads the platform's one predicate set and lives beside ops.gate
and caller, which is where this package is allowed to reach for the raw
request. It runs after the principal resolves and before anything is
written, so an unauthorised caller learns nothing about which appetites the
op accepts — the ordering ops.adopt already makes.
Disarming is deliberately left self-service: returning a model to shadow
cannot freeze a payment.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two facts reached the same answer at the self-serve credit door, and each
one reached the wrong one.
A scorer socket that cannot be DIALLED read as a scorer that is not there.
plane.Listening separates three facts — no listener, a listener, and a
socket present but unusable — and scorerUp folded the third into the first
by discarding the error. Absence is the one refusal the fail policy exempts,
so under fd exhaustion (EMFILE), a denied run directory (EACCES) or an
over-long one (ENAMETOOLONG) the probe answered "no scorer here" in
microseconds, against a scorer that was healthy, and a privileged top-up
settled unscored. The budget never caught it because nothing was slow.
scorerUp now returns the error and the seam refuses on it, so a genuine
absence still allows and an outage blocks.
A payment a RULE froze read as a scorer that had not answered. Since the
rule and the model were fused, Action and Refusal are independent: an armed
org with a warming model on a frozen payment answers {restrict, "warming"},
and a gate branching on the refusal alone called that a 503 "try again in a
moment" — inviting the retry that settles the payment it had just frozen.
The pair is the discriminator: riskUnavailable is the only producer of a
block that carries a refusal, and the scorer's own vocabulary tops out at
restrict, which TestActions_TheScorerNeverBlocks now holds closed
structurally.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A fresh account's first top-up is the one the model can never have an
opinion about: it has no history, so it is warming, so it declines, so
the credit door allowed it unscored. Ten million dollars from a
jurisdiction no anti-money-laundering supervision reaches is not
anomalous — it is unprecedented, which scores as nothing.
A second judge now answers beside the model, reading two stated facts:
the jurisdiction the payer acted from and the value moving. It does not
read the score, the cut or whether the model has warmed, so a refusing
model cannot soften it; the fused verdict is the severer of the two, and
a model with no opinion contributes an allow.
The jurisdiction listing is the one the screening engine already
maintains, in two tiers because the required response differs: a tier
called for countermeasures may freeze, a tier under monitoring may only
summon a person. A formal designation stays the AML plane's to make.
The gate states the jurisdiction our own edge resolved, under the same
trust rule the client address already uses — a direct caller's header is
the caller's own writing and is refused. The billing jurisdiction is the
one worth judging and no part of it reaches this process; the weaker
signal is documented as weak and replaced at one line when there is one.
Shadow is preserved whole. The finding is computed and recorded, the
outcome is left exactly where the model left it, and the cause says what
would have happened. No organisation is armed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`Deps` braided capabilities (a KMS client — a thing you USE) with policy (a
domain, a default model — a thing you DECIDE). Different owners, lifetimes and
failure modes: a client is wrong when it cannot connect; a policy is wrong when
it says the wrong thing, silently.
THE HEADLINE: `Domain` answered two questions. 18 readers want THIS PROCESS'S
API host; 5 want THE DEPLOYMENT'S APEX. The build outage was a reader in the
second group reading a field built for the first — selfGitHost = deps.Domain =
"api.hanzo.ai", so the forge at git.hanzo.ai (a SIBLING) was refused and every
native build fell back to GitHub.
dc84b46d fixed ONE derivation. There were THREE, with three algorithms:
platform.apexOf publicsuffix
sites.registrableDomain last-two-labels
git.defaultSSHHost TrimPrefix "api."
They diverge on 3 of 7 realistic hosts. defaultSSHHost("cloud.hanzo.ai") yields
git.cloud.hanzo.ai, which platform's allowlist then REFUSES — the identical
outage, still armed, needing only a different CLOUD_DOMAIN. And sites fed a bare
multi-label suffix (co.uk) into SelfDomains, claiming every domain under it.
All three now call brand.Apex / brand.Sibling — one implementation in the only
leaf all four consumers reach.
TWO MORE LIVE DEFECTS, measured:
IAMIssuer had TWO answers. With CLOUD_BRAND=lux and nothing pinned,
Config.IAMIssuer = "https://lux.id" while the package-level IAMIssuer()
read env raw with no brand fallback and returned "". Empty flows to IAMBase(),
which the sk- key resolver dials; unresolvable -> nil -> EVERY API-KEY REQUEST
AUTHENTICATES AS ANONYMOUS, silently, while JWT keeps working.
Domain was brand-blind where IAMIssuer was not: unpinned lux resolved
issuer=lux.id alongside domain=api.hanzo.ai.
DELETED, not migrated — no value gained a second home:
AIFallbackModel 1 reader, set by nobody, ever -> cloud.FallbackModel
AIDefaultModel -> cloud.DefaultModel; universe sets enso-flash and the
constant IS enso-flash, so behaviour is preserved. This made
agents.go's "model is required" branch provably dead.
Self ZERO readers, its whole life. selfID(cfg) is real and stays.
chart iamIssuer a value in NO brand row, rendered unconditionally, beating
the derivation on every chart deploy.
the "api.hanzo.ai" literal in apps/sites -> brand.APIHost, the same root
Config uses.
KEPT, each with a written justification in deps.go: Brand (whose deployment
this is, fixed at exec), Domain (this process's own API host — now documented as
NEVER the apex), IAMIssuer (the trust root; verifying identity precedes any
store you would read it from), Version/Env (boot facts).
deps_policy_test.go reflects over Deps and FAILS on any field without a
rationale, names the three evicted fields with why, and proves the env knobs are
inert — so a policy field cannot quietly re-enter through the dependency door.
build ./... 0, vet ./... 0.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The server halves for git_status and git_mirror, beside the import and inbound
ops already published here. Same rules as the others: the org is the caller's
plane identity, an anonymous caller is refused, an unmounted store answers 503.
A repo git does not know is simply absent from the reply, which is a real answer
and reads as not-imported.
It also closes a loop that was latent in the two existing handlers. planeInbound
and planeImport called cloud.InboundGitSync and cloud.ImportGitRepo, which now
fall through to the plane when the in-process importer is nil — so a handler
routing through them could dial its own socket and answer itself. It cannot
today, because Mount registers the importer before publishing the op, but that is
an ordering nobody is checking and exactly how the loop arrives later. Both now
call githubImporter directly, like the sync and tracker handlers.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Renames only. No behavior changes.
"Seam" named the thing badly. Every one of these is just one app calling
another, or a process calling itself, and saying so plainly is shorter than the
jargon was:
sync_seam.go -> sync.go
tracker_seam.go -> tracker.go
seam_absence_test.go -> absent_test.go
seamClassification -> kinds
crossApp / inProcess -> remote / local
absentSeamCalls -> calls
seamsInSource -> registers
clearSeams -> unregister
upsertIssueSeam -> upsertIssue
TestCrossAppSeamAbsenceIsAnError -> TestAbsentErrors
TestAbsenceIsDistinguishable -> TestAbsentIsNoPeer
TestEverySeamIsClassified -> TestAllListed
roundtrip_test.go is new and is the missing half of the argument. absent_test.go
shows a call errors when the app is gone, which on its own a call that always
failed would also satisfy. This stands the app up on a real socket and shows the
same call arrives with the caller's org and comes back with the app's answer —
through the public functions, not through Ask, so the op name, the wire types
and the org forwarding are all under test. Any of them drifting would leave
absent_test.go green and production broken.
The word survives in files this change does not own; it is house vocabulary
there and churning them would collide with work in flight.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The server halves for sync_run and tracker_upsert. Same callee pattern as
apps/git/files.go: the org is the CALLER's plane identity, an anonymous caller is
refused rather than defaulted, and an unmounted store answers 503 instead of
inventing a reply.
Each handler calls the LOCAL implementation directly — reconcileEvent and
upsertIssueSeam — never cloud.Sync or cloud.UpsertIssue. Those now fall through to
the plane when the in-process seam is nil, so a handler routing back through them
would let the process serving the op dial its own socket. The doc comments say so,
because the fall-through is what makes that the non-obvious failure mode.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The client halves landed in the previous commits; without these the calls reach a
live app and get "unknown op", which is louder than the silence but still not the
capability. These are the server halves.
projects_ownership the identity boundary asks who owns a claimed project
platform_push a landed push becomes a build
platform_release a proven image is patched onto its Service CR
Each follows the callee pattern already in the tree (apps/git/files.go,
apps/projects/key_rpc.go, apps/platform/rpc.go): the org comes from the CALLER's
plane identity, an anonymous caller is refused rather than defaulted, and an
unmounted store answers 503 instead of inventing a reply.
projects_ownership REFUSES when this process does not hold the project store,
rather than answering "nobody owns it". That distinction is the whole defect in
one line: a process that cannot look does not know, and the boundary reads
"not owned by another org" as permission to forward the claim.
exposePush reads the `mounted` global at call time instead of capturing the
service, matching the push builder registered directly above it — Shutdown sets
that global to nil and a captured pointer would keep building out of a torn-down
store.
plane.Built loses its speculative Matched field. Nothing consumes it and
buildFromPush does not return the count, so producing it would mean widening a
signature with six test call sites to compute a value no caller reads. ZAP fields
append at the end, so it can arrive the day something needs it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Nothing caught this defect class, which is why it kept shipping. Two tests, one
to check the rule and one to keep it checked.
TestCrossAppSeamAbsenceIsAnError calls every cross-app seam with nothing
registered, no socket and no router — the state a process is in when the owning
app is simply not beside it — and fails any that returns nil. Before this change
OnGitPush and OnServiceRelease returned nil there; that is now a failing test
rather than a passing one.
TestAbsenceIsDistinguishable additionally requires the error to wrap ErrNoPeer,
because "this app is not deployed here" and "it is here and the call failed" need
opposite responses, and a fleet that cannot tell them apart has already shipped
that mistake in both directions.
TestEverySeamIsClassified is the part that outlives today. It parses this package
for `func Register*` and fails if one is missing from seamClassification, so a new
seam cannot be added without its author recording in writing whether it crosses a
process boundary. A seam classified crossApp must also appear in absentSeamCalls
or the test fails as well — a classification that nothing exercises is a claim,
not a guarantee. Adding a silent seam now requires deleting a test that says not
to.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
projectIsForeign refused an X-Project-Id registered to ANOTHER org. It consulted
a registry list that `projects` fills at its Mount, and documented the empty case
as safe: "with no registry mounted, nothing is ever foreign, so the guard is a
no-op passthrough (an unmounted registry owns nothing)".
No registry is EVER mounted where this runs. The boundary is edge middleware in
every process; the registry belongs to projects, which runs as its own plugin. So
the list was empty everywhere, the guard concluded "not foreign" for every value,
and the cross-org project impersonation check was off fleet-wide. Measured with
projects absent: projectIsForeign(acme, "someone-elses-project") = false. Its own
tests passed because they exercised only the co-resident case.
Absence of an ANSWER may never read as permission. The registry is now ASKED over
the plane, and the outcomes stay distinct rather than collapsing into one bool:
mine keep provably the caller's own
other refuse a cross-org claim
neither, no error keep an unregistered within-org label, still free-form
ErrNoPeer keep no projects app exists anywhere in this fleet, so
no identifier is registered and none CAN be
foreign — the one case the old passthrough had
right, now proven instead of assumed
any other error REFUSE fail closed, as before
ProjectOwnership is the exported total form: co-resident registries answer
directly, otherwise projects is asked. It cannot decline to look — the only way
it yields no answer is by returning an error.
Cached per (org, project) on a detached context with a short TTL, the same shape
as the scope-rule read in middleware_ratelimit.go: this runs on every request
that asserts a project, and a client disconnect must not poison the cache.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
OnGitPush and OnServiceRelease returned a NIL ERROR when nothing was registered.
Not a wrong value — the absence of one. The push lands on git's embedded server
and the builder belongs to platform; the proving build and the Service CR control
plane are likewise different apps. Different apps are different processes, so
"nothing registered" was the state of every caller that ever ran, and every push
and every release reported success having done nothing. Measured, with the owner
absent: both returned nil.
A test asserted this. TestOnServiceReleaseNoop proved "the dispatch seam is a safe
no-op" and passed for as long as it existed; the no-op was never safe. It is
replaced by its opposite, plus a case proving the co-resident leg still wins so a
local releaser never pays a socket round trip to reach itself.
ServiceReleaserRegistered() is DELETED. A bool cannot carry "the app is
elsewhere": it read false both for a fleet with no control plane and for the
ordinary split fleet where platform is the next process over. apps/deploy turned
that false into a 503 that told operators the release plane did not exist while it
was up and reachable — a refusal whose stated cause was not the cause. Presence is
now answered by making the call, which is the only moment it is knowable at all,
since the router may start a lazy app on demand.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Four seams answered "not registered" for a capability running in the process next
door. Every plugin main mounts exactly ONE app and cmd/cloud mounts none, so the
registrant and the caller are never the same process: each seam was nil on
precisely the path that had work to do.
GitRepoStatuses the console repo list draws in integrations; the repos are
git's, so every repo rendered as never-imported
EnsureGitMirror the sync engine declares the target; git owns the repo that
pushes it, so configuring a mirror reconciled inbound forever
and pushed nothing back
Sync webhooks land on integrations and pushes on git; the engine is
the sync app, so no trigger could ever reach it
UpsertIssue integrations holds the GitHub App; the tracker owns the store,
so every mirrored issue and backfill row was refused
Each now takes the leg ImportGitRepo already took: the in-process call when the
owner is co-resident, the plane when it is not. No new mechanism — Ask is the one
mechanism and these are four more callers of it.
The Err*Unavailable sentinels stay for the case they always described honestly: a
registration this process was supposed to have and does not.
A map cannot cross ZAP, so the repo statuses travel as a slice whose rows carry
their own name.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The plane had no proof that event.log and event.span were reachable at all — the
reads had been repointed onto them while nothing was shown to fill them.
TestLiveEverySignalLandsInItsOwnTable drives one fact of each signal through the
SAME door and asserts each landed in exactly one table, with the columns that
table's ORDER BY is built on: service on the log, trace_id on the span, and a
shared trace across both, which is the correlation the identical envelope buys.
Two things it found:
The analytics manifest row had lost /v1/event. The row is the fleet router now,
so a missing prefix is not a tidier list — it is every beacon in the fleet
falling through to a bare /v1 catch-all that does not serve it. Restored with the
ingest doors named, and the comment says why they are load-bearing.
The round-trip readback waited for NINE groups and then asserted TEN rows. The
poll returned the moment the ninth distinct name landed and the tenth row was
simply not there yet — reproducible under load, invisible when the drain was
fast. Waiting for a smaller number than the assertion checks is a race, not a
shortcut; it now waits for what it asserts. The signal test had the same shape
and now awaits each table before counting it, because landing is a consumer and
a count taken the instant a sibling landed asserts the synchronous insert this
design removed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
An event is the FACT; a message is the container it travels in. The schema now
says so: one database named for what it holds, one table per KIND of fact, and
the subject a fact travels on carries the same name as the table it lands in.
event.event what someone did (kind = track | page | identify | group)
event.error what broke, grouped for issue lists
event.log what a service said
event.span what a service did, and how long
event.metric a sample; event.series the dimension it resolves through
The first fifteen columns are identical across event/error/log/span, so a
cross-kind read is a UNION ALL and not a translation layer. org leads every sort
key, so a single-tenant read is a primary-key seek. This retires three schemes at
once: a brand (hanzo.*), a numeronym (o11y_*), and two product names used as a
storage layer (insights, analytics).
Ingest is a bus, not a handler. The door accepts, normalizes, authorizes and
enriches, then publishes; fan-out happens on JetStream so adding a consumer never
touches the ingest path. Limits retention, one durable pull consumer per sink,
commit-the-sink-then-ack, idempotent on event.id because an unacked message is
redelivered. The subject carries taxonomy, never a tenant.
Two writes that could not land:
SETTINGS rendered AFTER VALUES in every warehouse INSERT. The Values input format
reads everything after VALUES as data, so a trailing SETTINGS is not a setting —
it is a row the parser cannot read, and the store answered
CANNOT_PARSE_INPUT_ASSERTION_FAILED for all four tables while the door still
returned 200 and the shape test still passed. Silent total loss behind a green
receipt. Rendered before VALUES now, with a test that pins the ordering rather
than the prefix, because a prefix and a placeholder count are both satisfied by a
statement the store refuses.
The Guide's funnel and marketing's cohorts still named the old table. They would
have gone on reading a table that no longer takes writes the moment the writer
moved — the same failure that left the observability reads pointed at databases
that had been dropped. Both now name apps/datastore's constant, so the plane has
one home for its names and a rename cannot leave a reader behind.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The unified telemetry plane had a schema and no writer. event.log and event.span
have sat at 0 rows since the databases were created, because nothing in any
shipped binary knew how to fill them — these three files existed only as
untracked working-tree state and had never been committed, built, or run.
fact.go the fact. ONE envelope (org, time, id, name, kind, product,
session/distinct/anonymous/person, url, path, attributes, el)
plus what each signal adds. kind is a COLUMN, not a table:
event.event carries track|page|identify|group, so a caller verb
never becomes a schema decision.
bus.go publish to the EVENT stream, subject per signal (event.<kind>).
LimitsPolicy, not WorkQueue — warehouse, alerts and replay are
independent consumers and each needs its own copy.
warehouse.go durable pull consumer, one per table, ACK only AFTER the insert
commits, idempotent on event.id, so a redelivery cannot double
count and a crash cannot silently drop.
Org rides in the signed envelope, never in the subject, so wildcard
subscriptions stay stable and tenancy stays an authorization concern.
Committed alone, out of a tree carrying 551 files of other in-flight work.
Builds and vets clean on its own package.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The prior comment said provisioning 'registered nothing there'. That is wrong:
provisioning.go:431-434 registers POST /v1/s3, GET /v1/s3, and GET+DELETE
/v1/s3/:name as the s3 slice of its 7-kind loop. Adversarial verification of the
doc sweep caught it.
Behaviour is unchanged either way — storage also claims bare /v1/s3 and won the
duplicate-pattern merge, so those four were already unreachable when declared.
But the comment asserted a fact about the code that is not true, and the four
dead routes are a real open question (provisioning creates S3 instances, storage
performs S3 operations) rather than a non-issue.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The router registers All(prefix)+All(prefix/*) per app and MERGES identical
patterns, appending the later handler behind the earlier one. The earlier is a
proxy that never calls Next(), so a duplicated prefix does not conflict loudly —
the later app just never runs. Two shipped that way, out of 241 prefixes:
/v1/s3 provisioning behind storage — pure shadow, provisioning registered
nothing under it. Removed.
/v1 zen behind commerce. Not a dead endpoint: zen mounts
Group("/v1", z.Claim()) — spend/claim MIDDLEWARE (apps/zen/zen.go:84)
— so the metering gate on the inference path is inert. Recorded in
knownShadowed rather than flipped, because turning it on turns
metering on and that needs a billing owner, not a refactor.
manifest_test.go asserted only that mounting does not panic, which is why both
shipped. TestNoShadowedPrefix asserts ownership, fails on any NEW collision, and
fails again if a knownShadowed entry stops colliding so the list cannot rot.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Add an error/Sentry sink to the canonical event fan-out (forward.go): every
type:'error' event is handed, o11y-free, to clients/o11y, which normalizes it
through the reused errortracking engine and ingests it via the embedded
Modules.Sentry into o11y_sentry_events + the o11y_issues lifecycle, so
/v1/event errors surface on sentry.hanzo.ai alongside the errors a Sentry SDK
posts to /v1/sentry directly.
The seam is the twin of the existing destinations SetSink: analytics imports
neither subsystem, each installs its own sink at Mount, and every sink is
detached + fail-soft + additive (a projection failure never touches the
hanzo.events write or the receipt). The org UUID is derived from the tenant
slug by the same UUIDv5 the o11y read side uses (iamidentn), so an error lands
under the exact org the console resolves; each org gets one canonical Sentry
project (get-or-create, cached). Default-on, gated by CLOUD_SENTRY_LENS;
active only when the in-process o11y runtime is up (Modules.Sentry).
Reuses the ONE datastore path already open (o11y's telemetrystore for the
events plane, its sqlstore for the issue lifecycle) via Modules.Sentry.Ingest;
opens no new connection.
Tests: error fan-out detection/extraction across every wire shape (folded
*Exception, native $exception map, bare error-typed); org-UUID pinned against
the o11y formula; ErrorEvent -> normalize -> fingerprinted occurrence; project
get-or-create + cache; and the pk_ ingest-key mint->use flow end-to-end
(POST /v1/ingest/keys -> POST /v1/event accepted, no 403), so anonymous site
ingest stops 403ing.
2026-07-23 02:52:36 -07:00
737 changed files with 69958 additions and 13789 deletions
[ -s "$work/images"]||{echo"orphans: no published versions found for ${IMAGE_PATH} — that is not a clean run, it is a read that returned nothing" >&2;exit 2;}
# ── what is tagged ───────────────────────────────────────────────────────────
# `git ls-remote` failing and a repo having no tags produce the same empty list,
# so each remote is checked for the READ succeeding, separately from what it
# returned. At least one remote must answer: a network that answered nothing
# would otherwise make every published image look untagged and turn a broken
comm -23 "$work/images""$work/tags" > "$work/untagged"
comm -23 "$work/untagged""$work/accepted" > "$work/new"
if[ -s "$work/new"];then
whileread -r v;do
echo"::error::${v} is published but no git tag names it — an image no commit can be traced to is not a release. Tag it at the commit it was built from, or record it in .hanzo/orphans.txt with the reason it cannot be."
done < "$work/new"
echo"orphans: $(wc -l < "$work/new") published image(s) with no receipt; $(wc -l < "$work/accepted") previously recorded" >&2
exit1
fi
echo"orphans: $(wc -l < "$work/images") published, $(wc -l < "$work/tags") tagged, $(wc -l < "$work/accepted") recorded — every published image has a receipt"
echo "v$NEXT is already published from our sha — skipping the build"
RESUMED=1
echo "v$NEXT is already published from our sha — skipping the build (whether it was SMOKED is a different question, asked below)"
BUILT=1
else
echo "::error::v$NEXT is a version this run OWNS (tag at ${SHA}) but the registry already serves bytes built from '${REV:-an unlabelled commit}'. Another lane pushed onto a name it did not hold. Nothing here may overwrite it — publish the intended bytes under a new number and delete the foreign image."
exit 1
@@ -532,22 +602,37 @@ jobs:
fi
echo "version=$NEXT" >> "$GITHUB_OUTPUT"
echo "resumed=$RESUMED" >> "$GITHUB_OUTPUT"
echo "built=$BUILT" >> "$GITHUB_OUTPUT"
echo "highest seen v$LAST (registry + git tags) -> v$NEXT, claimed at ${SHA} (document sha256:$SPEC_SHA)"
| tr -d '\r' | sed -n 's/^[Dd]ocker-[Cc]ontent-[Dd]igest: *//p' | head -1)
case "$DIGEST" in
sha256:*) : ;;
*) echo "::error::${img} resolves but the registry named no Docker-Content-Digest — these bytes have no name, so nothing may attest to them"; exit 1 ;;
esac
echo "digest=$DIGEST" >> "$GITHUB_OUTPUT"
echo "$img is $DIGEST"
# The receipt's address, derived ONCE. A ref name cannot hold a colon,
# and three places needing "the digest with the colon swapped" is three
-d "$(jq -nc --arg r "refs/${{ steps.img.outputs.receipt }}" --arg s "${{ github.sha }}" '{ref:$r, sha:$s}')")
case "$CODE" in
201) echo "recorded: ${{ steps.img.outputs.digest }} booted, built from ${{ github.sha }}" ;;
422) echo "already recorded by another run — same digest, same statement" ;;
*) echo "::error::could not record the smoke receipt for ${{ steps.img.outputs.digest }} (HTTP $CODE). These bytes booted and nothing can prove it, so a resume would smoke them again at best and skip them at worst — refusing to leave that state."
cat /tmp/receipt.json; exit 1 ;;
esac
# NOTHING LEAVES THIS JOB UNSMOKED, re-asked of the authority rather than
# remembered from a step output. Cheap, and it is the assertion `rollout`
# would otherwise have to trust: every later car declares `needs: image`, so
# this is the last place the question can still be answered before an
|| { echo "::error::no smoke receipt for ${{ steps.img.outputs.digest }} — these bytes have never been proven to boot. Nothing downstream may pin them."; exit 1; }
echo "${{ steps.img.outputs.digest }} is smoked"
# ══ CAR 2 ══ ROLL OUT, AND PROVE IT IS LIVE.
#
# The train used to stop at "pin pushed". A pin is not a release — the RUNNING
@@ -721,7 +910,7 @@ jobs:
# broker has, so it 404'd on every release since the car was written.
/tmp/universe/charts/app/values/hanzo/cloud.yaml | head -1)
if [ "$PINNED" != "${{ needs.image.outputs.digest }}" ]; then
echo "::error::pinned cloud.yaml names ${PINNED:-no digest}, but the bytes this run smoked are ${{ needs.image.outputs.digest }}. v${VERSION} resolved to a different image between the smoke and the pin — refusing to deploy an image nothing proved."
exit 1
fi
echo "pin names ${PINNED} — the digest this run smoked"
if git -C /tmp/universe push --quiet origin HEAD:main 2>/dev/null; then
echo "pinned cloud to v${VERSION}"; exit 0
echo "pinned cloud to v${VERSION} @ ${PINNED}"; exit 0
fi
echo "universe moved under us (attempt ${attempt}/5) — re-applying on the new tip"
@@ -764,7 +969,7 @@ jobs:
# ══ CAR 3 ══ THE DOCUMENT TELLS THE TRUTH ABOUT PRODUCTION.
#
# `surface-check` proves the document equals the code. Nothing proved the
# `check` proves the document equals the code. Nothing proved the
# address is REACHABLE, and three things break that independently: a stale
# subset, a prefix missing from manifest/apps.go, and an edge worker that
# intercepts before the origin. Every address here is a method in eight SDKs, a
@@ -893,7 +1098,7 @@ jobs:
# Same route correction as the pin above: no org in a KMS path.
echo "::error::FLEET_DISPATCH_TOKEN missing in KMS at deploy/ (env ${KMS_SECRET_ENV}, org from the KMS credential). Create it with contents:write + metadata:read on hanzoai/{python-sdk,js-sdk,java-sdk,cli} hanzo-go/sdk hanzo-rs/sdk hanzo-kotlin/sdk hanzo-cpp/sdk hanzo-docs/docs — six owners, so a fine-grained PAT cannot carry it; use a classic PAT with repo scope or a GitHub App installed on all six. Until it exists, every cloud release ships a document no client is regenerated from — which is the failure this car was built to end, so it fails rather than skipping."
exit 1
@@ -1020,6 +1225,10 @@ jobs:
runs-on:[hanzo-build-linux-amd64]
timeout-minutes:15
steps:
# This job used to need no source. It does now: the invariant re-checked at
# the end of it lives in .hanzo/scripts/orphans.sh, beside the list of
["${n:-0}" -gt 1]||{echo"SKILLS-GATE FAIL: apps/skills/catalog holds the ${n:-0}-skill fallback — the overlay missed the //go:embed path";exit 1;};\
j="$(find /src/webui/dist -type f -name '*.js'| wc -l)";\
["$j" -gt 0]||{echo"CONSOLE-GATE FAIL: webui/dist carries no JavaScript — the overlay missed the //go:embed path and the image would ship the fallback shell";exit 1;};\
# the shipped build carries, so the suite exercises the same schema surface.
TEST_TAGS:= sqlite_fts5
test:## Run unit + integration tests (pure-Go, with the FTS5 tag the image ships).
# The lifted prose is COMMITTED (zipdoc_gen.go) because bare `go build` cannot
# regenerate it; -check writes nothing and goes red when a lift no longer
# matches its source, which is the drift being committed makes possible.
# Go drops comments at compile time, so cmd/zipdoc is the ONLY path from a typed
# handler's prose to /v1/openapi.json — the document the SDK repos and the CLI
# read. Its output is COMMITTED, and that is what lets a bare `go build` (and the
# release image) produce a binary that still describes itself without anyone
# paying to lift the prose again. The image used to pay: `go generate -run zipdoc
# ./...` ran on the build's critical path for 355.9s of a 17-minute build, to
# reproduce 99 files that were already in the tree.
#
# Committed means it can go STALE, so exactly one thing has to stay true:
# regenerating from source changes nothing. This asserts it by running THE
# GENERATOR and diffing, rather than asking a -check mode for a second opinion —
# a gate must never be able to disagree with the tool it polices. It is also the
# only form that catches the case below.
#
# `git status --porcelain`, not `git diff`: a NEW package's zipdoc_gen.go is
# untracked and therefore invisible to a diff, which is the failure that matters
# most. The pathspec scopes it to the generator's own files, so an unrelated
# dirty tree neither hides a stale lift nor invents one.
zipdoc-check:## Regenerate the lifted prose FROM SOURCE and fail on any diff.
# Per PACKAGE, not ./...: the checker must load exactly the way `go generate`
# does, one package at a time — whole-module loading extracts differently
# (zap-proto/zip zipdoc: single-vs-module load divergence) and a gate must
@@ -273,14 +288,18 @@ test: ## Run unit + integration tests (pure-Go, with the FTS5 tag the image ship
# A dot-directory is not this module's source. An agent worktree at
# .claude/worktrees/<id>/ is a whole second checkout of this repository, and
# the walk read it: 203 packages where there are 104, and it went red on a
# copy's o11y while nothing here had changed. Same rule the source-walking
# gates in Go state (typed_request_gate_test.go, orgns_test.go).
@set -e;for d in $$(grep -rl '^//go:generate go run github.com/zap-proto/zip/cmd/zipdoc' --include='*.go' --exclude-dir='.?*' clients cmd . 2>/dev/null | xargs -n1 dirname | sort -u);do(cd$$d &&$(GO) run github.com/zap-proto/zip/cmd/zipdoc -check)||{echo"$$d/zipdoc_gen.go is stale — run: go generate -run zipdoc ./$$d/...";exit 1;};done
# copy's o11y while nothing here had changed.
@set -e;for d in $$(grep -rl '^//go:generate go run github.com/zap-proto/zip/cmd/zipdoc' --include='*.go' --exclude-dir='.?*' clients cmd . 2>/dev/null | xargs -n1 dirname | sort -u);do\
(cd$$d &&$(GO) run github.com/zap-proto/zip/cmd/zipdoc -check)||{echo"$$d/zipdoc_gen.go is stale — run: go generate -run zipdoc ./$$d/...";exit 1;};\
done
test:## Run unit + integration tests (pure-Go, with the FTS5 tag the image ships).
@echo ">> openapi.yaml — $$(grep -c '^ /' openapi.yaml) paths. The MCP tool list is NOT an artifact: POST /v1/mcp asks every subsystem."
test-cgo:## Prove the cgo build works too — forces the fork's pure-Go backend via -tags sqlite_purego so the embedded modernc importers don't double-register "sqlite".
Description:"Lists every affiliate across the fleet with its ORG exposed, plus a\nfleet summary of lifetime accrued, still-pending and paid commission in\ninteger cents.\n\nPLATFORM SUDO ONLY, and a non-admin is refused outright. This is the\ncross-tenant view and it names orgs — exactly what the partner-facing\nleaderboard refuses to do. There is deliberately no org-scoped variant of this\nread; a partner sees its own standing through its own dashboard. Bounded per\nrequest.",
Fields:map[string]string{
"page.limit":"Limit caps the rows returned. Absent or non-positive means the default of\n500; anything above 1000 is clamped to 1000.",
"adminAffiliateView.accruedCents":"AccruedCents is lifetime commission accrued, in cents. It only grows — a\npayout moves paidCents, never this.",
"adminAffiliateView.approvedAt":"ApprovedAt is when staff approved, Unix seconds UTC. 0 means never approved.",
"adminAffiliateView.code":"Code is the minted referral code, the slug the ?aff link carries. Empty until\napproval mints it. Codes are one global namespace across all affiliates.",
"adminAffiliateView.createdAt":"CreatedAt is when the org applied, Unix seconds UTC.",
"adminAffiliateView.id":"ID is the affiliate's server-minted handle, \"aff_\"-prefixed — the id the\napprove, suspend, rate and payout routes address.",
"adminAffiliateView.org":"Org is the partner's own org slug. It appears ONLY on this cross-tenant admin\nview; no partner-facing read ever names another org.",
"adminAffiliateView.paidCents":"PaidCents is lifetime commission already paid out, in cents — credits grants\nand record-only cash disbursements alike.",
"adminAffiliateView.pendingCents":"PendingCents is accrued minus paid, in cents: what is still owed, and the hard\nceiling the next payout is reserved against. Never negative.",
"adminAffiliateView.rateBps":"RateBps is this affiliate's DIRECT (level 1) commission rate in basis points\nOF Hanzo's margin (2000 = 20% of margin, never of the customer's bill). Levels\n2 and 3 are platform-wide switches and are not carried per affiliate.",
"adminAffiliateView.referredCount":"ReferredCount is how many orgs this affiliate is the DIRECT referrer of,\ncounted from the attribution edges. It is 0 on the single-affiliate answers\n(approve, suspend, rate, payout), which do not run the count.",
"adminAffiliateView.requestedCode":"RequestedCode is the vanity code the applicant asked for. A request, not an\nallocation: approval mints a different slug if this one was taken. Absent when\nnone was asked for.",
"adminAffiliateView.status":"Status is \"applied\", \"approved\" or \"suspended\". Only \"approved\" resolves for\nattribution and accrues; \"suspended\" stops future earning and claws nothing\nback.",
"adminAffiliateView.suspendedAt":"SuspendedAt is when staff suspended, Unix seconds UTC. 0 means never\nsuspended; it is not cleared by a later re-approval.",
"directoryData.affiliates":"Affiliates is one row per affiliate across the whole fleet, ORG EXPOSED,\noldest first and bounded by the request's limit.",
"directoryData.summary":"Summary tallies exactly the rows above — not the whole table — so a limit that\ntruncates the page truncates the tally with it.",
"directoryOut.data":"Data is the affiliate directory and its tally.",
"envelope.msg":"Msg is an operator-facing note. Empty on every success here — it exists\nbecause the console's admin unwrapper reads the shape cloud.OK writes.",
"envelope.status":"Status is \"ok\" on every 2xx from this surface; a failure is an HTTP error with\nzip's error body, not this envelope carrying a different word.",
"page.limit":"Limit caps the rows returned. Absent or non-positive means the default of\n500; anything above 1000 is clamped to 1000.",
"totals.accruedCents":"AccruedCents is lifetime commission accrued summed over those rows, in cents.",
"totals.applied":"Applied is how many of those rows are still awaiting approval — no code, no\naccrual yet.",
"totals.approved":"Approved is how many are approved: the only rows whose code resolves for\nattribution and whose balance can still grow.",
"totals.paidCents":"PaidCents is lifetime commission already paid out summed over those rows, in\ncents.",
"totals.pendingCents":"PendingCents is accrued minus paid summed over those rows, in cents — the\noutstanding liability across the page.",
"totals.suspended":"Suspended is how many were suspended. What they already accrued stays accrued\nand stays payable.",
"totals.total":"Total is how many affiliate rows this page covered, at every status. It is the\npage, not the table: a limit that truncates truncates this too.",
},
})
zip.Describe("GET /v1/admin/referrals",zip.Doc{
Description:"Answers the referral board: the top referrers by lifetime\ncommission, the funnel conversion rate (referred orgs that have actually\nproduced commission, over all referred orgs), and the accrual LIABILITY the\nplatform owes, broken out by upline level.\n\nRead the liability figure carefully — it is commission accrued and NOT yet\npaid, so it is money owed, not money spent, and the per-level split says how\nmuch of it comes from direct referrals versus the second and third levels.\n\nPLATFORM SUDO ONLY, cross-tenant, and it names orgs. It reads the SAME single\nattribution spine the accrual itself walks, so the board and the ledger cannot\ndisagree. Amounts are integer cents.",
Fields:map[string]string{
"envelope.msg":"Msg is an operator-facing note. Empty on every success here — it exists\nbecause the console's admin unwrapper reads the shape cloud.OK writes.",
"envelope.status":"Status is \"ok\" on every 2xx from this surface; a failure is an HTTP error with\nzip's error body, not this envelope carrying a different word.",
"funnel.convertedOrgs":"ConvertedOrgs is how many distinct referred orgs have produced positive\ncommission at least once — a referral that actually spent.",
"funnel.ratePct":"RatePct is convertedOrgs over referredOrgs as a PERCENTAGE, 0–100, and the one\nnon-integer figure on this board. It is 0 when nothing has been referred yet,\nnot undefined.",
"funnel.referredOrgs":"ReferredOrgs is how many attribution edges exist fleet-wide — one per referred\norg, first-touch, so it is also the count of distinct referred orgs.",
"levelSplit.l1Cents":"L1Cents is lifetime commission accrued to DIRECT referrers, in cents.",
"levelSplit.l2Cents":"L2Cents is lifetime commission accrued one step above the direct referrer, in\ncents, at the platform-wide level-2 rate.",
"levelSplit.l3Cents":"L3Cents is lifetime commission accrued two steps above, in cents. Nothing\naccrues past level 3, so l1+l2+l3 is the whole accrual.",
"referralBoard.accrualByLevel":"AccrualByLevel splits the lifetime accrual across the three upline levels —\nhow much of the liability comes from direct referrals versus the chain above.",
"referralBoard.conversion":"Conversion is the funnel: referred orgs against those that actually earned.",
"referralBoard.summary":"Summary is the fleet tally — population by status, and lifetime accrued, paid\nand still-owed commission.",
"referralBoard.topReferrers":"TopReferrers is the 25 affiliates with the most lifetime accrued commission,\ndescending, orgs named.",
"referralsOut.data":"Data is the referral board: leaders, funnel, tally and per-level liability.",
"referrerRow.accruedCents":"AccruedCents is lifetime commission accrued, in cents. The board is sorted by\nthis, descending.",
"referrerRow.code":"Code is that affiliate's minted referral code; empty if it is not approved.",
"referrerRow.org":"Org is the partner's own org slug. Named only here, on the SuperAdmin board —\nthe partner-facing leaderboard shows an opt-in handle and never an org.",
"referrerRow.pendingCents":"PendingCents is accrued minus paid, in cents — what is still owed to this\naffiliate. Never negative.",
"referrerRow.referredCount":"ReferredCount is how many orgs this affiliate is the DIRECT referrer of —\nits level-1 downline, not the whole three-level chain.",
"referrerRow.status":"Status is \"applied\", \"approved\" or \"suspended\".",
"tally.accruedLifetimeCents":"AccruedLifetimeCents is all commission ever accrued, summed across every\naffiliate, in cents. It only grows; a payout does not reduce it.",
"tally.affiliates":"Affiliates is how many affiliate rows the board read, at every status. The\nread is bounded at 1000 rows, so a larger fleet reports the bound.",
"tally.approved":"Approved is how many of those rows are approved — the only ones whose code\nresolves for attribution and whose balance can grow.",
"tally.paidLifetimeCents":"PaidLifetimeCents is all commission ever paid out, in cents: credits grants\nplus record-only cash disbursements.",
"tally.pendingLiabilityCents":"PendingLiabilityCents is accrued minus paid across every affiliate, in cents.\nRead it as money OWED and not yet disbursed — a liability, not spend.",
},
})
zip.Describe("GET /v1/affiliates",zip.Doc{
Description:"Answers the caller org's OWN affiliate standing: status, referral\ncode and share link, commission rate, how many orgs it has referred, and its\nlifetime accrued, still-pending and already-paid commission in integer cents,\nwith its payout history.\n\nAn org that never applied gets an honest `isAffiliate:false` and the default\nrate rather than a 404 — the console renders the apply form off that answer.\n\nThe affiliate is resolved from the VALIDATED org, never from a field, so this\ncan only ever read the caller's own row; without a principal it is refused. It\nis a PURE READ: nothing accrues until the sweep runs. Commission is earned on\nHanzo's MARGIN, never on the referred customer's bill, so nothing here changes\nwhat that customer pays.",
@@ -25,6 +76,8 @@ func init() {
"affiliateStanding.code":"Code is the minted referral code; empty until staff approve.",
"affiliateStanding.defaultRateBps":"DefaultRateBps is the direct rate a new affiliate would get, answered only\nto a caller that has not applied.",
"affiliateStanding.handle":"Handle is the opt-in public leaderboard name; empty means opted out.",
"affiliateStanding.id":"ID is the affiliate's server-minted handle, \"aff_\"-prefixed — what staff\napprove, suspend, re-rate and pay against. Absent until the org applies.",
"affiliateStanding.isAffiliate":"IsAffiliate says whether the caller org has an affiliate record at all. It is\nthe ONE field an org that never applied gets besides defaultRateBps: on false,\nread nothing else here — every other field is absent, not zero.",
"affiliateStanding.link":"Link is the shareable ?aff URL; empty until a code is minted.",
"affiliateStanding.marginBps":"MarginBps is the platform gross-margin fraction commission is a rate OF.",
"affiliateStanding.paidCents":"PaidCents is lifetime commission already paid out, in cents.",
@@ -32,79 +85,219 @@ func init() {
"affiliateStanding.pendingCents":"PendingCents is accrued minus paid — what the platform still owes.",
"affiliateStanding.rateBps":"RateBps is the affiliate's own direct commission rate, in basis points.",
"affiliateStanding.referredCount":"ReferredCount is how many orgs this affiliate has referred.",
"affiliateStanding.requestedCode":"RequestedCode is the vanity code asked for at apply time — a request, not an\nallocation. Approval mints `code`, which may be a different slug if this one\nwas already taken.",
"affiliateStanding.status":"Status is \"applied\", \"approved\" or \"suspended\". Only an approved affiliate has\na code that resolves for attribution and accrues commission; suspended keeps\nwhat it already earned but stops earning more.",
"remittance.amountCents":"AmountCents is the amount disbursed, in cents. It was reserved against pending\ncommission atomically when recorded, so it never exceeds what was owed.",
"remittance.createdAt":"CreatedAt is when the payout was recorded, Unix seconds UTC — when the balance\nmoved, not necessarily when the cash landed.",
"remittance.id":"ID is the payout row's server-minted handle, \"apo_\"-prefixed.",
"remittance.method":"Method is how it was settled. \"credits\" issued a commerce grant into the\naffiliate org's own wallet; any other value (wire, paypal, check, …) is a\nRECORD of cash a human moved out of band.",
"remittance.reference":"Reference is the operator's settlement note — a bank id, a ledger ref. Free\ntext, absent when none was given.",
"remittance.txn":"Txn is the commerce ledger transaction id, set ONLY where a \"credits\" payout\nactually issued the grant. Absent for cash methods, which write no ledger row.",
Description:"Answers the top affiliates by lifetime accrued commission, shown by\nOPT-IN HANDLE with aggregate figures only, plus the caller's own exact rank.\n\nIt never discloses an org identity and never a referred org's usage. An\naffiliate that has set no handle still OCCUPIES its rank but is not listed —\nso opting out hides the name, not the position, and the visible board must not\nbe read as a complete roster.\n\nThe caller's own row carries its exact GLOBAL rank, computed over the whole\napproved set rather than over the page, so it is right well outside the top of\nthe board. Only an approved affiliate has a rank. Requires a validated\nprincipal; a signed-in non-affiliate may read the board but gets no personal\nrow.",
Fields:map[string]string{
"affiliateBoard.leaders":"Leaders are the top opt-in affiliates, by handle and aggregate figures only.",
"affiliateBoard.total":"Total is the approved population where it is known; omitted where the top\npage truncated and the caller has no rank to derive it from.",
"affiliateBoard.you":"You is the caller's own row with its exact global rank; only an approved\naffiliate has one.",
"affiliateBoard.leaders":"Leaders are the top opt-in affiliates, by handle and aggregate figures only.",
"affiliateBoard.total":"Total is the approved population where it is known; omitted where the top\npage truncated and the caller has no rank to derive it from.",
"affiliateBoard.you":"You is the caller's own row with its exact global rank; only an approved\naffiliate has one.",
"leaderboardRow.accruedCents":"AccruedCents is that affiliate's lifetime commission accrued, in cents, and\nwhat the board is ordered by. An aggregate: no per-customer figure is exposed.",
"leaderboardRow.handle":"Handle is the affiliate's self-chosen display name — the only identity the\nboard ever carries. The org behind it is never disclosed.",
"leaderboardRow.isYou":"IsYou marks the caller's own row, so a client can highlight it without\nmatching on a handle. Absent on every other row.",
"leaderboardRow.rank":"Rank is the position in the GLOBAL approved set ordered by lifetime accrued\ncommission, 1-based. Affiliates that set no handle still occupy their rank and\nare simply not listed, so the visible ranks have gaps and the board is not a\ncomplete roster. On the caller's own row the rank is computed over the whole\nset, so it is exact well outside the top page.",
"leaderboardRow.referredCount":"ReferredCount is how many orgs that affiliate directly referred — a count\nonly, never which orgs.",
},
})
zip.Describe("GET /v1/affiliates/me",zip.Doc{
Description:"Answers the richer self-view: the same lifetime accrued, pending and paid\ncommission and payout history, plus the caller's downline broken out by upline\nLEVEL — direct, second, third — each with the rate paid at that level and how\nmany orgs sit there.\n\nCommission is MULTI-LEVEL: a referred org's spend pays up its referral chain,\nthree levels deep and no further. The direct level is the affiliate's own\nnegotiated rate; the second and third are platform-wide switches, read live,\nso the schedule shown is the one actually in force rather than one compiled\nin. A caller that has not applied still gets that schedule alongside\n`isAffiliate:false`, so the console can show what it would earn.\n\nScoped to the validated org and nothing else, and refused without a\nprincipal. A PURE READ — it reports the downline but accrues nothing.",
Fields:map[string]string{
"affiliateSelf.downlineTotal":"DownlineTotal counts every org in the caller's downline across the levels.",
"affiliateSelf.levels":"Levels is the caller's downline per upline level, with the rate paid there.",
"affiliateSelf.schedule":"Schedule is the rate schedule quoted to a caller that has not applied.",
"affiliateSelf.accruedCents":"AccruedCents is lifetime commission accrued, in cents. It only grows — a\npayout is recorded against paidCents and never reduces this.",
"affiliateSelf.code":"Code is the minted referral code, the slug the ?aff link carries. Absent until\nstaff approve; codes live in ONE global namespace across all affiliates.",
"affiliateSelf.defaultRateBps":"DefaultRateBps is the direct rate a new affiliate starts at, in basis points\nof margin (2000 = 20%). Answered ONLY to a caller that has not applied, as the\nquote beside `schedule`.",
"affiliateSelf.downlineTotal":"DownlineTotal counts every org in the caller's downline across the levels.",
"affiliateSelf.handle":"Handle is the opt-in public leaderboard name. Empty means opted out: the\ncaller keeps its rank and still sees its own row, it is just not listed.",
"affiliateSelf.id":"ID is the affiliate's server-minted handle, \"aff_\"-prefixed. Absent until the\norg applies.",
"affiliateSelf.isAffiliate":"IsAffiliate says whether the caller org has an affiliate record. On false the\nanswer carries the rate SCHEDULE and the default rate instead of a downline,\nso the console can show what the caller would earn.",
"affiliateSelf.levels":"Levels is the caller's downline per upline level, with the rate paid there.",
"affiliateSelf.link":"Link is the shareable ?aff URL built from the code. Empty until a code is\nminted, since there is nothing to share before approval.",
"affiliateSelf.marginBps":"MarginBps is the platform gross-margin fraction, in basis points, that every\nrate here is a rate OF. Read live per request, so it is the value in force\nnow, not the one that applied to commission already accrued.",
"affiliateSelf.paidCents":"PaidCents is lifetime commission already paid out, in cents — credits grants\nand record-only cash disbursements alike.",
"affiliateSelf.payouts":"Payouts is the payout history, newest first, bounded to the last 100 rows.",
"affiliateSelf.pendingCents":"PendingCents is accrued minus paid, in cents — what the platform still owes\nand the ceiling on the next payout. Never negative.",
"affiliateSelf.rateBps":"RateBps is the caller's OWN direct (level 1) commission rate, in basis points\nof margin. Levels 2 and 3 are platform-wide and appear in `levels`.",
"affiliateSelf.schedule":"Schedule is the rate schedule quoted to a caller that has not applied.",
"affiliateSelf.status":"Status is \"applied\", \"approved\" or \"suspended\"; absent for a caller that never\napplied. Only \"approved\" mints links and accrues.",
"levelView.downlineCount":"DownlineCount is how many orgs sit exactly this many hops below the caller. It\nis 0 in the schedule quoted to a caller that has not applied, which has no\ndownline to count.",
"levelView.level":"Level is the upline distance from the org whose spend is being shared: 1 is\nthe direct referrer, 2 and 3 the referrers above it. Nothing accrues past 3.",
"levelView.rateBps":"RateBps is the commission paid at this level, in basis points OF Hanzo's\nmargin (2000 = 20% of margin, never of the customer's bill). Level 1 is the\naffiliate's own negotiated rate; 2 and 3 are platform switches read live, so\nthis is the schedule actually in force, not one compiled in.",
"remittance.amountCents":"AmountCents is the amount disbursed, in cents. It was reserved against pending\ncommission atomically when recorded, so it never exceeds what was owed.",
"remittance.createdAt":"CreatedAt is when the payout was recorded, Unix seconds UTC — when the balance\nmoved, not necessarily when the cash landed.",
"remittance.id":"ID is the payout row's server-minted handle, \"apo_\"-prefixed.",
"remittance.method":"Method is how it was settled. \"credits\" issued a commerce grant into the\naffiliate org's own wallet; any other value (wire, paypal, check, …) is a\nRECORD of cash a human moved out of band.",
"remittance.reference":"Reference is the operator's settlement note — a bank id, a ledger ref. Free\ntext, absent when none was given.",
"remittance.txn":"Txn is the commerce ledger transaction id, set ONLY where a \"credits\" payout\nactually issued the grant. Absent for cash methods, which write no ledger row.",
Description:"Answers the caller's own commission ledger: per period, the margin it\nearned against and the commission taken from that margin; and per referred\norg, that referral's aggregate contribution. Integer cents throughout.\n\nThe per-org view deliberately carries the affiliate's OWN earned share and NOT\nthe referred org's spend or margin. An affiliate is entitled to what it\nearned, not to a restatement of its customer's usage — the period view is\nwhere the margin base appears, aggregated across every referral.\n\nScoped server-side to the validated caller's affiliate; a caller that is not\none gets `isAffiliate:false`.",
Fields:map[string]string{
"affiliateEarnings.accruedCents":"AccruedCents is lifetime commission accrued, in cents.",
"affiliateEarnings.byPeriod":"ByPeriod is the per-period ledger: the margin earned against and the\ncommission taken from it.",
"affiliateEarnings.byReferredOrg":"ByReferredOrg is each referral's aggregate contribution — the affiliate's\nOWN share, never the referred org's spend.",
"affiliateEarnings.marginBps":"MarginBps is the platform gross-margin fraction commission is a rate OF.",
"affiliateEarnings.paidCents":"PaidCents is lifetime commission already paid out, in cents.",
"affiliateEarnings.pendingCents":"PendingCents is accrued minus paid — what the platform still owes.",
"affiliateEarnings.accruedCents":"AccruedCents is lifetime commission accrued, in cents.",
"affiliateEarnings.byPeriod":"ByPeriod is the per-period ledger: the margin earned against and the\ncommission taken from it.",
"affiliateEarnings.byReferredOrg":"ByReferredOrg is each referral's aggregate contribution — the affiliate's\nOWN share, never the referred org's spend.",
"affiliateEarnings.isAffiliate":"IsAffiliate says whether the caller org has an affiliate record. On false it\nis the ONLY field present — there is no ledger to report, and the zeros you\nmight expect are absent rather than reported as earnings of nothing.",
"affiliateEarnings.marginBps":"MarginBps is the platform gross-margin fraction commission is a rate OF.",
"affiliateEarnings.paidCents":"PaidCents is lifetime commission already paid out, in cents.",
"affiliateEarnings.pendingCents":"PendingCents is accrued minus paid — what the platform still owes.",
"orgEarningView.commissionCents":"CommissionCents is what the caller earned from that org across ALL periods, in\ncents. Deliberately the caller's own share and nothing else: that org's spend\nand the margin on it are not restated here.",
"orgEarningView.referredOrg":"ReferredOrg is the org slug this contribution came from — one the caller\nreferred, directly or up to three levels down.",
"periodEarningView.commissionCents":"CommissionCents is what the caller earned that period, in cents: the sum over\neach referred org and upline level of margin × that level's rate. Always ≤\nmarginCents, by construction.",
"periodEarningView.marginCents":"MarginCents is the margin Hanzo earned in that period on the spend of every\norg the caller referred, in cents — the base commission is a rate OF. It is\nthe aggregate base, never any one customer's bill.",
"periodEarningView.period":"Period is the accrual bucket: the UTC year-month, \"YYYY-MM\". Commission is\nlatched at most once per referred org per period, so one row is one month.",
Description:"Answers the caller's share links, each with its URL and its funnel:\nclicks tracked, signups — orgs attributed with that code — and conversions,\nmeaning how many of those signups have actually produced commission.\n\nSignups and conversions are DERIVED from the commission ledger and never\nstored, so they cannot drift from the money. Clicks are the one stored counter\nand the one that is pure vanity.\n\nAny pending public click pings are folded into the store before the read, in\none batch — which is how the counters stay current without a database write\nper click. Scoped to the validated caller's own affiliate; a non-affiliate\ngets `isAffiliate:false` and the link cap.",
Fields:map[string]string{
"affiliateLinks.links":"Links is the caller's share links, each with its URL and funnel.",
"affiliateLinks.maxLinks":"MaxLinks is how many share links one affiliate may hold.",
"affiliateLinks.isAffiliate":"IsAffiliate says whether the caller org has an affiliate record. On false only\nmaxLinks comes back — there are no links, and there is no link to mint until\nthe org applies and is approved.",
"affiliateLinks.links":"Links is the caller's share links, each with its URL and funnel.",
"affiliateLinks.maxLinks":"MaxLinks is how many share links one affiliate may hold.",
"affiliateLinks.status":"Status is the caller's affiliate status: \"applied\", \"approved\" or\n\"suspended\"; absent for a non-affiliate. Minting a link requires \"approved\",\nbecause a link that cannot accrue quietly loses the referral.",
"codeView.clicks":"Clicks is how many pings this code has taken. The one STORED counter here and\npure vanity: no accrual or payout reads it, pings are coalesced in memory and\nflushed in batches, and a dropped tally is accepted rather than contending\nwith the money write path. Do not reconcile it against anything.",
"codeView.code":"Code is the link's slug — 3–32 chars of a–z, 0–9 and hyphen — unique across\nthe WHOLE directory, so any affiliate's code resolves an attribution.",
"codeView.conversions":"Conversions is how many of those signups have actually produced positive\ncommission for the caller. Also derived, from the accrual rows, so it is\n≤ signups and lags a referral until the first sweep after it spends.",
"codeView.createdAt":"CreatedAt is when the link was minted, Unix seconds UTC.",
"codeView.label":"Label is the caller's own note for the link (\"twitter\", \"newsletter\").\nCosmetic: trimmed, stripped of control characters, capped at 48 bytes, and\nnever part of the code. \"primary\" on the link mirrored at approval.",
"codeView.signups":"Signups is how many orgs were attributed with this code — DERIVED by counting\nattribution edges, never stored, so it cannot drift from the ledger.",
"codeView.url":"URL is the full shareable link, the brand host plus ?aff=<code>. The host is\nthe deployment's own brand, so a Lux or Zoo install never mints a hanzo.ai\nlink.",
Description:"Approves an affiliate and MINTS its referral code — the moment\nthe partner has a working share link and starts accruing.\n\nThe code is taken from the body if one is given, else the vanity code the\napplicant requested, else a slug derived for them. Codes are ONE global\nnamespace, so a taken code is a 409 and nothing is approved. The minted code\nis also mirrored as a link row so click tracking is uniform across every code\nthe affiliate holds; that mirror is best-effort and its failure never fails\nthe approval.\n\nApproval is what makes an affiliate eligible: before it, attribution against\nits code does not resolve and no sweep accrues to it. PLATFORM SUDO ONLY.\nAudited.",
Fields:map[string]string{
"approval.code":"Code overrides the minted code; else the requested vanity code, else a\nderived slug.",
"approval.id":"ID is the affiliate to approve, from the path.",
"adminAffiliateView.accruedCents":"AccruedCents is lifetime commission accrued, in cents. It only grows — a\npayout moves paidCents, never this.",
"adminAffiliateView.approvedAt":"ApprovedAt is when staff approved, Unix seconds UTC. 0 means never approved.",
"adminAffiliateView.code":"Code is the minted referral code, the slug the ?aff link carries. Empty until\napproval mints it. Codes are one global namespace across all affiliates.",
"adminAffiliateView.createdAt":"CreatedAt is when the org applied, Unix seconds UTC.",
"adminAffiliateView.id":"ID is the affiliate's server-minted handle, \"aff_\"-prefixed — the id the\napprove, suspend, rate and payout routes address.",
"adminAffiliateView.org":"Org is the partner's own org slug. It appears ONLY on this cross-tenant admin\nview; no partner-facing read ever names another org.",
"adminAffiliateView.paidCents":"PaidCents is lifetime commission already paid out, in cents — credits grants\nand record-only cash disbursements alike.",
"adminAffiliateView.pendingCents":"PendingCents is accrued minus paid, in cents: what is still owed, and the hard\nceiling the next payout is reserved against. Never negative.",
"adminAffiliateView.rateBps":"RateBps is this affiliate's DIRECT (level 1) commission rate in basis points\nOF Hanzo's margin (2000 = 20% of margin, never of the customer's bill). Levels\n2 and 3 are platform-wide switches and are not carried per affiliate.",
"adminAffiliateView.referredCount":"ReferredCount is how many orgs this affiliate is the DIRECT referrer of,\ncounted from the attribution edges. It is 0 on the single-affiliate answers\n(approve, suspend, rate, payout), which do not run the count.",
"adminAffiliateView.requestedCode":"RequestedCode is the vanity code the applicant asked for. A request, not an\nallocation: approval mints a different slug if this one was taken. Absent when\nnone was asked for.",
"adminAffiliateView.status":"Status is \"applied\", \"approved\" or \"suspended\". Only \"approved\" resolves for\nattribution and accrues; \"suspended\" stops future earning and claws nothing\nback.",
"adminAffiliateView.suspendedAt":"SuspendedAt is when staff suspended, Unix seconds UTC. 0 means never\nsuspended; it is not cleared by a later re-approval.",
"affiliateData.affiliate":"Affiliate is the row as it stands AFTER the action that returned it. Its\nreferredCount is 0 here: these single-affiliate answers do not run the count.",
"affiliateOut.data":"Data carries the affiliate row the action just wrote.",
"approval.code":"Code overrides the minted code; else the requested vanity code, else a\nderived slug.",
"approval.id":"ID is the affiliate to approve, from the path.",
"envelope.msg":"Msg is an operator-facing note. Empty on every success here — it exists\nbecause the console's admin unwrapper reads the shape cloud.OK writes.",
"envelope.status":"Status is \"ok\" on every 2xx from this surface; a failure is an HTTP error with\nzip's error body, not this envelope carrying a different word.",
Description:"Pays out accrued commission and answers the payout row with the\naffiliate's updated balances.\n\nThe amount is reserved atomically against the affiliate's PENDING commission —\naccrued minus paid — so a payout can never exceed what is owed. The METHOD\ndecides whether money actually moves: `credits` issues a commerce grant into\nthe affiliate ORG's own wallet, tagged so the ledger can tell an affiliate\npayout apart from an admin or referral grant; every other method — wire,\npaypal and the rest — is RECORD-ONLY: the payout row and the balances move,\nthe cash is disbursed out of band.\n\nThe amount is integer cents and must be positive. PLATFORM SUDO ONLY.\nAudited.",
Fields:map[string]string{
"disbursal.amountCents":"AmountCents is the payout, integer cents; it must be positive and can\nnever exceed the affiliate's pending commission. Body-only (`url:\"-\"`,\nlike every money field here): a payout must never ride the URL into\naccess logs, and the raw handler read only the body.",
"disbursal.id":"ID is the affiliate to pay, from the path.",
"disbursal.method":"Method decides whether money moves: `credits` issues a commerce grant,\nevery other method (wire, paypal, …) is record-only.",
"disbursal.reference":"Reference is the operator's settlement note (a bank id, a ledger ref).",
"adminAffiliateView.accruedCents":"AccruedCents is lifetime commission accrued, in cents. It only grows — a\npayout moves paidCents, never this.",
"adminAffiliateView.approvedAt":"ApprovedAt is when staff approved, Unix seconds UTC. 0 means never approved.",
"adminAffiliateView.code":"Code is the minted referral code, the slug the ?aff link carries. Empty until\napproval mints it. Codes are one global namespace across all affiliates.",
"adminAffiliateView.createdAt":"CreatedAt is when the org applied, Unix seconds UTC.",
"adminAffiliateView.id":"ID is the affiliate's server-minted handle, \"aff_\"-prefixed — the id the\napprove, suspend, rate and payout routes address.",
"adminAffiliateView.org":"Org is the partner's own org slug. It appears ONLY on this cross-tenant admin\nview; no partner-facing read ever names another org.",
"adminAffiliateView.paidCents":"PaidCents is lifetime commission already paid out, in cents — credits grants\nand record-only cash disbursements alike.",
"adminAffiliateView.pendingCents":"PendingCents is accrued minus paid, in cents: what is still owed, and the hard\nceiling the next payout is reserved against. Never negative.",
"adminAffiliateView.rateBps":"RateBps is this affiliate's DIRECT (level 1) commission rate in basis points\nOF Hanzo's margin (2000 = 20% of margin, never of the customer's bill). Levels\n2 and 3 are platform-wide switches and are not carried per affiliate.",
"adminAffiliateView.referredCount":"ReferredCount is how many orgs this affiliate is the DIRECT referrer of,\ncounted from the attribution edges. It is 0 on the single-affiliate answers\n(approve, suspend, rate, payout), which do not run the count.",
"adminAffiliateView.requestedCode":"RequestedCode is the vanity code the applicant asked for. A request, not an\nallocation: approval mints a different slug if this one was taken. Absent when\nnone was asked for.",
"adminAffiliateView.status":"Status is \"applied\", \"approved\" or \"suspended\". Only \"approved\" resolves for\nattribution and accrues; \"suspended\" stops future earning and claws nothing\nback.",
"adminAffiliateView.suspendedAt":"SuspendedAt is when staff suspended, Unix seconds UTC. 0 means never\nsuspended; it is not cleared by a later re-approval.",
"disbursal.amountCents":"AmountCents is the payout, integer cents; it must be positive and can\nnever exceed the affiliate's pending commission. Body-only (`url:\"-\"`,\nlike every money field here): a payout must never ride the URL into\naccess logs, and the raw handler read only the body.",
"disbursal.id":"ID is the affiliate to pay, from the path.",
"disbursal.method":"Method decides whether money moves: `credits` issues a commerce grant,\nevery other method (wire, paypal, …) is record-only.",
"disbursal.reference":"Reference is the operator's settlement note (a bank id, a ledger ref).",
"envelope.msg":"Msg is an operator-facing note. Empty on every success here — it exists\nbecause the console's admin unwrapper reads the shape cloud.OK writes.",
"envelope.status":"Status is \"ok\" on every 2xx from this surface; a failure is an HTTP error with\nzip's error body, not this envelope carrying a different word.",
"payoutOut.data":"Data is the recorded payout and the balances it left behind.",
"remittance.amountCents":"AmountCents is the amount disbursed, in cents. It was reserved against pending\ncommission atomically when recorded, so it never exceeds what was owed.",
"remittance.createdAt":"CreatedAt is when the payout was recorded, Unix seconds UTC — when the balance\nmoved, not necessarily when the cash landed.",
"remittance.id":"ID is the payout row's server-minted handle, \"apo_\"-prefixed.",
"remittance.method":"Method is how it was settled. \"credits\" issued a commerce grant into the\naffiliate org's own wallet; any other value (wire, paypal, check, …) is a\nRECORD of cash a human moved out of band.",
"remittance.reference":"Reference is the operator's settlement note — a bank id, a ledger ref. Free\ntext, absent when none was given.",
"remittance.txn":"Txn is the commerce ledger transaction id, set ONLY where a \"credits\" payout\nactually issued the grant. Absent for cash methods, which write no ledger row.",
"settlement.affiliate":"Affiliate is the row re-read AFTER the payout, so its paidCents and\npendingCents already account for the row beside it.",
"settlement.payout":"Payout is the payout row just recorded.",
Description:"Sets one affiliate's DIRECT commission rate, in basis points of\nHanzo's margin.\n\nThe rate is CAPPED so that the direct rate plus the platform-wide second- and\nthird-level rates can never exceed the whole margin — the structural guarantee\nthat everything paid on one source event stays inside the margin actually\nearned. The cap is resolved from the rates in force at the moment of the call\nand quoted in the refusal, because those switches move; a hardcoded bound\nwould start lying the moment somebody edits the schedule.\n\nOnly the direct level is per-affiliate. The second and third levels are\nplatform switches and are not settable here. The change applies to FUTURE\naccruals — commission already latched for a period is not recomputed. PLATFORM\nSUDO ONLY. Audited.",
Fields:map[string]string{
"rateSet.id":"ID is the affiliate whose direct rate moves, from the path.",
"rateSet.rateBps":"RateBps is the direct commission rate, in basis points of Hanzo's margin;\ncapped so the whole L1+L2+L3 schedule never exceeds the margin. Body-only\n(`url:\"-\"`): a money parameter must never ride the URL into access logs.",
"adminAffiliateView.accruedCents":"AccruedCents is lifetime commission accrued, in cents. It only grows — a\npayout moves paidCents, never this.",
"adminAffiliateView.approvedAt":"ApprovedAt is when staff approved, Unix seconds UTC. 0 means never approved.",
"adminAffiliateView.code":"Code is the minted referral code, the slug the ?aff link carries. Empty until\napproval mints it. Codes are one global namespace across all affiliates.",
"adminAffiliateView.createdAt":"CreatedAt is when the org applied, Unix seconds UTC.",
"adminAffiliateView.id":"ID is the affiliate's server-minted handle, \"aff_\"-prefixed — the id the\napprove, suspend, rate and payout routes address.",
"adminAffiliateView.org":"Org is the partner's own org slug. It appears ONLY on this cross-tenant admin\nview; no partner-facing read ever names another org.",
"adminAffiliateView.paidCents":"PaidCents is lifetime commission already paid out, in cents — credits grants\nand record-only cash disbursements alike.",
"adminAffiliateView.pendingCents":"PendingCents is accrued minus paid, in cents: what is still owed, and the hard\nceiling the next payout is reserved against. Never negative.",
"adminAffiliateView.rateBps":"RateBps is this affiliate's DIRECT (level 1) commission rate in basis points\nOF Hanzo's margin (2000 = 20% of margin, never of the customer's bill). Levels\n2 and 3 are platform-wide switches and are not carried per affiliate.",
"adminAffiliateView.referredCount":"ReferredCount is how many orgs this affiliate is the DIRECT referrer of,\ncounted from the attribution edges. It is 0 on the single-affiliate answers\n(approve, suspend, rate, payout), which do not run the count.",
"adminAffiliateView.requestedCode":"RequestedCode is the vanity code the applicant asked for. A request, not an\nallocation: approval mints a different slug if this one was taken. Absent when\nnone was asked for.",
"adminAffiliateView.status":"Status is \"applied\", \"approved\" or \"suspended\". Only \"approved\" resolves for\nattribution and accrues; \"suspended\" stops future earning and claws nothing\nback.",
"adminAffiliateView.suspendedAt":"SuspendedAt is when staff suspended, Unix seconds UTC. 0 means never\nsuspended; it is not cleared by a later re-approval.",
"affiliateData.affiliate":"Affiliate is the row as it stands AFTER the action that returned it. Its\nreferredCount is 0 here: these single-affiliate answers do not run the count.",
"affiliateOut.data":"Data carries the affiliate row the action just wrote.",
"envelope.msg":"Msg is an operator-facing note. Empty on every success here — it exists\nbecause the console's admin unwrapper reads the shape cloud.OK writes.",
"envelope.status":"Status is \"ok\" on every 2xx from this surface; a failure is an HTTP error with\nzip's error body, not this envelope carrying a different word.",
"rateSet.id":"ID is the affiliate whose direct rate moves, from the path.",
"rateSet.rateBps":"RateBps is the direct commission rate, in basis points of Hanzo's margin;\ncapped so the whole L1+L2+L3 schedule never exceeds the margin. Body-only\n(`url:\"-\"`): a money parameter must never ride the URL into access logs.",
Description:"Suspends an affiliate: it stops accruing on the next sweep, and\nits code stops resolving for new attributions.\n\nIt CLAWS NOTHING BACK. Commission already accrued stays accrued and stays\npayable, and existing attribution edges are left standing — suspension ends\nearning, it does not unwind history. PLATFORM SUDO ONLY. Audited.",
Fields:map[string]string{
"affiliateRef.id":"ID is the affiliate's server-minted handle, \"aff_\"-prefixed.",
"adminAffiliateView.accruedCents":"AccruedCents is lifetime commission accrued, in cents. It only grows — a\npayout moves paidCents, never this.",
"adminAffiliateView.approvedAt":"ApprovedAt is when staff approved, Unix seconds UTC. 0 means never approved.",
"adminAffiliateView.code":"Code is the minted referral code, the slug the ?aff link carries. Empty until\napproval mints it. Codes are one global namespace across all affiliates.",
"adminAffiliateView.createdAt":"CreatedAt is when the org applied, Unix seconds UTC.",
"adminAffiliateView.id":"ID is the affiliate's server-minted handle, \"aff_\"-prefixed — the id the\napprove, suspend, rate and payout routes address.",
"adminAffiliateView.org":"Org is the partner's own org slug. It appears ONLY on this cross-tenant admin\nview; no partner-facing read ever names another org.",
"adminAffiliateView.paidCents":"PaidCents is lifetime commission already paid out, in cents — credits grants\nand record-only cash disbursements alike.",
"adminAffiliateView.pendingCents":"PendingCents is accrued minus paid, in cents: what is still owed, and the hard\nceiling the next payout is reserved against. Never negative.",
"adminAffiliateView.rateBps":"RateBps is this affiliate's DIRECT (level 1) commission rate in basis points\nOF Hanzo's margin (2000 = 20% of margin, never of the customer's bill). Levels\n2 and 3 are platform-wide switches and are not carried per affiliate.",
"adminAffiliateView.referredCount":"ReferredCount is how many orgs this affiliate is the DIRECT referrer of,\ncounted from the attribution edges. It is 0 on the single-affiliate answers\n(approve, suspend, rate, payout), which do not run the count.",
"adminAffiliateView.requestedCode":"RequestedCode is the vanity code the applicant asked for. A request, not an\nallocation: approval mints a different slug if this one was taken. Absent when\nnone was asked for.",
"adminAffiliateView.status":"Status is \"applied\", \"approved\" or \"suspended\". Only \"approved\" resolves for\nattribution and accrues; \"suspended\" stops future earning and claws nothing\nback.",
"adminAffiliateView.suspendedAt":"SuspendedAt is when staff suspended, Unix seconds UTC. 0 means never\nsuspended; it is not cleared by a later re-approval.",
"affiliateData.affiliate":"Affiliate is the row as it stands AFTER the action that returned it. Its\nreferredCount is 0 here: these single-affiliate answers do not run the count.",
"affiliateOut.data":"Data carries the affiliate row the action just wrote.",
"affiliateRef.id":"ID is the affiliate's server-minted handle, \"aff_\"-prefixed.",
"envelope.msg":"Msg is an operator-facing note. Empty on every success here — it exists\nbecause the console's admin unwrapper reads the shape cloud.OK writes.",
"envelope.status":"Status is \"ok\" on every 2xx from this surface; a failure is an HTTP error with\nzip's error body, not this envelope carrying a different word.",
Description:"Runs the accrual: for each referred org it reads that org's metered\nspend for the current period and accrues commission to every affiliate up its\nreferral chain, then answers how many sources were swept and how many NEW\naccruals landed.\n\nThis is the cron path, and it is LATCHED at most once per affiliate, source\norg and period — so re-running it inside the same period accrues nothing\nfurther. Safe to retry, and safe to run by hand beside the schedule.\n\nCommission is a rate of Hanzo's MARGIN on that spend, never of the customer's\ngross bill, so every level's share summed over one source event stays within\nthe margin actually earned and the customer's charge is untouched. Nothing\naccrues past the third upline level, and only an APPROVED affiliate accrues at\nall.\n\nThe same spend read drives the OSS author royalty — one read, both programs —\nso the answer reports royalties accrued alongside. PLATFORM SUDO ONLY. Bounded\nper run; a source whose spend cannot be read is skipped and picked up next\ntime, never half-accrued.",
Fields:map[string]string{
"accruals.accrued":"Accrued is how many NEW commission accruals this run created, counted across\nevery upline level. The accrual is latched at most once per (affiliate, source\norg, period), so a re-run inside the same month reports 0 having changed\nnothing — 0 means \"already accrued\", not \"failed\".",
"accruals.royaltiesAccrued":"RoyaltiesAccrued is how many OSS-author royalty accruals the SAME spend read\nproduced in the sibling authors program. One read drives both.",
"accruals.royaltyFailures":"RoyaltyFailures is reported, not swallowed: a sweep that could not reach\nthe royalty store must not read as one that found nothing owed. The count\nwas already computed and then dropped on the floor, which is the same\nsilence the typed leg was added to end.",
"accruals.swept":"Swept is how many source (referred) orgs the run visited, bounded at 500 per\nrun. A source with no spend this period, or one whose spend could not be read,\nstill counts as swept.",
"accrualsOut.data":"Data is what the run did: sources visited, new accruals, royalties alongside.",
"envelope.msg":"Msg is an operator-facing note. Empty on every success here — it exists\nbecause the console's admin unwrapper reads the shape cloud.OK writes.",
"envelope.status":"Status is \"ok\" on every 2xx from this surface; a failure is an HTTP error with\nzip's error body, not this envelope carrying a different word.",
},
})
zip.Describe("POST /v1/affiliates/apply",zip.Doc{
Description:"Enrolls the caller's OWN org as an affiliate at status `applied`,\noptionally requesting a vanity code, and answers the record — 201 on the first\napply, 200 with `created:false` afterwards.\n\nIDEMPOTENT, first apply wins: one affiliate per org, so re-applying never\ncreates a second row and never resets an existing approval. Applying is not\njoining — no code is minted and nothing accrues until staff approve, which is\nwhere both the code and the commission rate come from.\n\nThe org is the validated caller's, never a field. A malformed vanity code is\nrefused up front; the code is only REQUESTED here, and approval may mint a\ndifferent one if the requested code is taken.",
Fields:map[string]string{
"application.code":"Code is the minted referral code. Empty on a first apply — applying does not\nmint a code, approval does; a re-apply echoes whatever the row already holds.",
"application.created":"Created says whether THIS call made the row. false means the org had already\napplied and nothing changed — no second row, no reset of an existing approval.\nThe HTTP status states the same fact: 201 when true, 200 when false.",
"application.id":"ID is the affiliate's server-minted handle, \"aff_\"-prefixed — the id staff\napprove, suspend, re-rate and pay against.",
"application.rateBps":"RateBps is the direct (level 1) commission rate the row carries, in basis\npoints OF Hanzo's margin (2000 = 20% of margin, never of the customer's bill).",
"application.requestedCode":"RequestedCode echoes the vanity code asked for, normalized to lower case. It\nis a request only: approval mints a different slug if this one is taken.",
"application.status":"Status is \"applied\" for a row this call created. A re-apply echoes the\nexisting row's status, which may already be \"approved\" or \"suspended\".",
"applyRequest.requestedCode":"RequestedCode is the vanity code the applicant asks for; approval may mint\na different one if it is taken. Body-only: the URL cannot supply it.",
Description:"Records the first-touch edge every later commission is computed\nfrom: the caller's org was referred by the affiliate that owns this code.\n\nThe REFERRED org is the validated caller, never a field. A caller that could\nname the referred org could attach itself to somebody else's revenue. The\naffiliate is resolved from the code, and only an APPROVED affiliate's code\nresolves.\n\nFIRST TOUCH WINS, set once: one affiliate per referred org, so a re-post\nanswers the existing edge with `created:false` rather than moving the\nattribution. Self-attribution is refused, and so is a code that would make a\ncycle in the upline chain. An unknown code is a 404, deliberately: an\naffiliate code IS a public shareable link, so whether one is real is public by\ndesign, and the caller legitimately needs to know its link resolved.\n\nA user-level mirror of the edge is written best-effort; a conflict there never\nfails the org attribution, which is the money-bearing one.",
Fields:map[string]string{
"attributeRequest.code":"Code is the affiliate code the referred org arrived with. Body-only: the\nURL cannot supply it.",
"attribution.code":"Code is the affiliate code the edge was recorded under, normalized to lower\ncase. On a re-post it is the code of the STANDING edge, which may differ from\nthe one just sent — first touch wins.",
"attribution.created":"Created says whether THIS call made the edge. false means the caller org was\nalready attributed and nothing moved. The HTTP status says the same: 201 when\ntrue, 200 when false.",
"attribution.createdAt":"CreatedAt is when the edge was FIRST recorded, Unix seconds UTC. On a re-post\nit is the original time, not now.",
"attribution.id":"ID is the attribution edge's server-minted handle, \"afr_\"-prefixed.",
},
Example:json.RawMessage(`{"code":"acme"}`),
})
zip.Describe("POST /v1/affiliates/click",zip.Doc{
Description:"Counts a click on a share link. PUBLIC — it takes no principal, because\na visitor clicking a shareable link has no session yet.\n\nThe ping folds into an in-memory buffer and NEVER writes the money database\nsynchronously, so a click flood cannot contend with the accrual and payout\nwrite path; tallies are flushed in one batch on the next authenticated links\nread and at shutdown. Clicks are a vanity metric: no accrual and no payout\never reads them — those key on real metered spend — so click inflation cannot\nmove money.\n\nAny well-formed code is accepted WITHOUT checking that it exists,\ndeliberately: this is not a code-existence oracle. `counted` reports that the\nbuffer took the ping, not that the code is real; an unknown code simply no-ops\nat flush time.",
Fields:map[string]string{
"clickRequest.code":"Code is the share-link code that was clicked. Body-only: the URL cannot\nsupply it.",
"clickCount.counted":"Counted says the in-memory buffer took the ping. It does NOT say the code\nexists — this is deliberately not a code-existence oracle, and an unknown code\nsimply no-ops at flush time. false means the buffer was full and the ping was\ndropped, which is harmless: clicks are vanity and move no money.",
"clickRequest.code":"Code is the share-link code that was clicked. Body-only: the URL cannot\nsupply it.",
},
Example:json.RawMessage(`{"code":"acme"}`),
})
@@ -127,14 +325,23 @@ func init() {
Description:"Sets the caller's public leaderboard display name, or clears it.\n\nThe handle IS the opt-in. An empty handle opts out: the affiliate keeps its\nrank and can still see its own row, it simply stops being listed to anyone\nelse. That is the whole privacy control — there is no separate visibility\nflag, and no way to be listed without choosing a name.\n\nRequires a validated principal and an existing affiliate record; apply first.\nThe handle is bounded and restricted to letters, digits, space, hyphen,\nunderscore and dot.",
Fields:map[string]string{
"handleRequest.handle":"Handle is the public leaderboard display name; empty opts out. Body-only:\nthe URL cannot supply it.",
"handleSet.handle":"Handle is the display name as STORED, echoed back after trimming. Empty means\nthe caller opted out: it keeps its rank and still sees its own row, it is just\nno longer listed to anyone else.",
Description:"Mints a new share link for the caller's own affiliate and answers it\nwith its full URL, 201.\n\nAPPROVAL IS REQUIRED: an org that has applied but is not approved is refused,\nbecause a link that cannot accrue is a link that quietly loses the referral. A\nrequested vanity code must be valid and free across the WHOLE directory —\ncodes are one global namespace, so a taken code is a 409 rather than a silent\nalias. Omit the code and a random one is minted.\n\nBounded per affiliate. The label is cosmetic: it is trimmed, stripped of\ncontrol characters and capped, and it is never part of a code.",
Fields:map[string]string{
"codeView.clicks":"Clicks is how many pings this code has taken. The one STORED counter here and\npure vanity: no accrual or payout reads it, pings are coalesced in memory and\nflushed in batches, and a dropped tally is accepted rather than contending\nwith the money write path. Do not reconcile it against anything.",
"codeView.code":"Code is the link's slug — 3–32 chars of a–z, 0–9 and hyphen — unique across\nthe WHOLE directory, so any affiliate's code resolves an attribution.",
"codeView.conversions":"Conversions is how many of those signups have actually produced positive\ncommission for the caller. Also derived, from the accrual rows, so it is\n≤ signups and lags a referral until the first sweep after it spends.",
"codeView.createdAt":"CreatedAt is when the link was minted, Unix seconds UTC.",
"codeView.label":"Label is the caller's own note for the link (\"twitter\", \"newsletter\").\nCosmetic: trimmed, stripped of control characters, capped at 48 bytes, and\nnever part of the code. \"primary\" on the link mirrored at approval.",
"codeView.signups":"Signups is how many orgs were attributed with this code — DERIVED by counting\nattribution edges, never stored, so it cannot drift from the ledger.",
"codeView.url":"URL is the full shareable link, the brand host plus ?aff=<code>. The host is\nthe deployment's own brand, so a Lux or Zoo install never mints a hanzo.ai\nlink.",
"createLinkRequest.code":"Code is an optional vanity code; it must be free across the whole\ndirectory, and omitting it mints a random one. Body-only.",
"createLinkRequest.label":"Label is cosmetic — trimmed, stripped of control characters, capped — and\nnever part of a code. Body-only: the URL cannot supply it.",
"linkMint.link":"Link is the link just minted, with its full shareable URL. Its funnel counters\nall start at zero — nothing has clicked or signed up through it yet.",
Description:"Returns one agent with its system prompt and its 20 most recent runs.\nThe ref is the agent's public id or its org-unique name — a created agent is\nimmediately gettable by whatever create handed back.",
Fields:map[string]string{
"agentRef.ref":"Ref is the agent's public id (the agent_… handle create and list return) or\nits org-unique name, from the path. Either resolves the same agent.",
"agentRef.ref":"Ref is the agent's public id (the agent_… handle create and list return) or\nits org-unique name, from the path. Either resolves the same agent.",
"agentRunView.agent":"What an operator needs to answer \"what ran, for whom, and what did it do\" —\nand, through traceId, to leave this record for the waterfall of the very\nsame run rather than a search that hopefully lands near it.\n\nAgent is on the row because the org-wide feed lists runs across agents, and\na run that cannot name its agent is an orphan in exactly the view built to\nmake sense of many of them. Every field is omitempty: a run recorded before\nthese columns existed reports absence rather than a zero it never measured.",
},
Example:json.RawMessage(`{"ref":"helper"}`),
})
zip.Describe("GET /v1/agents/:ref/runs",zip.Doc{
Description:"Returns one agent's execution history, newest first — each run's\ninput, its output or its error, and how long it took. Every row is a run that\nactually happened.",
Fields:map[string]string{
"runList.runs":"Runs is the agent's executions, newest first.",
"runsQuery.limit":"Limit caps how many runs come back, newest first. Absent, zero or out of\nrange (1..200) reads as 50.",
"runsQuery.ref":"Ref is the agent's public id or its org-unique name, from the path.",
"agentRunView.agent":"What an operator needs to answer \"what ran, for whom, and what did it do\" —\nand, through traceId, to leave this record for the waterfall of the very\nsame run rather than a search that hopefully lands near it.\n\nAgent is on the row because the org-wide feed lists runs across agents, and\na run that cannot name its agent is an orphan in exactly the view built to\nmake sense of many of them. Every field is omitempty: a run recorded before\nthese columns existed reports absence rather than a zero it never measured.",
"runList.runs":"Runs is the agent's executions, newest first.",
"runsQuery.limit":"Limit caps how many runs come back, newest first. Absent, zero or out of\nrange (1..200) reads as 50.",
"runsQuery.ref":"Ref is the agent's public id or its org-unique name, from the path.",
Description:"Returns the org's agent runs across EVERY agent, newest first —\nwhat ran here, for whom, on which model, how long it took, and why it failed.\n\nIt is the feed the per-agent history could not be: an operator asking \"what is\nthis tenant's agent plane doing\" does not start out knowing an agent ref, and\nanswering by listing the agents and then paging each one's history is N+1 round\ntrips to reconstruct one ordering the database already has (RunsSince, ordered\nby created_at over the org index).\n\nThe org is the CALLER's, resolved from identity by tenantStore — never a\nparameter. There is deliberately no org field on orgRunsQuery to forge: run\nhistory is the tenant's own record, and the only tenant this can answer for is\nthe one asking.",
Fields:map[string]string{
"agentRunView.agent":"What an operator needs to answer \"what ran, for whom, and what did it do\" —\nand, through traceId, to leave this record for the waterfall of the very\nsame run rather than a search that hopefully lands near it.\n\nAgent is on the row because the org-wide feed lists runs across agents, and\na run that cannot name its agent is an orphan in exactly the view built to\nmake sense of many of them. Every field is omitempty: a run recorded before\nthese columns existed reports absence rather than a zero it never measured.",
"orgRunsQuery.limit":"Limit caps how many runs come back, newest first. Absent, zero or out of\nrange (1..200) reads as 50.",
"orgRunsQuery.status":"Status keeps only runs with this outcome (\"ok\" or \"error\"). Empty keeps\nboth. It is the filter an operator reaches for first — \"show me what broke\"\n— and answering it here rather than by paging the whole history client-side\nis the difference between a usable feed and a download.",
"runList.runs":"Runs is the agent's executions, newest first.",
Description:"Returns the caller org's live sessions, newest first — each with\nits event count, its direct-child count and a one-line preview of its latest\nevent. With no filter it returns ROOT sessions only, so a dashboard shows one\nrow per flow rather than one per subagent; ?root= or ?parent= descends.",
Description:"Answers a bridge's turn.\n\nUnlike the session ops, the org travels IN the request rather than being taken\nfrom the caller's plane identity: the tenant here is the one that connected the\nSlack workspace, resolved by the bridge from the signed team_id, and the bridge\nplugin's own identity is not it. That is safe because this op only SPENDS the\nnamed org's own balance under its own agent — it reads nothing across tenants —\nand because the subject must be a link the bridge already proved.\n\nAn empty subject is refused rather than defaulted. A turn that lost its caller\nmust not run AS THE ORG: that would bill the tenant for an unattributable act\nand hand an unlinked user the org's agent.",
Fields:map[string]string{
"RunOnBehalfIn.input":"Input is the user's message, already stripped of the leading @mention.",
"RunOnBehalfIn.model":"Model is the ASKER's own choice, empty when they have not made one. It is a\npreference of the person, not a property of the agent, which is why it rides\nthe turn instead of being written into an agent row: two people in one\nworkspace can prefer different models of the same assistant.",
"RunOnBehalfIn.org":"Org is the isolation gate, the tenant, and the balance the run bills.",
"RunOnBehalfIn.ref":"Ref names the agent to run.",
"RunOnBehalfIn.subject":"Subject is the caller's LINKED Hanzo identity, unqualified. Attribution and\nauthorization both hang off it, so a turn can never run as nobody: the\nanswering side refuses an empty subject rather than falling back to the org.",
Description:"Tears down every live session of the CALLER's org matching\nthe revoking subject, and reports how many it stopped.\n\nThe org is the caller's plane identity and never the argument — plane\n.SessionMatchIn has no org field, deliberately, because this op STOPS things\nand a caller able to state the org could stop a co-tenant's work. Anonymous is\nrefused rather than defaulted: a teardown arriving with no principal must\nfail, not pick a tenant.\n\nThe actor is built HERE, from the org the plane proved and the subject the\ncaller names, so the HIGH-1 actor scoping (a revoke stops only that user's own\nsessions) is enforced by the side that owns the store rather than trusted from\nthe wire.\n\nA named handler, not a closure, so zipdoc can lift this prose into the registry.",
Fields:map[string]string{
"SessionMatchIn.host":"Host/Provider/Account narrow WITHIN the actor's own sessions; empty is any.",
"SessionMatchIn.subject":"Subject is the revoking user, unqualified. The answering side qualifies it.",
},
})
zip.Describe("POST /v1/agents",zip.Doc{
Description:"Defines an agent in the caller's org: a model, a system prompt\n(instructions) and a set of tool names. The name must be unique in the org and\nmatch ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$. An omitted model takes the\ndeployment's configured default; a named one is checked against the gateway's\nserved catalog, so a model this deployment never serves is refused here rather\nthan failing at run time. A long-running agent must carry a 5-field cron\nschedule (the scheduler would otherwise never fire it) and counts against a\nper-org cap on scheduled agents.",
for_,q:=range[]string{"what's my MRR?","how long is my runway","are we profitable?","how much cash do we have","what's my gross margin","show me the P&L","how much did we make"}{
ifreg.Match(q)==nil{
t.Fatalf("financial question %q must match the books contributor",q)
// TestClassifierRoutesEachDomainsVocab locks the classifiers: each domain's vocabulary routes to
// it, and off-topic questions match nothing at all rather than being swept into whichever domain
{"books",[]string{"what's my MRR?","how long is my runway","are we profitable?","how much cash do we have","what's my gross margin","show me the P&L","how much did we make"}},
{"projects",[]string{"what have I deployed?","which projects are live","what sites have I published","what is running in production","what did we ship"}},
{"git",[]string{"how many repositories do I have?","what changed recently","how much code do we have","list my repos","which branches are there"}},
}{
for_,q:=rangetc.questions{
c:=reg.Match(q)
ifc==nil{
t.Fatalf("%q must match the %s contributor, matched nothing",q,tc.want)
}
ifc.Name()!=tc.want{
t.Fatalf("%q must route to %s, routed to %s",q,tc.want,c.Name())
}
}
}
for_,q:=range[]string{"what's the weather","how many users signed up","deploy the app"}{
for_,q:=range[]string{"what's the weather","how many users signed up","who is the CEO of France"}{
ifc:=reg.Match(q);c!=nil{
t.Fatalf("off-topic question %q must NOT match any domain, matched %q",q,c.Name())
Description:"Returns the platform's whole author program — every org's author\nrecord, not the caller's — with each one's repository and deploy counts and a\nfleet roll-up of the money accrued, pending and paid.\n\nIt is a Hanzo platform operation: a caller who is not a SuperAdmin gets 403. It\nexposes the owning org of each author, which no tenant-facing read ever does.",
Fields:map[string]string{
"adminBook.data":"Data is the book.",
"adminBook.msg":"Msg is the envelope's message slot, empty on success.",
"adminBook.status":"Status is \"ok\" — the operator console's envelope discriminator.",
"adminBookData.authors":"Authors are the author records, with each one's repository and deploy counts.",
"adminBookData.summary":"Summary is the fleet roll-up: how many authors at each status and the money\naccrued, pending and paid across all of them.",
"adminLimit.limit":"Limit bounds the page. 0 or less means the default of 500; anything above\n1000 is clamped to 1000.",
"adminAuthorView.accruedCents":"AccruedCents is lifetime royalty accrued, in integer USD cents: the sum of\nevery latched accrual (spend × shareBps / 10000). It only ever rises — a\npayout is recorded against paidCents and never reduces this.",
"adminAuthorView.approvedAt":"ApprovedAt is unix seconds of the first approval, and 0 means never approved —\nwhich is also \"has never been able to accrue\". Re-approving to renegotiate the\nshare leaves it at the original date.",
"adminAuthorView.createdAt":"CreatedAt is unix seconds at the FIRST connect. Re-connecting re-links the\nlogin and leaves this alone, so it dates the enrolment, not the latest link.",
"adminAuthorView.deployCount":"DeployCount is how many attribution edges point at this author — one per\n(repository, project, deploying org), so re-deploying the same project adds\nnone. It includes self-deploys, which are recorded for provenance and excluded\nfrom accrual, so it measures reach, not the earning set.",
"adminAuthorView.githubLogin":"GithubLogin is the linked forge account, lowercased. It comes from IAM's\nlinked account when the connect had one — which is also what sets verified —\nand otherwise from the login the caller declared. The treasury author carries\n\"<brand>-maintainers\".",
"adminAuthorView.id":"ID is the author record's server-minted handle, \"aut_\"-prefixed. It is the id\nthe approve, suspend, payout and admin-basis routes address.",
"adminAuthorView.org":"Org is the tenant org that owns this author record — UNIQUE, one author per\norg. It is exposed HERE and nowhere else (Author.Org is json:\"-\" on the tenant\nsurface), and it is the org excluded from this author's own accrual: deploying\nyour own repo earns you nothing.",
"adminAuthorView.paidCents":"PaidCents is lifetime royalty RECORDED as paid, in integer USD cents. It rises\nthe moment a payout reserves against pending — recording, not settling; a human\nmoves the money out of band — and falls back only when a payout is voided.",
"adminAuthorView.pendingCents":"PendingCents is what a payout may still draw against — accrued − paid, floored\nat zero. It is derived for each response, never stored, and it is the exact\nfigure the atomic payout guard refuses to exceed.",
"adminAuthorView.repoCount":"RepoCount is how many of this author's repository claims are VERIFIED, counted\nfor this response in one GROUP BY over the whole table rather than a query per\nrow. The single-author replies from approve, suspend and payout report 0: they\ncarry the mutated row, not a re-listing.",
"adminAuthorView.shareBps":"ShareBps is the royalty rate accrual applies, in basis points of a deploying\norg's metered spend for the period: 2000 (the platform default) is 20%, 10000\nwould be the entire spend. The platform keeps 10000 − shareBps. Changing it\nnever rewrites history — each ledger row keeps the rate it was written with.",
"adminAuthorView.status":"Status is connected, approved or suspended. Only an approved author accrues;\na connected one may verify repos and collect deploy edges but earns nothing\nuntil a reviewer admits it.",
"adminAuthorView.suspendedAt":"SuspendedAt is unix seconds of the most recent suspension. 0 means the author\nis not suspended: either never was, or was and has since been approved again,\nwhich clears this back to 0.",
"adminAuthorView.verified":"Verified is IDENTITY proof of the login, NOT proof of any repository: true\nwhen the connect took the login from IAM's linked forge account (and for the\nseeded treasury author), false when the caller merely declared it. A false\nhere still earns — repository ownership is proven separately, per claim.",
"adminBook.data":"Data is the book.",
"adminBook.msg":"Msg is the envelope's message slot, empty on success.",
"adminBook.status":"Status is \"ok\" — the operator console's envelope discriminator.",
"adminBookData.authors":"Authors are the author records, with each one's repository and deploy counts.",
"adminBookData.summary":"Summary is the fleet roll-up: how many authors at each status and the money\naccrued, pending and paid across all of them.",
"adminLimit.limit":"Limit bounds the page. 0 or less means the default of 500; anything above\n1000 is clamped to 1000.",
"authorProgramSummary.accruedCents":"AccruedCents is the page's lifetime royalty accrued, in integer USD cents.",
"authorProgramSummary.approved":"Approved is how many are admitted and accruing.",
"authorProgramSummary.connected":"Connected is how many of those are enrolled but not yet admitted to earning.",
"authorProgramSummary.paidCents":"PaidCents is what has been RECORDED as paid across the page, in integer USD\ncents. Recorded, not settled: the money leaves in a human's hands.",
"authorProgramSummary.pendingCents":"PendingCents is what the platform still owes across the page, in integer USD\ncents — the sum of each author's own accrued − paid, each floored at zero.",
"authorProgramSummary.suspended":"Suspended is how many have been stopped from accruing further. An author holds\nexactly one status, so the three buckets never overlap and connected +\napproved + suspended = total.",
"authorProgramSummary.total":"Total is how many author records this response actually carried. The roll-up\nis folded over the SAME page as authors — newest first, bounded by limit\n(default 500, ceiling 1000) — so on a program larger than the page it\nsummarizes that page, not the fleet.",
Description:"Admits one author to EARNING, optionally on a negotiated royalty\nshare. Until this runs, a connected author accrues nothing however many verified\nrepositories they have.\n\nA share override applies from here forward only — existing ledger rows keep the\nshare that was applied when they were written, because a rate change must never\nrewrite what was already owed.\n\nA Hanzo platform operation: a caller who is not a SuperAdmin gets 403.",
Fields:map[string]string{
"approveRequest.id":"ID is the author to approve, from the path.",
"approveRequest.shareBps":"ShareBps overrides this author's royalty share, in basis points (0–10000).\n0 keeps the platform default. A share change never rewrites history: existing\nledger rows keep theshare that was applied when they were written.",
"authorData.author":"Author is the author record after the change. Its repository and deploy counts\nare 0 here — this is the mutated row, not a re-listing.",
"authorResult.data":"Data carries the author.",
"authorResult.msg":"Msg is the envelope's message slot, empty on success.",
"authorResult.status":"Status is \"ok\".",
"adminAuthorView.accruedCents":"AccruedCents is lifetime royalty accrued, in integer USD cents: the sum of\nevery latched accrual (spend × shareBps / 10000). It only ever rises — a\npayout is recorded against paidCents and never reduces this.",
"adminAuthorView.approvedAt":"ApprovedAt is unix seconds of the first approval, and 0 means never approved —\nwhich is also \"has never been able to accrue\". Re-approving to renegotiate the\nshare leaves it at the original date.",
"adminAuthorView.createdAt":"CreatedAt is unix seconds at the FIRST connect. Re-connecting re-links the\nlogin and leaves this alone, so it dates the enrolment, not the latest link.",
"adminAuthorView.deployCount":"DeployCount is how many attribution edges point at this author — one per\n(repository, project, deploying org), so re-deploying the same project adds\nnone. It includes self-deploys, which are recorded for provenance and excluded\nfrom accrual, so it measures reach, not the earning set.",
"adminAuthorView.githubLogin":"GithubLogin is the linked forge account, lowercased. It comes from IAM's\nlinked account when the connect had one — which is also what sets verified —\nand otherwise from the login the caller declared. The treasury author carries\n\"<brand>-maintainers\".",
"adminAuthorView.id":"ID is the author record's server-minted handle, \"aut_\"-prefixed. It is the id\nthe approve, suspend, payout and admin-basis routes address.",
"adminAuthorView.org":"Org is the tenant org that owns this author record — UNIQUE, one author per\norg. It is exposed HERE and nowhere else (Author.Org is json:\"-\" on the tenant\nsurface), and it is the org excluded from this author's own accrual: deploying\nyour own repo earns you nothing.",
"adminAuthorView.paidCents":"PaidCents is lifetime royalty RECORDED as paid, in integer USD cents. It rises\nthe moment a payout reserves against pending — recording, not settling; a human\nmoves the money out of band — and falls back only when a payout is voided.",
"adminAuthorView.pendingCents":"PendingCents is what a payout may still draw against — accrued − paid, floored\nat zero. It is derived for each response, never stored, and it is the exact\nfigure the atomic payout guard refuses to exceed.",
"adminAuthorView.repoCount":"RepoCount is how many of this author's repository claims are VERIFIED, counted\nfor this response in one GROUP BY over the whole table rather than a query per\nrow. The single-author replies from approve, suspend and payout report 0: they\ncarry the mutated row, not a re-listing.",
"adminAuthorView.shareBps":"ShareBps is the royalty rate accrual applies, in basis points of a deploying\norg's metered spend for the period: 2000 (the platform default) is 20%, 10000\nwould be the entire spend. The platform keeps 10000 − shareBps. Changing it\nnever rewrites history — each ledger row keeps the rate it was written with.",
"adminAuthorView.status":"Status is connected, approved or suspended. Only an approved author accrues;\na connected one may verify repos and collect deploy edges but earns nothing\nuntil a reviewer admits it.",
"adminAuthorView.suspendedAt":"SuspendedAt is unix seconds of the most recent suspension. 0 means the author\nis not suspended: either never was, or was and has since been approved again,\nwhich clears this back to 0.",
"adminAuthorView.verified":"Verified is IDENTITY proof of the login, NOT proof of any repository: true\nwhen the connect took the login from IAM's linked forge account (and for the\nseeded treasury author), false when the caller merely declared it. A false\nhere still earns — repository ownership is proven separately, per claim.",
"approveRequest.id":"ID is the author to approve, from the path.",
"approveRequest.shareBps":"ShareBps overrides this author's royalty share, in basis points (0–10000).\n0 keeps the platform default. A share change never rewrites history: existing\nledger rows keep the share that was applied when they were written.",
"authorData.author":"Author is the author record after the change. Its repository and deploy counts\nare 0 here — this is the mutated row, not a re-listing.",
"authorResult.data":"Data carries the author.",
"authorResult.msg":"Msg is the envelope's message slot, empty on success.",
Description:"Records a payout of accrued royalty and settles it.\n\nThe amount is RESERVED against the author's pending royalty atomically before\nanything is paid, so a payout can never exceed what is owed even under concurrent\ncalls. An external author's payout is then BACKED against the platform reserve\nfund — a second, independent guard — and refused with 402 if the reserve cannot\ncover it, with the reservation voided. A \"credits\" method issues the actual wallet\ngrant after both guards; a cash method is record-only. A first-party (treasury)\nauthor's royalty is realized into Hanzo's own reserve instead of an external\nwallet, and every payout row discloses which of the three it was.\n\nA Hanzo platform operation: a caller who is not a SuperAdmin gets 403.",
Fields:map[string]string{
"payoutData.author":"Author is the author record after the payout, with the balances updated.",
"payoutData.payout":"Payout is the recorded payout, including where it settled.",
"payoutRequest.amountCents":"AmountCents is how much to pay, in cents. Must be positive and can never\nexceed the author's pending royalty (accrued minus paid).",
"payoutRequest.id":"ID is the author to pay, from the path.",
"payoutRequest.method":"Method is how it settles: \"credits\" issues a grant into the author's wallet;\nwire, paypal and the like are record-only. Required.",
"payoutRequest.reference":"Reference is the operator's external reference for a cash settlement — a wire\nconfirmation, a PayPal transaction id.",
"payoutResult.data":"Data carries the payout and the author.",
"payoutResult.msg":"Msg is the envelope's message slot, empty on success.",
"payoutResult.status":"Status is \"ok\".",
"payoutView.settlement":"Settlement discloses treasury-vs-wallet-vs-cash on every payout, to the author\nand to the admin mirror alike — the disclosure that keeps a first-party\nsettlement legible as internal accounting.",
"adminAuthorView.accruedCents":"AccruedCents is lifetime royalty accrued, in integer USD cents: the sum of\nevery latched accrual (spend × shareBps / 10000). It only ever rises — a\npayout is recorded against paidCents and never reduces this.",
"adminAuthorView.approvedAt":"ApprovedAt is unix seconds of the first approval, and 0 means never approved —\nwhich is also \"has never been able to accrue\". Re-approving to renegotiate the\nshare leaves it at the original date.",
"adminAuthorView.createdAt":"CreatedAt is unix seconds at the FIRST connect. Re-connecting re-links the\nlogin and leaves this alone, so it dates the enrolment, not the latest link.",
"adminAuthorView.deployCount":"DeployCount is how many attribution edges point at this author — one per\n(repository, project, deploying org), so re-deploying the same project adds\nnone. It includes self-deploys, which are recorded for provenance and excluded\nfrom accrual, so it measures reach, not the earning set.",
"adminAuthorView.githubLogin":"GithubLogin is the linked forge account, lowercased. It comes from IAM's\nlinked account when the connect had one — which is also what sets verified —\nand otherwise from the login the caller declared. The treasury author carries\n\"<brand>-maintainers\".",
"adminAuthorView.id":"ID is the author record's server-minted handle, \"aut_\"-prefixed. It is the id\nthe approve, suspend, payout and admin-basis routes address.",
"adminAuthorView.org":"Org is the tenant org that owns this author record — UNIQUE, one author per\norg. It is exposed HERE and nowhere else (Author.Org is json:\"-\" on the tenant\nsurface), and it is the org excluded from this author's own accrual: deploying\nyour own repo earns you nothing.",
"adminAuthorView.paidCents":"PaidCents is lifetime royalty RECORDED as paid, in integer USD cents. It rises\nthe moment a payout reserves against pending — recording, not settling; a human\nmoves the money out of band — and falls back only when a payout is voided.",
"adminAuthorView.pendingCents":"PendingCents is what a payout may still draw against — accrued − paid, floored\nat zero. It is derived for each response, never stored, and it is the exact\nfigure the atomic payout guard refuses to exceed.",
"adminAuthorView.repoCount":"RepoCount is how many of this author's repository claims are VERIFIED, counted\nfor this response in one GROUP BY over the whole table rather than a query per\nrow. The single-author replies from approve, suspend and payout report 0: they\ncarry the mutated row, not a re-listing.",
"adminAuthorView.shareBps":"ShareBps is the royalty rate accrual applies, in basis points of a deploying\norg's metered spend for the period: 2000 (the platform default) is 20%, 10000\nwould be the entire spend. The platform keeps 10000 − shareBps. Changing it\nnever rewrites history — each ledger row keeps the rate it was written with.",
"adminAuthorView.status":"Status is connected, approved or suspended. Only an approved author accrues;\na connected one may verify repos and collect deploy edges but earns nothing\nuntil a reviewer admits it.",
"adminAuthorView.suspendedAt":"SuspendedAt is unix seconds of the most recent suspension. 0 means the author\nis not suspended: either never was, or was and has since been approved again,\nwhich clears this back to 0.",
"adminAuthorView.verified":"Verified is IDENTITY proof of the login, NOT proof of any repository: true\nwhen the connect took the login from IAM's linked forge account (and for the\nseeded treasury author), false when the caller merely declared it. A false\nhere still earns — repository ownership is proven separately, per claim.",
"payoutData.author":"Author is the author record after the payout, with the balances updated.",
"payoutData.payout":"Payout is the recorded payout, including where it settled.",
"payoutRequest.amountCents":"AmountCents is how much to pay, in cents. Must be positive and can never\nexceed the author's pending royalty (accrued minus paid).",
"payoutRequest.id":"ID is the author to pay, from the path.",
"payoutRequest.method":"Method is how it settles: \"credits\" issues a grant into the author's wallet;\nwire, paypal and the like are record-only. Required.",
"payoutRequest.reference":"Reference is the operator's external reference for a cash settlement — a wire\nconfirmation, a PayPal transaction id.",
"payoutResult.data":"Data carries the payout and the author.",
"payoutResult.msg":"Msg is the envelope's message slot, empty on success.",
"payoutResult.status":"Status is \"ok\".",
"payoutView.amountCents":"AmountCents is the amount RESERVED against pending royalty, in integer USD\ncents, always positive. The reservation is atomic and can never exceed\naccrued − paid, so this is owed money moved out of pending — not money moved.",
"payoutView.createdAt":"CreatedAt is unix seconds when the payout was RECORDED — the moment the amount\nleft pending, not the moment a human moved the money.",
"payoutView.id":"ID is the payout row's server-minted handle, \"apo_\"-prefixed. A caller never\nsupplies it; it is what an operator quotes when reconciling a settlement.",
"payoutView.method":"Method is how the operator says this settles, lowercased as recorded.\n\"credits\" is the one method that means the author's own wallet; anything else\n— wire, paypal, check — is a cash disbursement a human performs. Recording it\npays nobody either way.",
"payoutView.reference":"Reference is the operator's external handle for the settlement: a wire\nconfirmation, a PayPal transaction id. Absent when none was given.",
"payoutView.settlement":"Settlement discloses treasury-vs-wallet-vs-cash on every payout, to the author\nand to the admin mirror alike — the disclosure that keeps a first-party\nsettlement legible as internal accounting.",
"payoutView.txn":"Txn is the commerce ledger transaction id of a SETTLED credits payout, and it\nis absent on every payout this service records. Recording moves no money, and\nauthors asks the money plane exactly one question — what has this org spent? —\nwith no write to answer it with, so there is no receipt to carry. It fills in\nonly when a settlement stamps its transaction back onto the row.",
Description:"Stops one author earning. Their record, verified claims and ledger\nare untouched — suspension halts future accrual, it does not erase what was already\nowed, and it does not delete the evidence behind it.\n\nA Hanzo platform operation: a caller who is not a SuperAdmin gets 403.",
Fields:map[string]string{
"authorData.author":"Author is the author record after the change. Its repository and deploy counts\nare 0 here — this is the mutated row, not a re-listing.",
"authorRef.id":"ID is the author record's handle, \"aut_\"-prefixed.",
"authorResult.data":"Data carries the author.",
"authorResult.msg":"Msg is the envelope's message slot, empty on success.",
"authorResult.status":"Status is \"ok\".",
"adminAuthorView.accruedCents":"AccruedCents is lifetime royalty accrued, in integer USD cents: the sum of\nevery latched accrual (spend × shareBps / 10000). It only ever rises — a\npayout is recorded against paidCents and never reduces this.",
"adminAuthorView.approvedAt":"ApprovedAt is unix seconds of the first approval, and 0 means never approved —\nwhich is also \"has never been able to accrue\". Re-approving to renegotiate the\nshare leaves it at the original date.",
"adminAuthorView.createdAt":"CreatedAt is unix seconds at the FIRST connect. Re-connecting re-links the\nlogin and leaves this alone, so it dates the enrolment, not the latest link.",
"adminAuthorView.deployCount":"DeployCount is how many attribution edges point at this author — one per\n(repository, project, deploying org), so re-deploying the same project adds\nnone. It includes self-deploys, which are recorded for provenance and excluded\nfrom accrual, so it measures reach, not the earning set.",
"adminAuthorView.githubLogin":"GithubLogin is the linked forge account, lowercased. It comes from IAM's\nlinked account when the connect had one — which is also what sets verified —\nand otherwise from the login the caller declared. The treasury author carries\n\"<brand>-maintainers\".",
"adminAuthorView.id":"ID is the author record's server-minted handle, \"aut_\"-prefixed. It is the id\nthe approve, suspend, payout and admin-basis routes address.",
"adminAuthorView.org":"Org is the tenant org that owns this author record — UNIQUE, one author per\norg. It is exposed HERE and nowhere else (Author.Org is json:\"-\" on the tenant\nsurface), and it is the org excluded from this author's own accrual: deploying\nyour own repo earns you nothing.",
"adminAuthorView.paidCents":"PaidCents is lifetime royalty RECORDED as paid, in integer USD cents. It rises\nthe moment a payout reserves against pending — recording, not settling; a human\nmoves the money out of band — and falls back only when a payout is voided.",
"adminAuthorView.pendingCents":"PendingCents is what a payout may still draw against — accrued − paid, floored\nat zero. It is derived for each response, never stored, and it is the exact\nfigure the atomic payout guard refuses to exceed.",
"adminAuthorView.repoCount":"RepoCount is how many of this author's repository claims are VERIFIED, counted\nfor this response in one GROUP BY over the whole table rather than a query per\nrow. The single-author replies from approve, suspend and payout report 0: they\ncarry the mutated row, not a re-listing.",
"adminAuthorView.shareBps":"ShareBps is the royalty rate accrual applies, in basis points of a deploying\norg's metered spend for the period: 2000 (the platform default) is 20%, 10000\nwould be the entire spend. The platform keeps 10000 − shareBps. Changing it\nnever rewrites history — each ledger row keeps the rate it was written with.",
"adminAuthorView.status":"Status is connected, approved or suspended. Only an approved author accrues;\na connected one may verify repos and collect deploy edges but earns nothing\nuntil a reviewer admits it.",
"adminAuthorView.suspendedAt":"SuspendedAt is unix seconds of the most recent suspension. 0 means the author\nis not suspended: either never was, or was and has since been approved again,\nwhich clears this back to 0.",
"adminAuthorView.verified":"Verified is IDENTITY proof of the login, NOT proof of any repository: true\nwhen the connect took the login from IAM's linked forge account (and for the\nseeded treasury author), false when the caller merely declared it. A false\nhere still earns — repository ownership is proven separately, per claim.",
"authorData.author":"Author is the author record after the change. Its repository and deploy counts\nare 0 here — this is the mutated row, not a re-listing.",
"authorRef.id":"ID is the author record's handle, \"aut_\"-prefixed.",
"authorResult.data":"Data carries the author.",
"authorResult.msg":"Msg is the envelope's message slot, empty on success.",
Description:"Proves that the caller owns a repository — or a whole OWNER — and\nrecords the claim, which is what makes deploys of that code earn royalty.\n\nOwnership is proven the SAME two ways in both cases, tried in order: an IAM-linked\nforge token with admin or push permission, or a hanzo.json on the default branch\ncarrying the author's verify code. Claiming an OWNER proves it against that\nowner's \".github\" control repository, and is exactly as strong as a per-repository\nclaim — an owner the caller cannot prove is refused with 422, never assumed.\n\nA per-repository claim wins over an owner-wide one, so a specifically-claimed\nrepository always earns for its own author. A repository another author has\nalready verified is a 409. The org must have connected first.\n\nAnswers 201 when it recorded a new claim and 200 when the claim already existed.",
Fields:map[string]string{
"claim.created":"Created reports whether this call recorded a new claim (201) or found an\nexisting one (200).",
"claim.org":"Org is the verified owner-wide claim, present when an owner was claimed. It\ncovers every repository the author publishes under that owner.",
"claim.repo":"Repo is the verified repository claim, present when a repository was claimed.",
"verifyRequest.repoUrl":"RepoURL is what to claim: a repository (github.com/owner/name) or a whole\nOWNER (github.com/owner, no repository segment). gitlab.com is accepted too.",
"authorRepo.badgeMarkdown":"BadgeMarkdown is the ready-to-paste README snippet, DERIVED for each response\nfrom this deployment's badge host and never stored: a \"Deploy on Hanzo\" image\nlinking to the one-click import of this repository. Re-hosting the builder\nchanges every badge without touching a row.",
"authorRepo.createdAt":"CreatedAt is unix seconds when the claim was first recorded. It equals\nverifiedAt on the first proof and then stays put while verifiedAt moves, so the\npair reads as \"claimed since / last proven\".",
"authorRepo.method":"Method is HOW ownership was proven: \"oauth\" — an IAM-linked forge token showed\nadmin or push on the repository; \"file\" — a hanzo.json on the default branch\ncarried this author's verify code; or \"maintainer\" — the repository sits in a\nfirst-party namespace, where ownership is intrinsic and the treasury author\nholds it with no proof step. Omitted on a row written before the method was\nrecorded.",
"authorRepo.repoUrl":"RepoURL is the claim key in canonical form — lowercased \"host/owner/name\",\nno scheme, no .git, host ∈ {github.com, gitlab.com}. A deploy's source repo is\nnormalized through the same function before attribution, so the two sides can\nnever miss on a cosmetic difference. UNIQUE across every author: first proven\nclaim wins.",
"authorRepo.verified":"Verified reports that ownership was proven. Only a proven claim is ever\nwritten, so it is true on every row this surface returns; the deploy path\nre-reads it regardless, because an unverified claim attributes nothing.",
"authorRepo.verifiedAt":"VerifiedAt is unix seconds of the most recent successful proof. Re-verifying\nrefreshes it, and the method beside it, in place.",
"claim.created":"Created reports whether this call recorded a new claim (201) or found an\nexisting one (200).",
"claim.org":"Org is the verified owner-wide claim, present when an owner was claimed. It\ncovers every repository the author publishes under that owner.",
"claim.repo":"Repo is the verified repository claim, present when a repository was claimed.",
"orgView.badgeMarkdown":"BadgeMarkdown is the ready-to-paste README snippet, DERIVED for each response\nfrom this deployment's badge host and never stored — here it deep-links the\nOWNER's template import rather than one repository's.",
"orgView.createdAt":"CreatedAt is unix seconds when the owner claim was first recorded — equal to\nverifiedAt on the first proof, then fixed while verifiedAt moves.",
"orgView.method":"Method is HOW the owner was proven, always against its \".github\" control\nrepository: \"oauth\" — an IAM-linked forge token showed admin or push on it; or\n\"file\" — a hanzo.json on its default branch carried this author's verify code.\nThe \"maintainer\" shortcut is a per-repository attribution and never appears\nhere. Omitted on a row written before the method was recorded.",
"orgView.ownerUrl":"OwnerURL is the claim key in canonical form — lowercased \"host/owner\" with NO\nrepository segment, host ∈ {github.com, gitlab.com}. It covers every repository\nunder that owner, so code with no claim of its own still earns; a per-repository\nclaim outranks it. UNIQUE across every author: first proven claim wins.",
"orgView.verified":"Verified reports that ownership of the WHOLE owner was proven — against that\nowner's \".github\" control repository, which is exactly as strong as a\nper-repository claim. Only a proven claim is written, so every row returned\nhere is true.",
"orgView.verifiedAt":"VerifiedAt is unix seconds of the most recent successful proof of the owner;\nre-verifying refreshes it, and the method beside it, in place.",
"verifyRequest.repoUrl":"RepoURL is what to claim: a repository (github.com/owner/name) or a whole\nOWNER (github.com/owner, no repository segment). gitlab.com is accepted too.",
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
zeekay
hanzo-devandzeekay
z