ghcr.io/hanzoai/billing:1.0.25
sha256:337719abf3c65215706b3766264e5301d68143283067eef51ca15fe01498ddf9
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 36083
All three passed against a stub, and — far worse — against the real thing
while sign-in redirected to `https://door.example.com`, a placeholder host that
does not resolve. Clicking sign in landed on a browser error page, and nothing
reported it for as long as that has been shipping.
Every assertion had an escape:
- `hasAuth = … || body.length > 0`. Any non-empty body passed, which made
the three clauses before it decorative.
- `if (url.startsWith('chrome-error'))` fell back to fetching the ORIGIN,
which is healthy — converting the one signal that the browser could not
reach the page into a reason to measure something else.
- "API health check" accepted `status < 500` across `/health`, `/api/health`
and `/`. Visor is an SPA and answers 200 with its shell for every unknown
path, so a 404 route and a healthy one were indistinguishable.
Now: the surface names itself in its title; sign-in must reach a RESOLVABLE
hanzo.id authorize naming `hanzo-visor`, with ERR_NAME_NOT_RESOLVED watched
explicitly because that failure is otherwise invisible — the browser sits on
chrome-error:// and every text assertion reads an empty document; and the API
test asserts JSON from `/v1/regions`, since content-type is what separates a
real route from the shell.
Production: 2 pass, 1 red — the red is the live login, fixed in hanzoai/visor
`web/src/Conf.js`. Against the stub: 3 passing became 0.
Two verbs the code calls and the Role did not grant, so both calls have been
failing at Warn in production since the namespace went up:
pods list reap.go orphans(). A pod whose row is gone is claimed by nobody,
so no store read can name it — asking the API server what is
running here is the only way to see it. The sweep runs every
minute, logs, and returns; orphaned pods accumulate forever.
pvc delete runtime.go purge(), reached only by
DELETE /v1/sandboxes/:id?purge=1. Withholding the destructive
verb did not make purge safe — it made purge answer 204 and
leave the disk. A caller who asked to delete their volume was
told it was done.
Measured before:
kubectl auth can-i list pods --as system:serviceaccount:hanzo:cloud \
-n hanzo-sandboxes -> no
kubectl auth can-i delete persistentvolumeclaims ... -> no
No watch. Nothing in apps/sandbox opens one — the sweep lists once per pass —
and this file's rule is that the verbs are what the code calls and nothing
else. The header said "no list, no watch" and "NO delete"; the code stopped
matching that, so the prose is corrected here rather than left contradicting
the rules it explains.
`the OIDC scope is not duplicated` read `page.url()` straight after loading the
insights landing — a page with no query string at all. So its parts list was
empty, and empty trivially equals its own Set: it passed by measuring nothing.
Meanwhile the defect it is named for is LIVE:
scope=openid+email+profile+openid+profile+email
Every value twice, on the real hanzo-insights authorize. A test written for a
specific bug, green while that exact bug ships, is worse than no test — it also
tells you not to look. It now reads the authorize URL, requires a scope to
exist before judging it for duplicates, and is correctly RED.
Six more here were absence checks with no floor. "renders a real UI, no
overflow, monochrome, honest" asserted `text.length > 20` and then tested for
things that were NOT there — no sideways scroll, no purple, no overclaim. An
absence only means something once you know what you are looking at, and twenty
characters is not a UI: analytics renders 33 in total and cleared it with 13 to
spare. Each lens now declares the words that identify it.
The header test had the same shape — no dead anchors, not more than one Sign in
— both satisfied perfectly by a page with no header. It counts the anchors
first.
`insights` no longer bounces to IAM; it serves its own landing with a Sign in.
The test asserting the bounce has been red against production and is rewritten
to the surface that exists rather than deleted — its replacement proves the
handoff still opens a complete authorize for hanzo-insights.
Production: 10 passed / 2 failed, the same count as before but both failures
now real (the ERRORS-shell red is pre-existing). Against the stub: 8 passing
became 0.
Nine of this file's tests passed against a server that answers 200 on every
path and implements nothing. Three shapes, one mistake — asking whether
something happened rather than asserting it:
- `toBeLessThan(500)`, named "host is reachable (non-5xx)". A 404 satisfies
it. So does anything that answers at all.
- "IAM authorize URL carries OIDC params (WHEN REACHED)" — when the redirect
had not happened it wrote an annotation and returned, passing in exactly
the case it existed to catch.
- The landing CTA test held THREE escapes: no link found → return, href
already points at hanzo.id → return, then `reachedIam || reachedAuth`,
which any URL containing "auth" satisfies.
Measured first, then asserted. Every entry point into this product — the
landing CTAs and all three playground hosts — arrives at a hanzo.id authorize
naming `client_id=hanzo-bot`, `response_type=code`, `scope` with `openid` and
a `redirect_uri`. That is now one shared assertion applied whole and
unconditionally, so the "when reached" hedge is gone: the redirect IS the
contract. Naming the client is what makes it this product's gate rather than
some sign-in page somewhere.
Three tests per host became one. The two removed proved nothing the survivor
does not, and both passed against the stub.
The console route needed BOTH halves and neither alone: `toBe(200)` still
survived the stub (a server answering 200 everywhere answers this too), and
gating alone proves nothing about the route, because an SPA redirects an
unknown path to sign-in as readily as a real one. Asserting existence and
protection together is what a stub cannot fake.
5 pass against production, 5 fail against the stub (was 9 passing there).
Same shape as the lux and zoo fleets: luxfi/traffic v0.2.0, TRAFFIC_KEY
from lux-deployer/LUX_MNEMONIC, 100-wallet circulation, chains.json
fanning to all five hanzod-mv pods (no mempool gossip on 36963).
Wallet 0 funded 300 AI from the faucet; chain-traffic scaled to 0 and
retired to killswitch duty.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The allowlist was derived from principals observed writing Secrets. cert-manager
writes only when a certificate renews, so a point-in-time sample could not see it
and it was omitted. ns/zen holds three live TLS secrets and zen-landing-tls
renews 2026-09-12 — the Deny binding would have refused that renewal SILENTLY,
five weeks from now, long after anyone would connect the failure to this policy.
Measured before the fix: creating a Secret in zen as
system:serviceaccount:cert-manager:cert-manager returned Forbidden. It was the
ONLY missing writer — every other SA holding secret-create in zen was already
listed, checked by enumerating them rather than by guessing.
The general failure is worth more than the entry: an allowlist derived from
observed writes is only as complete as its observation window, and anything that
writes on a TIMER — renewals, rotations, scheduled backups — is invisible to a
live sample and then fails at the least convenient moment. Enumerate RBAC, do not
sample traffic.
ghcr.io/hanzoai/app:v1.42.342
sha256:e7233e8d57a5ec56666846ba962de3e430bac39c327b61a77d8e36a1b5036a1d
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 35800
ghcr.io/hanzoai/app:v1.42.342
sha256:59282fe0ecf5bea8a267f1314429c06eb28fa7cd1fd6264b26b7c64969a89192
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 35814
/v1/sandboxes answers and then cannot do the one thing it exists for. A sandbox
IS a pod in hanzo-sandboxes, and cloud's ServiceAccount has no verb on pods
anywhere — its ClusterRole grants none deliberately, so that a compromised cloud
process cannot create or exec into a pod in kube-system or a tenant namespace.
Measured before this commit:
kubectl auth can-i create pods --namespace hanzo-sandboxes \
--as system:serviceaccount:hanzo:cloud
no
So the grant is a namespaced Role bound in hanzo-sandboxes and nowhere else, on
the SAME identity — one binary, one ServiceAccount, one more place it may act.
The verbs are what apps/sandbox calls and nothing more: pods get/create/delete to
lease and end a lease (the reaper ends them on expiry and on an hour idle),
pods/exec create because all three verbs travel over the exec subresource, and
persistentvolumeclaims get/create for the project disk. No PVC delete: a lease
ending must not take the checkout with it, and purge is opt-in — the destructive
verb is not granted rather than granted and avoided. No list and no watch: the
reaper reads the store for what is running, not the API server.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ghcr.io/hanzoai/app:v1.42.341
sha256:f91d974d385ae1382dc63dcba67188c5286e5ff4f793df2b16ad7f28509978aa
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 35747
It was evicted, and the kubelet named the cause:
The node was low on resource: memory. Container hanzocd-repo-server was
using 198560Ki, request is 128Mi, has larger consumption of memory.
A pod over its own request is in the first eviction class. For this pod that is
a bad trade twice over: 128Mi is below what it steadily uses (151Mi now, 194Mi
at eviction), so it is permanently in that class, and evicting it frees ~70Mi
while stopping git delivery for ~320 Applications.
That failure is silent, which is what makes it worth a commit. With no repo-
server able to read a new revision, every Application keeps reporting
Synced/Healthy against the last revision it managed to fetch — the dashboard
stays green while nothing ships. Observed today: an app sat Synced/Healthy at
0f9ea04f8 with the real branch tip two commits further on.
256Mi: above the 194Mi seen at eviction, under the untouched 512Mi limit.
It reported 131 tests as unable to fail. The real number is 35. The other 96
were talking to PRODUCTION for the whole run, so whether they passed said
nothing about them, and the harness had no way to tell the two apart.
Two ways a spec escapes redirection, and the second is the one that fooled me
twice:
1. A hardcoded host. 63 of 117 specs carry one — `09-iam` pins
`const HANZO_ID_URL = 'https://hanzo.id'` with no override, so eight of
its twelve "unfailable" tests were reading the live IdP.
2. A host BUILT from a var the harness cannot set. `helpers/app.ts`
composes `https://<slug>.${E2E_APP_DOMAIN}`; a domain used that way has
no local address at all, so 163 app tests ran against the live fleet
while the report called 38 of them unfailable. No hardcoded literal
appears anywhere in that file — a scan for string literals misses it
entirely, which is why the first correction was still wrong.
`scripts/measurable.cjs` applies both rules statically and prove.sh now prints
the unmeasurable set beside the verdict, so the two can never again be read as
one number. The stub also counts the requests it actually received: zero hits
means nothing below it is evidence.
The per-spec findings already fixed stand — those were confirmed redirected
test-by-test (the stub's address appears in each test's own title), not
inferred from the aggregate. What was wrong was only the total.
200Gi ephemeral was set before anyone read the meter. Read now, per pod, from
kubelet stats/summary across all 10 live runners:
6.9 8.2 8.5 9.5 13.3 14.2 14.5 21.9 24.3 42.6 GiB
Peak 42.6. The reservation was ~4.7x the worst real runner, and the cost is not
waste but placement: 200Gi against ~1141Gi allocatable bounds a 1TB node at five
runners, and both nodes sit at 98% ephemeral requested while using ~8%. The pool
cannot grow without buying nodes to satisfy a number nobody had checked.
96Gi, moved together with the docker-storage sizeLimit as that block requires.
Chosen against the two real numbers on this pod rather than rounded: 1.7x the
56.5G eviction already recorded here, 2.25x today's live peak. 64Gi was
considered and rejected for sitting 1.13x above a size this fleet has already
reached once — that reintroduces the eviction the 200Gi cap existed to stop.
Effect: five runners per node becomes eleven, node ephemeral requests 98% -> ~45%.
cpu 500m and memory 4Gi are deliberately UNTOUCHED. Neither is the binding
constraint (nothing is Pending on either), and memory is the one number here
measured to be correct: live usage peaks at 3620Mi against the 4Gi request, so
cutting it buys nothing and makes a runner first to be evicted under pressure.
A `df` inside these pods reports the NODE filesystem, not the pod, and reading
it that way understates a 42.6Gi runner as ~24KB. The command that does not lie
is recorded next to the value.
buildkitd-node is live and healthy (8/8) but no client can dial it. The
namespace egress CNP denies every internal CIDR, and the only hole punched
for a builder names `app: buildkitd` -- the hand-applied spike StatefulSet,
not the DaemonSet's `app: buildkitd-node`. So the 2.68x is deployed and
unreachable: cut a scheduler over today and every build hangs 20s on a
silent drop, then fails.
Measured, from a pod on runner-pool-32g-3m84nd (a node WITH a local daemon
and a Ready endpoint, so this is policy, not placement):
before: dial tcp 10.124.63.67🔢 i/o timeout [20.01s, build fails]
after: buildctl debug workers -> s18ze40s1skcv0ymr9dfrmxgf linux/amd64
cache-mount build: CACHE MISS 2.0s -> CACHE HIT 0.0s on rerun
The client in that test ran privileged:false,
allowPrivilegeEscalation:false -- delegating to the daemon is what lets the
build pod drop privilege, so this rule is also what makes that possible.
Added to the existing policy rather than as a second one: "what may a build
pod egress to" is one question, and two files answer it twice. Scoped to
the daemon's label and its single gRPC port, so no other internal
destination opens up.
Supersedes the untracked `buildkitd-client-egress` CNP; delete that
alongside the spike.
Every pod of the DaemonSet logged FailedToRetrieveImagePullSecret for
ghcr-secret, which is not present in hanzo-build. The chart defaults
imagePullSecrets to ghcr-secret (charts/app/values.yaml:41); this file
never overrode it.
Harmless today only because moby/buildkit is public and pinned by digest.
Overriding to [] lets the namespace default SA supply registry-hanzo and
hanzo instead - both of which exist - so the pod gains working
credentials rather than losing them.
ghcr.io/hanzoai/app:v1.42.340
sha256:a998432a5af155d2a8f43857157238d72bd73a21cb7a0474877e85d89e23288d
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 35718
ghcr.io/hanzoai/app:v1.42.340
sha256:8b6c3c3e57f2255c9cc908fa1e76db600709091624655f39942f7aa3444af2cc
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 35675
0.1.0 refused every request on every host. Its jail built the chroot by binding
each allowed path at its own absolute path and THEN mounting the writable tmpfs
at <root>/tmp, which covered every bind under /tmp — where the staging directory
lives. The self-test masked its own working tree, chdir returned ENOENT, and the
child exited 126:
JAIL SELF-TEST FAILED — refusing every request
err="jail: serve canary did not run: exit status 126"
The fail-closed spine did exactly its job on it: /readyz 503, no endpoints.
0.1.1 orders the mounts shallowest path first, so an ancestor mounted after a
descendant is not representable, and execs its canary rather than reporting from
inside the process that built the jail. Measured on this pool under runsc: serve
`socket=denied`, fetch `socket=open`.
Thirteen commits, one image.
SLACK CHAT WORKS. deps.AI dialed zip.SocketPath("ai") and spoke cleartext HTTP
to it. Wrong twice: the socket is served by zaphttp (ZAP is framed binary, so
plaintext is unintelligible and the peer closes -> EOF), and what binds there is
the app's PLANE, not its routes — framed correctly it still 404s /v1/models.
Measured with controls: over ai.sock, health / models / a bogus path / chat all
EOF'd on plain HTTP and all 404'd on ZAP, and agents.sock and commerce.sock did
the same. So this was never about `ai`, and fixing only the wire would have
turned EOF into 404 — the changed-error-message trap.
Model calls now enter the fleet ROUTER's own listener on loopback, which owns
the route table that sends /v1/* to ai and owns starting a cold app. No DNS, no
Service hop, no trip out through Cloudflare back to the pod's own address.
Verified on the running pod BEFORE this deploy, same credential deps.AI mints:
GET /v1/models -> 200 {"data":[...]} (control)
GET /v1/bogus-not-real -> 404 (control)
POST /v1/chat/completions -> 200 "Hello from Enso."
Why it hid for a day: agents_run_on_behalf returned 200 in 205ms with a NIL
error, so the bridge fell into `run.Status != "ok"` — the one branch that logged
nothing. Every layer reported success while no work happened. That branch now
logs status and run id.
MCP: 1,189 flat tools in 977 KB (~244k tokens just to list) became 116 tools in
107 KB. One hanzo_<app> per subsystem with the op as an enum, plus hanzo_describe
to fetch a schema on demand. Clients truncate at 128, so 1,061 ops were
unreachable at any ordering. refuse() is untouched and still the only gate.
o11y: PromQL stops reading o11y_metrics, a database that does not exist — 264 of
264 logged queries failed on it. The data was never missing: event.metric holds
145.7M rows at 3 seconds of lag.
Build: 17m00s of work, of which 356s regenerated 99 files git already had and
221s exported a cache mode=max bought 4.6s from. Both gone; layers are zstd.
/zen is coresident and got a binary it could never execute: -165 MB.
Also carries three billing fixes and a fleet-door fix from other lanes.
Every `--mount=type=cache` in every Dockerfile is decorative today. Builds run
as one-shot Jobs, a Job is a fresh pod, so the mount is created empty and the
compile pays full cost every run. The registry cache already works and is kept,
but it restores LAYERS — and a layer only helps when the whole step is
unchanged. It cannot restore the inside of a cache mount, which is where Go's
build cache and module cache live. The expensive half of a build was never
cached at all.
A daemon that outlives the build fixes that, and nothing else does.
MEASURED here, ghcr.io/hanzoai/cloud, two consecutive commits against one
daemon (19010dbb cold, then 0e57f39c warm — different commits deliberately; an
identical rebuild would only prove the layer cache works):
wall clock 19m36s -> 7m19s 2.68x, -62.7%
#18 go mod download 54.4s -> 1.0s 54.40x cache mount /go/pkg/mod
#24 go generate 588.5s -> 88.5s 6.65x both cache mounts
#25 16.0s -> 2.3s 6.96x
#23 12.9s -> 2.1s 6.14x
#27 220.7s -> 114.5s 1.93x
Every step carrying a cache mount got faster. Two steps got slower warm
(COPY --from=build /plugins/ 13.5s -> 194.9s, /smoke 0.1s -> 11.5s): cold those
layers returned whole from the registry cache, warm the rebuilt stage had to
materialise 12.07GB for real. Net 12m17s saved. The honest number is the wall
clock, not the six steps that flatter it.
SIZED FROM MEASURED DATA. One cloud build is 34.28GB in the store, 22.92GB
reclaimable, of which the mounts that matter are 12.04GB (go-build 7.10 +
gomod 4.94). A DaemonSet has one spec across pools of different sizes, so the
GC budget is sized for the smallest node it lands on — runner-pool-32g at
94.9GB allocatable. 24GB protected for cache mounts, 40GB global ceiling.
CONTENT-ADDRESSED, and the manifest says so at length so nobody optimizes it
away. Sharing a daemon is safe because the layer store is keyed by DIGEST: a
layer cannot masquerade, its name is its content. Cache MOUNTS are the
exception and the whole hazard — `id=NAME` is a named mutable directory, and
per-pod isolation prevented cross-build poisoning by accident. Hence: ids stay
namespaced per repo (cloud already does, `cloud-gomod-v4`), first-party repos
only, never collapse ids to a generic `gomod` to "share more".
Rootless, so this drops `privileged`. VERIFIED on runner-pool-1tb-3m8cmh with
the exact securityContext and TOML in this file: uid=1000, `auto snapshotter:
using overlayfs`, process-mode no-sandbox, GC rule#0 loaded with the configured
filters, server listening on 1234.
That probe also found a real defect before it shipped. allowPrivilegeEscalation:
false — the obvious hardening flag, and the first thing I wrote — makes the
daemon fail to start:
[rootlesskit:parent] error: failed to setup UID/GID map:
newuidmap 16 [0 1000 1 1 100000 65536] failed: newuidmap: Could not set caps
RootlessKit builds the user namespace with the setuid helper newuidmap, and
that flag sets no_new_privs, which is exactly a promise that no setuid binary
gains anything. It is omitted, with the error recorded inline so nobody adds it
back on inspection. Upstream's own Kubernetes example omits it for this reason.
ADDITIVE. Named buildkitd-node, not buildkitd, because an untracked
hand-applied StatefulSet `buildkitd` with a HEADLESS Service of that name is
live in the namespace right now. Colliding would hard-fail — clusterIP is
immutable — and sharing its `app: buildkitd` label would be worse than failing,
silently splitting traffic across two daemons with two caches. Nothing points
at this yet; the existing one-shot Job path is untouched. Rollback is "stop
pointing at it".
chart 0.1.9 -> 0.1.10 adds service.internalTrafficPolicy, which the chart had
no way to express. A per-node daemon needs Local: the default round-robin finds
a given repo's warm cache 1/N of the time, so the cache would be shared badly
enough to stop being warm. Local drops traffic when the local node has no
endpoint, so the nodeAffinity must cover every node a client runs on — verified
against live pod placement (build Jobs on runner-pool-32g, git-runner on 1tb).
Verified: helm template renders 4 objects; `kubectl apply --dry-run=server`
accepts all 4 (nothing written); all 121 existing values files still render
against the modified chart.
cd-owns-the-fleet and the split kms-only-secrets were written, pushed, and
reported as landed while being applied by nobody. infra/k8s belongs to the
manual-sync `universe` app, sitting OutOfSync at 57/240, so the policies enforced
nothing and `kubectl get` returned NotFound for both. Declared and not real —
which is precisely the shape those policies exist to catch, committed by the
person writing them.
Scoped to three FILES, not the directory. infra/k8s/zt also holds the zero-trust
controller, edge, routers and zrok; pointing an automated app at the whole path
would turn "let the policies reach the cluster" into "auto-sync the zero-trust
stack", which is a different and much larger decision.
Own project rather than sharing hanzo-scheduling, because a project fence refuses
anything outside its whitelist and one refusal deadlocks every other resource in
the app. The whitelist is exactly two kinds, so a misconfiguration here can reach
admission policy and nothing else.
prune:false like every sibling, and it matters more here: pruning an admission
policy removes a guard while the cluster keeps working — a green surface over a
control that stopped.
Still hand-applied, deliberately. The hanzo-cd kustomization keeps Applications
out of `resources` so a deleted one cannot silently return, and that reason is
good enough to leave alone.
ghcr.io/hanzoai/app:v1.42.339
sha256:e7ee1532658fdf3f7e3b1aabe76815c8f964cc1d12b22937d928e94b65e881d0
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 35609
ghcr.io/hanzoai/app:v1.42.339
sha256:7a627b1b7300d91ceff59259dbb4932c5053703f0e6fdaf4501d3e815eb93e0d
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 35621
The committed node set was worker-pool: 8 nodes autoscaling to 16, untainted,
carrying the fleet's ordinary workloads. That was the right call when it was
made — code-exec-pool was an inventory comment in node-pools.tf and nothing
else, so worker-pool was the only real pool to aim at. The premise expired.
MEASURED on hanzo-k8s (do-sfo3-hanzo-k8s), today:
* code-exec-pool exists: 2 Ready nodes, labelled
doks.digitalocean.com/node-pool=code-exec-pool by DOKS itself, tainted
dedicated=code-exec:NoSchedule.
* gvisor-installer is 2/2 there. Its initContainer log: runsc
release-20260622.0, both sha512s OK, containerd restart dispatched.
* On BOTH nodes, read through the host mount namespace: runsc and
containerd-shim-runsc-v1 present in /usr/local/bin, containerd active,
imports = ["/etc/containerd/conf.d/*.toml"], and `crictl info` lists CRI
handlers "runc" AND "runsc". That last one is the fact that was missing —
the handler is REGISTERED, not merely declared to Kubernetes.
* A Pod with runtimeClassName: gvisor, carrying this commit's selector and
toleration, reached Running on code-exec-pool-3m8k4d and reported kernel
4.19.0-gvisor against a 6.12.73 host, dmesg "Starting gVisor...".
That pod was deleted; it was a measurement, not a workload.
Installing rewrites containerd's runtimes table and RESTARTS CONTAINERD. On
worker-pool every pod on 8-16 nodes pays that for a runtime none of them uses.
code-exec-pool is tainted, so it holds nothing but DaemonSets and the restart
costs nothing — which is the entire reason to spend a dedicated pool rather
than a label. Blast radius is the argument.
TOLERATIONS ARE NOW LOAD-BEARING and are checked in two places. While the
target was untainted, a missing toleration granted nothing. On a tainted pool
it inverts: scheduling.nodeSelector is merged into every pod naming the class
at admission, so the selector PINS a sandboxed pod to the pool and the taint
then refuses it — Pending forever, with runsc installed and idle three feet
away. So ci.yml's "gVisor node set is declared once" now compares tolerations
as well as nodeSelector (normalised — order is not meaning), and guard.yaml
grows a fourth live assertion for the same pair. Selector and toleration are
one fact; they were already two objects, and two objects with one fact is
exactly what both checks exist to hold together.
Not done here, and deliberately: code-exec-pool is NOT in node-pools.tf as a
resource — only in the inventory comment. The pool exists on the cluster and
Terraform does not know it, so its size, labels and taint are undeclared. That
wants an import, not a fresh resource block, and an import is its own change.
ghcr.io/hanzoai/app:v1.42.338
sha256:5210890f29597a3dce2085a3140a9759e8f32ec799d97fb6b9f60e2417d44c3f
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 35469
charts/app/values/hanzo/lsp.yaml — ghcr.io/hanzoai/lsp on the code-exec gVisor
pool (runtimeClassName gvisor, RO rootfs, drop ALL caps, no SA token, PVC
/var/lib/lsp, egress-allowlist to the go module proxy). memory request 4Gi /
limit 6Gi to FIT the pool as sized (was 8Gi = unschedulable). LSP_KEY from KMS
hanzo/prod:/lsp/LSP_KEY.
cloud.yaml — same LSP_KEY env so the /v1/code/lsp thin proxy authenticates.
HELD: digest is a placeholder until ghcr.io/hanzoai/lsp:0.1.0 builds (forge run
#3 running); pins.py gate rejects the zero-digest, so this is NOT pushed yet.
The storefront shipped pk_aGFuem8.woN0IELSfgLQF4xg837XiQ — the retired
pk_<b64 org>.<sig> HMAC family. Cloud's one publishable spelling is pk- with a
hyphen (PublishablePrefix, auth_identity.go), and the check is a PREFIX test, so
the old shape does not read as a wrong key: it reads as no key at all, and
/v1/event answers 401 ingest_key_required. Confirmed against api.hanzo.ai on both
the Authorization: Bearer and ?ingest_key= carriers, and karma.style is still
serving the dead value in /config.json today.
While the keyless lane existed this was invisible — those events landed under
$public and nobody could read them. Since that lane was retired on Aug 4 the same
requests are refused outright, so this storefront has been dropping 100% of its
telemetry: view_item, add_to_cart, begin_checkout, purchase.
The replacement is the hanzo org's publishable key, the same one docs.hanzo.ai,
hanzo.ai and hanzo.app already ship. An org key resolves through IAM and carries
no project, so nothing rewrites `product` — a project key would overwrite it with
the project's name.
Still a literal rather than a secretKeyRef: the value is published to every
browser in /config.json, so sealing it would protect nothing and only hide which
org this storefront reports as.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Pointed at a stub that implements nothing, 13 of this file's tests passed —
including every "authorize renders sign-in (real UI)" row. Three separate
reasons, each measured:
The drive sent no PKCE. Every app here is a public client and the IdP
enforces it: authorize without a code_challenge answers
`invalid_request / "PKCE is required for public clients"` and bounces to the
app's callback. So layer 2 was reading an error page and asking whether it
looked like a login. The file asserts elsewhere that each issuer advertises a
"code+PKCE" endpoint and then sent no challenge. With PKCE, lux.id serves its
login normally — one of the failures was the test's own doing.
Layer 2 claimed to be a redirect gate on the premise that a compliant server
refuses to render login for a bad redirect. Measured: hanzo.id renders a
byte-identical page (134 chars) for console.hanzo.ai's callback and for
e2e-unallowed.invalid/steal. It has no discriminating power over redirects
and now says so — it is a reachability check, which is the outage this file
was written for. Layer 1 is the redirect gate.
Its affordance matcher accepted buttons, and a brochure has buttons. The
stub's "Continue with Hanzo" satisfied it. Now a credential INPUT: measured
live, hanzo.id and lux.id each render one password and one text input, while
zoolabs.id and pars.id render zero of anything because their authorize is
erroring — which is the finding, not a reason to loosen the matcher.
Adds layer 3 for the redirect property that IS spec-mandated and reachable
unauthenticated: RFC 6749 4.1.2.1's "MUST NOT automatically redirect the
user-agent to the invalid redirection URI". All four issuers pass. It
compares HOSTNAMES — a substring match reported all four as handing the
user-agent to the evil host, because the authorize URL carries that host in
its own query string and percent-encoding leaves it intact. Four false
alarms on the one assertion here that reads as a vulnerability. It also
carries a coverage floor, since "did not redirect anywhere bad" is trivially
true of a server that does nothing.
Prod is unchanged at 10 failed / 32 passed; the 10 are pre-existing and are
what the gate exists to report. Against the stub: 13 -> 0.
hanzoai/lsp holds immutable per-(org, repo, commit) trees on a volume and
answers position questions about them. The answer it exists for is a definition
that leaves the repository and lands in a dependency — which means fetching that
dependency and type-checking both, which means running gopls, and `go list`
under it, over untrusted bytes. That clause is why this is a Deployment and not
code in the cloud pod, and everything in the values file is downstream of it:
runsc outside, the daemon's own namespaces + chroot + seccomp inside, non-root
at a fixed uid, a read-only rootfs, no capabilities, no service-account token,
and an egress policy whose `except` list is the part that matters.
WHAT IS TRUE TODAY, so nobody reads this as shipped:
* The image tag and digest are PLACEHOLDERS. hanzoai/lsp's first CI run is
queued on the git.hanzo.ai runners; this file gets the real pair in a
follow-up commit, and until then the Deployment cannot pull. That is the
right failure — a values file pinned to a tag that does not exist is an
ImagePullBackOff you have to go and read, and this one says so in the diff.
* The pod will be Pending after that. RuntimeClass/gvisor schedules onto
worker-pool, whose s-4vcpu-8gb nodes have ~6.2Gi allocatable, and this
daemon asks for 8Gi — 4 live roots, a 6 GiB address-space rlimit per
language server, and a memory-backed 2 GiB tmpfs inside each jail. The
number is stated rather than shaved to fit, because a daemon that OOM-kills
gopls mid-index reads as flaky for weeks and a Pending pod reads as itself.
* gvisor-installer is still 0/0/0. `runtimeClassName: gvisor` is a promise the
cluster cannot keep yet; the daemon's own boot-time probe is what refuses to
trade on it — /readyz stays 503 and the Service keeps no endpoints until the
jail proves out on the actual host.
The key is ONE secret with TWO readers: hanzo/lsp/LSP_KEY@prod, synced here as
`lsp-env` and read by cloud's /v1/code/lsp proxy as X-API-Key. cloud takes it
`optional: true` so the fleet's front door does not crash-loop on a sibling
release that has not synced; the daemon is the end that fails closed, refusing
every request when the key is empty rather than serving whoever asks.
No nodeSelector and no tolerations in this file, deliberately — see the chart
commit: the RuntimeClass merges the pool at admission and the node set has
exactly one home.
Neither policy that enforces "changes arrive through cd" was itself managed by
cd. kms-only-secrets and pipeline-only-releases were applied by hand and declared
nowhere — the only trace in this repo was a COMMENT describing one of them. Any
kubeconfig could have deleted the control that refused a hand-made release build
this morning, and nothing would have put it back.
pipeline-only-releases is adopted VERBATIM: kubectl diff against the live object
is empty, so this changes nothing about its behaviour and everything about who
can remove it.
kms-only-secrets is split, because it was two policies wearing one name. It
carried validations[0] admitting `grandfathered` and validations[1] asserting
!(user in grandfathered) — the second refusing exactly the twelve the first
admits. VAP requires ALL validations to pass and validationActions is set per
BINDING, so under [Warn] the contradiction is quiet telemetry and under [Deny]
the grandfather list inverts into a deny-list. It would have denied
hanzocd-application-controller, and a policy that stops the reconciler cannot be
reverted by the reconciler.
The two jobs were never one job. Enforcement asks who may write a Secret; debt
asks which of those should have migrated to a KMSSecret. Braided they cancel;
separated each is trivially correct and each gets its own binding, which is the
only place an action can be set:
kms-only-secrets one validation, allowlist, [Deny] scoped to ns/zen
kms-migration-debt failurePolicy Ignore + [Warn] — structurally unable to
block, so it can be wrong without being dangerous
That property is the point of the split: the enforcing policy must never be able
to be wrong quietly, and the reporting policy must never be able to be dangerous.
ghcr.io/hanzoai/iam:v1.34.23
sha256:0d21bcb8ef8e14deab8a765c188b2acc56238123e90c1a7cdf2dc5e895e551e8
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: hand
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GET /v1/billing/{transactions,credit-balance,accounts} and
accounts/:id/members 404'd in production, so billing.hanzo.ai's
Transactions, Credits, Team and Settings tabs were permanently empty.
It read as a stale image and was not one. commerce declares all four on
its api.Route() `user` group, but the co-resident embed registers on the
HOST's router and never compiles that table — a commerce route reaches
production only if apps/commerce/mount.go names it, and none of these
was named in any image ever built. The pinned module already carried
every handler (v1.50.11 has all four symbols), which is why grepping the
library found them wired while the tabs stayed blank.
Forward: 0dfe88d35cdf is an ancestor of 1fa6f964c on hanzo-inc/cloud.
Verified in the BYTES rather than the tag, with a control: the string
/v1/billing/transactions is absent from sha-0dfe88d35cdf's commerce
binary (0 hits) and present in sha-1fa6f964cb4d's (1 hit), while
/v1/billing/tier is present in BOTH — so the probe can tell the images
apart and is not merely failing to match.
ghcr.io/hanzoai/billing:1.0.24
sha256:f069025b9b9b3861d6d8d1bb8d8bba8b01d2f8343c4534b5c9f37f071bee6cfa
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 35491
Two keys, and the first is why the second exists.
runtimeClassName was simply not expressible. charts/app sets
`additionalProperties: false`, so a values file naming it fails the schema — and
the workload it is needed for is not exotic: anything that parses tenant bytes
runs under runsc, and the fleet is about to have two such daemons. Without this
they would have gone to raw manifests outside CD, which is how the gVisor
DaemonSet and RuntimeClass themselves spent 33 days live and in no repository.
It carries the NODE SET too, and that is the part worth knowing: RuntimeClass
merges its own scheduling.nodeSelector and tolerations into the pod at
admission. So a values file that names a RuntimeClass must NOT restate the pool
— the node set is declared once, in infra/k8s/gvisor/runtimeclass.yaml beside
the DaemonSet that installs the runtime there, with a CI gate that fails when
the two disagree. A third copy in a values file is a third thing to be wrong,
and the day a dedicated pool exists the copies conflict and the pod is REJECTED.
The template comment says this where someone about to add nodeSelector will read
it.
automountServiceAccountToken defaults to NULL, not false, and that is not
timidity. The API treats an absent field and `true` as the same thing; a diff
does not. Defaulting it would write the field into every rendered pod in the
fleet and every one would read as changed. `with` cannot express "emit only when
stated" — it treats false as empty — so the guard tests the KIND.
Proven inert: all 106 values files render, and exactly one emits either key.
Two of the three chat tests passed against a server that answers 200 on
every path and implements nothing. One asserted only `status < 500`, which
a 404 satisfies; the other asserted `composer || signinButton || loginCopy`
— an OR any page with a textarea or the word "continue" satisfies. Neither
could fail, so neither protected anything.
scripts/stub.cjs is that server, kept deliberately generous: a stub failing
every selector would flatter the suite. scripts/prove.sh points every
overridable service URL at it and lists the tests that still pass. Read a
pass as "cannot tell the product from a stub".
tests/65b-chat-controls.spec.ts replaces the deleted UI tests: the settings
tabs, the exact roster of switches on each (asserted as a SET, so a new
control with no test fails the build), every toggle flipping and persisting,
the dock rendering a CSP-allowlisted frame, and the nav-rail regression.
7 pass against production, 7 fail against the stub.
08-links gains a coverage floor. "0 dead links" is only good news if the
crawl reached the surface — collapse the seeds to a login wall and it
reports clean having checked four urls, which reads exactly like health.
helpers/models: families were read off `body.families`, a key the gateway
has never emitted, so the list was always empty and `families[0]` always
undefined. Every assertion on it failed identically whatever the catalog
held — red through both the bug and the fix, so its colour meant nothing.
Now derived from the ids in catalog order, which makes the remaining red a
real finding: the catalog is alphabetical, so enso sits 27th behind
all-mini-lm-l6-v2 and anthropic-claude-opus-5.
ghcr.io/hanzoai/app:v1.42.337
sha256:16eb47caffb690c7d4ce3b473699fb675c3d2231d0c781552ba34c9d69cdb703
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 35366
The containment shipped two commits ago selected `hanzo.ai/box-class` in
namespace `hanzo-boxes`. The subsystem that creates the pods labels them
`hanzo.ai/sandbox-class` in `hanzo-sandboxes`. So the policy would have matched
nothing — the third instance this week of an object that exists, reads green,
and enforces nothing, after the netpol that selected no pod and the RuntimeClass
that scheduled onto a node pool nobody created.
One vocabulary, chosen to match the API rather than the implementation: the
surface is /v1/sandboxes, so the namespace is hanzo-sandboxes and the label is
hanzo.ai/sandbox-class. "Box" was the container-era name for the same thing and
keeping it would have left the wire, the policy and the product each using a
different word for one object.
This is also why the word matters more than it looks: /v1/machines already
exists and is visor's — whole GPU and VPS instances, sized, quoted and billed.
A sandbox is a pod that runs somebody's code. Two products cannot share a noun.
The previous pin stated the tenant inside the agents op, which is reached over
the plane — a real request. zip reads a stated caller only where there is NO
request (caller.go:352-356), so the statement was discarded and every @hanzo
turn still failed with "authorize: no org on the call".
The bridge now states it on a detached context before the hop, which is where
Caller.headers can render it onto the wire.
Also widens the install URL's scope request to match the app manifest: it asked
for 9 of 13, so an install through it produced a token with no `commands`.
The installer for runsc existed ONLY as live state. `kubectl get ds
gvisor-installer -n hanzo` answered, the ConfigMap answered, the ServiceAccount
and RuntimeClass answered, and all four carried
`app.kubernetes.io/managed-by: universe` while no repository held any of them.
Nothing reconciled them, a cluster rebuild lost them, and every fix anyone
applied was invisible to review. runsc IS the boundary around
customer-submitted code, so "the boundary is live state nobody can read" is the
part that mattered.
It was also installing on nothing. The DaemonSet selected `workload: code-exec`
and tolerated `dedicated=code-exec:NoSchedule`. Measured against all 20 nodes of
hanzo-k8s: ZERO carry a `workload` label of any value, and ZERO carry a
`dedicated=code-exec` taint. Both named code-exec-pool, which
infra/terraform/doks/node-pools.tf still lists in its inventory comment and
which does not exist on the cluster. So it read 0 desired / 0 ready / Healthy
for 33 days -- 0 being the correct answer to "how many nodes match", agreed with
by every readback there is. The same shape as a NetworkPolicy selecting no pod.
WHY worker-pool AND NOT THE TAINTS THAT EXIST. `dedicated=data` holds the
datastores and `dedicated=ci-runner` holds CI; tolerating them installs a
runtime, and restarts containerd, on nodes that will never run a sandbox.
Labelling nodes `workload=code-exec` by hand is the disease being cured --
worker-pool is doctl-managed and not in Terraform state, so the label would be
an unreviewable side effect that the next autoscaled node does not inherit.
`doks.digitalocean.com/node-pool` is set by DOKS from the pool definition, so it
needs no hand-labelling and every node the autoscaler adds carries it on
arrival. All 8 worker-pool nodes are untainted, so the toleration is dropped
rather than replaced. MEASURED with an inert probe DaemonSet (a schedulerName no
scheduler answers to, so its pods stayed Pending on no node): this exact
selector reports desiredNumberScheduled 8.
RuntimeClass/gvisor carried the same phantom node set, and it is the half that
decides where sandboxed pods land -- scheduling.nodeSelector is merged in at
admission. Left pointing at `workload: code-exec` it would pin every gVisor pod
to nowhere, forever. One node set, written in two objects that must agree, with
a CI step that fails the build when only one is edited.
PLATFORM PINNED TO systrap. The script chose `kvm` if /dev/kvm exists, and
/dev/kvm DOES exist on DOKS nodes, so every node took kvm silently. Measured on
our real workload: kvm 98805ms vs systrap 81789ms. The auto-detect was a ~21%
regression wearing the costume of a hardware optimisation.
overlay2 = "root:self" added. It keeps the container's writable layer in the
Sentry instead of crossing the gofer for every write and metadata op, which is
the measured hot path (npm install writing 30k files 1.58x, git status lstat
storm 7.63x). NOT MEASURED HERE -- added on gVisor's documented behaviour for
this access pattern, and a separate benchmark is quantifying it.
The script is otherwise byte-identical, including the sha512 fail-closed pin. It
now lives as a real .sh file rather than an escaped blob, generated into the
ConfigMap WITH the name hash: the installer only runs at pod start, so a
same-named ConfigMap would update in place and no node would ever re-run it.
Raw manifests and not charts/app, for two reasons that are not taste: hostPID is
load-bearing (nsenter --target 1 reaches the node's init, and without it the
script configures the pod) and charts/app sets additionalProperties: false with
no hostPID key; and RuntimeClass is cluster-scoped, where one kind outside the
AppProject whitelist refuses the ENTIRE Application. s3-csi, the fleet's other
privileged hostPID node agent, lives here for the same reason.
guard.yaml is new and is the point of the exercise: an hourly CronJob that FAILS
when desiredNumberScheduled is 0, when no node carries the selector, when nodes
are ready short of desired, or when the RuntimeClass and the installer disagree.
It reads the selector off the live object rather than repeating it, so it
follows an edit instead of decaying past one. It reports and does not mutate --
converging restarts containerd on every node it reaches, which is an operator's
decision to schedule.
NOT APPLIED. Bringing this to the 8 worker-pool nodes restarts containerd under
204 running pods; that is a scheduled maintenance, not a side effect of landing
a commit.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ghcr.io/hanzoai/app:v1.42.336
sha256:c04aa43320941845845fb907751205bac30c4e93028bd00b9b21aa89ae5e9340
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 35197
Two claims in this file outlived the migration they described, and both fail in
the direction that costs a deploy.
1. "luxfi/universe and zooai/universe are PULL MIRRORS of the GitHub repos —
GitHub is still where Lux and Zoo are written." Not any more. Measured on the
live plane: lux-fleet sources git.hanzo.ai/luxfi/universe, zoo-fleet sources
git.hanzo.ai/zooai/universe, and across every Application there is not one
github.com repoURL (the only non-forge source is oci.hanzo.ai/charts). All
three universes are canonical on the forge and private; CI/CD is native
(git.hanzo.ai Actions + platform.hanzo.ai builds), so GitHub is out of the
loop rather than upstream of it. Acting on the old text would push Lux or Zoo
changes to a repo nothing reconciles — the exact silent no-op this file opens
by warning about, in the one place it claimed an exception.
2. "the GitHub copy is archived (the server refuses the push)" — offered as one
of two guards making that no-op structural. github.com/hanzoai/universe is
archived=false, private, and was pushed to today: it is the live destination
of a push mirror OUT, so it must accept writes. The guard does not exist. One
remains, `.git/hooks/pre-push`, and hooks are not copied by git clone — so a
fresh checkout now has NO protection where it used to inherit the archive.
Recorded rather than quietly corrected, because a guard that has silently
stopped existing is worse than one never claimed: the reader stops checking.
Also notes, next to the lux/zoo ignoreDifferences difference, that the same
entry on `fleet` is why /spec/replicas is applied at CREATE and never
reconciled again — the footgun that let visor.yaml read 1 while the Deployment
ran 2.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ghcr.io/hanzoai/billing:1.0.23
sha256:3f4af2668a7b25815de5d64f2c15380142886828999b12399e230b848dda5598
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 35188
Three layers stand between a change and production. Two exist and both work:
BUILD pipeline-only-releases [Deny] refuses an image that claims a v<N>
name without going through cicd.yml. It refused one today.
DELIVER pin.sh is semver-only and cd watches the repo, not the registry.
It refused a sha pin today.
MUTATE nothing. Any kubeconfig can edit a live Deployment.
Every out-of-band change this estate has suffered went through the third gap:
kubectl-authored Secrets, hand-built images pinned by hand, a ConfigMap edit
undone minutes later by the reconciler it was fighting. The problem is not
carelessness — it is that layer 3 has no refusal in it, so a mistake there is
invisible until a reconcile reverts it, or worse, until it does not, because the
resource is one nothing reconciles.
This closes it: workloads may be written by cd's reconciler, by the operators
that render their own CRs (declared in git one hop down), and by the control
plane. Everyone else is refused with a message naming the right door — a refusal
that does not say where to go instead just gets routed around. Reads are
untouched; the goal is to stop undeclared CHANGE, not to stop looking.
BOUND AS Warn, NOT Deny, and that is not timidity. The sibling policy
kms-only-secrets carries two validations that contradict — the first admits a
grandfathered principal, the second refuses exactly those — which is harmless
telemetry under Warn and inverts the allowlist into a deny-list the instant the
binding says Deny. It would have denied hanzo-cd's own application controller,
and a policy that takes down the reconciler cannot be reverted BY the
reconciler. So this policy carries exactly ONE validation and will not grow a
second; a new rule gets a new policy with its own binding.
The allowlist is derived from what actually writes to this cluster, but it was
derived under a kubectl that returned empty on the largest queries, so it is
measured-but-incomplete by construction. Soak the warnings for one full CD
resync and one operator reconcile, add whatever legitimate writer they name,
then flip validationActions to ["Deny","Audit"]. One line each way.
Delivered through cd rather than kubectl, because a policy asserting that the
fleet is reconciled and not edited has no business being applied by hand.
A validator doing a full sync was still OOMKilled at the 5Gi ceiling with a
4GiB soft target, while its steady-state peers sat at 673Mi-2.3Gi. The gap
between the soft target and the hard limit is what absorbs NON-heap growth —
DB mmap, cgo, per-peer buffers — and a syncing node allocates far more of that
than a running one.
The ceiling cannot go up: the hanzo-val node has 6.26Gi allocatable shared with
33 other pods. So shrink the heap target instead of asking for memory that is
not there. 2Gi of slack below the kill line.
Measured after: restarts 1 -> 0 sustained, memory plateaus at ~3.3Gi with GC
holding it at target rather than sailing into the cgroup ceiling.
This is also the honest reason bootstrap "did not work" and
--skip-bootstrap=true became the default: a validator that OOMs while syncing
can never finish, so the flag hid a sizing bug rather than a protocol one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A Slack turn reached agents over the plane and died with
"authorize: no org on the call". commerce/balance_rpc.go takes the org from
cloud.Who(ctx).Org and refuses when it is empty, which is deliberate: the org
is the CALLER'S and can never be named in the input. A plane-dispatched run
has no inbound request carrying that identity, so there was none to take.
onbehalf_rpc.go now calls cloud.For(ctx, in.Org) before the run. That is the
one construct for this: it states the tenant a BACKGROUND call acts for,
supplying an identity where there is none, and it cannot launder one — zip
prefers a gateway assertion when one exists. in.Org comes from the Slack
install→org map keyed by the Slack-verified team_id, never from the payload.
Tag and digest move together; the kubelet honours the digest.
1.0.20 was named for a commit it did not contain. The builder clones
refs/heads/main from GITHUB while the tag is composed from the git.hanzo.ai
event sha, and GitHub was seven commits behind — so the image asserted
315cb42 and shipped the code from 4f404a0d. Verified by content, not by
name: 1.0.21 carries the audience/interval controls and the comparison
matrix, 1.0.20 carries none of them.
ghcr.io/hanzoai/insights:1.52.80
sha256:779c0f9c24a4d89df66d84fa699e0e296859718db42d10a938aee02957af5fb8
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 34922
ghcr.io/hanzoai/insights:1.52.80
sha256:779c0f9c24a4d89df66d84fa699e0e296859718db42d10a938aee02957af5fb8
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 34922
tag v1.801.486 -> sha-9c4c861384b6
digest b62be7f5... -> sha256:ac9f359ec08e13be0bd98b6d0197e79650bff3061b7b297a67dc74039f329d86
A sha- tag, not a v*: the admission policy reserves v* for the pipeline that
claims refs/tags/v<N> as a compare-and-swap before it builds, so a namespace
build cannot mint one. That is the tag-is-a-receipt invariant working.
BUILT-IN DEFAULT AGENT — the actual cause of "agents: agent not found", which
is what @hanzo said once the plugin-boundary fix let the turn through. Store
.Resolve is a plain row lookup with NO seeding anywhere, and the bridges ask
for the conventional ref "hanzo", so an org that connected Slack and did
nothing else had no agent at all. @hanzo could not work out of the box in ANY
workspace. The ref now resolves to a built-in default; an org's own row still
wins, and an unknown ref stays a miss.
CHAT BRAIN = enso, the auto-routing SKU. I had used cloud.FallbackModel, whose
own doc says it "keeps a bot's reply landing when the flash tier is saturated;
the interactive chat path never uses it" — the degraded tier, wired as the
default. BRIDGE_AGENT_MODEL overrides.
APP HOME — model selector, mode, connected-as, getting started, replacing
Slack's "this is still a work in progress" placeholder. Both controls write the
SAME userLink the chat path reads, and the turn carries that model per-request:
it is a preference of the person asking, not a property of the agent.
Also: a bounded, actor-attributed tool-calling loop in the agent run, and a
curated tool projection in fleet.
Since hanzoai/base v1.5.4 -> v1.5.16 landed on superbase main, no build off main
could boot:
panic: kms: "https://kms.hanzo.ai" is an HTTP endpoint — Base speaks native
ZAP to KMS; set the KMS endpoint to zap://host:9999 (or host:9999)
Base v1.5.16 rejects an http(s) KMS endpoint at construction. main has been
un-deployable since that bump; nothing surfaced it because nothing had rebuilt
superbase since.
These two edits are ONE change and must never be split:
- 0.3.4 (base v1.5.4) speaks HTTP, so pointing it at zap:// breaks the pod
that is serving right now.
- 0.3.9 (base v1.5.16) rejects https://, so bumping the image alone
reproduces the panic above.
The endpoint is not a new dependency. kms.hanzo.ai already routes to the cloud
Service — the standalone kmsd is retired and KMS is embedded in cloud — and that
Service already exposes zap:9653 alongside http:8000. Verified reachable from
ns hanzo-superbase before this was written.
Proven on a canary of the exact digest with this exact endpoint, run beside the
live pod (emptyDir for the RWO volume the live pod holds): boots to "Server
started at http://0.0.0.0:8090" with the ZAP transport listening and no panic.
The previous attempt at this shipped straight to production and crash-looped;
this one did not.
0.3.9 is main as it stands — the seam change is reverted out of it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Ships the incident remediation (forge 315cb42, rebased on the crypto-address
containment ed272ea): /topup reads its token from localStorage instead of a
cookie nothing sets, the Square card field mounts and stays mounted on a
direct visit (the provider no longer remounts children 3x and the hook no
longer destroys the card it just attached), and a failed read renders as
'couldn't load' instead of a false $0. Pinned by digest — the build's bytes
are proven pullable; the semver tag label follows.
A live low-value charge remains the one unproven step, to run once this rolls.
The 02:41 pin to 1.0.95 (red-team markdown image-exfiltration fix + 1.0.94's
dock/signature) never reached the pod: the reconciler restarted ~08:40 and
processes only commits it sees live, so it caught the ~09:00 iam pin but not
chat's earlier one, leaving chat.hanzo.ai on 1.0.93 with the HIGH fix declared
but undeployed for ~7h. Re-affirming the same pin as a fresh commit is the
deploy trigger; running converges to the git-declared digest.
Begin an authorize on iam.hanzo.ai/auth.hanzo.ai and the host-only hanzo_fed
bind cookie was set there, but beginFederation registers the Google callback
at the pinned hanzo.id — so the cookie was absent at the callback and every
social sign-in begun on an alias failed with 'the federation session could
not be verified'. v1.34.22 relocates the request to the issuer (307) before
anything is set. Digest-pinned; verified 200 on ghcr.
The code-exec NetworkPolicy was written for the right reasons and enforced
nothing. It lived in namespace `hanzo` matching `app: code-exec`; apps/sandbox
schedules boxes into `hanzo-boxes` labelled `hanzo.ai/box-class`, and the
Deployment it was written for never shipped. Its own header said "the isolation
has to already be in place the moment the workload comes back" — a selector that
matches no pod is not in place, and the day boxes deploy it would have stayed
not in place with nothing to notice.
That gap is load-bearing rather than theoretical. The box API key is ONE SHARED
KEY across the pool, so the credential a box hands the code it runs is the
credential that opens every other tenant's box. Theft of it from /proc was
reproduced end to end before boxd started refusing to run without uid
separation. Network reach is what decides whether a stolen key is useful, and it
is the control that holds if the other is misconfigured — so it has to select a
real pod.
ONE SELECTOR, BOTH SHAPES. A box is a box whether apps/sandbox scheduled it for
one project or a Deployment holds it in the shared exec pool for hanzo.chat;
both carry hanzo.ai/box-class, so Exists governs every executor in the namespace
and a new class is contained the day it is added rather than the day someone
remembers to widen a list. That is also why the shared pool moves out of `hanzo`
— apps/sandbox's pool already records that a box must not sit beside the
datastores it is forbidden to reach, and the exec pool runs the same binary on
the same submitted code.
Egress is a whitelist, so box→box on 8000, box→datastore and box→apiserver are
denied by omission. The one opening is 80/443 to public address space, which a
dev box needs to clone and install; the RFC1918 and link-local excepts are what
keep that from being a route back in, since a ClusterIP is just an address in
10/8 and the metadata endpoint is at 169.254.169.254.
Policy and namespace only. The executor Deployment lands here when the box image
publishes — pinning a tag that does not exist deploys an ImagePullBackOff.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The resource change is unchanged and still right; two comments justifying it
described kubelet's ranking wrongly, and a wrong mechanism in a comment is how
the next person fixes the wrong thing.
kubelet ranks memory-pressure eviction per POD: whether the pod's working set
exceeds the SUM of its containers' requests. Not per container. So "a
container that requests 0 is always over its request" is not the mechanism --
what a 0-request container does is contribute nothing to the sum while still
consuming, lowering the bar the whole pod has to clear. Same conclusion, and
it is why both halves of the fix were needed, but it gets there honestly.
Also: the order containers appear in an eviction message is report ordering by
consumption, not the ranking. "replicate" appearing ahead of "chat" is not
evidence about which one caused the eviction.
helm template renders 640Mi/32Mi/32Mi as intended; no other diff.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The 256Mi -> 512Mi bump earlier today did not hold. The pod it shipped in,
chat-77cbb6dbd6-fcl69, was evicted 36 minutes after it started:
06:42:01Z chat-74797cf9db-v4n8q using 500556Ki, request is 256Mi
replicate using 13468Ki, request is 0
07:27:52Z chat-77cbb6dbd6-fcl69 using 542816Ki, request is 512Mi
replicate using 18904Ki, request is 0
Two separate things are wrong and only one of them was chat's own request.
kubelet ranks eviction victims by how far a pod is over its request. The
replicate sidecar shipped with no resources block at all, so its request is
zero, and a container that requests zero and uses anything is over its request
always. That drags the whole pod into the first-to-evict bucket regardless of
what the chat container asks for -- which is why both eviction messages name
replicate first, ahead of chat. No amount of raising chat's request could have
fixed that half.
So: give replicate a request (32Mi, ~1.7x its 18904Ki peak), give the restore
init the same, and put chat's request ABOVE its working set rather than at it.
The peak climbed 488Mi -> 530Mi across the two measured evictions; 512Mi was
already under it when it shipped. 640Mi leaves real headroom.
An init container's request does not add to the pod's steady-state footprint
(kubelet takes max(init, sum(app))), so replicate-restore's costs nothing and
keeps a cold-starting pod from being picked off before it reaches Ready.
Net change to the pod: +128Mi on chat, +32Mi on replicate. One replica.
Not fixed here, and the reason the nodes are under pressure in the first place:
otel-agent requests 192Mi, is configured with a memory_limiter ceiling of
800 MiB, and was measured at 967000Ki (944Mi) -- five times its request, on all
19 nodes. Five hanzo-val-* nodes sit at MemoryPressure=True because of it.
Reconciling that request with that limiter is a fleet-wide scheduling decision,
not a chat one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The exchange SPAs have shipped SPA_RPC_HOST_HANZO =
https://api.hanzo.network/v1/bc/C/rpc for a while, and that host does not
exist: no ingress, no DNS record. Only api.hanzo.ai (an unrelated app) and
api.hanzo.team were served here, so every Hanzo RPC call from a browser died
at DNS. lux and zoo both have their public edge; hanzo never did.
Backed by hanzod-rpc rather than a new gateway. lux runs an api-gateway with
five explicit luxd backends, but hanzod-rpc already selects only validator pods
marked hanzo.ai/rpc=fresh — adding a gateway here would be a second way to do
the same thing.
Carries the same upstream-hash-by pinning as the lux hosts. nginx-ingress
balances across Service endpoints itself rather than deferring to kube-proxy,
so hashing the client address really does pin a caller to one validator — which
is what stops a broadcast and its receipt poll landing on different nodes and
manufacturing "receipt not mined in time" against a healthy chain.
Verified: a request with Host: api.hanzo.network against the ingress LB
(129.212.164.5) is matched and redirected to HTTPS, so routing is live. Still
needs the DNS A record -> 129.212.164.5; the CLOUDFLARE_API_TOKEN in the
environment is rejected (code 1000, Invalid API Token), so that one step is
blocked on a working credential.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Standing doors to the top-up page (the Credit Balance tile, an Add-credit
button on the Credits section, the zero-balance banner). The dashboard stops
printing a Developer plan and a 30-day renewal for a zero-subscription org:
no plan reads as Prepaid credit.
_helpers.tpl renders repository:tag@digest and the kubelet resolves the DIGEST,
so a values file whose tag moved and whose digest did not deploys the old image
and reports success. Every readback agrees with you, because they all read the
tag.
Three writers, three different failures:
pin.sh already writes both from one lookup and refuses when it cannot resolve.
Unchanged.
live2values.py could only ever write one. It split repo:tag@digest on '@' and
threw the TAG away, and a live pod on a bare tag yielded no digest at all -- so
every file adopted through it started life violating the invariant. It now emits
the pair or reports the workload unfit, which is the mechanism the script already
had for a field the chart has no home for. A live pod on a bare tag IS an
unmovable pin; writing it into a values file launders it into something that
looks converged. Measured on all three shapes: before, all three wrote a file and
the CORRECT one lost its tag; after, only the pair writes.
The human-edit path had a check -- pins.py --resolve, the one gate in the fleet
that asks the registry anything -- and it was continue-on-error. The comment
narrowed it to a single live finding and said to delete the flag once that
cleared. It has: iam pinned v1.34.4@9681afd5 against a re-pushed tag, and today
pins v1.34.17@5c17d06d which the registry serves as 5c17d06d. A concession that
outlives its reason is how a check stops being one. Mutation-tested by moving
iam's tag to v1.34.16 and leaving the digest: the gate now says "digest does not
match the tag ... runs sha256:5c17d06d while reporting 'v1.34.16'".
And the gate had the same disease it was written to cure. It reported "no new
violations" over 43 pins it never fetched -- an unread pin fell through to the
clean path, so reading nothing and finding nothing wrong were one exit code.
Three answers now, the distinction cloud's .hanzo/scripts/orphans.sh already
draws: 0 read and clean, 1 a contradiction, 2 a source that did not answer. Not
narrowed to "verified nothing at all", because a mis-scoped token blinds it to
exactly the private repos worth checking while the public ones keep it green.
The offline shape gate is red on main with 9 violations and was before this
commit; pins_test.sh fails the same 4 cases either side of it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ghcr.io/hanzoai/cloud:v1.801.486
sha256:b62be7f5b557f41cfe7774e2990be38280eed378be9cb16f97c3f7ba5bded20c
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: hand
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Moving tag: alone changed nothing a customer could see. The rendered image is
tag@digest and Kubernetes resolves the DIGEST, so pay kept running the bytes
built from 59d886c while its tag claimed 98e0f066 — a pin that reads as the new
release and serves the old one, which is worse than an obviously stale tag
because every surface reports the new name.
Both move together or neither does.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The pinned image was built from 59d886c, fourteen commits behind, so the page it
served still gated crypto and wire behind 'coming soon' — a claim that stopped
being true once the wire address was fixed (commerce v1.50.7) and MPC keygen
started completing (luxfi/mpc v1.17.22).
98e0f066 is the single lineage: the app code with no comingSoon left, built by
the Dockerfile into ghcr.io/hanzoai/static and served behind hanzoai/ingress.
The Cloudflare Worker that used to answer this host is deleted in that commit,
so this pin is now the ONLY thing that decides what a customer sees.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
0.3.8 CrashLoopBackOff:
panic: kms: "https://kms.hanzo.ai" is an HTTP endpoint — Base speaks native
ZAP to KMS; set the KMS endpoint to zap://host:9999 (or host:9999)
Not the seam change that motivated the build. hanzoai/base v1.5.4 -> v1.5.16
landed on superbase main separately and requires a ZAP endpoint; this values
file still supplies an HTTP URL. 0.3.8 was simply the first image to carry that
bump to production, so an unrelated dependency upgrade shipped under a change
about Kubernetes clients.
Back to the last image known to serve. The fix is a config change (KMS endpoint
-> zap://kms.hanzo.ai:9999) verified against the new Base BEFORE re-pinning,
not a re-roll of the same image.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Ships 7379671: plugin/tenant no longer links client-go or controller-runtime.
The CR writes go through the same fail-closed seam cloud-oss uses, and Apply
patches first (MergePatch, Create on NotFound) instead of read-modify-write, so
the operator's status is preserved by construction rather than by re-reading it.
0.3.8 and not 0.3.5: GHCR already carried 0.3.5, 0.3.6 and 0.3.7 while
production ran 0.3.4 — images built and never pinned. Next free patch wins;
the git tags, which stop at v0.3.4, are not the authority on what the registry
already holds.
Digest read from the build log's "pushing manifest" line and confirmed against
GHCR independently, because platform leaves build_job.imageDigest null on
success. Tag and digest move together.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Adds e50609ab94 (cached miss must never overwrite a durable block) on
top of the .57 race fix. Applied deliberately after the in-flight mv-2
resync completes — this tree is hand-applied by design.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Five Jobs published ghcr.io/hanzoai/cloud:v* out of ns hanzo-build — 485, 484
and 476 by kubectl, 478 and 477 by platform. None claimed a version, so none
minted a tag; 477 built the OSS split under the private tree's release name and
its failure is why that tag has no image. All five resolved #refs/heads/main at
fetch time and passed no REVISION, so the bytes can be neither traced nor rebuilt.
image-build-job.yaml already said in a comment that it is not the canonical
build path. That is what the five jobs are made of.
A credential split cannot carry this rule: GHCR gates per package, never per
tag, so any token that lets this namespace push cloud:sha-6c1b4a6 — which the
delegate lane does legitimately every push — also lets it push cloud:v1.801.485.
Labels cannot carry it either; build-cloud-476c already set managed-by=platform
by hand. The output ref can: it is in the pod spec, it is the bytes that will
execute, and admission runs after authz on every request, so it binds
cluster-admin where RBAC does not.
Scoped to the repos that have DECLARED one release owner (cloud, in hanzo.yml),
not to the v-tag shape: 13 of the 18 live v-tag builds here are luxfi/node,
luxfi/mpc, hanzoai/replicate, visor and sensei-group releasing exactly this way,
and a rule that reddened them is a rule someone switches off.
Prevention. .hanzo/scripts/orphans.sh in cloud's receipt job stays detection.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ghcr.io/hanzoai/insights:1.52.79
sha256:8460f345784b7a776e79e846bf190f658c30746b907d554781417995fe441835
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 34298
ghcr.io/hanzoai/insights:1.52.79
sha256:8460f345784b7a776e79e846bf190f658c30746b907d554781417995fe441835
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 34298
ghcr.io/hanzoai/insights:1.52.79
sha256:ec14abcce65942d8f3152e0238112cd9e99f05931680735aa4f182bb83502076
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 34295
ghcr.io/hanzoai/insights:1.52.79
sha256:ec14abcce65942d8f3152e0238112cd9e99f05931680735aa4f182bb83502076
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 34295
ghcr.io/hanzoai/insights-plugin:0.1.8
sha256:d1781d998428fbfe906dd0740e68b425cd7afd322647614d446c60ae42ee9424
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 34297
ghcr.io/hanzoai/insights-livestream:1.0.2
sha256:58a1209561fef8d7ad7cefec2b482b474d59ecc178c13fc62209530150d35e4e
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 34296
tag v1.801.484 -> v1.801.485
digest a7832789... -> sha256:a82d5c68bb7dee66e5d8e6b43bb4d5d43c2f530e611e8a9157e080ccca4f7196
Tag and digest move together; digest from GHCR's docker-content-digest.
Fixes the reason every Slack DM answered "Sorry — I couldn't reach your Hanzo
account just now": kms.ErrSecretNotFound does not survive the plugin wire (a
plugin is a process; the error is rebuilt from its STRING), so errors.Is was
false and an UNLINKED user took the broken branch instead of being handed the
link prompt. Repaired at kmsGet, the one door onto the store.
Also publishes a real App Home on app_home_opened instead of leaving Slack's
"this is still a work in progress" placeholder.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
tag v1.801.483 -> v1.801.484
digest c3a88a72... -> sha256:a78327894f4c81f4236f5c12557ca7a08024b6628fe5e05280fd81e060bbe15e
Tag and digest move together; digest read from GHCR's docker-content-digest for
that tag, not from the build log.
Ships:
- assistant:write scope + assistant_thread_started/_context_changed events,
the two code-side parts of Slack's Agents & AI Apps surface (the third is
the toggle in the app config). Without them @hanzo answers mentions and DMs
but is never OFFERED as an agent.
- GET /v1/integrations/slack/install -> 302 to Slack's consent URL. Slack
refuses a slack.com URL in the Direct install URL field and requires one of
ours that 302s to slack.com, because the field is an attribution hook.
Built from github.com/hanzo-inc/cloud, which is what the BuildKit lane's
--opt=context still points at. The same change is on the forge (1df91068) as a
separate commit; the two trees have diverged (forge +125 / inc +50) and
reconciling them, then repointing the context at the forge, is #194.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GATEWAY_BODY_LIMIT defaulted to 16 MiB (cloud config.go), a number chosen for
CHAT PROMPTS: the framework default of 4 MiB silently capped the context window,
since a 1M-token prompt serializes to ~4.3 MB of JSON, so 16 MiB was picked to
give that ~3.7x headroom. Site publishing then inherited it.
The site subsystem's own caps are an order of magnitude larger -- blob.go allows
512 MiB total and 64 MiB per file -- so a zip deploy was throttled by a constant
that was never about zips. Two of 24 built exports could not be posted at all,
and the failure is unreadable: fasthttp refuses an oversize body BEFORE any
handler runs, returning the opaque 400 'Error when parsing request', which reads
like a malformed payload rather than a size cap.
100 MiB clears trillerfest.com (76.7 MiB). It does NOT fix hanzo.ai, which has
8,536 files against blob.go's separate maxFiles=5000 -- a different cap that no
body limit reaches. GOMEMLIMIT is 9GiB, so the headroom is there.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ghcr.io/hanzoai/billing:1.0.18
sha256:2b8f908c975a2805a2b014d69f88873d5e10d7aa604e7d17578942c9f4747a1f
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 34118
The pod runs ~500Mi steady; requesting 256Mi made it the first eviction
target when its node ran dry (evicted tonight, ~3 minutes of 503). The
request now matches reality so scheduling accounts for it honestly.
The Ingress declared TLS for a host that resolves nowhere, so now that
cert-manager is reconciling again the ingress-shim mints a Certificate on
every pass that cannot be issued by either challenge type: DNS-01 needs a
Cloudflare zone we do not have, HTTP-01 needs an A record pointing here.
pars.ai is on GoDaddy nameservers (ns51/ns52.domaincontrol.com) parked at
15.197.148.33, and auth.pars.ai is NXDOMAIN. docs/CANONICAL.md:354 already
said so; nothing acted on it because the failure was invisible while
cert-manager was down.
What it was meant to be is in the header it sat under:
auth.pars.ai -> pars.id
a branded alias mirroring auth.hanzo.ai -> hanzo.id, assuming Pars has a .ai
the way Hanzo does. It does not. Pars has pars.id and pars.network, both on
our Cloudflare, with pars.id already A'd to this ingress at 129.212.164.5
and already carrying the pars-console IAM app. The canonical host was never
missing — only the alias was imaginary, so the alias goes rather than being
repointed.
Also out of the IAM CORS origin list, where it was an entry that could never
appear as a real Origin header, and out of the CANONICAL alias table.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Ships 51e3923: visor links zero k8s.io packages (the cluster autoscaler that had
never run is deleted; billing election moved to the ha.Membership seam) and
registers ha.Static — the single-writer claim that makes hourly metering
exactly-once.
Ordering, because this pair can double-bill: the Deployment was scaled to 1 by
hand BEFORE this image goes out. `replicas: 1` in this file could not do it —
the Application ignores /spec/replicas fleet-wide, so it said 1 while the live
Deployment ran 2 and CD reported Synced throughout. The image field is NOT
ignored, so this bump does reconcile.
Tag and digest move together. Built through the platform lane
(POST /v1/runner -> BuildKit in hanzo-build); digest read from the job log's
"pushing manifest" line and confirmed independently against GHCR, since
platform leaves build_job.imageDigest null even on success.
Two things blocked this and are worth recording. The builder clones
github.com/hanzoai/visor, NOT the forge that repo calls canonical — and that
repo was ARCHIVED, so the tag could not be pushed there and BuildKit failed in
11s with "repository does not contain ref". That is why v1.108.18 through
v1.108.21 all exist as tags with no image in GHCR. The repo is unarchived.
Second, enqueue is idempotent on (repo, sha, target) regardless of status, so
the failed row had to be deleted before the same sha would build.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ghcr.io/hanzoai/app:v1.42.335
sha256:ca9685be04f52ba79ecbb0637519295cb9c7b3719444340f55398b508b7e18c5
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 34020
ghcr.io/hanzoai/app:v1.42.335
sha256:5621c135e45956084af8bcd6d733e7553e4854a4555d92aa72b29da59bd33d19
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 34019
github.com/hanzoai/mobile signs in with PKCE as hanzo-mobile; type: desktop
derives its one redirect (hanzo://oauth/mobile) from the org scheme, and the
app spells the same URI through the same convention, so neither side can
drift. A month of refresh, the CLI's reasoning on the CLI's shape.
The Application was refused whole on first sync:
namespace kube-system is not permitted in project 'hanzo-cert-manager'
cert-manager and cainjector take their leader election in kube-system, so the
chart renders four objects outside its own namespace — a Role and a RoleBinding
each, both named :leaderelection. Nothing else leaves cert-manager.
Omitting the destination did not scope the project down. It stopped it deploying
at all, which is the same whole-Application refusal the sibling projects keep
recording for one disallowed thing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The previous comment said "MUST stay 1" and left the reader to assume the file
made it so. It does not. The fleet Application carries
ignoreDifferences: {group: apps, kind: Deployment,
jsonPointers: [/spec/replicas]}
with RespectIgnoreDifferences=true, so CD applies replicas on CREATE and never
reconciles it again — deliberate, so an HPA-owned count does not read as
permanent drift.
Measured, not inferred: this file said 1, the live Deployment ran 2, and CD
reported Deployment/visor Synced the whole time. A normal refresh and a hard
refresh both advanced the revision and ran no sync; the last actual sync
operation was five days old.
That matters more than usual here, because main.go now registers ha.Static — the
claim "I am the only replica" — and at 2 replicas that bills every customer
twice an hour. A declarative line that moves nothing is exactly the wrong place
to hold a money invariant.
The Deployment was scaled to 1 by hand before the ha.Static image ships. The
comment now names the check that actually answers the question.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ghcr.io/hanzoai/app:v1.42.334
sha256:5aa1ca2191f3db1a42e83b3367ef6f091d41eecd2aa92e218757d8f08bb40a41
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 33974
ghcr.io/hanzoai/app:v1.42.334
sha256:b0bf6cbd6f2fb7867000925159e363778be921598bc784382252036ec290cb8a
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 33973
ghcr.io/hanzoai/app:v1.42.333
sha256:d6a11e317f5f88faaf49712b2717a9efa83cefe4f58c5af4e23230a54f75592a
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 33912
ghcr.io/hanzoai/app:v1.42.332
sha256:35ea0d67afa38cfdb2f2c7f80a77b2a26d9e4c0958bbf128b3d8d361821c187e
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 33880
ghcr.io/hanzoai/billing:1.0.17
sha256:6a8e2883dc19061d634e6862bfad4f33c73888293c6cb5ba0ff1425d98b37bec
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 33860
Slack's MCP Servers page connects an app to an external MCP server over OAuth.
It needs a client registered HERE (client_id hanzo-slack, HIP-0111) to get a
token for api.hanzo.ai/v1/mcp.
Opposite direction from hanzo/integrations/slack/CLIENT_ID+CLIENT_SECRET, which
are SLACK's credentials that cloud uses to call Slack. Pasting that pair into
Slack's form is the natural mistake and cannot work.
Its own app rather than a redirect on hanzo-mcp: that one is loopback PKCE for
MCP clients on a developer machine; this is a server-side client on slack.com.
redirects is EMPTY deliberately. The callback is Slack's URL and only Slack's
own page states it; this document REPLACES redirectUris every converge, so a
guessed URI is a redirect_uri_mismatch at consent while an omitted one is a
single converge away. The client and its secret exist now.
Also documents the auth shape Slack needs: IAM advertises client_secret_basic
first (matches Slack's HTTP Basic toggle) and S256 (matches its PKCE toggle).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ghcr.io/hanzoai/app:v1.42.331
sha256:73582bcd37e9fa85313788aedd107a2a0c751af6d4caa27e1ba9cfc860457aba
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 33752
ghcr.io/hanzoai/cloud:v1.801.483
sha256:c3a88a726d0662fce7c4ef4745368f48f1bc9fb44ca9ab2ce9d39eaacb3d3f69
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: hand
ghcr.io/hanzoai/app:v1.42.330
sha256:54f610cb6fe4d144a653eeeef221a45582192eb69861a57b4fcdcc0fadb3ee9f
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 33343
ghcr.io/hanzoai/cloud:v1.801.482
sha256:ee4a81d85a5164da13e612a3ffc9d825da128418beab37821b158dc791949552
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: hand
ghcr.io/hanzoai/app:v1.42.329
sha256:da8a5b89d3ad1eda5c750284bdca0de95854973fcc2b71fa53e336064d839bdd
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 33337
ghcr.io/hanzoai/app:v1.42.329
sha256:0568d1924b5b4ff347156113ff3522c15f3ae20cfc0b074c2812f1a64c9b112d
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 33333
ghcr.io/hanzoai/cloud:v1.801.481
sha256:6b7d4610cfce1454454c4f364de54163f2cb47525925c819f32019e3ba10457c
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: hand
The vendoring step I added one commit ago fails on a fresh runner:
Error: no repository definition for https://charts.jetstack.io.
Please add the missing repos via 'helm repo add'
`helm dependency build` resolves a repository by NAME out of the runner's local
repo list. It does not fetch the URL written in Chart.yaml, and a runner that
starts from a clean image has an empty list — so the step only ever worked on a
machine where somebody had already run `helm repo add` by hand. That is the same
"works because of what the working directory happened to hold" failure the
vendoring step exists to remove, one level up.
The alias is derived from the URL rather than invented, so it is stable across
runs and two charts sharing a repository cannot fight over the name.
Verified against an empty repo list: both umbrellas in this tree vendor and
render — cert-manager pulls jetstack v1.19.6 and renders 52 objects, and dash
pulls grafana. dash has the same latent defect and this fixes it too.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Recovery + upgrade in one cycle: the 3-vs-2 split at 6698/6700 cannot
heal in place (cert catch-up adopts nothing, certAccepted=0), so the
behind nodes restart-bootstrap onto the peer frontier while adopting
the fixed binary.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The running 0.1.3 image predated 21f05a1 and still served three client
testimonials nobody ever gave us. 0.1.4 is built from agency main, which
carries the deletion.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzo-k8s has had no cert-manager since 2026-08-02T00:15:31Z. The namespace, the
helm release and five of six CRDs were deleted; the sixth is still stuck
Terminating on a Challenge finalizer no controller remains to clear. 317 TLS
Secrets carry cert-manager annotations and nothing can renew any of them.
Nothing in this repository removed it. There is no such commit, and hanzo-k8s
never had a cert-manager Application to prune, so no sync could have. It was a
hand teardown and it was not recorded anywhere.
It also was not a decision this repository agrees with. Everything written the
week before builds ON cert-manager: LLM.md names "cert-manager Certificate ->
Secret" as the remainder path in the ONE fix for the ingress restart-outage
class; ingress/wildcard-cert-loaders.yaml exists to publish cert-manager's
wildcards into the Traefik TLS store; and project-platform.yaml was widened on
07-27 specifically to admit cert-manager.io/Certificate so those wildcards could
stop being hand-applied. The kms-only-secrets policy, authored 07-28, lists
`system:serviceaccount:cert-manager:cert-manager` among its four CANONICAL
principals — four days before the teardown, this cluster's secret policy named
cert-manager as a thing allowed to author Secrets at all. It is also Audit+Warn,
never Deny, so it could not have blocked anything either.
What this actually broke, beyond TLS: `universe-ingress` cannot sync AT ALL,
because one invalid task fails the whole operation —
one or more synchronization tasks are not valid: failed to discover server
resources for group version cert-manager.io/v1
That Application also owns routes.yaml, the only routing plane on this cluster.
The missing issuer froze ingress GitOps.
THE ISSUER FILE WAS WRONG AND RESTORING IT VERBATIM WOULD HAVE BEEN WORSE THAN
RESTORING NOTHING. It declared one issuer with a single http01 solver. The live
letsencrypt-prod was not that — it solved dns01 over Cloudflare with http01 only
as fallback. `wildcard-hanzo-ai-tls` is annotated `issuer-name: letsencrypt-prod`
with subject `CN=*.hanzo.ai`, and HTTP-01 cannot issue a wildcard at all; ingress/
host-certs.yaml says it in words. Applying the old file would have made every
wildcard in the fleet permanently unissuable while looking like a faithful
restore. `letsencrypt-prod-cf` was never in git at all and 59 Secrets name it.
Solvers are stated as a predicate, not a list: dns01 is the default because the
question is "can we create a TXT record here", and the answer is yes exactly when
the zone is in our Cloudflare account. http01 carries the short exception list —
domains pointed at this cluster whose DNS is someone else's. The inverse (dns01
enumerating all 267 zones we own) is ~800 stale-by-tomorrow lines whose failure
mode is a silently unissuable wildcard.
Ordering is the rate-limit control. This installs CRDs and the controller and
nothing that issues. The issuers live under `universe`, which has no automated
block and moves only when a human syncs it, so when the CRDs land
universe-ingress recreates its 11 Certificates and every one parks on "issuer not
found" — Let's Encrypt is never contacted. letsencrypt-staging proves the path
first. hanzo.ai holds 193 distinct certificates against a 50-per-week ceiling, so
a mistake against production issuance is a week-long lockout, not a retry.
Credential: --cluster-resource-namespace=hanzo, so the issuers read the
Cloudflare key that already syncs there from KMS and that the ingress already
uses. One credential, one KMS path, no second copy to rotate.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`helm package` does not fetch dependencies. It packages whatever is already in
the chart's charts/ directory, so an umbrella whose dependency was never
vendored packages CLEANLY into an artifact that renders to nothing.
Every signal says that release is fine. The job is green, the version resolves
at the registry, `helm show chart` returns the right metadata, and the
Application reports the pinned revision. Only the object count gives it away,
and nothing looks at the object count.
charts/cert-manager is the first umbrella here with a real dependency, which is
how this surfaced. The published 0.1.0 happens to contain its subchart because
it was pushed from a laptop where deps had been built — the failure this
workflow's own header warns about, one level down: not "the published artifact
was whatever someone last pushed", but "the published artifact was whatever
someone's working directory happened to hold".
`build` and not `update`: build installs exactly what Chart.lock pins, so what
ships matches the lock that was reviewed. update would re-resolve the constraint
and could vendor something newer than the lock. A Chart.lock that disagrees with
Chart.yaml now fails the job, which is the point.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
58 values files opened with:
# Generated from the <svc> App CR by hack/app2values.py.
Neither side of that sentence is true. 0c7b5ea36 deleted
infra/k8s/operator/crs — the App CRs it read — and hack/app2values.py is not
in the tree either. So the header named a generator nobody can run, over an
input that is absent, on files that are now hand-maintained and are themselves
the source of truth the fleet ApplicationSet reads.
A false "generated" banner is worse than no banner. It reads as "do not edit",
so the reader either goes looking for a source that does not exist, or edits
the file expecting to be overwritten. Both happened here: this pass nearly
abandoned a correct one-line change to visor.yaml on the assumption it would be
regenerated away.
The replacement says what is true — the file IS the declaration, edit it
directly — and keeps one line of provenance so the next person who greps for
app2values finds the explanation rather than the ghost.
Comment-only: 0 non-comment lines change across all 58 files, each still parses.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
visor's hourly metering is exactly-once by virtue of there being ONE writer:
its main.go registers ha.Static, the claim "I am the only replica". Under the
Base backend (the default, and what production runs — nothing sets
STORAGE_BACKEND) Shared() is a pod-local _global SQLite coord, so the
insert-once lease PK does not span pods. At 2 replicas both pods would make
that claim against their own coord and every customer would be billed once per
replica per hour.
This half and the ha.Static registration are ONE change. Raising replicas
requires registering a real membership source in the same commit; the comment
here and the one in visor main.go each point at the other.
No metering capacity is lost: only one replica was ever the elected billing
owner. What goes is HTTP HA during a rollout.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
imagePullPolicy is Always, so a tag alone is re-resolved on every pod start:
a retag upstream changes what the estate's single entry point runs at the next
reschedule, with nothing in git recording the change. That is the pin shape
behind the api.hanzo.ai outage. The kubelet honours the digest; the tag stays
so a human can still read which release this is.
Digest is the one the kubelet already resolved for 1.10.6, so this pins what
is running rather than asking for something new.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The chart env is already right and already applied -- the running pod carries
PUBLISHABLE_KEY=pk-live-... -- but the pinned image (v0.19.18) predates the
rename and looks for the retired spellings, so studio.hanzo.ai still serves
window.HZ.pk empty. Measured just now: 6848 bytes, zero pk-live tokens.
Pins hanzoai/studio a154b33c, which is _PUBLISHABLE_KEY_ENV = ("PUBLISHABLE_KEY",)
by construction. Tag and digest move together; digest taken from GHCR's
docker-content-digest, not a build log.
This supersedes a v0.19.19 pin someone landed while this was in flight. That
tag is a DIFFERENT image (sha256:f17605fc... vs sha256:19979195...) and there
is no v0.19.* git tag in hanzoai/studio, so nothing could establish whether it
contains the fix -- and a tag whose content cannot be established is not a
thing to deploy a fix on. Verify by BEHAVIOUR after rollout:
curl -s https://studio.hanzo.ai/ | grep -o 'window.HZ={[^}]*}'
should show a pk-live value rather than "pk":"".
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ghcr.io/hanzoai/studio:v0.19.19
sha256:f17605fcd211ff980c9196c66f5444a88caa3bb335cd2e7ccc0eb5e2b56db141
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 33062
ghcr.io/hanzoai/app:v1.42.328
sha256:4ddfec802db037e5546fc1d9cbc373ad4d0b4511d9a880e8e8aad4268ed65057
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 33058
hanzod-mv-1 was OOMKilled (exit 137) 21 times in a row, holding the fleet
at 4/5 — exactly the BFT threshold — and the C-Chain sat frozen at block
1193 with 9372 un-receipted transactions queued behind it.
The comment above these values had already worked out the right number:
each validator owns a 6.26Gi-allocatable hanzo-val node, so 5Gi leaves
~1.2Gi for system pods. The file still said 4Gi, so that decision never
actually landed.
It also cites a 2474Mi peak, which was measured on a quiet chain. Under
the backlog mv-0 and mv-4 were sitting at 3260Mi and 3422Mi — both next
in line to OOM, so this was one node away from dropping below quorum.
GOMEMLIMIT 3GiB -> 4GiB so the Go runtime still GCs hard below the cgroup
ceiling rather than sailing into a SIGKILL. requests left at 900Mi: the
comment records that raising the RESERVATION made mv-4 permanently
Pending while changing nothing about the kill, because the limit is what
the cgroup enforces.
Measured after the roll: mv-1 restarts 23 -> 0, and the C-Chain moved
1193 -> 1352 -> 2008 -> 4177 -> 6037.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`serve --zap :9653` is in iam's command and has been for as long as the flag
has existed. The listener comes up on every rollout. Nothing could reach it.
Measured against the live cluster, from a cloud pod, just now:
nc iam.hanzo.svc 80 -> OPEN the only published port
nc iam.hanzo.svc 9653 -> CLOSED no such port on the Service
nc 10.125.8.184 9653 -> OPEN the listener, answering, on the pod
and inside the pod /proc/net/tcp shows three listeners in state 0A -- 1F40
(8000), 2382 (9090) and 25B5 (9653). The ZAP door was open the whole time and
the Service published port 80 alone, so the only thing any caller could observe
was a refusal.
That refusal is the root of a fleet-wide pattern. A door that is bound and
unroutable is worse than one never opened, because every caller reads "iam does
not speak ZAP" and writes an HTTP client instead -- which is what happened six
times independently: base (25+ sites to https://hanzo.id), commerce (20), ai
(9), visor, gateway (JWKS via the public edge) and kms (its token exchange).
Most of those hairpin out through Cloudflare to reach a pod one hop away. Every
one of them was a rational response to the only evidence available.
The fix is one ports[] entry. containerPort with no servicePort publishes
9653 -> 9653 (charts/app/templates/service.yaml renders
`port: .servicePort | default .containerPort`), which is exactly how cloud
already publishes its own zap port. Rendered and checked:
ports:
- name: http port: 80 targetPort: 8000
- name: zap port: 9653 targetPort: 9653
Purely additive -- port 80 is untouched, so no existing caller changes -- and it
does not by itself move anyone onto ZAP. It makes moving possible, which it
was not before.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Four replicas were evicted off the memory-pressured hanzo-val nodes, which run
validators beside general workloads on 6.5Gi machines. The kubelet ranks
Burstable pods for eviction by how far usage exceeds REQUEST, not by absolute
size, and ingress idles at 130-163Mi against a 128Mi request — permanently
above its own request, so permanently first in line. The estate's front door
was shed ahead of every batch job sharing the machine.
The request now sits above real usage so the pod ranks below its neighbours;
the 1Gi limit is untouched. priorityClassName says the rest out loud: every
host reaches its Service through this pod, so a node under pressure must shed
something else first.
The S3_* env this file already carries had never reached the cluster — the
deployed pods ran 1.10.4 with no object-store credentials, so every staticFiles
middleware failed closed and hanzo.ai, www.hanzo.ai, gallery.hanzo.ai and
hanzo.agency all answered 404 with their bytes already staged. Applied.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The previous commit added the file and CD reported infra/k8s/hanzo-cd Synced at
that exact revision with the CronJob nowhere in the cluster. This directory
renders through a kustomization with an explicit resources list, so an unlisted
file is not applied and nothing says so -- the state the note above
sync-truth-guard already describes, reproduced within ten minutes by the commit
that was supposed to catch it.
Verified with `kustomize build`, per that note's own instruction: exit 0, 35
objects, forge-actions-guard among them.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzo.agency served a ten-day-old build while its fix sat on the forge at the
right commit. Nothing was broken in the way anything reports: the mirror was
current, the workflow was correct, the runner was up, and the repo's Actions tab
was empty in the way an idle repo is empty.
The forge only runs a repo's workflows if that repo has the Actions UNIT on, and
notifier_helper.go answers a push to a repo without it by returning nil -- no
run, no error, no log. So the one state that stops every build is also the one
state that produces no evidence of itself.
Worth naming what this is NOT, because it is the obvious suspicion and it is
wrong: mirror syncs DO trigger `on: push`. SyncPushCommits raises the same
HookEventPush with the same payload as a human push, measured on hanzoai/platform
-- GitHub HEAD 811d5e01d8f4 arrived by sync and produced run 32556, event=push,
on that sha. A mirror-sync nudge was added in July and dropped a week later; it
would not have helped, and should not come back for this.
The predicate is `.hanzo/workflows` + unit off, and it is deliberately narrower
than "has workflows". 43 first-party repos have workflows with the unit off, but
38 are forks carrying upstream's .github/workflows -- langflow's 36 files,
frappe's 17 -- where off is correct and nobody wants upstream's release workflow
running on our runners with our secrets. .hanzo/workflows is only read by this
forge, so a file there was written to run here. Across the whole forge the
correlation is total: every repo matching this has push=0 for its entire life,
every repo with the unit on builds on push.
It finds 13 today, in three orgs, two of which have never run anything at all.
It reports and does not mutate, on the same terms as cd-sync-truth-guard beside
it: enabling Actions starts running code on our runners with our secrets, which
is a decision rather than a repair. It also could not mutate if it wanted to --
the read token it shares with that guard answers 403 to a PATCH.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
v0.1.6 answers /ai, /help, /blog/all and /case-study/3 with the homepage
— the same 1,894,622 bytes as /. Only the front page works.
The base image tag is the cause, both times. Clean-URL fallback
(/ai -> ai.html) and the -404 flag arrived together in hanzoai/static
44ff6d7, released in v0.5.1. This image was built on 0.4.1, which has
neither: -404 aborts the process ("flag provided but not defined"),
which is the v0.1.3 crash loop, and with no .html fallback -spa is both
the only flag that boots AND the thing that swallows every route.
v0.1.7 builds on static v0.5.6. Verified on an isolated pod before this
pin moved: /ai, /help, /blog, /blog/all, /case-study/{,3,10} each serve
their own page, / resolves in zero redirects (0.4.1 was leaking a 301 to
/index.html), and an unknown path returns 404 with the real 404 page.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The chart set EVENT_INGEST_KEY and its comment claimed studio read it. The
DEPLOYED studio does not: studio.hanzo.ai serves zero pk- tokens right now, so
window.HZ.pk is empty and studio's events are unattributed -- exactly the bug
the comment says it fixed.
Fixed on both sides: hanzoai/studio a154b33c collapses _PUBLISHABLE_KEY_ENV to
("PUBLISHABLE_KEY",) with no fallbacks, and this chart now sets that name.
Takes effect once studio is rebuilt past that commit; the comment claiming
"no rebuild is involved" was corrected rather than left to mislead.
Also renames NEXT_PUBLIC_EVENT_INGEST_KEY in the docs website analytics
component. Fleet-wide the key is PUBLISHABLE_KEY -- it is a pk- publishable
value, and the name now says so.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The Pages project `hanzo-agency` was a direct wrangler upload with no source
repo recorded, so finding the source was the work. It is hanzoai/agency,
established by rebuilding it and matching the live <title> and all 34
react-router routes.
Serving moves to the site plane: middleware `hanzo-agency-static` over
s3://cdn/hanzo-agency, router `hanzo-agency`, no pods. spaMode is TRUE — this
is one index.html with a client router, the opposite of hanzo-ai-static, where
a real HTML file exists per route and a fallback would hide a miss.
The point of the move is the key. hanzo.agency could have been flipped straight
at the existing agency Deployment, which already claimed the host and serves the
same site — but its image ghcr.io/hanzoai/agency:0.1.3 carries NO ingest key
(assets/index-C6KAO3Lp.js: @hanzo/observe wired, zero `pk-` bytes). That origin
answers 200 and reports nothing, which is the silent-dark failure being fixed,
and it cannot be fixed here because images are built by CI. The rebuilt bundle
inlines the key and is asserted to contain it before it ships.
The key had to be discovered, not assumed: this app reads VITE_PUBLISHABLE_KEY,
hanzo.ai reads NEXT_PUBLIC_EVENT_INGEST_KEY and @hanzogui/telemetry reads
*_HANZO_INGEST_KEY. Guessing yields a green build and a keyless bundle, so the
gate is on the built output, not on the command line.
@hanzo/observe was absent entirely and is added at 0.1.7. Below 0.1.6 the engine
runs and measures no coordinate, so clicks carry no $x and heat maps stay empty
while every other signal looks healthy.
hanzo.agency also comes off the agency Deployment's ingress host list, so exactly
one router claims the host. agency.hanzo.ai and www.hanzo.agency stay there and
still serve the keyless bundle; they are a follow-up. www.hanzo.agency is not
claimed by the new router at all because it still CNAMEs to Vercel.
No ACME: agency-tls already carries DNS:hanzo.agency.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Tag and digest move together. The kubelet honors the digest, so moving the
tag alone would have reported a new version while serving the old bytes --
that exact split caused an api.hanzo.ai outage today.
Verified before pinning: ghcr manifest for 0.1.15 resolves to
sha256:9b5ecc08 under the in-cluster ghcr-secret creds, and a scratch pod
pulled it by digest. Serving diff vs live 0.1.14 is og:image (absent
entirely before), og:image:alt/width/height, and a twitter card rewritten
off the stale "Hanzo -- the AI cloud" title onto the current one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The image carries zen 1.4.11, which routes every chat and vision SKU (zen5-mini,
zen5-flash, zen5, zen5-coder, zen5-pro, zen-vl, zen-guard) through openrouter
instead of DigitalOcean. Cheaper on every rung — zen5 quotes 2.28/7.26 per MTok
where it quoted 4.176/13.2, at the same 3x margin.
OPENROUTER_API_KEY rides in the SAME commit deliberately. zen's resolver reads
env first and KMS only as a fallback, so an image with these routes and no key in
the environment does not degrade — it 401s on every chat. Pin and credential are
one change or they are a broken window.
Every route was tested against openrouter before this landed, including that the
new vision model reads an actual image.
Four SKUs stay on DigitalOcean because openrouter has no endpoint for them:
zen-embedding, zen-image, zen-video, zen-rerank.
One CI run reported 196 "no space left on device" failures and every reading
said there was space: the runner's docker volume held 36.8 GB against a 200 GiB
limit, on a node with 1.17 TB free.
The path in the error is what tells them apart:
write /dev/shm/hanzo-sqlite-plain-2867704271/db: no space left on device
/dev/shm, not the disk. Docker gives every container a 64 MiB shared-memory
tmpfs and act_runner takes that default, while hanzoai/cloud's suite opens its
SQLite databases there. Across ~200 packages the segment fills, and every package
after the fill point dies — 196 of them, all alphabetically LATE (translate,
treasury, validators, wallets, webhooks, world, x402) and all in hundredths of a
second, because they fail OPENING a database rather than running a test.
That shape is why this was read as disk exhaustion three times: late packages,
instant failures, "no space left on device". It is a 64 MiB tmpfs on a 1.2 TB
node.
tmpfs is RAM, so this is charged to the pod's 26Gi memory limit rather than to
disk. 8 GiB is far beyond what the suite's databases need and still leaves the
linker the headroom runner.envs' GOFLAGS -p was tuned for.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ghcr.io/hanzoai/cloud:v1.801.479
sha256:6e27bd85ed30202edfe08a5217c69c74204f5e570d6dc5fa98c7db2e946348a6
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: hand
Tag and digest move together (bumping the tag alone renders repo:newtag@olddigest
and the kubelet honours the digest). Digest verified against GHCR's
docker-content-digest, not the build log.
Applied directly on forge/main because hanzo-cd reads
https://git.hanzo.ai/hanzo/universe and the forge and GitHub mains have
DIVERGED (GitHub ahead by 5128). Reconciling that is a separate, deliberate
merge -- this is the two-line pin only.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
commerce is a plugin of the cloud binary — there is no commerce backend
pod — and the embed carried the store and billing routes but no resource
bundle. So product, collection, variant, webhook, saleschannel and
stocklocation 404'd, and commerce-admin, correctly pointed at
api.hanzo.ai/v1/commerce, had nothing to read.
404 vs 401 is the tell and it is what located this: /v1/store/current
answered 401 (mounted, gated) while /v1/commerce/product answered 404
(absent). A gate returns 401; a missing route returns 404.
Built from hanzo-inc/cloud at refs/heads/main = a10583c5b. That repo, not
the forge and not hanzoai/cloud — three diverged lineages answer to the
name "cloud" and only this one is what BuildKit clones.
Also corrects the verification recipe this file prescribed. It said to
read org.opencontainers.image.revision via `crane config`; that label is
"unknown" on .466, .476 and .478 alike, because the builder passes
build-arg:VERSION and no revision. The command answers for every image
and distinguishes none of them, so following it yields a confident
nothing. The build job's own --opt=context= arg is the real input.
Carries hanzoai/ingress 52b6bbd. 1.10.5 gated the static-export route ladder on
path.Ext, which reads a version number as an extension — "glm-5.2" has ext ".2",
"llama-3.2-1b-instruct" has ".2-1b-instruct". Measured live on the hanzo.ai
cutover: 181 of 773 routes 404d, every model page carrying a version, while the
dotless routes served. DNS was rolled back to Pages within the minute and this
is the fix.
The ladder now runs for every miss. It asks for one exact file per candidate and
never falls back to a shell, so a missing chunk still answers a bare 404 — by
construction now rather than by guessing what a path means.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ghcr.io/hanzoai/app:v1.42.327
sha256:152e12fa8e95f835a693534057342e36cabae5cc921f297f6304066b2484abfb
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 32590
ghcr.io/hanzoai/app:v1.42.327
sha256:f7e63ce4e7990ca46f7882d2f70cb4fea44f4fc2f5fb58b4ded3eb7f6a42e4c4
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 32589
sensei.group has been serving a generated placeholder built from
hanzo-apps/sensei.group — meta author "Hanzo AI", a stock 404, a
generated-asset favicon, 853 bytes of it. The actual site lives in
sensei-group/sensei.group and had never shipped an image.
v0.1.3 is that site: a Next 14 static export, no API routes, no
middleware, no auth in the module graph at all. Tag and digest move
together, because the kubelet honors the digest and a tag moved alone
reports a new version while serving the old bytes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
script-src named hanzo.app so https://hanzo.app/edit.js would execute.
connect-src did not, so every fetch that script made was refused, and the
'edit this page' control has been dead on every route since.
Measured on the live site with a real headless Chromium at 1280x900 and
390x844, four console errors on each of /, /docs, /docs/getting-started and
/docs/api:
Connecting to 'https://hanzo.app/v1/edit/config' violates the following
Content Security Policy directive: "connect-src 'self' https://api.hanzo.aihttps://analytics.hanzo.ai". The action has been blocked.
and the same pair for https://hanzo.app/v1/me. Users also saw the widget paint
an empty ~28px ring at the bottom-right of the viewport and then remove itself
— it is in no DOM element by t+7s, because the component tears down when its
config fetch fails.
Same change already landed on github.com/hanzoai/universe main (1a3c28ad9).
This lineage is what cd.hanzo.ai actually reads — Application hanzo-docs,
repoURL https://git.hanzo.ai/hanzo/universe, targetRevision main — so the fix
was committed and never deployed. The two lineages have diverged (github main
is 5127 commits ahead, this one 17); reconciling them is not this commit's job.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Carries hanzoai/ingress e2be32b. staticFiles resolved only the literal key, so a
prerendered site answered 404 on every route but "/" — measured against
s3://cdn/hanzo-ai, where 772 exported routes were staged and all but the root
were unreachable. It now tries "<path>.html" then "<path>/index.html" before any
fallback (the ladder cloud already uses in apps/sites), and serves a directory
index instead of 301-ing to it.
Same image also unblocks the repo: hanzoai/commerce v1.49.29 had been re-tagged
under a published version, so go.sum no longer matched and no ingress image
could be built at all.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The console IS static: the image is the SPA behind hanzoai/static, whose
server listens on 3000 (Dockerfile: CMD -port=3000). These numbers said
4000, the port of the Next.js server image that Dockerfile REPLACED —
so the Service targeted a port nothing listened on and both probes
dialled it. v8.5.59, the first image ever built from the current
Dockerfile, came up serving /srv on 3000, failed six liveness dials at
4000 and was killed at exactly three minutes, four times over.
The SERVICE keeps 4000: the ingress routes to it and must not move.
Only the container side follows the image.
ghcr.io/hanzoai/console:v8.5.59
sha256:5fc1aebe79769308db83705cc996f7eb6feca6710c7b5e2b3b9631d2164cd714
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: hand
The endpoint is right from this image on. It answers 404 until cloud ships the
plugin mount (forge/main e173ea030) — but so did the old address, permanently,
so this is a strict improvement: same behaviour today, correct the moment the
release lands, and no second cutover to remember.
Raising docker-storage to 200Gi without moving the ephemeral-storage request
left the scheduler believing a runner needs 38Gi while the pod may take 200Gi.
A request is a reservation; the cap is the real appetite. With the two apart,
the scheduler will place more runners on a node than that node's disk can
serve — and the first symptom is the eviction this change exists to end, only
now with the cap looking innocent.
200Gi against ~1141Gi allocatable bounds placement at five runners per node.
Two is what actually lands; five is the ceiling if the pool is ever scaled
down underneath them.
My omission, from the commit before this one.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The runners evict at 38Gi and cloud's test suite was measured at 56.5G of
/var/lib/docker, so the heaviest build in the fleet could not finish on the
fleet. The kubelet enforces an emptyDir cap by killing the pod, and a killed
pod writes no error — the log simply stops. That signature is 94 of the 111
failing tests in cloud gate run 836 (`no space left on device`), run 829's
truncated log, and the docs build dying at "Finalizing page optimization":
three separate investigations, one cap.
The cap could not be raised where the runners were standing. runner-pool-32g
is g-8vcpu-32gb with a 100GB disk and two runners to a node, so anything past
~44Gi meant one runner per node and half the CI fleet. So the nodes move:
runner-pool-1tb is so-8vcpu-64gb-intel — 1200GB of NVMe, and 64GB of RAM,
which matters second and independently, because the runner's memory LIMIT is
26Gi and a 32GB node could never honour two of them (node memory evictions
were already being seen). Same 8 vCPU, so scheduling is unchanged.
200Gi is from the measurement, not rounded to feel safe: two caps plus the
~14GB image store is ~444GB of 1200GB, and the heaviest observed build sits
at a twentieth of the disk. The bug was a cap with no headroom; a close fit
would be the same bug with a larger number.
A PersistentVolumeClaim is the better shape and was tried in 925984f, then
reverted in 7b561ae1: volumeClaimTemplates is IMMUTABLE on an existing
StatefulSet, so the API refuses the update and landing it costs a
delete/recreate of the whole CI fleet. sizeLimit and nodeSelector are both
spec.template fields and apply in place. Revisit the PVC alongside a planned
StatefulSet replacement, not before.
This does not replace the prune CronJob. A leak fills any volume; a bigger
cap only changes how long that takes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Serves the marketing apex from s3://cdn/hanzo-ai through the staticFiles
middleware — no pod, no image, no tag to bump. It is a Next.js `output: export`,
so every one of its 772 routes is already a file; there was never anything to
render.
This retires BOTH mechanisms that were aimed at this host and neither of which
ever served it:
- Cloudflare Pages, which is where hanzo.ai actually lived. It serves every
content-hashed chunk `public, max-age=0, must-revalidate` and answers a
missing .js with 30KB of branded HTML, which is most of why the apex threw
ChunkLoadErrors. Here the hashed types are immutable for a year and a miss
is a bare 404.
- the `hanzo-ai-www` image plane, whose router in routes.yaml pointed at a
Service that was never created. Anything reaching the cluster for this host
got a 502; it went unnoticed because DNS still said Pages. Deleted rather
than left inert — a file-provider router outranks the CRD one, so leaving it
would have served that 502 the moment DNS moved.
spaMode is off and no errorPage404 is set: with a real file per route there is
no client router to fall back to, and an SPA fallback would answer 200 with the
homepage for every missing chunk. Cache policy is keyed by extension because
that is what the middleware matches; verified against the built export, which
puts all 332 of its .js/.css/.woff2 under /_next/static/ and leaves nothing
hashed outside it.
DNS still points at Pages on this commit. The route is inert until it moves.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ghcr.io/hanzoai/cloud:v1.801.476
sha256:0743f907a3f12b469a538d52b7685c43cd2ffdb9cfc53a4c06e2ccb773a8f173
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: hand
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The staticFiles middleware fails closed. Absent S3_ENDPOINT it refuses to build
("needs S3_ENDPOINT in the ingress environment"), absent either credential
likewise, and a middleware that will not build takes its router down with it.
charts/app/values/hanzo/static-sites.yaml has said since it was written that the
plane "requires ingress >= 1.10.0 with S3_ENDPOINT/S3_REGION +
AWS_ACCESS_KEY_ID/SECRET from s3-credentials in its env". That env was never
committed here, so gallery-static — the only staticFiles middleware in the fleet
— has never once built, and gallery.hanzo.ai has been served the whole time by
the older cdn-filer routers in routes.yaml. The zero-pod site plane existed on
paper and nowhere else.
No behaviour changes on this commit: gallery keeps its file-provider routers,
which outrank the CRD route 150 to 100. This only makes the plane buildable.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The size was never the defect. Measured on a runner that STARTED CLEAN and ran
the full suite: 18.1G against the existing 38Gi cap, with 20G of headroom unused.
Every eviction traced to a runner that began its build already carrying 19-25G of
another job's leftovers — the leak, which the prune CronJob now clears between
runs.
So this bought nothing the prune had not already bought, and it cost three things:
volumeClaimTemplates are IMMUTABLE, so it could only land by deleting and
recreating the StatefulSet — which is 10 runners draining at 31 minutes apiece,
on the critical path of a cutover; 10 x 1Ti of always-on block storage is
~$1,024/month for CI scratch that is regenerable by definition; and until that
recreate happened the CD application would have sat in permanent sync failure
against a field it cannot patch.
The right version of this is a separate piece of work — an autoscaled runner pool
with EPHEMERAL disk per runner and the cluster autoscaler behind it — built
deliberately rather than threaded through a release.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The host was declared twice: hanzo-domains routes it with the path split
(/v1/commerce and /v1/billing to cloud, everything else to this Service),
and pay.yaml declared its own Ingress for the same host carrying only '/'.
Two Ingress objects for one Host means which one answers a /v1 request is
decided by rule length, not by design -- it happened to work.
The subset one goes. The Service is unchanged; hanzo-domains points / at it.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud-api never bound a listener. kms and pubsub fail closed on a missing
master key ('cek: no master key'), each failure burns a 60s start timeout,
and the liveness probe SIGTERMs the process at exactly initialDelay 15 +
30x6 = 180s -- before boot finishes. 393 restarts over 35h with zero
listening sockets, and the whole boot log only 32 lines.
The key is now in KMS at org=hanzo env=test path=/cloud
name=CLOUD_KMS_MASTER_KEY -- a sibling of the coordinate prod and devnet
read, written through /v1/kms with the platform machine identity and verified
to decode to 32 bytes. env=test and not prod on purpose: a testnet plane
sharing prod's at-rest key makes the blast radius of a testnet compromise
production data.
Consumed the canonical way, a KMSSecret the operator materialises, not a
hand-made Secret. Every other namespace's copy of this shape is owner=<none>
because someone kubectl-created it, which is exactly what the
kms-only-secrets admission policy exists to stop: 'Secrets must originate
from KMS. Author a KMSSecret instead.'
ONE-WAY DOOR, stated in the file: the first store open after this key becomes
visible ENCRYPTS those stores in place, and a ciphertext store whose key is
lost is unrecoverable by any key. So this coordinate is never re-pointed or
re-minted -- rotation is a re-encrypt, not an overwrite.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
/var/lib/docker was a 38Gi emptyDir, which is NODE ephemeral storage, and the
kubelet enforces that limit by EVICTING the pod. A build that outgrew it was
killed mid-run with its log simply stopping — no error line, nothing to read —
so the failure looked random and followed no repository. Twice in one hour, and
it had been happening for far longer than anyone had attributed to it.
The limit could not just be raised. An emptyDir comes out of the node's ~88Gi and
these nodes carry two runners, so anything above ~44Gi means one runner per node:
headroom bought with half the fleet.
A PersistentVolumeClaim is not the node's disk. The size becomes a property of
the runner rather than of what it was scheduled beside, and two runners still
share a node. 1Ti against a heaviest-measured build of ~20Gi is deliberate
overprovisioning — the point is that disk stops being something anyone reasons
about. Cost is on record: 10 x 1Ti of do-block-storage is ~$1,024/month.
It does NOT retire the prune CronJob. A leak fills any volume eventually; 1Ti
only changes how long that takes, and a runner that never sheds its build caches
would still get there. Belt and braces, deliberately.
The ephemeral-storage REQUEST goes with it: the node was reserving 38Gi per
runner for storage that no longer lives on the node.
NOTE for whoever applies this: volumeClaimTemplates are IMMUTABLE, so the
StatefulSet must be recreated (delete --cascade=orphan, re-apply, then roll the
pods) — a values edit alone will report synced and change nothing.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Address generation was dead fleet-wide behind a health endpoint that reported
signing_quorum: true, for two reasons that only bite together:
a node counted ITSELF as a peer, because filterSelf compared node IDs while
the --peer flags carry addresses, so the self entry never matched and every
node dialled itself;
the readiness gate compared the peer count with == rather than >=, so the one
extra connection that bug produced put the count permanently past the value
the gate would accept.
The cluster ran v1.17.15 while this manifest still pinned v1.17.12 — drifted,
so the pin is now what runs rather than a third value nobody deploys. This is
what makes the crypto rail's 'temporarily unavailable' stop being true.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two corrections to the shape this landed in, both from measuring rather than
assuming.
IT MUST NOT RUN UNDER A LIVE BUILD. Measured on a runner mid-suite: the
docker-storage volume held 17.4G while the runner's own rootfs held 2.7G, and
nearly all of that 17.4G was the RUNNING job container's writable layer — the Go
build cache across ~120 packages. That is not reclaimable and must not be, so
pruning there frees almost nothing while converting a legible "no space left on
device" into a build that fails for reasons nobody can reconstruct. It now skips
any runner with a container in flight and prunes between jobs.
CONTAINERS FIRST. An image cannot be removed while a container references it, so
`docker image prune -af` on a runner carrying 24G reclaimed 0B until the finished
job's container went — and the container is exactly where the build cache lives
once the job ends. Measured with the corrected order: 22733M -> 704M on one
runner, 25481M -> 4385M on another.
The threshold moves 15G -> 8G for the same reason the skip exists: an idle runner
sits near 2G once this has run, so anything above 8G is leftovers, and it leaves
the heaviest build measured (~20G) a clear 30G inside the 38Gi cap.
It reports reclaimed bytes per runner, because "the job exists" is not evidence
that anything was freed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Every runner is its own Docker host with /var/lib/docker on an emptyDir capped
at 38Gi, and an emptyDir lives as long as the POD, not the job. So each build
leaves its images and build cache behind and the next build starts higher than
the last: measured across the fleet at one moment, 1.6G on a pod minutes old and
24.5G on one four hours old, for the same work.
The cap is enforced by EVICTION. The kubelet kills the pod mid-build, the job log
STOPS — no error, no last line — and the run reports failure with nothing in it
to read. Twice in one hour (git-runner-6 20:44:35, git-runner-5 21:15:31), and
because the build that dies is simply the one that arrives last, it followed no
repository and looked random.
Reclaiming by hand returned ~65G across ten runners and the next build passed on
a freshly restarted runner — the same build that had been evicted twice. The wall
was the leak, not the build. Raising the cap would have halved the fleet (nodes
hold ~88Gi and run two runners each, so anything above ~44Gi is one per node) to
buy time against something that refills whatever it is given.
It removes only what nothing is using, so it is safe while a build runs — proven
against a live job that kept running while 7G came back underneath it. It fires
on a threshold rather than every tick, so an idle fleet keeps its caches warm.
Two details that are the difference between working and silently doing nothing:
DOCKER_CONFIG is redirected to a writable path, because the runner mounts
/root/.docker read-only and buildx refuses to start without somewhere to create
its own directory — `docker builder prune` then fails with "mkdir
/root/.docker/buildx: read-only file system" and reclaims zero. And it prunes
IMAGES with -a: without it only dangling images go, which on these runners is
nothing at all.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ghcr.io/hanzoai/app:v1.42.326
sha256:8c6e0259e022f8997dd6bd6675ec3bca53cefa23d09f9bab54c980a93cad8f34
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 32248
ghcr.io/hanzoai/app:v1.42.325
sha256:9a8161c0c7c1cb4e247e263611ac6321a5db657b412cdbd259007c9e3699b6ab
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 32212
Both SPAs sign in through the hanzo-cloud client and return to a callback under
their own base path — /tracker/callback and /meet/callback, baked into each
built bundle and into the PKCE request it sends. Neither is /auth/callback, so
neither can be derived from `hosts:`; they are literal `redirects:`.
Measured on the live IdP first: authorize with either URI answers 400, the same
answer an unregistered URI gets, while console.hanzo.ai/auth/callback answers
302. The same probe found https://social.hanzo.ai/auth/callback — sitting in
init_data.json — also answering 400, which is the difference between the two
files stated plainly: init_data seeds a FRESH database, provision converges the
LIVE one. A redirect URI added to init_data.json for an IdP that already exists
reaches nothing.
Landed ALONE, ahead of any routing change and ahead of the binary that serves
these hosts. Registering a callback grants nothing by itself — no host resolves
to cloud yet — but a host flipped before its callback is registered serves the
app and then fails sign-in at the door.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ghcr.io/hanzoai/insights:1.52.78
sha256:0fe05abbeb9132f5d11cc2056e41947324435f92c14a5d1fefa6bd35ccc8439a
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 32159
ghcr.io/hanzoai/insights:1.52.78
sha256:0fe05abbeb9132f5d11cc2056e41947324435f92c14a5d1fefa6bd35ccc8439a
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 32159
ghcr.io/hanzoai/insights-plugin:0.1.7
sha256:fcfd918b91b219d173867d87481a1cb7bff9c1c057ce6ce073a2483cf70ab5fc
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 32161
ghcr.io/hanzoai/insights-livestream:1.0.1
sha256:64aeb0c4198b12623dc63caaaee28369664660f1f1067d03a78b496fa8d08ee5
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 32160
ghcr.io/hanzoai/cloud-www:0.1.14
sha256:ba48c0c2a7130924089a43e27d36b47601d6248ded808e46a38fc3bb469ff3c2
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: hand
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two deploys that were built and merged but never became real.
insights-plugin 0.1.6 is insights e89a5b62a3b — org-derived project, token
path deleted. It has been in GHCR since 20:43Z while production served
0.1.4 from 08-03, because plugin.yml's `Ship it` step reads the pin token
from /v1/kms/orgs/<org>/secrets/..., which the embedded KMS does not serve.
curl -fsS exits 22 on that 404 under set -e, so the step died before its own
"missing token" guard could name the problem: two releases pinned nothing
and said only "curl: (22)". Pinned by digest because both forge mirrors
pushed tag 0.1.6 a minute apart with different bytes.
studio gets EVENT_INGEST_KEY. The image already knows how to use it —
_publishable() stamps window.HZ.pk from it, behind a pk- allowlist — but the
variable was unset, so the branch that runs is the one whose comment says
events go unattributed. Measured: zero pk- tokens in every root studio
serves. Runtime-read, so the pinned image is unchanged.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
insights.hanzo.ai reached insights-web only through the priority:1
insights-hanzo-ai-web router in the Traefik file provider, so the host had no
declaration in the chart that owns the rest of the service. Enable the chart
ingress on the host instead: same backend (insights-web:8000), rendered by the
shared template, with an exact-SNI cert from letsencrypt-prod.
The rendered rule outranks the superseded file-provider router by rule length
and stays under the priority:150 cloud-ingest carve-out, so /e, /batch,
/capture and /v1/e still divert to /v1/event.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
<service>.<env>.hanzo.ai, so *.dev is devnet, *.test is testnet and *.main is
mainnet. Every service in an environment sorts under it (api.dev,
console.dev, ...) and no name has to spell out "devnet". dev.hanzo.ai was
mine and was wrong the other way: it named the environment and dropped the
service.
TLS arrives with the declaration, which is worth knowing before the next host
is added: cert-manager is NOT installed here -- the Certificate objects in
infra/k8s/ingress/wildcard-certs.yaml are inert, the API has no such resource
type -- so Traefik's own ACME issues per host, which is why
api.devnet.hanzo.ai holds a cert whose CN is exactly itself. The one wildcard
that exists, *.hanzo.ai, does not cover a two-label name: this host failed TLS
outright until it was declared.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The runner resolved every type:boolean workflow_call input to false —
explicitly passed true included — which is how 38 green runs in 16
repos published nothing and cloud's gates never once executed. The
reusable workflows were fixed to strings hours ago; this rolls the
runner fix itself, which also covers the ~1,400 mirrored repos whose
workflows we do not control. Grace 1860s lets in-flight jobs finish;
the queue is idle at roll time.
Carries the typed sweep: platform 32/32, billing and affiliates 24, the
tail 16 with notify's sends typed at last, commerce's probe, and every
surviving raw door naming its reason in place. Availability now records
into the framework's own registry beside the old meter path. Every typed
op projects to REST, OpenAPI, MCP and native ZAP by construction.
A stale pin here does not degrade gracefully — it refuses everything. The Job
ran ghcr.io/hanzoai/iam:sha-ba43c54 while the live server runs v1.34.17 and
provision.yaml had gained `owner:` (5c6990c4b). The provisioner parses
STRICTLY, so that one unknown field made `iam provision` reject the whole
document — `parse: [99:5] unknown field "owner"` — and converge nothing. Every
edit to provision.yaml since has silently reached the live IdP not at all, which
is exactly the failure this file's own comment warns about and then fell into.
Pinned to v1.34.17, the build the server runs. Converged: 15 apps, 0 failed, and
verified live with a control — commerce.hanzo.ai/admin/callback now 302s,
evil.example.com/pwn still 400s, so the allowlist is honouring the new URI
without having gone permissive.
commerce.hanzo.ai could not sign anyone in. Driving it gives client_id=
hanzo-commerce, redirect_uri=https://commerce.hanzo.ai/admin/callback, PKCE S256
-> 400 invalid redirect_uri. The live client, seeded from init_data.json, lists
only /callback and localhost:8090/callback.
The allowlist is genuinely enforced rather than failing open: a bogus
https://evil.example.com/pwn control 400s identically. So this was a total
sign-in outage on that surface, not a bad secret.
The path carries /admin because the app is MOUNTED there — the export is built
with Next basePath: '/admin', so every route it owns, its callback included,
lives under that prefix. A bare /callback is a path the app does not serve.
Declared in provision.yaml rather than init_data.json, which only seeds on boot
with newOnly=true: an app already present is never revised there, so a redirect
added to that file would have reached nothing.
Migrations run in a migrate init container — after restore, before
serve — with no image key, so the chart's inheritance rule pins it to
the app's own build and the runner can never drift from the binary that
serves the result. serve drops --automigrate: a bad migration now fails
the pod at Init, named, and the app never serves a half-applied schema.
The 0.39.16 recursion ran inside the serving process's boot automigrate;
this is the structural end of that class.
ghcr.io/hanzoai/billing:1.0.15
sha256:d6d8107c255fb377f1c25494fd4a40caf12066149e10f1ca659e9d11f13c0709
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: hand
The .network host was retired 2026-07-27 and no longer resolves at all (a
request to it now returns nothing, not an error), so every reference to it named
an address that cannot answer. The brand host is .cloud, matching kms.zoo.cloud
and the rest of the white-label convention.
Left alone deliberately, because rewriting them would invert what they say:
the LLM.md line that RECORDS the retirement, the e2e spec that ASSERTS
kms.lux.network must not resolve, and the recorded applies + cluster backups,
which are faithful accounts of what was applied and are not config.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Rolling back a break I shipped. The image built from current main serves the
export's PRE-COMPRESSED index.html.gz as a file: content-type application/x-gzip,
no content-encoding: gzip, and x-content-type-options: nosniff already set — so a
browser is handed what it reads as a gzip download and refuses to render it. The
body is correct (it decodes to the new admin); the framing is not.
0.0.1 is stale but renders. The pin follows main again once the static layer
either stops emitting .gz siblings or serves them with content-encoding.
The ship script replaced whatever tag the CR held, and the CR held a newer
release than the one my watcher was gating on — another lane had already
shipped past it. v1.801.466 contains the console 8.5.50 pin commit and
everything after it, so forward is simply the tag that was already there.
The doc said hanzo-app does not replicate, conflating the volumeless web
workload with its Base data plane. The Base replicates now (cf839494b):
sync green, objects sealed, restore proven with -o inside the sidecar,
and the two plaintext snapshots the old sidecar left are quarantined.
Carries console 8.5.50: the assistant's streamed completion rides the
client's one authorized door instead of a bare fetch, preferences live on
cloud's /v1/prefs, a 401 with a live token stops claiming the session
expired, and exactly one composer mounts per viewport.
ghcr.io/hanzoai/chat:1.0.84
sha256:fc75ed6d0626e9a0ff90420dfeabc1e12faf5b6b97698ab9c38b5bbaa4ffb645
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: hand
chat and dataroom replicate; the Base behind hanzo.app did not — no
sidecar, no restore, a Retain policy standing in for a backup. Same
shape as the other two: WAL to SeaweedFS (bucket hanzo-app-db, prefix
hanzo-app/app), age-encrypted client-side, restore-if-absent on boot,
v0.9.10 so Base rewriting the file cannot starve the bookkeeping.
Only data.db — auxiliary.db is the log store, derivable by design.
The age keypair is the one provisioned 2026-07-22 and never connected;
nothing was ever sealed to it, so it is reused, not rotated.
The sequence is the load-bearing part. An encrypted lineage had to exist and
had to restore clean BEFORE the surviving plaintext was touched, because until
that moment the plaintext was the only restorable copy dataroom had -- frozen
at txid 0x20 since 2026-08-02. Moved, not deleted; sizes verified before each
source was removed; restore re-run afterwards against the encrypted-only
lineage and passed again.
Also states the thing that would otherwise be read wrong: 65 tables, zero rows
in every business table. dataroom is deployed and unused. The backup is now
correct and there is nothing in it.
ghcr.io/hanzoai/id:0.2.32
sha256:4c06b69e3cfad0b9cb74ed0c425d97519e4412355b7791a1a5c19124d6dc3e17
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: hand
af64aeb5b reverted the collections declaration mid-incident on the theory
that mounting these files crash-looped the pod. It did not. 0.39.16 recursed
in core.(*Collection).UnmarshalJSON on ANY unapplied migration, because
`type alias *Collection` is a defined POINTER alias and jsonv2 resolves the
element's UnmarshalJSON through it and re-enters forever.
The revert deleted only the DECLARATION. prune is false, so the ConfigMap
stayed, the Deployment went on mounting an object no repo owned, and the
collections were applied anyway by the 0.39.11 rollback pod. That left the
app one `kubectl delete cm` away from a pod that cannot start, and the
Application permanently OutOfSync on the one resource nothing declared.
Restored verbatim on 0.39.17: both files hash identically to what is already
in the cluster (projects a66d49e44a6f, images a8622b86e270), and both are
already recorded in _migrations, so --automigrate is a no-op for them. The
pod rolls once to pick up the checksum annotation.
Measured on prod bytes under GOEXPERIMENT=jsonv2 — same data, same unapplied
migration: 0.39.16 exits 2 with "fatal error: stack overflow", 0.39.17
applies it and serves.
Two faults, one dead backup. Both are recorded in the file because both recur.
1. The sidecar had failed every checkpoint for three days -- 777 of them --
on "reacquire read lock: no such table: _replicate_seq", while the pod read
2/2 Running. replicate keeps two bookkeeping tables in the file it guards
and, up to 0.9.9, created them once in init(). dataroom boots
`prisma db push --accept-data-loss`, which reconciles the whole file against
the app's schema and drops every table that schema does not declare. It
announced it in its own boot log, one second after the sidecar created them:
You are about to drop the `_replicate_seq` table, which is not empty (1 rows)
Whichever container finishes last wins, and they start together, so this was
a restart away the whole time. Fixed in replicate 0.9.10 at the layer that
owns the invariant: the schema is asserted on every read-lock acquisition
instead of once at boot. Fixing dataroom's CMD would have fixed one caller
-- drizzle-kit push and atlas drop unknown tables too, and no app will ever
declare a table it does not know about.
2. There was no age block, so all five objects in the bucket are plaintext LTX,
including the L9 full-database snapshot. The keypair is now supplied from
KMS and every new object is sealed. The existing plaintext stays where it is
until 0.9.10 has written a full encrypted lineage AND that lineage has been
restored and integrity-checked -- right now it is the only restorable copy
dataroom has, frozen at txid 0x20 on 2026-08-02.
The config itself moves into universe. It had existed ONLY in the cluster,
applied by hand, which is precisely the condition that lets a second config for
the same prefix exist -- the failure this repo just cleaned up three times over.
chat.yaml's fleet note is corrected while here: it said dataroom and hanzo-app
both replicate without age. dataroom did and no longer does; hanzo-app does not
replicate at all -- no sidecar, no config, no mounted volume -- so nothing there
was ever going to fail closed. Its bucket holds two plaintext snapshots from
2026-07-29, left behind when its sidecar was removed. And the runbook that
comment says "still reads" now actually exists.
ChatReplicaSyncFailing matched `body CONTAINS 'monitor error'`. dataroom's
sidecar logged "sync error" 777 times over three days with a dead replica, so
the one rule written to catch a silent replica could never have fired for it.
They are different loops, not synonyms: "monitor error" is replica.go's replica
monitor, "sync error" is db.go's per-database sync monitor. A failure that
wedges the database's own sync -- which is what a dropped bookkeeping table
does -- only ever produces the second. The filter now matches both.
Renamed off "Chat" while here. The rule was never chat-specific: it filters on
the container name and groups by namespace and service, so it covers every
replicate sidecar in the estate, and a page reading ChatReplicaSyncFailing for
a dataroom outage is a rule lying about its own scope. Free to rename because
nothing is deployed to orphan -- apply.sh matches on .alert, and there is no
o11y admin token anywhere in KMS, so this directory has never been applied.
Which is the thing to fix next, and it is bigger than this file: the rule still
does not exist in production. It is JSON in git that nothing reconciles.
`./apply.sh` needs O11Y_TOKEN and is run by hand or not at all -- and "not at
all" is the state that let dataroom go quiet for three days. Validated here
(`./apply.sh --check`: 19 rules, 3 channels, ok) which is as far as this commit
can get on its own.
The DR doc described three layers and stopped. The fourth -- continuous WAL
replication to SeaweedFS, which is the ONLY recovery path chat and dataroom
have -- was invisible here, so a responder reading this file would not have
known it existed. Worse, the SQLite quick-reference told them to `.restore`
chat from a logical dump that no CronJob produces.
The recovery procedure is now stated, and it is one procedure: the service's
OWN replicate config, with `-o` redirecting the output. Never a second config.
A config carries the age block as well as the bucket, and a second one written
to "just restore something" reliably omits it, because the author is reading,
not writing. replicate resolves one config for both, so that pod writes
plaintext to the live prefix the moment it compacts. recover-config,
recover-config-chat and recover-config-hanzo-app were exactly that, and the
chat one left two unencrypted full-database snapshots holding Balance and
Transaction rows under chat/app/0009/. All three are deleted from the cluster;
nothing referenced them (checked every deployment, statefulset, daemonset, job,
cronjob and pod in every namespace).
Verified before writing it down: restoring chat's encrypted replica to a
scratch path passes `-integrity-check full` and yields 28 users, 13 balances,
358 transactions, 350 messages.
Also corrects a claim this file has been making while it was false. It says
every critical PV is Retain; chat-app-db, dataroom-app-db and hanzo-app-app-db
were all Delete -- three PVs created 2026-07-29, after the patch pass that
this table records, so it never reached them. Chat's is the file holding the
money tables, one PVC delete from gone. All three are patched back to Retain,
and the table now says out loud that it drifts and how to re-audit it.
ghcr.io/hanzoai/base:0.39.17
sha256:8d05b245c4d290d1f7d5bb0d1b188376513097d896abc0ce9330f47406864b11
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: hand
dataroom's replicate sidecar has no age block, so every object under
s3://dataroom-db/dataroom/app is plaintext LTX -- all five of them read
`LTX1` in the clear, including the L9 full-database snapshot.
The keypair to fix that has been sitting in the cluster for 31 days as a
hand-made Secret named dataroom-replicate-age, provisioned and never
connected to anything. It is reused rather than rotated: it round-trips
under luxfi/age v1.6.0, the library replicate actually links, and nothing
has ever been sealed to it, so a rotation buys nothing and puts a second
keypair in flight. What it lacked was a home -- it is now in KMS at
hanzo/dataroom/{AGE_IDENTITY,AGE_RECIPIENT}@prod, and this sync is what
makes KMS the source of truth rather than a Secret somebody typed once.
This lands ALONE, ahead of the config that consumes it. From 0.9.9 on
RequireEncryption defaults true: a sidecar that resolves an empty identity
refuses to start rather than quietly writing cleartext. That is the correct
behaviour and it means the key must be present first, not in the same sync.
ghcr.io/hanzoai/chat:1.0.83
sha256:899913aace2c9ce4b731fec8c32384603caa62c24271fd3f2e7d4478a76db5dd
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: hand
ghcr.io/hanzoai/app:v1.42.324
sha256:04452094b2e2ecb0aeefab2f39bb5fd8a4092bb06fa71a7a8ec53fb00c870ce3
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 30494
ghcr.io/hanzoai/app:v1.42.323
sha256:9359a72dad0f571a8a769e2c84dc4515b90fb60ef9095e584d919b378d141ca1
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 30444
ghcr.io/hanzoai/app:v1.42.323
sha256:d33f6dc5e9d745fc0ff0c6409c8af7bb32bacf5f7bc4a437a4f46b3becc842bd
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 30448
ghcr.io/hanzoai/app:v1.42.322
sha256:ef8b81acf7b8c0f1409cae7cf59c8463bff02c973be11ddf24924a5b364943b4
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: ci run 30405
0.9.9 is running and the replica is green: txid.replica tracks txid.db at 0xd7f,
zero errors, and every object it writes opens "age-encryption.org/v1" instead of
LTX1. Restoring that lineage to a scratch path reproduces the live database
exactly -- integrity ok, 35/35 tables, User/Message/Conversation/Balance/
Transaction/Session all equal. The live chat.db was never opened for write.
The two plaintext snapshots are moved, not deleted, to chat/.quarantine/0009/
with sizes and mtimes intact. They stayed put until the encrypted lineage had
been written AND restored from, because until then they were the only copy that
could be read back.
Also corrects the note on -if-replica-exists. The gate did not fail -- it said
truthfully that it found no replica it could read. Blaming the flag would point
the next reader at the wrong thing; the replica behind it was the fault, and the
alarm is the control that catches that.
ai v1.832.30 puts web_search / fetch_url / deep_research in the builtin tool
registry that until now held five TIME tools. An agent on /v1/responses asking
to search the web had nothing to call while /v1/websearch, /v1/crawl and /v1/ask
were live the whole time. This host installs the websearch backend in-process —
never over api.hanzo.ai, which validates a customer credential and 401s a
service.
deep_research is declared but NOT installed, deliberately: it carries a 25-cent
per-answer fee charged through Bill.Gate, and a tool call has no payer, so a
direct seam would be unbilled 25-cent calls an agent could loop. It answers
'unavailable in this deployment' rather than an empty result — an agent told 'no
results' concludes the web holds nothing and answers from memory.
Also the widget rule on /v1/ask, so a comparison arrives as a table and a
procedure as steps. The client validates and drops anything malformed, keeping
the prose.
Digest read from the registry with crane.
Both halves of age support land here. Write: s3.ReplicaClient.WriteLTXFile now
dates the object with internal.ExtractLTXTimestamp, which sniffs the age intro
instead of parsing ciphertext as LTX. Read: the three decrypt sites go through
internal.DecryptIfSealed, so a plaintext object no longer dies on "unexpected
intro: LTX1" -- which is what the bucket holds and what made restore a no-op.
Built from main (f864603), not a tag: v0.9.6 carries the write half and nothing
through v0.9.8 carries the read half. Digest-pinned, since the point of this
change is that a tag sitting here proved nothing about what was running.
It also arms the fail-closed default -- RequireEncryption is true unless a
config says age.allow-plaintext, and every write path refuses plaintext. That
is what stops recover-config-chat re-leaking cleartext into this prefix.
The comment I just wrote claimed the pinned image carried the ExtractLTXTimestamp
fix. It does not -- the pin is still 0.8.0 and hanzoai/replicate has no published
build with it (highest is 0.9.4, which predates both halves). Declared state
saying a thing is fixed when it is not is worse than saying nothing, so it now
says which pin is broken and what has to replace it.
Records the other half too: 0.8.0 decrypts unconditionally once identities are
set, so restore dies reading a PLAINTEXT object ("unexpected intro: LTX1").
Read and write were each assuming the other's encryption state. DecryptIfSealed
is in main and in no tag through v0.9.8, so the image has to come from main.
Measured, not inferred: `restore -if-replica-exists` at this pin exits 0 having
written no file. The gate swallows the decrypt error, so losing the volume today
boots chat on an empty database in silence. ChatReplicaSyncFailing watches the
sidecar's "monitor error" lines for exactly that -- logs, because the metrics in
db.go are never exposed (no addr, nothing scrapes). It will fire the moment it
is applied, which is correct: the condition is true right now.
chat-replicate-config was mounted by name and declared nowhere, so the backup
configuration for the only copy of chat's data was reviewed by no one and a
rebuild did not reproduce it. Adopted verbatim -- the rendered data is byte
identical to what is running (426 bytes), so this is a declaration, not a
change.
The comment above it now records why the sidecar never replicated: 0.8.0
encrypts in Replica.WriteLTXFile and then reads the resulting CIPHERTEXT with
ltx.PeekHeader to date the S3 object, so every sync died on "invalid LTX file"
and txid.replica never left 0. Also records the second writer -- recover-config-chat
targets this same bucket and prefix with no age block, which is where the
plaintext snapshots under chat/app/0009/ came from.
BuildKit jobs fetch source over the public host, git.hanzo.ai, which resolves to
this cluster's own LoadBalancer. DO's LB does not hairpin, so the connection did
not fail -- it hung, for 134678 ms, and the build died on a timeout that reads
like a network fault somewhere else entirely. The s3 lane hit the identical wall
at 133755 ms and artifact-publish-egress.yaml already documents it.
The in-cluster path is the hanzo-git Service, and ciliumnetworkpolicy.yaml denies
the build namespace every 10/8 address -- correctly, since a build runs a repo's
own Dockerfile. But that blanket denial has no exception for fetching the source
being built, so "use the native forge" was structurally impossible: not
misconfigured, unreachable by construction.
Measured from a pod in hanzo-build, before: the Service, its ClusterIP, and the
backend pod IP all time out at 10s. From a pod in hanzo, the same backend answers
303 in 0.016s. After: 303 in 0.023s via the Service, and `git ls-remote` against
hanzoai/cloud and hanzo/universe both return a SHA.
The port is 3000, the container's, not the Service's 80 -- Cilium translates the
Service before it evaluates egress, so naming 80 would look right and deny.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ghcr.io/hanzoai/chat:1.0.81
sha256:c34ae170714763cef10170c379367d536ef828895ee0dc13673342e6ffe4c4e7
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
Pinned-by: hand
Task asked whether chat could take replicas:2 + RollingUpdate to stop every
release being a ~3min 503. It cannot, and the values file never said why, so
the question keeps getting re-asked and the answer keeps getting re-derived.
app-db is not vestigial. chat.db on that claim holds 28 users, 155
conversations, 348 messages and 356 transactions, in WAL, with the txid
advancing under live traffic. Four independent things pin it to one writer --
an in-process synchronous better-sqlite3 handle with no flock or leader,
read-modify-write updates in JS that lose a write silently under two pods,
a deferred-BEGIN bulkWrite that hits the one busy error busy_timeout will not
retry, and a ReadWriteOnce claim over 23 nodes. Replicas:2 would trade a
visible 503 for silent Balance and Transaction corruption.
Records the exit too: the store abstracts collections behind CollectionSpec,
so a networked handle against Hanzo Base drops the claim and makes RollingUpdate
legal. That is the fix; raising replicas is not.
And records what the audit turned up next to it: the replicate sidecar that is
supposed to be the second copy of that database has failed every sync since it
started and reports txid.replica=0, `replicate ltx` lists nothing, so the
restore init container skips and chat would boot EMPTY if the volume were lost.
The two objects that are in the bucket carry a plaintext LTX1 magic while the
config declares age encryption. Its ConfigMap is also mounted by name and
declared nowhere in this repo. Needs its own fix -- noted here because it is
what makes the single copy load-bearing.
Comments only -- the parsed values are byte-identical to the previous commit.
ghcr.io/hanzoai/cloud:v1.801.460
sha256:bd2a8786e52566857bee804ddb9ab0dad67c97100eea36bbef8eaeddbc4da15e
Tag and digest are one registry answer, resolved together and written in one
edit, so the name a human reads and the bytes the kubelet pulls cannot be two
different releases. cd.hanzo.ai reconciles the change from here.
WHAT MOVES. /v1/catalog has served {"data":[],"total":0} since catalog and
index became separate processes, and hanzo.app's /community and /templates have
rendered that empty ever since. Nothing was wiped and no ingestion was missing:
the hourly reconcile assembled the corpus correctly and then handed it to
index.Reconcile, which serves out of a process-level global that is nil in the
catalog process. It was never written once. projects.LiveSites had the same gap
and reports nil when unmounted, so the live-site half vanished with no error at
all.
cloud a8b5bd5 + 74b7430 publish index_reconcile and sites_live on the internal
plane and give catalog the second leg on both — in-process first, then the
plane, the same shape the read already had. The swap still executes in the
process that owns the SQLite; only the request for it crosses. Measured against
live GitHub before release: 559 rows (product 358, third-party 110, community
69, template 22).
WHY BY HAND. The rollout car reads UNIVERSE_PIN_TOKEN from
/v1/kms/orgs/<org>/secrets/... and the broker has no such route — there is no
org in a KMS path, the store root comes from the validated claim. So nothing has
pinned since v1.801.454 and .455/.457 built and were never deployed. cloud
11a6ce2 fixes the path, but it only takes effect from the release after the one
carrying it, so this pin is manual.
Pinned-by: hand
2026-08-05 01:18:19 -07:00
149 changed files with 6295 additions and 608 deletions
description:The one Hanzo application chart — a container, a Service, and the things every service needs (ingress, PDB, HPA, probes, KMS-backed secrets). Replaces the App CRD.
"description":"Which kernel this pod gets. Empty = the node default (runc). 'gvisor' is the runsc user-space kernel and is the boundary for any workload that parses tenant bytes. A RuntimeClass merges its own scheduling.nodeSelector and tolerations into the pod at admission, so a values file setting this must NOT restate that node set - the two would disagree the day a dedicated pool exists, and the pod would be rejected."
},
"automountServiceAccountToken":{
"type":[
"boolean",
"null"
],
"title":"Mount a service-account token",
"description":"null (default) omits the field, so the API default applies and no existing render changes. false refuses the pod a kubeconfig it has no use for - mandatory for anything running untrusted code, where a mounted token sits inside the blast radius."
},
"nodeSelector":{
"type":"object",
"additionalProperties":{
@@ -492,6 +505,11 @@
"headless":{
"type":"boolean"
},
"internalTrafficPolicy":{
"type":"string",
"enum":["","Cluster","Local"],
"description":"Local routes a client to the endpoint on its OWN node — required for a per-node DaemonSet whose cache is node-local. Drops traffic when the local node has no ready endpoint; it does not fall back. Inert on a headless Service."
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
hanzo-ci
hanzo-dev
a