The automatic token reads only a package its repository was granted, so a
caller with no GH_PAT whose image is an org package it was never linked to
read nothing published, and with no declared version it had no version at
all. The KMS GitHub token is the rung the registry logins already take
between the two, and imgver now climbs the same ladder.
The KMS step exported HANZO_DEPLOY_TOKEN for any caller that declares a
site, in both roles, so the test role held it while it ran the repo's own
code, and a branch pushed for a pull request could print it. The token is
sealed once per org and publishes every site of that org. The site lane
reads it only in the publish role off a pull request, so the fetch is
there and nowhere else.
uv venv makes a venv without pip, so the clean install before upload
answered "No module named pip" for every wheel and the lane published
nothing (hanzoai/python-sdk v8.5.162, run 36508815355). The install and
import move into alone(), which bin/alone_test.sh lifts out of the
workflow and runs against wheels it builds.
Ubuntu's ldconfig cache lists libclang-18.so.18 and libclang-18.so,
never a bare libclang.so, so the probe missed the copy the runner image
carries and fell through to sudo apt-get, which the runner refuses
("effective uid is not 0"). hanzoai/decision run 36307074308 died there.
clang-sys searches libclang-*.so itself, so no LIBCLANG_PATH is needed.
Claude-Session: https://claude.ai/code/session_018Gt5TvrE2rxXuZikWSauv3
The suite printed only a summary line, and the reusable's test tally,
which counts per-result lines, read that as a runner that ran nothing
and failed main.
Claude-Session: https://claude.ai/code/session_01HuMDfgFssPUptYUzgZ6Cp8
The fake clock rode on bash's ticking SECONDS, so a starved runner
spent 14 real seconds of the 30-minute budget and the suite failed
at 1786. Unset, SECONDS is a plain variable only the fake sleep moves.
Claude-Session: https://claude.ai/code/session_01HuMDfgFssPUptYUzgZ6Cp8
A fixed 30-second poll for 20 minutes was measured short: hanzoai/cd
asked 41 times on 2026-09-23 and got 429 every time, then failed a
publish whose tests had passed. Every waiting repo of an org also hit
the door in the same second a slot freed.
The wait now doubles from 15s to a 120s ceiling with jitter, takes a
numeric Retry-After as a floor, and stops at 30 minutes, since a build
holds its slot for up to ~39. bin/enqueue_test.sh lifts the function
out of build.yml and runs it against a faked wire and clock.
Claude-Session: https://claude.ai/code/session_01HuMDfgFssPUptYUzgZ6Cp8
curl -L keeps every hop's headers, and the check took the last
content-length it saw. A root that redirected once before a
chunked page was measured against the redirect's 17 bytes, which
failed a healthy publish of dev-hanzo-ai three times over.
Claude-Session: https://claude.ai/code/session_01TUxvfuta1Z44KVNYuAz71F
ghcr.io/hanzoai/git-runner is linked to no repository, so this repo's
workflow token cannot list its versions and bin/imgver would publish
9.0.0 over and over. The source label links it on the next push.
Claude-Session: https://claude.ai/code/session_01HuMDfgFssPUptYUzgZ6Cp8
hanzo-inc/cloud's gates name their caches with ${GOCACHE:-...}, so they
built into a directory nothing restored and nothing saved. Restore now
exports the two directories it used, and every later step reads them.
Claude-Session: https://claude.ai/code/session_01HuMDfgFssPUptYUzgZ6Cp8
The publish job held its runner up to 40 minutes polling the registry so
it could write the version tags itself: 289 runner-minutes in 48 hours
spent waiting. The door now writes those tags in the same push and says
so in its answer, so the job ends on the 202 and names the build to
watch. A tag the door does not promise is still written here, and
`wait: 'true'` holds the job for a caller that uses the image next.
The first ref is taken by expansion: `echo | head -1` under pipefail
failed hanzoai/ci's own publish with a broken pipe.
Claude-Session: https://claude.ai/code/session_01HuMDfgFssPUptYUzgZ6Cp8
A runner lives one job, so no Go cache was ever restored: 142 Go-gate
minutes in 48 hours were 31 downloading modules and 69 compiling, and a
warm pass compiled commerce's vet in 15s against 198s cold.
The cache lives in the org's own bucket behind cloud's /v1/s3, reached
with the IAM token the KMS step already holds, so the server keeps one
org out of another's and CI holds no bucket key. Go trusts a restored
cache without re-hashing it, so only the default branch and tags write.
setup-go's own cache is off: it is GitHub's store, and the forge has none.
Claude-Session: https://claude.ai/code/session_01HuMDfgFssPUptYUzgZ6Cp8
Every job downloaded Go and Node through setup-go and setup-node, 110
runner-minutes in 48 hours, because nothing was in the tool cache they
read. Go 1.27.1 and Node 22 now sit there in their layout, beside
corepack's pinned pnpm, helm, crane and zstd.
The image is an images: entry built at the build door on every push.
Its forge lane stopped when Actions went off on the mirror, and a
runner has no docker daemon to run it anyway.
Claude-Session: https://claude.ai/code/session_01HuMDfgFssPUptYUzgZ6Cp8
`secrets: inherit` passes nothing to a reusable workflow in another
org, and a caller may only name a secret the reusable declares. Every
luxfi, zooai, lux-apps, hanzo-inc and hanzo-apps caller therefore
reached the KMS step with KMS_CLIENT_ID empty and delegate refused on
an empty HANZO_API_TOKEN. All five are optional, so a hanzoai caller's
`inherit` is unchanged.
Claude-Session: https://claude.ai/code/session_01HuMDfgFssPUptYUzgZ6Cp8
Hanzo CI/CD / cicd (push) Blocked by required conditions
Hanzo CI/CD / test (linux-amd64) (push) Skipped
Hanzo CI/CD / test (linux-arm64) (push) Skipped
Hanzo CI/CD / test-amd64 (push) Waiting to run
Hanzo CI/CD / test-advisory (push) Skipped
CI/CD / gate (push) Waiting to run
Every caller ran its tests on both architectures, which doubled runner
demand and queued the whole estate behind the smaller arm64 pool for
hours, while the arm64 leg caught almost nothing amd64 did not. A repo
with architecture-specific code names both in test-runners.
Claude-Session: https://claude.ai/code/session_01HuMDfgFssPUptYUzgZ6Cp8
Hanzo CI/CD / test (linux-arm64) (push) Canceled after 0s
Hanzo CI/CD / test (linux-amd64) (push) Canceled after 0s
Hanzo Git evaluates a job's if with no job in scope, and cancelled() reads the
job's status, so a condition calling it panicked the forge's job resolver the
moment both of cicd's needs were decided. always() alone lets the condition be
asked; naming a success is what refuses when both gating jobs were skipped.
Co-authored-by: Hanzo Dev <[email protected]>
test-runners selects the gating jobs: both architectures (the default, the
`test` matrix as before) or linux-amd64 alone (`test-amd64`). advisory-runners
runs the same test on one more architecture in `test-advisory`, which nothing
needs, so a slow or red leg there reports without holding a release. The choice
is a job per value, selected by `if`, because Hanzo Git does not evaluate inputs
in strategy.matrix. cicd needs both gating jobs and publishes only when the one
that ran succeeded, so a caller that passes neither input behaves exactly as
before.
Co-authored-by: Hanzo Dev <[email protected]>
Hanzo CI/CD / cicd (push) Blocked by required conditions
Hanzo CI/CD / test (linux-amd64) (push) Waiting to run
Hanzo CI/CD / test (linux-arm64) (push) Waiting to run
CI/CD / gate (push) Waiting to run
hanzoai/openapi is public, and git.hanzo.ai refuses the org's
machine identity because no account is linked to it, so the lane
failed on every client repo. It now clones from github.com with
the runner's git config set aside, and needs no credential.
Claude-Session: https://claude.ai/code/session_01TUxvfuta1Z44KVNYuAz71F
Hanzo CI/CD / test (linux-amd64) (push) Successful in 3m47s
Hanzo CI/CD / test (linux-arm64) (push) Successful in 8m51s
Hanzo CI/CD / cicd (push) Successful in 1m37s
CI/CD / gate (push) Successful in 12m40s
A static export inlines its ingest key at build just as an image
bakes one, but only images[] were read, so a site had no way to
take a pk- key from KMS and fell back to committing a literal.
Claude-Session: https://claude.ai/code/session_01TUxvfuta1Z44KVNYuAz71F
Hanzo CI/CD / test (linux-amd64) (push) Successful in 2m43s
Hanzo CI/CD / test (linux-arm64) (push) Successful in 9m27s
Hanzo CI/CD / cicd (push) Failing after 21m27s
CI/CD / gate (push) Failing after 1h2m53s
Each org's CI holds its own <org>-ci identity and the build door confines it to
that org's registry namespaces, so a 403 prints the token's org beside its
subject. The README says which identity each GitHub org holds and which
namespaces it reaches.
Co-authored-by: Hanzo Dev <[email protected]>
Hanzo CI/CD / test (linux-amd64) (push) Successful in 3m28s
Hanzo CI/CD / test (linux-arm64) (push) Successful in 8m17s
CI/CD / gate (push) Successful in 42m34s
Hanzo CI/CD / cicd (push) Successful in 1m41s
When a release handed its number down (spec-version), the cut tagged that
number but skipped rewriting the file hanzo.yml names in client.version: the
rewrite sat under the derived-patch branch alone. A fanout from cloud v8.5.189
tagged rust-sdk v8.5.189 over a Cargo.toml still at 8.5.156, so its release
refused the tag, and the npm, PyPI and Maven publishers saw an unchanged
version and published nothing.
The choice and the rewrite are now one function, `cut`, that writes whichever
number it picked. A release's number must be x.y.z before it reaches the tag
or sed: `8.5.189/;e cmd;#` ran cmd through GNU sed's e command.
bin/cut_test.sh lifts `cut` out of build.yml and runs it; it is a gate in
hanzo.yml.
Co-authored-by: Hanzo Dev <[email protected]>
Hanzo CI/CD / test (linux-amd64) (push) Successful in 4m48s
Hanzo CI/CD / test (linux-arm64) (push) Successful in 6m51s
Hanzo CI/CD / cicd (push) Successful in 3m30s
CI/CD / gate (push) Successful in 33m0s
The door scopes a build to a context directory and reads the Dockerfile at
its root-relative path, so an image whose context is a subdirectory builds
on this lane instead of being refused.
Co-authored-by: Hanzo Dev <[email protected]>
Hanzo CI/CD / test (linux-arm64) (push) Successful in 6m3s
Hanzo CI/CD / test (linux-amd64) (push) Successful in 2m14s
Hanzo CI/CD / cicd (push) Successful in 1m34s
CI/CD / gate (push) Successful in 34m48s
cloud restarts on every deploy and is back within a minute, and a publish that
reached the door during one failed on Cloudflare's 502 — the canary's did, at
09:17:40, inside a restart from 09:17:19 to 09:18:14, after its tests had
passed on both architectures. No answer and 502/503/504 now wait on the same
budget as the per-org ceiling.
Co-authored-by: Hanzo Dev <[email protected]>
Hanzo CI/CD / test (linux-amd64) (push) Successful in 9m18s
Hanzo CI/CD / test (linux-arm64) (push) Successful in 9m14s
Hanzo CI/CD / cicd (push) Failing after 5m39s
CI/CD / gate (push) Failing after 19m36s
Under pipefail, a pipe into `grep -q` reports the writer's status too, and
grep quits at its first match, so the answer depends on whether the writer
finished before the pipe closed. The imgtags suite failed that way on an amd64
runner and passed on the arm64 one at the same commit. A here-string has no
second process to lose.
Co-authored-by: Hanzo Dev <[email protected]>
Hanzo CI/CD / test (linux-amd64) (push) Failing after 4m53s
Hanzo CI/CD / test (linux-arm64) (push) Successful in 9m14s
Hanzo CI/CD / cicd (push) Skipped
CI/CD / gate (push) Failing after 9m14s
The door holds one build per org, and a 429 is its queue. The delegate lane
refused on it, so two repos publishing together failed the second; the canary's
publish job died that way with both test legs green. It waits now, on the same
20-minute budget cloud's own image job gives the same answer.
Co-authored-by: Hanzo Dev <[email protected]>
Hanzo CI/CD / test (linux-amd64) (push) Successful in 4m49s
Hanzo CI/CD / test (linux-arm64) (push) Successful in 8m33s
Hanzo CI/CD / cicd (push) Failing after 2m14s
CI/CD / gate (push) Failing after 22m46s
The gate list refuses a *_test.sh that reports no result, and this suite ended
with `bin/kms: ok`, which carries none. Every other suite here prints
`<name>: N passed, M failed`; this one does now. It was hidden behind the brand
gate, which failed first on every run.
Co-authored-by: Hanzo Dev <[email protected]>
Hanzo CI/CD / test (linux-amd64) (push) Failing after 1m44s
Hanzo CI/CD / test (linux-arm64) (push) Failing after 7m30s
Hanzo CI/CD / cicd (push) Skipped
CI/CD / gate (push) Failing after 19m17s
The verdict must carry no `if:`, so no input can silence it; the role guard I
gave it broke exactly that, and the self-check caught it on the first run. It
now runs everywhere: in the test job it reads the receipt as before, and in the
publish job, which holds no receipt, it refuses unless the test job answered
success. The self-check asserts the two-job shape — the gate and its opt-out
guarded by the role, the publish job behind `needs: test`.
Co-authored-by: Hanzo Dev <[email protected]>
cicd.yml / test (${{ fromJson(inputs.runner) }}) (push) Canceled after 0s
Hanzo CI/CD / test (linux-arm64) (push) Failing after 2m27s
Hanzo CI/CD / test (linux-amd64) (push) Failing after 3m53s
Hanzo Git expands a reusable workflow without evaluating `inputs` inside
`strategy.matrix`, so on the forge the test job became one leg labelled with the
unevaluated expression and waited for a runner that cannot exist. The legs are
now linux-amd64 and linux-arm64, written out — every image ships for both, so
both are always gated — and `runner` names the publishing runner again, as it
did before the split, so no caller changes.
Co-authored-by: Hanzo Dev <[email protected]>
cicd.yml / test (${{ fromJson(inputs.runner) }}) (push) Canceled after 0s
Hanzo CI/CD / cicd (push) Canceled after 0s
CI/CD / gate (push) Canceled after 0s
The bytes here already were brand's current sheet; the hash named the 1.4.9
Geist one, because brand changed the file without a version. Brand is 1.4.10
now and this pins it.
Co-authored-by: Hanzo Dev <[email protected]>
cicd.yml / test (${{ fromJson(inputs.runner) }}) (push) Canceled after 0s
Hanzo CI/CD / cicd (push) Canceled after 0s
CI/CD / gate (push) Canceled after 0s
Every image ships for amd64 and arm64, and until now its tests ran on amd64
alone, one runner at a time on the AWS cluster's single amd64 sandbox node while
the arm64 one sat idle. `test` now runs one leg per runner in `runner:` at once,
linux-amd64 and linux-arm64 by default, and `cicd` needs it: a red leg on either
architecture leaves no image, tag or release. Publishing is a job of its own
because a matrix leg cannot wait for its sibling. Both jobs share one step list
through an anchor, and HANZO_CI_ROLE picks which steps run in each.
Co-authored-by: Hanzo Dev <[email protected]>
The fallbacks downloaded the x86_64 builds by name. crane is not in the runner
image, so the delegate lane fetches it every run to alias version tags onto the
door's digest — and on an arm64 runner that is an exec format error at the one
step that makes a build deployable.
Co-authored-by: Hanzo Dev <[email protected]>
A runner runs whatever a repository says, so the AWS cluster walls it off: egress
is DNS and 80/443 to public addresses, every RFC1918 range excepted. cloud's
Service address is inside that wall, so a delegated build was refused in
milliseconds before the door saw it — every image on that cluster, cloud's own
included, has failed there since the wall went up. api.hanzo.ai is the name a
runner is meant to use, and from a runner it answers.
Co-authored-by: Hanzo Dev <[email protected]>
This lane was written for our own pools — it names linux-amd64 and linux-arm64 —
and filed under .github/workflows, where nothing executes it. So it never ran:
the arm64 half was published by hand and the amd64 half of the current runner was
never built at all, which is what left the fleet pinned to an image that predates
the variable rename.
The entrypoint's refusal is asserted by the name it actually prints. Asserting
the old one meant the smoke test would have failed on the very image it was
meant to prove, and called it "failed for the wrong reason".
Co-authored-by: Hanzo Dev <[email protected]>
Version 7 was published before the rename landed, so the image under that tag
still asks for the old variable and exits when it is unset. A running fleet
cannot tell those two apart by the tag, so the renamed runner gets a version of
its own.
Co-authored-by: Hanzo Dev <[email protected]>
Modules declare go 1.27.1 and the build images follow. go fix applied the
modernizers; omitzero is not applied, since it changes what the JSON says.
Co-authored-by: Hanzo Dev <[email protected]>
The read-back demands HTTP 200 from the live URL. A client preview cannot give
one: hanzo-apps hosts are guarded, so they answer an API caller 401 and a browser
a redirect to hanzo.id. The same workflow's "every host is gated" step REFUSES to
publish any host that answers otherwise, so the two steps asked one deploy to be
two contradictory things.
It cost 27 of 29 client sites. The first slug of each repo uploaded and flipped
live, the read-back called it unreadable, and `set -e` took the remaining sites
down with it -- leaving hosts that resolve and gate with nothing behind them.
A browser-shaped probe that does not follow tells a gate from a failure: a gate
redirects to hanzo.id naming that host as its own callback, which proves the host
is routed and guarded. Its bytes still are not read back, so the run says so
rather than counting it verified. Every other failure path is unchanged: an empty
body, a short read and a plain non-200 all still fail.
Co-authored-by: Hanzo Dev <[email protected]>
Hanzo Git is the product; act_runner was the upstream binary's name and
GITEA_ was its env prefix. The binary installs as git-runner and the two
variables cloud sets are GIT_RUNNER_URL and GIT_RUNNER_TOKEN.
Co-authored-by: Hanzo Dev <[email protected]>
zeekay
Hanzo Dev