Hanzo CI/CD / test (linux-arm64) (push) Canceled after 0s
Hanzo CI/CD / test (linux-amd64) (push) Canceled after 0s
Hanzo Git evaluates a job's if with no job in scope, and cancelled() reads the
job's status, so a condition calling it panicked the forge's job resolver the
moment both of cicd's needs were decided. always() alone lets the condition be
asked; naming a success is what refuses when both gating jobs were skipped.
Co-authored-by: Hanzo Dev <[email protected]>
test-runners selects the gating jobs: both architectures (the default, the
`test` matrix as before) or linux-amd64 alone (`test-amd64`). advisory-runners
runs the same test on one more architecture in `test-advisory`, which nothing
needs, so a slow or red leg there reports without holding a release. The choice
is a job per value, selected by `if`, because Hanzo Git does not evaluate inputs
in strategy.matrix. cicd needs both gating jobs and publishes only when the one
that ran succeeded, so a caller that passes neither input behaves exactly as
before.
Co-authored-by: Hanzo Dev <[email protected]>
Hanzo CI/CD / cicd (push) Blocked by required conditions
Hanzo CI/CD / test (linux-amd64) (push) Waiting to run
Hanzo CI/CD / test (linux-arm64) (push) Waiting to run
CI/CD / gate (push) Waiting to run
hanzoai/openapi is public, and git.hanzo.ai refuses the org's
machine identity because no account is linked to it, so the lane
failed on every client repo. It now clones from github.com with
the runner's git config set aside, and needs no credential.
Claude-Session: https://claude.ai/code/session_01TUxvfuta1Z44KVNYuAz71F
Hanzo CI/CD / test (linux-amd64) (push) Successful in 3m47s
Hanzo CI/CD / test (linux-arm64) (push) Successful in 8m51s
Hanzo CI/CD / cicd (push) Successful in 1m37s
CI/CD / gate (push) Successful in 12m40s
A static export inlines its ingest key at build just as an image
bakes one, but only images[] were read, so a site had no way to
take a pk- key from KMS and fell back to committing a literal.
Claude-Session: https://claude.ai/code/session_01TUxvfuta1Z44KVNYuAz71F
Hanzo CI/CD / test (linux-amd64) (push) Successful in 2m43s
Hanzo CI/CD / test (linux-arm64) (push) Successful in 9m27s
Hanzo CI/CD / cicd (push) Failing after 21m27s
CI/CD / gate (push) Failing after 1h2m53s
Each org's CI holds its own <org>-ci identity and the build door confines it to
that org's registry namespaces, so a 403 prints the token's org beside its
subject. The README says which identity each GitHub org holds and which
namespaces it reaches.
Co-authored-by: Hanzo Dev <[email protected]>
Hanzo CI/CD / test (linux-amd64) (push) Successful in 3m28s
Hanzo CI/CD / test (linux-arm64) (push) Successful in 8m17s
CI/CD / gate (push) Successful in 42m34s
Hanzo CI/CD / cicd (push) Successful in 1m41s
When a release handed its number down (spec-version), the cut tagged that
number but skipped rewriting the file hanzo.yml names in client.version: the
rewrite sat under the derived-patch branch alone. A fanout from cloud v8.5.189
tagged rust-sdk v8.5.189 over a Cargo.toml still at 8.5.156, so its release
refused the tag, and the npm, PyPI and Maven publishers saw an unchanged
version and published nothing.
The choice and the rewrite are now one function, `cut`, that writes whichever
number it picked. A release's number must be x.y.z before it reaches the tag
or sed: `8.5.189/;e cmd;#` ran cmd through GNU sed's e command.
bin/cut_test.sh lifts `cut` out of build.yml and runs it; it is a gate in
hanzo.yml.
Co-authored-by: Hanzo Dev <[email protected]>
Hanzo CI/CD / test (linux-amd64) (push) Successful in 4m48s
Hanzo CI/CD / test (linux-arm64) (push) Successful in 6m51s
Hanzo CI/CD / cicd (push) Successful in 3m30s
CI/CD / gate (push) Successful in 33m0s
The door scopes a build to a context directory and reads the Dockerfile at
its root-relative path, so an image whose context is a subdirectory builds
on this lane instead of being refused.
Co-authored-by: Hanzo Dev <[email protected]>
Hanzo CI/CD / test (linux-arm64) (push) Successful in 6m3s
Hanzo CI/CD / test (linux-amd64) (push) Successful in 2m14s
Hanzo CI/CD / cicd (push) Successful in 1m34s
CI/CD / gate (push) Successful in 34m48s
cloud restarts on every deploy and is back within a minute, and a publish that
reached the door during one failed on Cloudflare's 502 — the canary's did, at
09:17:40, inside a restart from 09:17:19 to 09:18:14, after its tests had
passed on both architectures. No answer and 502/503/504 now wait on the same
budget as the per-org ceiling.
Co-authored-by: Hanzo Dev <[email protected]>
Hanzo CI/CD / test (linux-amd64) (push) Successful in 9m18s
Hanzo CI/CD / test (linux-arm64) (push) Successful in 9m14s
Hanzo CI/CD / cicd (push) Failing after 5m39s
CI/CD / gate (push) Failing after 19m36s
Under pipefail, a pipe into `grep -q` reports the writer's status too, and
grep quits at its first match, so the answer depends on whether the writer
finished before the pipe closed. The imgtags suite failed that way on an amd64
runner and passed on the arm64 one at the same commit. A here-string has no
second process to lose.
Co-authored-by: Hanzo Dev <[email protected]>
Hanzo CI/CD / test (linux-amd64) (push) Failing after 4m53s
Hanzo CI/CD / test (linux-arm64) (push) Successful in 9m14s
Hanzo CI/CD / cicd (push) Skipped
CI/CD / gate (push) Failing after 9m14s
The door holds one build per org, and a 429 is its queue. The delegate lane
refused on it, so two repos publishing together failed the second; the canary's
publish job died that way with both test legs green. It waits now, on the same
20-minute budget cloud's own image job gives the same answer.
Co-authored-by: Hanzo Dev <[email protected]>
Hanzo CI/CD / test (linux-amd64) (push) Successful in 4m49s
Hanzo CI/CD / test (linux-arm64) (push) Successful in 8m33s
Hanzo CI/CD / cicd (push) Failing after 2m14s
CI/CD / gate (push) Failing after 22m46s
The gate list refuses a *_test.sh that reports no result, and this suite ended
with `bin/kms: ok`, which carries none. Every other suite here prints
`<name>: N passed, M failed`; this one does now. It was hidden behind the brand
gate, which failed first on every run.
Co-authored-by: Hanzo Dev <[email protected]>
Hanzo CI/CD / test (linux-amd64) (push) Failing after 1m44s
Hanzo CI/CD / test (linux-arm64) (push) Failing after 7m30s
Hanzo CI/CD / cicd (push) Skipped
CI/CD / gate (push) Failing after 19m17s
The verdict must carry no `if:`, so no input can silence it; the role guard I
gave it broke exactly that, and the self-check caught it on the first run. It
now runs everywhere: in the test job it reads the receipt as before, and in the
publish job, which holds no receipt, it refuses unless the test job answered
success. The self-check asserts the two-job shape — the gate and its opt-out
guarded by the role, the publish job behind `needs: test`.
Co-authored-by: Hanzo Dev <[email protected]>
cicd.yml / test (${{ fromJson(inputs.runner) }}) (push) Canceled after 0s
Hanzo CI/CD / test (linux-arm64) (push) Failing after 2m27s
Hanzo CI/CD / test (linux-amd64) (push) Failing after 3m53s
Hanzo Git expands a reusable workflow without evaluating `inputs` inside
`strategy.matrix`, so on the forge the test job became one leg labelled with the
unevaluated expression and waited for a runner that cannot exist. The legs are
now linux-amd64 and linux-arm64, written out — every image ships for both, so
both are always gated — and `runner` names the publishing runner again, as it
did before the split, so no caller changes.
Co-authored-by: Hanzo Dev <[email protected]>
cicd.yml / test (${{ fromJson(inputs.runner) }}) (push) Canceled after 0s
Hanzo CI/CD / cicd (push) Canceled after 0s
CI/CD / gate (push) Canceled after 0s
The bytes here already were brand's current sheet; the hash named the 1.4.9
Geist one, because brand changed the file without a version. Brand is 1.4.10
now and this pins it.
Co-authored-by: Hanzo Dev <[email protected]>
cicd.yml / test (${{ fromJson(inputs.runner) }}) (push) Canceled after 0s
Hanzo CI/CD / cicd (push) Canceled after 0s
CI/CD / gate (push) Canceled after 0s
Every image ships for amd64 and arm64, and until now its tests ran on amd64
alone, one runner at a time on the AWS cluster's single amd64 sandbox node while
the arm64 one sat idle. `test` now runs one leg per runner in `runner:` at once,
linux-amd64 and linux-arm64 by default, and `cicd` needs it: a red leg on either
architecture leaves no image, tag or release. Publishing is a job of its own
because a matrix leg cannot wait for its sibling. Both jobs share one step list
through an anchor, and HANZO_CI_ROLE picks which steps run in each.
Co-authored-by: Hanzo Dev <[email protected]>
The fallbacks downloaded the x86_64 builds by name. crane is not in the runner
image, so the delegate lane fetches it every run to alias version tags onto the
door's digest — and on an arm64 runner that is an exec format error at the one
step that makes a build deployable.
Co-authored-by: Hanzo Dev <[email protected]>
A runner runs whatever a repository says, so the AWS cluster walls it off: egress
is DNS and 80/443 to public addresses, every RFC1918 range excepted. cloud's
Service address is inside that wall, so a delegated build was refused in
milliseconds before the door saw it — every image on that cluster, cloud's own
included, has failed there since the wall went up. api.hanzo.ai is the name a
runner is meant to use, and from a runner it answers.
Co-authored-by: Hanzo Dev <[email protected]>
This lane was written for our own pools — it names linux-amd64 and linux-arm64 —
and filed under .github/workflows, where nothing executes it. So it never ran:
the arm64 half was published by hand and the amd64 half of the current runner was
never built at all, which is what left the fleet pinned to an image that predates
the variable rename.
The entrypoint's refusal is asserted by the name it actually prints. Asserting
the old one meant the smoke test would have failed on the very image it was
meant to prove, and called it "failed for the wrong reason".
Co-authored-by: Hanzo Dev <[email protected]>
Version 7 was published before the rename landed, so the image under that tag
still asks for the old variable and exits when it is unset. A running fleet
cannot tell those two apart by the tag, so the renamed runner gets a version of
its own.
Co-authored-by: Hanzo Dev <[email protected]>
Modules declare go 1.27.1 and the build images follow. go fix applied the
modernizers; omitzero is not applied, since it changes what the JSON says.
Co-authored-by: Hanzo Dev <[email protected]>
The read-back demands HTTP 200 from the live URL. A client preview cannot give
one: hanzo-apps hosts are guarded, so they answer an API caller 401 and a browser
a redirect to hanzo.id. The same workflow's "every host is gated" step REFUSES to
publish any host that answers otherwise, so the two steps asked one deploy to be
two contradictory things.
It cost 27 of 29 client sites. The first slug of each repo uploaded and flipped
live, the read-back called it unreadable, and `set -e` took the remaining sites
down with it -- leaving hosts that resolve and gate with nothing behind them.
A browser-shaped probe that does not follow tells a gate from a failure: a gate
redirects to hanzo.id naming that host as its own callback, which proves the host
is routed and guarded. Its bytes still are not read back, so the run says so
rather than counting it verified. Every other failure path is unchanged: an empty
body, a short read and a plain non-200 all still fail.
Co-authored-by: Hanzo Dev <[email protected]>
Hanzo Git is the product; act_runner was the upstream binary's name and
GITEA_ was its env prefix. The binary installs as git-runner and the two
variables cloud sets are GIT_RUNNER_URL and GIT_RUNNER_TOKEN.
Co-authored-by: Hanzo Dev <[email protected]>
The default was linux/amd64, "because that is what every image in the estate
serves", and that sentence was the problem: this fleet is two nodes and two
architectures, so an image with one half is a service that cannot move. Eleven
of the twelve images on the critical path had only amd64, and when the amd64
node saturated its disk on the 15th neither the edge nor the npm registry every
JS build resolves through could be scheduled on the machine that was up.
Both lanes now read the same default — both architectures — so a repo that says
nothing inherits both on its next release. The door builds each natively; the
buildx lane still refuses to emulate, and its refusal now names the third way
out: declare one platform on purpose.
A deliberate single architecture is allowed and warned about, because the cost
lands somewhere else — the workload can be scheduled on one half of the fleet
and its pin needs a nodeSelector to say so.
Co-authored-by: Hanzo Dev <[email protected]>
sudo is setuid and the runner's filesystem is nosuid, so a job cannot
install anything: every repo with a client lane died at `sudo apt-get`
before it built. The image ships the JDK, libssl and libclang the reusable
build looks for, and each of those steps short-circuits on its own check.
Co-authored-by: Hanzo Dev <[email protected]>
bin/site typed every .md as application/octet-stream, and the Sites
plane serves the type stored at upload when its own table has none.
So the markdown twins on hanzoskills.com and hanzo.ai reached agents
with no usable Content-Type.
Co-authored-by: Hanzo Dev <[email protected]>
gVisor builds the VDSO for both architectures in one bazel invocation, so a
job building it on amd64 needs aarch64-linux-gnu-gcc and on arm64 needs
x86_64-linux-gnu-gcc. Runner jobs execute inside the visor sandbox, which
does not honour setuid, so a job cannot install either for itself.
The smoke step runs that compiler on each arch rather than testing for it.
Co-authored-by: Hanzo Dev <[email protected]>
hanzo-agent 0.1.0 went to PyPI importing numpy and httpx and declaring
neither. A monorepo supplies every sibling, so its own tests all passed and
`pip install hanzo-agent` failed at import for everybody else. An outside
adopter hit it, gave up on three of our packages, and wrote their own clients.
It is the most expensive kind of bug we publish, because nothing inside the
tree can see it.
So between build and upload, each wheel is installed ALONE into a clean
environment and imported. A clean environment is the only thing that can tell a
declared dependency from one the workspace happened to have, and
--no-cache-dir stops a wheel cached from the build above standing in for a
missing declaration. Here rather than in each repo's own lane: one check, every
package that publishes through this pipeline.
The import name comes from the wheel, not from the source tree. Guessing from
the tree picked `examples` for hanzo-agent — that directory has an __init__.py
and sorts first — and a check that imports the wrong module can pass while the
package is broken, which is worse than not checking. The installed
distribution's own top-level list is what a consumer gets, so it is what gets
imported.
Verified against three published wheels: hanzo-agent resolves to `agents`,
hanzo-mcp to `hanzo_mcp`, hanzoai to `hanzoai`, and all three import.
Co-authored-by: Hanzo Dev <[email protected]>
build.yml cannot build the image it runs inside without depending on the
artifact it is producing; runner.yml already cuts it per arch.
Co-authored-by: Hanzo Dev <[email protected]>
oci.hanzo.ai/hanzoai/git-runner:5 was cut by hand and amd64 alone, and
cloud hands it to every runner Job. An arm64 job therefore lands on spark
and the kubelet answers "failed to load /usr/bin/tini: exec format
error", so that lane has never once started — which is why the builder
could not build arm64 either.
One job per architecture on its own native pool, because --load refuses a
manifest list and testing each half is the point: a wrong-arch binary
passes every test -x and fails the instant it is executed, so the smoke
runs tini, both agents and the entrypoint rather than inspecting them.
Its own file, not build.yml: the reusable cannot build the image it runs
inside without depending on what it is producing.
Co-authored-by: Hanzo Dev <[email protected]>
CI/CD / gate (push) Blocked by required conditions
Hanzo CI/CD / cicd (push) Waiting to run
github.job_workflow_sha is the documented name for this and arrives empty
here -- hanzoai/base run 34563223211 printed job_workflow_sha=<unset> -- so
the previous fix fell straight back to the caller's ref and a release still
tried to clone hanzoai/[email protected].
The caller is checked out and GITHUB_WORKFLOW_REF names which of its files
started the run, so the `uses:` line is on disk and says exactly which ref
the author asked for.
Co-authored-by: Hanzo Dev <[email protected]>
`mode` chooses WHERE an image is built, so a caller with no image to
build reaches whichever lane is the default and has to be let through
it. buildx has said so since it was written; delegate never had to,
because nothing defaulted to it.
Without the sentence a test-only caller -- 75 of the 117 manifests in
the estate -- dies at a step it does not use, and every gate, binary
and distribution below it is skipped because of it.
Co-authored-by: Hanzo Dev <[email protected]>
`docker pull ghcr.io/hanzoai/base` asks for :latest, which no lane writes, so
it still served 1.5.22 -- sixty-nine patches behind, amd64 alone. A branch
build still writes none; the leading v already tells the two apart.
Co-authored-by: Hanzo Dev <[email protected]>
A runner builds natively for itself and refuses the other platform rather than
emulating it, so `mode: buildx` — the default — published one architecture and
said so nowhere. Against two nodes that is an image the arm64 machine either
cannot pull, or, where the image is a bare manifest rather than an index, runs
under qemu until it faults. 144 of the 185 images this estate runs carry amd64
alone and every one of them took this default. The door schedules a Job per
platform on a node of that platform and joins them under one index, so it is
the lane that can do this, and it is now the one callers get.
Three things had to be true first. `client:` and `binaries:` were conditioned on
the mode, which has nothing to do with where an image builds — they publish
nothing under the new default until that is untangled, and 13 repos declare one
or the other. The delegate lane sent no `args:`, so seven repos that declare
them would have built from the Dockerfile's ARG defaults: green, correctly
tagged, and different from what buildx makes of the same commit. And the two
lanes have to expand the same five spellings, or one commit means two images.
Five repos cannot use the door and say so at the call site: four declare
build_secrets, which the door has no KMS to resolve, and papers builds from a
subdirectory context the door cannot express.
Co-authored-by: Hanzo Dev <[email protected]>
Nothing built git-runner: it was cut by hand, amd64 alone, and cloud
hands it to every runner Job. An arm64 job therefore lands on spark and
the kubelet answers "failed to load /usr/bin/tini: exec format error",
so that lane has never once started.
The listener leak is the other half. A just-in-time configuration is
minted for one already-queued job, but run.sh has no timeout for a job
that never arrives: two sat in "Listening for Jobs" for 151 and 73
minutes, holding two of the cluster's six runner slots while the queue
behind them was refused.
Co-authored-by: Hanzo Dev <[email protected]>
hanzoai/python-sdk carried the estate's only PyPI publisher, as a
per-repo workflow, so every other Python repo had none. hanzoai/agent
built a wheel, installed it, imported it and threw it away, which is why
0.1.0 has sat on the index unimportable for months with the fix merged.
`pypi:` is a list of directory globs; the tag decides which of them go.
A glob rather than a list of names because the retired workflow's
hardcoded list could not notice a package added beside it. The
distribution name is read from each pyproject rather than from the
directory it sits in, so a rename cannot keep publishing under the old
name.
Co-authored-by: Hanzo Dev <[email protected]>
github.com resolves a reusable out of .github/workflows and the forge
reads it from there as well, so one file already serves both planes and
the copy under .hanzo/workflows is a second address to keep in step. It
had drifted: the concurrency group, the per-runner docker config and the
whole tools resolution were only in one of them. `one-definition` has
refused it all along, which is why this repo's own gate is red.
Co-authored-by: Hanzo Dev <[email protected]>
cloud registers the build endpoint at /v1/build and nowhere else
(apps/platform/platform.go). The nested address this defaulted to
answers 404, so every delegated build failed on a reply naming no
address, which reads like an outage rather than a wrong door.
Co-authored-by: Hanzo Dev <[email protected]>
GITHUB_WORKFLOW_REF names the CALLER's ref, never the reusable's. A caller
pushing v1.5.94 made the step clone hanzoai/[email protected], which names nothing
here, so every release in the estate died at step one having cut its Go module
and built no image. A caller pushing main cloned ci@main, so a pin of @v2 ran
v2's pipeline against main's bin/ -- the skew this step exists to prevent.
github.job_workflow_sha is the one name GitHub gives a job for the reusable it
is running. The caller's ref stays as a second candidate, so a host that knows
only the old answer still answers.
Co-authored-by: Hanzo Dev <[email protected]>
Release and bump commits were authored by the automation address, so the work never reached the owner's contribution graph.
Co-authored-by: Hanzo Dev <[email protected]>
The door became its own service at /v1/build; this default still named
/v1/platform/runner, so every delegated build was refused 404 and no repo
using this lane could produce an image.
Co-authored-by: Hanzo Dev <[email protected]>
The lane creates the project before it publishes, so a slug that has never
deployed needs nothing out of band. It reads a refusal of that create as a
refusal of the whole publish, and those are not the same thing: a key can be
allowed to publish to a project it may not open, which is the ordinary shape of
a deploy credential.
A refusal now asks the question it actually cares about — is the project there —
and one GET answers it. Present means publish; absent means the create really
was needed and the error stands.
Measured on luxfi/cdn: the project answers 200 and every publish had been
failing at the create above it.
Co-authored-by: Hanzo Dev <[email protected]>
The 2,591 per-repo hooks are gone; webhook 3035 is the only row in the forge's
webhook table. This delivery therefore has exactly one possible source.
Co-authored-by: Hanzo Dev <[email protected]>
Empty on purpose: this push carries no tree change, only the delivery. The
forge's per-repo hooks all named cloud's /v1/git/webhook, which answers 204
without dispatching, so a push here has never built. Forge-wide system
webhook 3035 now delivers to platform's /v1/git-webhook instead.
Co-authored-by: Hanzo Dev <[email protected]>
The delegate lane schedules a Job per platform on a node of that
architecture, natively — it has always been able to do this and defaulted
to amd64 alone. Measured today: of 211 running pods, 148 were pinned to
amd64 across 118 distinct images, not one of which published an arm64
variant. So 163 pods sat on one node while a 20-core machine idled at 19%,
and a taint on that node left amd64 CI with nowhere to run at all.
The buildx lane keeps its amd64 default: one job holds one runner, and a
second architecture there can only be emulated.
Co-authored-by: Hanzo Dev <[email protected]>
The 2,591 per-repo hooks are gone; webhook 3035 is the only row in the forge's
webhook table. This delivery therefore has exactly one possible source.
Co-authored-by: Hanzo Dev <[email protected]>
zeekay