The two files are one reusable and eleven repos import them by path, but only
the .hanzo copy was moved onto the embedded KMS. The .github copy — the one
docs, gateway, tasks, deploy and openapi import — still defaulted to the
standalone host and still read /v1/kms/orgs/<org>/... with .secret.value, a
shape the embedded service answers with nothing. Every one of those builds
resolved an empty credential, or the stale value the standalone held, and said
nothing about it.
They are byte-identical again: the embedded url and response shape, the
standalone fallback, and the refusal to build with an unreadable declared
build secret.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A repo that lists `build_secrets` in hanzo.yml has asserted its image is not
valid without them. The fetch treated every failure as survivable and exported
nothing, so buildx baked an empty value and the run went green — the artifact
was broken and only production showed it, as silence.
The declaration is now read before the two short-circuits, so an unconfigured
or unreachable KMS is an error for a repo that declares secrets instead of an
empty result that looks like "not in KMS". An unresolvable name is an error at
fetch, and the build step refuses a name that is still unset when it assembles
--build-arg.
A repo that declares no build_secrets keeps the best-effort path exactly as it
was: KMS stays optional, GHCR push still works on the workflow token.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
KMS is embedded in hanzoai/cloud now (HIP-0106, cloud/apps/kms). This reusable
still spoke the standalone contract, and both differences fail SILENTLY -- the
result is an empty string, which every caller here already treats as "not in
KMS", so a build that lost its secret looked exactly like one that never
declared it.
url /v1/kms/secrets/<path>/<name> -- the org comes from the TOKEN, not the
url. The old /v1/kms/orgs/<org>/... form 404s, and `curl -sf` turns that
into empty output.
body {"name","env","value"} -- flat. `.secret.value` selects nothing.
Reading the OLD host is worse than reading nothing: the two instances hold
DIFFERENT values under the same name. The standalone copy of
deploy/EVENT_INGEST_KEY is stale, and cloud rejects it at ingest with 403 --
so a build that "found" a key still shipped one that cannot write, and every
pageview from that property was filed under the reserved $public tenant, which
our own org cannot read.
Tries the embedded shape first and falls back to the standalone one, so a repo
still pointed at the old host by vars.KMS_ENDPOINT keeps working while the fleet
moves. Default endpoint flipped to api.hanzo.ai.
forge/main sat 43 commits off this history and could not fast-forward, so every
push to the mirror was rejected and the two hosts have been drifting apart while
saying nothing about it.
Nothing on it is unlanded. `git cherry origin/main forge/main` finds 42 of 42
commits already applied here by patch equivalence and none unique; no file exists
on that branch and not on this one; and the whole textual difference is prose this
side has since rewritten — the registry.hanzo.ai mirror leg, crane, corepack
provisioning and the delegate lane are all present here in their later wording. It
is this repo's own history, absorbed once already at a43f5a3 and then left behind
by 89 commits.
So this takes the lineage and NOT the tree (-s ours): the merge is a no-op on
content — verified, the resulting tree object is byte-identical to its first
parent — and its only effect is that the mirror can fast-forward again. The
alternative was a force-push, which would delete 43 commits of real history to
assert something a merge can state without deleting anything.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
df6c025 made a gate that reports success without running a test fail the build.
It asks each gate that question by looping over hanzo.yml's `test:` block — so it
cannot ask the repo that declares no block at all. There the loop simply does not
run, and the build goes green having tested nothing: the exact shape that commit
exists to end, reached by declaring less rather than more.
23 repos in the fleet are in that state today, so failing here would paint the
fleet red without fixing one of them. It warns instead, into the ::warning::
annotation and the step summary this build already writes, which puts the absence
ON THE RUN rather than in a file only a reader of hanzo.yml would ever open.
The gate list is now read ONCE and asked both questions — "is it empty" and "run
each" — instead of yq being invoked twice for one fact.
Both copies of build.yml, byte-identical as aeb6adf requires (one document, two
paths, one tag serving both forges). Checked: the two files compare equal after
the edit, both parse, the step's shell passes `bash -n`, and the three shapes
behave — absent `test:` warns, `test: []` warns, a populated block runs its gates
and does not warn.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/commerce ran 49 ginkgo suites green and then died on its last gate:
::group::test admin-typecheck
bash: line 3: pnpm: command not found
exitcode 127
skipping post step for "Provision Node toolchain (JS test gates)";
main step was skipped
That last line is the whole story. The step exists precisely to stop a JS gate
dying at 127 on a runner image with no Node — and it had SKIPPED, so it could
not.
Its guard was `hashFiles('package.json')`, which only ever looks at the repo
ROOT. commerce is a Go repo whose front ends live in app/, and its gate is
`cd app && pnpm install --frozen-lockfile && ... turbo run typecheck`. No root
package.json, so the guard said "not a JS caller" about a repo with four JS
workspaces.
The guard now also matches a nested one. Both patterns are passed rather than
relying on `**/` to match zero segments: this reusable has 13 callers, and a
guard that stopped matching a root package.json would break every JS repo at
once. Widening a provisioning step is safe in the other direction — a repo that
gains a Node toolchain it does not use loses nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The `runner` default has always been '["hanzo-build-linux-amd64"]', and every
runner carrying that label is StatefulSet/git-runner in ns hanzo, registered to
hanzo-git.hanzo.svc. The prose called it "the Hanzo cloud arc pool" anyway. That
is not a cosmetic mismatch: arc (arcd) is being retired, and a default described
in terms of a dead system is how the dead system comes back — the next reader
provisions one because the doc says the pipeline needs it. 156 files across three
orgs ask for this label and not one of them was ever served by arcd, which
advertised 48 labels and zero amd64.
The delegate lane was worse than stale, it was broken. `mode: delegate` POSTs to
PLATFORM_ENQUEUE_URL, defaulting to platform.hanzo.ai/v1/arcd/enqueue. Platform
renamed that route to /v1/runner; the old path now falls through to the catch-all,
which answers 401 {"message":"Unauthorized"} — distinguishable from the real route
only by the body, since /v1/runner answers 401 {"message":"Invalid enqueue token"}
and so does a path that never existed. A delegated build has therefore been
failing auth rather than enqueueing. Nothing noticed because no repo passes
`mode: delegate`, which is also why this was safe to correct in place.
Both published copies edited identically; the byte-equality gate passes at 1321
lines. The remaining occurrence of the word is deliberate — a one-line tombstone
in the input description, so the next person to look for the arc pool is told it
was retired instead of concluding the doc is merely out of date.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The guard now resolves SPEC_TOKEN-or-GH_PAT, but the generator runs in a child
process and reads SPEC_TOKEN itself:
generate.sh: no hanzoai/openapi checkout at .openapi and no SPEC_TOKEN to
clone one
So resolving the fallback only in the guard moved the failure one line down
instead of fixing it. Exporting the resolved value under the name the script
already reads means one credential is chosen once and every consumer in the lane
sees the same one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lane failed every run with
::error::the client: lane needs SPEC_TOKEN (contents:read on hanzoai/cloud)
and there was nothing to do about it: SPEC_TOKEN is provisioned on NO org and NO
repo on this forge. Measured — orgs/hanzoai holds GHCR_USER, GHCR_TOKEN, GH_PAT,
OCI_USER, OCI_TOKEN, KMS_CLIENT_ID, KMS_CLIENT_SECRET, REGISTRY_TOKEN, and
repos/hanzoai/cli holds nothing. A missing org secret is "", not an error, so the
guard fired on every build and named a secret that does not exist anywhere.
SPEC_TOKEN stays PREFERRED: it is the least-privilege choice, a fine-grained
token scoped contents:read on one private repo, and that is worth keeping when
someone mints it. GH_PAT is the fallback because it is provisioned and already
reads private hanzoai repos.
The step now prints `spec credential: SPEC_TOKEN|GH_PAT`. A fallback that does
not say which credential it took is how a wrong one goes unnoticed — the same
defect that made luxfi/trader read as a registry problem for weeks.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ground truth from the run, once the step was made to report:
GITHUB_WORKFLOW_REF=<unset>
❌ Failure - Main Check out this reusable`s own tools
Nothing between them. The step died on the very next line, before any clone:
ref="${GITHUB_WORKFLOW_REF##*@}"
This forge does not set GITHUB_WORKFLOW_REF, and since bash 4.4 a `##`
expansion of an unset variable under `set -u` is an unbound-variable error. So
the step exited 1 while every host still looked blameless, and the fallback that
would have produced the right answer -- `ref="${ref:-v1}"`, two expansions later
-- was unreachable.
Reading `${GITHUB_WORKFLOW_REF:-}` into a local first makes the whole chain safe
and still derives v1, which is what every caller pins.
⚠️ Not reproducible on macOS: /bin/bash is 3.2, where the old form survives with
an empty result. Testing it there says the code is fine. The forge`s log is the
only thing that settled this, which is why the previous commit made the step
print what it derived.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The step sent both clone attempts to /dev/null and printed only
::error::could not fetch hanzoai/ci@$ref (bin/imgver)
which names neither the ref it derived nor which host refused, so the cause can
only be guessed at from source — and guessing produced one wrong fix already.
Now it echoes GITHUB_WORKFLOW_REF, the derived ref, and the outcome of each host
in its own group.
Also tries git.hanzo.ai/hanzoai/ci as a third host. The two org axes on that
forge are orthogonal: `hanzo` is the tenant org and `hanzoai` is the mirror
namespace, and this reusable lives under BOTH. Only the tenant path was listed.
Keeps the refs/tags/ strip from v1.0.20: every caller pins @v1, so the imported
ref is refs/tags/v1 and `--branch refs/tags/v1` names no branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every caller pins `@v1`, so GITHUB_WORKFLOW_REF ends in `@refs/tags/v1`. The ref
derivation stripped only `refs/heads/`, leaving `refs/tags/v1` — and
`git clone --branch refs/tags/v1` is not a valid branch name, so BOTH clone
attempts failed and the step exited:
::error::could not fetch hanzoai/ci@refs/tags/v1 (bin/imgver)
It worked for a branch caller and could never work for a tag one, which is every
caller this reusable actually has. The clone itself was fine: github.com/hanzoai/ci
is public and an anonymous `--branch v1` clone returns bin/imgver.
Carried at both paths, which this file requires — git.hanzo.ai resolves only
.hanzo/workflows and github.com only .github/workflows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first cut walked up from the Dockerfile to the nearest go.mod. That is
right for a subdirectory that is its own module built from its own directory,
and wrong for the shape hanzoai/s3 actually ships: Dockerfiles under
test/kafka/ built with `context: ../..` that `COPY go.mod go.sum ./` and
compile the ROOT module. Judged by test/kafka/go.mod the floor reads 1.25.0;
the truth is 1.26.5. That is the difference between "fine" and "cannot
build", and the gate was reporting the first.
So when a Dockerfile copies the context's own go.mod — `COPY go.mod ...`, the
overwhelmingly common shape — the context's module is what gets compiled and
its floor is the one that counts. The walk-up stays as the fallback, because
it is the correct answer for s3's own s3-rdma-sidecar/ and telemetry/server/,
which are separate modules built from their own directories.
Found by checking the gate against every Dockerfile in the orgs rather than
trusting it: re-running the 461-file corpus flags 57 where the first cut
flagged 54, catches both s3 test/kafka cases that had been identified by hand,
and un-flags nothing. A gate that mis-attributes is worse than no gate,
because the number it prints looks like an answer.
Three cases pinned in the suite: context-wins-when-copied, own-directory
context keeps its own floor, and a Dockerfile copying a SUBDIRECTORY's go.mod
is still judged by the nearest one (otherwise every multi-module repo reports
false alarms).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
mode=delegate moves WHERE an image is built — platform.hanzo.ai's arcd pool
instead of the runner — but not whether the base image can compile the
module. A mismatch enqueued to platform still dies with "go.mod requires go
>= X (running Y; GOTOOLCHAIN=local)", on a machine whose logs the GitHub run
never shows, minutes after this job reported success.
The check is cheap exactly where the checkout already is, and the delegate
lane resolves the same $df and $ctx the buildx lane does, so it is the same
one line at the other call site.
Coverage boundary, stated plainly: this reaches callers that declare an
`images:` block. Repos whose images are built outside this reusable —
luxfi/node (platform reads its hanzo.yml directly over the webhook) and
hanzoai/cloud (its own cicd.yml `image` job, kept single-owner on purpose) —
do not pass through either lane and are NOT gated by this.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The official golang images set GOTOOLCHAIN=local, so a go.mod requiring a
newer Go than the base image does not degrade to a download — it dies
mid-build:
go: go.mod requires go >= 1.26.5 (running go 1.26.4; GOTOOLCHAIN=local)
visor v1.108.16 shipped that failure. The reason it reached a release build
is that nothing local can see it: a dev box runs GOTOOLCHAIN=auto and simply
fetches what the module asks for, so `go build` is green right up until the
image build. The break is introduced by editing go.mod, a file with nothing
to do with Docker.
It is not one repo's problem. Sweeping every Dockerfile across the orgs —
all variants, all subdirectories, not just repo roots — found 54 builder
stages already below their own go.mod in 23 repos, and only 7 of 223 Go
stages setting GOTOOLCHAIN=auto. Fixing those 54 fixes today; this makes the
55th impossible, which is the part worth having.
bin/gover is that check, run by the build lane before any build work so a
mismatch costs seconds instead of a binfmt install and a layer cache. It
refuses only what cannot build: an image BELOW the module floor. Pinning
ahead of go.mod is valid and stays silent, because a newer toolchain
compiling an older directive always works. A floating tag against a
patch-pinned go.mod warns and passes — it builds today, and a gate that
fails what builds is a gate people learn to skip.
The error names the remediation rather than just saying no: pin the base to
the version go.mod asks for, and add ENV GOTOOLCHAIN=auto so the next bump
downloads its toolchain instead of failing.
This repo's own module and image move to go1.26.5 and adopt that same
GOTOOLCHAIN=auto line, so the pipeline holds itself to the rule it enforces.
bin/gover_test.sh pins both directions — the refusals and the allowances
(newer-than-floor, alpine suffix vs Go patch, ARG defaults, and a
multi-module repo judged by its nearest go.mod). Offline, no registry.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GITHUB_WORKFLOW_REF is set by GitHub Actions and by nothing else. The step runs
under `set -euo pipefail`, so on the forge the bare expansion aborted before the
clone loop it was written for — "unbound variable", exit 1, and with it every
step the pipeline had left: build, test, image, deploy, all reported as skipped.
A caller saw only that the reusable's own tools could not be checked out, which
reads like a missing repo and is not one.
Defaulting the variable is the whole fix. Where it IS set the behaviour is
unchanged — a caller on a branch still gets that branch's imgver, a caller on a
tag still gets the tag's. Where it is not, the fallback that was already written
(`${ref:-v1}`) is finally the thing that runs, and the GitHub clone ahead of it
in the loop carries bin/imgver.
The forge fallback URL named the `hanzo` org; the repo is `hanzoai/ci`. Its
first line was never reached, so the wrong name never surfaced.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two mechanisms did one job, and the one that lived here could never do it. The
forge's copy of this repo predated the file, so the workflow that would pull
GitHub's commits was itself one of the commits it had not pulled. A syncer that
has to already be synced to run is not a mechanism, it is a deadlock — and it
held: the forge sat 18 commits behind with `v1` frozen on a pipeline from months
earlier, while this file sat on GitHub looking like the answer.
hanzoai/mirrors owns "the forge is current", for every repo, from ONE table.
`ci` is in it now, and sync.py carries branches AND tags — including the `vN`
channels this file was the only place to force. Copying a workflow into fifty
repos is fifty things to keep in step; the table is one.
What survives is hanzoai/mirrors' own copy, and only because a syncer cannot
sync itself into existence — the same bootstrap exception, stated once, in the
repo it applies to.
One consequence to watch rather than hide: this file dispatched build.yml after
a fast-forward, because a push made with the WORKFLOW token does not fire other
workflows. sync.py pushes as the instance admin, which is a different identity
and not subject to that loop prevention. If it turns out commits arrive on the
forge and no build fires, the fix is one dispatch call in sync.py — named here
so it is looked for, not discovered.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The fleet's whole pipeline is published twice — `.github/workflows/build.yml`
for github.com and `.hanzo/workflows/build.yml` for git.hanzo.ai, because each
forge resolves only its own directory. Two copies of one artifact, and nothing
compared them. The `.hanzo` copy had drifted nine lines: a truncated second
`on: workflow_call: inputs: runner: description:` block and a second top-level
`name:` key, wedged between line 7 and the real body. Invisible, because the
only reader of that file is a forge no push from here reaches.
They are one file now, and `build-yml-is-one-file` in hanzo.yml says so: it
normalises the ONE spelling that may legitimately differ — the path each copy
names for itself — and demands byte equality of the other 1,251 lines. Proven
both ways locally: PASS on this tree, FAIL on the tree as committed an hour ago.
And the channel. `sync-from-github.yml` carried the tag mirror as the LAST STEP
of the fast-forward job, under `set -euo pipefail`. So the moment main diverged
— two empty commits pushed straight to the forge — the job exited 1 at the
ancestry check and the tag step never ran again. A branch nobody could merge
froze `@v1` for every caller in three orgs, and a stale pipeline runs green, so
no caller could tell.
Measured while writing this:
github.com hanzoai/ci v1 -> 830171c client: lane present, Rust toolchain present
git.hanzo.ai hanzoai/ci v1 -> 522aa9e neither
forge run hanzoai/cli #21 (a6ddd9f, main): step list has no "Client —
regenerate from the release document" at all, and `Test (per hanzo.yml)`
dies at `bash: line 1: cargo: command not found`, exit 127.
That is D1 in its final form: hanzoai/cli HAS the drift gate, HAS hanzo.yml,
HAS .spec-lock, and the gate still cannot run, because the pipeline the forge
hands it predates the lane that would run it. Same for all eight client repos.
A branch and a tag are independent facts, so they are two jobs with no `needs:`
between them. A diverged branch is now one red job about that branch, and the
channels keep moving.
The merge below carries the forge's two empty commits so the fast-forward has
somewhere to go.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A projection can stop being true two ways and both needed the same document, so
they are one step with two modes rather than two implementations.
push / pull_request CHECK the committed projection against the document its
OWN .spec-lock names, writing nothing.
spec-update MOVE it onto the document the release named, then commit
and cut.
The check half replaces the hand-rolled codegen-drift-check / spec-drift-check
step that four repos each carried a copy of, and fixes what none of those copies
could see: they regenerated from whatever hanzoai/openapi's main happened to be,
so two runs of one commit could disagree, and a change nobody in that lineage
made turned a client red. A pinned ref plus a pinned digest cannot.
And on a check the LOCK is itself a gate: the ref is pinned, so the bytes behind
it must be too. A digest that moved under a pinned ref means someone moved a tag,
and no amount of regenerating makes that safe.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Three languages, three honest answers, one key. A file to rewrite (npm, cargo,
cmake, pyproject); the tag itself, because a Go module has nothing to rewrite;
or nothing at all, for a repo whose version is not x.y.z and from which no patch
can be derived. Its projection still lands and is still gated — only the cut
waits for a human, instead of this lane tagging bytes under a number nobody
chose.
Inventing a VERSION file for the Go case, or a fake 0.0.1 for the gradle case,
would each be a second place a version could be wrong.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two corrections to the shape, both about ordering and both the same mistake:
declaring an assertion twice, and acting before the assertions ran.
NO build: KEY. Every client repo already says how it proves itself, in test: —
compiling the client and its examples is exactly what those blocks do. A build:
here would be that assertion written a second time, free to drift from the one a
plain push runs, and only one of the two would gate anything.
THE COMMIT AND THE TAG MOVED AFTER test:. They were running before it, so the
lane pushed a tag — which starts a publish, the one artifact in this workflow
nobody can take back — while the gate that would have refused it had not run.
version: is now optional. A Go module's version IS its tag; there is nothing to
rewrite, and inventing a VERSION file for those repos would be a second place a
version could be wrong. With no version: the current one is read from the tags.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A branch build published `sha-<short7>` and nothing else, so an image only
earned a semver when a human remembered to cut a git tag. That is why 14 of
the fleet's 117 universe pins name a commit instead of a release — not drift,
just the only tag CI ever offered them.
bin/imgver is that number, derived and never typed, monotonic against two
floors: the repo's own manifest (package.json / Cargo.toml / VERSION /
pyproject.toml, or an explicit `version:`) and the highest semver already at
the registry for that image. max + a patch, so one tag can never come to cover
two digests — deriving from the manifest alone re-publishes the same number
until someone edits the file, and a node on imagePullPolicy: IfNotPresent
never picks up the second one. universe's images.yml learned that rule on
iam-secret-sync; this is that rule, for every repo.
It is a SCRIPT, not inline shell, because the fleet has two build front doors:
this reusable, and the hand-rolled .hanzo/workflows/deploy.yml that 11 repos
carry instead of importing it. Both need the identical number, and written
twice it would be right twice and then wrong once. .github/actions/imgver is
the composite action those 11 call; build.yml calls the same script.
Also: the semver tag is proven resolvable before the run goes green (buildx
can exit 0 while the manifest is not yet servable, and a pin onto a phantom
tag is an ImagePullBackOff), and the run summary prints tag and digest
TOGETHER — universe pins repo:tag@digest and the kubelet honours the digest,
so a new tag beside an old digest reports the new version while serving the
old bytes.
22 cases in bin/imgver_test.sh, wired into hanzo.yml's test gate so it runs.
Both build.yml copies (.github/ for GitHub, .hanzo/ for the forge) stay
byte-identical.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two lanes were asking one question twice and the second could never answer it:
generate rewrites the tree, so a test: gate running after it judges the bytes
the lane just wrote, not the bytes the repo committed. Split them by event —
the projection moves when the DOCUMENT moves (repository_dispatch spec-update),
and a plain push runs the repo's own drift gate against the document its
.spec-lock already names.
Same reason the manual re-run no longer defaults to main: pressing 'run
workflow' must not drag a client forward onto an undeployed document. With no
payload it re-asks the document this tree already names, which makes the run a
no-op instead of a release.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A generated SDK is a projection of one document at one version, and until now
the fleet had no place that said so. Seven repos in seven languages either
hand-rolled the same eight lines (java, kotlin, cpp: codegen-drift-check) or had
nothing at all (python, js, go, rust had a generate.yml that fired twice ever,
both red) — and none of them could answer which document they were generated
from.
client: is that place. It fires on repository_dispatch spec-update, fetches
openapi.yaml AT THE SHA THE RELEASE NAMES, and refuses if the bytes hash to
anything but the digest the release published. That refusal is the gate: every
projection of one release comes from one document. Reading a live host instead
would be a lie about which deploy the client describes — at fanout time the
document is a git object and the host is whatever it happens to be serving.
The lane then compiles the client AND its examples (a regeneration that builds
but breaks the example flows has changed the surface out from under every
consumer), writes .spec-lock beside the code so anyone can ask a repo which
document it is without running a generator, and on a delta commits, bumps the
PATCH — derived from the current version, never typed — and pushes the tag. The
repo's own tag lane publishes, so the registry credential stays where the
publish is.
Two toolchain holes closed with it, both the same class this workflow already
refuses everywhere else: no rustup on a stock arc runner, so hanzoai/cli's
genspec --check gate would have been exit 127 — a declared gate that never
runs; and no JDK, without which openapi-generator cannot start.
Credential: SPEC_TOKEN, contents:read on the spec repo. The three existing
generate.yml already name this exact secret for the repo this lane replaces.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The 2,591 per-repo hooks are gone; webhook 3035 is the only row in the forge's
webhook table. This delivery therefore has exactly one possible source.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Empty on purpose: this push carries no tree change, only the delivery. The
forge's per-repo hooks all named cloud's /v1/git/webhook, which answers 204
without dispatching, so a push here has never built. Forge-wide system
webhook 3035 now delivers to platform's /v1/git-webhook instead.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
actions/checkout defaults to fetching no submodules, and this workflow took that
default, so a caller whose source includes submodules built without them. That
failure is silent by construction: the directory is empty, every other file
compiles, and the artifact ships with a hole where a section used to be. Nothing
in the run says so.
hanzo-docs/docs is the case that surfaced it. Doc sections live in submodules
under apps/docs/content/docs/ — studio, from the public hanzo-docs/studio-docs —
and its export gate names those pages, so on this lane the build fails the gate
every time while the forge lane, which has passed `submodules: recursive` to its
own checkout all along, builds the same commit clean. Two lanes, two different
checkouts of one repo; the gate was right and the checkout was wrong.
An input rather than always-on. A build that omits part of its own source is
wrong, but flipping the default turns a submodule the job token cannot read into
a checkout failure for a caller that builds today — and this workflow cannot know
which of those any given repo has. The default is '', which is checkout's own
(false, and false for nested), so every existing caller keeps byte-for-byte the
checkout it has now; the repo that needs recursive says recursive.
Both copies, since aeb6adf carries this file at .github/ and .hanzo/ and every
tag serves both forges. Bodies stay byte-identical — only the headers differ.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`vX.Y.Z` is a release and `vX` is a channel, and the mirror only ever handled the
first. `git push --tags` cannot move a tag — repointing one is a non-fast-forward,
the push is rejected, and the `|| true` on that line swallows it — so the forge's
v1 and v2 froze at 522aa9e / ddf1234 the day they were first pushed, while
GitHub's v1 moved on. A channel that cannot move is a pin wearing a channel's
name, and the 12 repos that import `.hanzo/workflows/build.yml@v1|@v2` (cloud,
commerce, git, console, tasks, this repo) have been running a months-old pipeline
without one of them being able to see it from their own config.
Concretely today: they do not have the assertion that fails a gate reporting
success over ZERO tests (df6c025). Putting a fix in "one place" only works if the
one place is the ref the callers actually resolve.
So push the channels explicitly, forced, by name. `git tag -l | grep -xE 'v[0-9]+'`
is the whole rule: vX moves, vX.Y.Z never does. Releases keep the exact behaviour
they had (the unforced --tags push above still skips any that exists), and there
is one way to move a channel instead of none.
Also corrects this file's sibling claim in build.yml's header, which is now false
in a way that costs a run: it said the `uses:` PATH is part of the version and
`.github/...@v1` / `.hanzo/...@v2` must be paired. aeb6adf carries build.yml at
BOTH paths, so every tag since serves both forges and all four combinations
resolve the same pipeline. The real hazard is the one it named and misplaced — a
`uses:` that does not resolve is a SILENT no-run on this plane, the same failure
shape as a green gate over zero tests.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
git.hanzo.ai/hanzoai/ci and github.com/hanzoai/ci had both grown commits since
0b976af, so sync-from-github's ff-only check failed every 10 minutes with
"DIVERGED: resolve by hand" — by design, it refuses to force-push either side.
Nothing had resolved it, so the sync had carried NOTHING across in the meantime:
neither main nor tags. The forge is missing v1.0.12 for the same reason (the tag
mirror runs after the ff step, which was exiting 1).
That is not a cosmetic drift. The forge's build.yml is the one 12 callers import
(`hanzoai/ci/.hanzo/workflows/build.yml@v1` x5, `@v2` x7 — cloud, commerce, git,
console among them), and it does NOT contain the non-zero-test assertion added in
df6c025. A fix that lives in "one place" reaches those repos only if the one
place is the one they actually resolve.
Resolved as a MERGE, not a rebase or a force-push: a rebase mints new SHAs, so
the forge's head would still not be an ancestor of GitHub's and the ff check
would keep failing. A merge makes it one, which is exactly what the sync asks for
and lets the existing mechanism carry it the rest of the way.
Three conflicts, all from the same change landing on both sides
independently (8c54cfe here / 05b75b2 there, "scope on the verified org"):
.gitignore, hanzo.yml — additive prose on the forge side only; kept both.
render.go — the forge replaced the hand-copied :root block with
<style>{{css}}</style> against the vendored @hanzo/brand sheet (7ad9222).
Kept the forge's: the copy had already drifted off the house palette, and
render_test.go (which arrives with it) pins the sheet's hash and fails any
colour the page names for itself.
ci — modify/delete: this side re-committed the 12MB darwin/arm64 binary in
8c54cfe, the forge deleted it in fffae20. Honored the delete. The Dockerfile
builds linux/amd64 from source (`go build -o /build/ci .`), so the tracked
artifact was never an input, and /ci is now ignored.
Gate after the merge: go build, go vet, go test -count=1 ./... all green, 9 tests
(scope_test.go + the arriving render_test.go), up from 5.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
A green build over ZERO tests is the worst failure mode a gate has, because it is
indistinguishable from a healthy one. Two ways to reach it were live in this
fleet, both silent, both exit 0:
$ go test ./... # nothing in the tree has a _test.go
? example.com/notests [no test files]
$ echo $?
0
$ go test -tags skipCi ./... # every _test.go is //go:build !skipCi
? example.com/notests [no test files]
$ echo $?
0
The second is how hanzoai/gateway hid: its whole fixture suite sits behind
`//go:build legacy` and its Makefile passed no -tags. When the tag was finally
passed, 12 subtests failed on an untouched main — including router_redirect
returning 404, which reproduced on the shipping config with the real binary.
hanzoai/iam runs `-tags skipCi` against files guarded `//go:build !skipCi` today.
Per-repo vigilance is not a mechanism, so the assertion lives here, once, and
every caller inherits it the moment it re-imports this workflow. The rule is the
runners' OWN words, not a heuristic: a gate fails when it SAYS it ran nothing
(Go `[no test files]` / `[no tests to run]`, pytest `collected 0 items` /
`no tests ran`, jest `No tests found` / `Tests: 0 total`, cargo `running 0 tests`,
mocha `0 passing`) AND nothing in its output shows a test having run. A gate that
is not a test gate — vet, lint, a build, a codegen-freshness check — says neither
and is untouched. The one way to satisfy it is to make the gate run a test.
Verified by pointing the step at suites that run nothing, not by reasoning about
it: a module with no test files FAILS, the same module with its tests excluded by
a build tag FAILS, `go vet ./...` and `go build ./...` PASS, a real suite PASSES,
and a genuinely failing suite still fails (the assertion masks nothing). Checked
against real fleet output too: this repo's own gate, hanzoai/git's
`go test ./modules/setting/...` (which legitimately prints `[no test files]` for
one sub-package while another runs), and hanzoai/gateway's full `make test`.
Carried at both workflow paths, since GitHub resolves only .github/workflows and
git.hanzo.ai only .hanzo/workflows; the bodies stay byte-identical.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The dashboard carried its own :root block — a hand-copy of the house
palette, and being a copy it had already drifted off it. The status
colours were GitHub Primer's (#3fb950 / #f85149 / #d29922) where the
house says #10b981 / #ef4444 / #f59e0b. Every surface black was a shade
wrong and blue-tinted (#0b0b0d) against a scale that is strictly neutral
(--surface-0 #080808). The hairline border was an opaque #25252b where
the house hairline is a 6% white wash. The font stacks named neither
Geist Sans nor JetBrains Mono. Of nine colours exactly one — the accent
— still matched. That is what a second component source looks like a few
months in, and it is why this page had to stop being one.
The values now come from @hanzo/brand, and they arrive as that package's
own published artifact rather than as hex codes retyped here:
styles/variables.css, which it ships as a plain custom-property sheet
(exports["./styles/*"], documented for a bare <link>), vendored verbatim
and go:embed-ed. dashboard.css holds what is left over — layout: what is
a row, what sticks, what collapses on a phone — and names no colour,
radius or type size of its own.
Not a @hanzo/gui port, deliberately. gui is React over Tamagui and needs
a bundler, which would put npm and a JS build on the path that ships the
board you read when the builds are broken, and would trade one request
that returns the answer for a shell that fetches it a second later. What
this page ever needed from the design system was its token vocabulary,
not its components — and gui's own shell reads these same var(--hanzo-*)
names, so a Go binary and a React app now spend one vocabulary from one
source. go.mod stays empty; the image stays the binary and a CA bundle.
Two offline gates keep it honest, because vendoring alone would only
move the copy rather than end it:
- TestBrandCSSIsUpstreamBytes pins the sheet to the sha256 of the
version it claims to be. Without it, "just darken that one border"
is a one-character local edit that silently rebuilds the second
palette and nothing ever catches it. This is go.sum's argument.
- TestDashboardCSSNamesNoColours fails on any hex or rgb() the page
writes for itself. The old :root block did not arrive wrong; it
arrived one reasonable exception at a time.
Neither gate touches the network, so proving we use one design system
costs the pipeline no npm and no registry.
TestRenderedPageShowsOnlyTheViewersOrg additionally pins at the HTML
layer what scope_test.go pins at the predicate layer: a lux viewer's
rendered page contains no other org's rows and no other org's name in
the nav. The renderer is where that leak lived, and it is now the
renderer that is asserted.
Rendered and checked in a browser, not just compiled: every token
resolves (surface-0 #080808, text-primary #fafafa, accent-muted #a78bfa,
success/error/warning #10b981/#ef4444/#f59e0b, hairline rgba(255,255,255,
.06), radius-card 8px, JetBrains Mono), color-scheme comes out dark via
the sheet's own .dark hook, and header, chips, nav, table and footer now
sit on one --space-6 gutter instead of the table drifting 12px left.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`go build ./...` writes the binary into the package directory under the
package's own name, so a routine build followed by `git add -A` committed
12MB of darwin/arm64 Mach-O to a repo whose image is built linux/amd64
from source by the Dockerfile. It has been dead weight in the build
context ever since, and `COPY . .` was shipping it to the builder only
for `go build` to overwrite it.
The artifact is never an input. Only the source is.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The sync fast-forwarded main from GitHub but never carried tags, and a git
tag is the ONLY trigger for every publish workflow (twine/npm/go release).
So every release this session — cloud v1.801.299, cli v1.9.5, python
hanzoai-v3.1.3, js v2.0.1 — was tagged on GitHub and never reached the
forge that runs the publish: tagged, never published. Mirror tags in the
same job that syncs the branch. Idempotent; only a new tag fires a publish.
NOTE: this is the code half. The runner fleet must also be up — js-sdk
reported 716/717 hanzo-build-linux-amd64 runners offline, which no code
change fixes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/ci published the reusable workflow every other repo imports but
had no way to build its own image. build.yml is workflow_call-only, so
ghcr.io/hanzoai/ci:v0.1.0 was produced out of band and there was no
repeatable path to a second one — a CI system that cannot build itself
is not finished.
Adds the two pieces every other caller already has: a root hanzo.yml
(image + go vet + go test ./...) and a ~7-line caller pinned to
hanzoai/ci/.hanzo/workflows/build.yml@v2. Self-referential on purpose,
and pinned to the TAG rather than the working tree — that is what stops
a broken edit to build.yml from also breaking the build that would have
caught it.
The test gate is load-bearing here rather than decorative: scope_test.go
asserts the dashboard refuses a request carrying no X-Org-Id and that
`?org=` can only narrow. Those properties were absent once and the
service disclosed every org's build metadata to the internet, so a red
gate must block the image.
No `deploy:` — rollout stays a reviewed tag pin in hanzoai/universe
(crs/ci.yaml), the rule cloud and git follow. A pipeline that builds AND
rolls itself out can put an unreviewed image on a public host, and
cd.hanzo.ai's selfHeal reverts a direct patch regardless.
Also corrects build.yml's own header, which documented the @v1 form
(`.github/workflows/build.yml`) from inside the v2 file. Both forms are
valid — the path is part of the version, since the file moved between
tags — so the header now says which is which and who is on each. Pairing
them wrongly 404s the reference, and on this plane that is a silent
no-run.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
This service shipped with /v1/runs answering 200 to anyone on the
internet, disclosing repo names, workflow names, branches, commit SHAs,
actor logins and pass/fail across EVERY org. The cause was a category
error, not a missing check: `?org=` narrowed what was rendered and read
like tenancy, so it looked like the surface had one. A query parameter
is a request for a view. It can never be the authority for one.
The authority is now X-Org-Id, minted by admin-guard from the
IAM-verified `owner` claim and written onto the request by the ingress
middleware's authResponseHeaders (Traefik overwrites any client-sent
value, so it cannot be forged on the wired path).
Three properties, each with a test that fails without it:
- ABSENCE IS FATAL. No X-Org-Id => 403, never "no filter". Defaulting
an absent scope to "everything" is precisely the bug; absence means
the request did not come through the gate, so refusing is the only
honest answer. The refusal body carries no repo names.
- THE PARAMETER CAN ONLY NARROW. Permission is applied first, then
`?org=` selects within it. A lux viewer asking ?org=hanzo gets an
empty list, not hanzo's builds.
- THE ORG LIST IS SCOPED TOO. A tenant sees only its own org in the
nav. Hiding the runs but listing every org still discloses the set of
orgs that build on the platform.
The admin org keeps the cross-tenant fleet view, matched to
admin-guard's IAM_ADMIN_ORG via CI_ADMIN_ORG — the guard decides who
gets in, this decides who sees everything, and the two must name the
same org or the fleet view silently collapses (or, set too wide,
promotes a tenant into it).
renderDashboard now takes the viewer and is handed only rows that
already passed v.visible. A template that can see everything is one edit
away from showing it.
Mutation-verified: restoring `visible` to the old filter-as-gate
behaviour fails TestTenantCannotWidenWithQueryParam and the end-to-end
handler test.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
GitHub Actions resolves a reusable workflow ONLY from .github/workflows/ —
that is a platform rule, not a preference. git.hanzo.ai reads .hanzo/workflows/.
Callers are split across both, so a single tag can only serve everyone if the
file exists at both paths. This is what lets v1 be the one tag and v2 go away.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzoai/ci published the reusable workflow every other repo imports but
had no way to build its own image. build.yml is workflow_call-only, so
ghcr.io/hanzoai/ci:v0.1.0 was produced out of band and there was no
repeatable path to a second one — a CI system that cannot build itself
is not finished.
Adds the two pieces every other caller already has: a root hanzo.yml
(image + go vet + go test ./...) and a ~7-line caller pinned to
hanzoai/ci/.hanzo/workflows/build.yml@v2. Self-referential on purpose,
and pinned to the TAG rather than the working tree — that is what stops
a broken edit to build.yml from also breaking the build that would have
caught it.
The test gate is load-bearing here rather than decorative: scope_test.go
asserts the dashboard refuses a request carrying no X-Org-Id and that
`?org=` can only narrow. Those properties were absent once and the
service disclosed every org's build metadata to the internet, so a red
gate must block the image.
No `deploy:` — rollout stays a reviewed tag pin in hanzoai/universe
(crs/ci.yaml), the rule cloud and git follow. A pipeline that builds AND
rolls itself out can put an unreviewed image on a public host, and
cd.hanzo.ai's selfHeal reverts a direct patch regardless.
Also corrects build.yml's own header, which documented the @v1 form
(`.github/workflows/build.yml`) from inside the v2 file. Both forms are
valid — the path is part of the version, since the file moved between
tags — so the header now says which is which and who is on each. Pairing
them wrongly 404s the reference, and on this plane that is a silent
no-run.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
This service shipped with /v1/runs answering 200 to anyone on the
internet, disclosing repo names, workflow names, branches, commit SHAs,
actor logins and pass/fail across EVERY org. The cause was a category
error, not a missing check: `?org=` narrowed what was rendered and read
like tenancy, so it looked like the surface had one. A query parameter
is a request for a view. It can never be the authority for one.
The authority is now X-Org-Id, minted by admin-guard from the
IAM-verified `owner` claim and written onto the request by the ingress
middleware's authResponseHeaders (Traefik overwrites any client-sent
value, so it cannot be forged on the wired path).
Three properties, each with a test that fails without it:
- ABSENCE IS FATAL. No X-Org-Id => 403, never "no filter". Defaulting
an absent scope to "everything" is precisely the bug; absence means
the request did not come through the gate, so refusing is the only
honest answer. The refusal body carries no repo names.
- THE PARAMETER CAN ONLY NARROW. Permission is applied first, then
`?org=` selects within it. A lux viewer asking ?org=hanzo gets an
empty list, not hanzo's builds.
- THE ORG LIST IS SCOPED TOO. A tenant sees only its own org in the
nav. Hiding the runs but listing every org still discloses the set of
orgs that build on the platform.
The admin org keeps the cross-tenant fleet view, matched to
admin-guard's IAM_ADMIN_ORG via CI_ADMIN_ORG — the guard decides who
gets in, this decides who sees everything, and the two must name the
same org or the fleet view silently collapses (or, set too wide,
promotes a tenant into it).
renderDashboard now takes the viewer and is handed only rows that
already passed v.visible. A template that can see everything is one edit
away from showing it.
Mutation-verified: restoring `visible` to the old filter-as-gate
behaviour fails TestTenantCannotWidenWithQueryParam and the end-to-end
handler test.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`binaries:` published only to a GitHub release, which is storage on a quota we
do not own — 400MiB per cloud release, hundreds of releases. A top-level
`bucket:` now sends the same artifacts and the same binaries.json to hanzoai/s3
instead. Nothing about the format changes; only the url the index carries.
Credentials are the S3_ADMIN_* names the services already read (clients/s3admin),
pulled from KMS at run time — not a CI-only copy nobody remembers to rotate.
A declared bucket with no credential fails the publish rather than shipping an
index whose artifacts are absent, and the lane proves the index is readable
UNAUTHENTICATED before calling it published: a host fetches it with no
credentials, so a private object is an index that resolves every app to a 403.
Granting that read is the bucket's job, one policy once, which is why this
checks rather than sets it. Artifacts upload first and the index last, so it
never names an object that is not there yet.
Also adds `tests:` (default true, so every existing caller is byte-identical).
false asserts the gate already ran on this exact commit — it does not mean ship
untested. hanzoai/cloud mints its v* tag only after that SHA passed the gate on
main and built and smoked (clients/platform/release.go), so a tag build re-tests
a proven commit and pays a 3108-package link storm for no new information. The
publish-after-gating invariant is unchanged: it is enforced once, not twice.
Verified end to end against a real hanzoai/s3: both platforms cross-compiled,
signed SigV4 PUTs, index last, anonymous GET 200, digest of the fetched bits
equal to the index. Four paths exercised — non-tag builds without publishing,
missing credential refuses, private bucket refuses naming the remedy, public
bucket publishes.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`binaries:` is the artifact half of hanzo.yml — the lane that ships an executable
a running host installs, as opposed to `images:`, which ships an OCI image a
cluster runs. It read exactly one recipe shape: `main:` + `platforms:` + a
hardcoded `go build`. So a repo whose artifact is an npm tarball, a wheel, or a
Rust binary could declare nothing at all, and the honest answer to "how do I
publish this" was "you cannot, write a Dockerfile and ship an image instead" —
which is a different artifact with a different consumer.
Two optional fields close that, in the SAME block, with no second format:
run: the command that builds — any toolchain
out: the glob of what it produced
`main:` is unchanged and stays the zero-config Go lane, so every existing repo
builds byte-for-byte as before. A `run:` entry indexes as os/arch "any", because
an npm tarball is not per-platform and an index entry that claimed one would be a
lie a host acts on. Declaring `run:` without `out:` fails the build rather than
publishing an empty index — nothing else names what the command produced.
This is the GitHub half of a contract now implemented on both sides: hanzoai/
cloud's POST /v1/runner reads the identical block and builds it in-cluster, one
initContainer per entry in the toolchain image `image:` names (the one field this
lane reads past — here the toolchain IS the runner). Same recipe, same
binaries.json, same URL; two front doors, as with `images:`.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
ci is one of 13 non-mirror repos in the hanzoai org, and non-mirror is the gap
between the two mechanisms that keep git.hanzo.ai current: hanzoai/mirrors
reconcile.py only creates pull-mirrors and leaves existing repos alone, so a
repo already present as a NON-mirror is skipped forever; the per-repo pull
covers repos that are native-canonical and must not be mirrors. A non-mirror
without the file has neither.
Measured across the 12 non-empty non-mirror hanzoai repos: every one carrying
the file sits 0-1 commits behind GitHub; ci and cloud, the two without it, sat
3 and 10 behind. None was ever AHEAD.
No credential — this repo is public, so the fetch is anonymous. Fast-forward
only.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
`images:` ships an OCI image a CLUSTER runs. There was no lane for the other
artifact we ship: an executable a RUNNING host installs, which is what a zip
plugin is — zip.Plugin{URL, Sum} fetches it, verifies the SHA-256 BEFORE the
file is made executable, and caches it by digest. Every repo that needed one
grew its own release.yml, and no two agree on where the digest comes from.
One block in the hanzo.yml a repo already has:
binaries:
- name: billing
main: ./cmd/billing
platforms: [linux/amd64, linux/arm64]
Same job, same runner, same KMS token ladder — not a second pipeline. Two rules
of its own: BUILT on every push, so an arm64 cross-compile that breaks fails the
PR that broke it rather than the release; PUBLISHED on a tag and AFTER the test
gate, because a host installs an artifact unattended (an image is rolled out by
a reviewed pin — an artifact is not). CGO_ENABLED=0 -trimpath is not taste: the
host runs these bits on whatever base image the host is, and the digest must be
a function of the source, not of the checkout path.
binaries.json ships with them — {name,os,arch,url,sha256} per artifact — so the
bits and the digest that authorizes them are one release and a host reads both
from one place. The job summary prints the zip.Load() to paste: a digest a
human retypes is a digest a human gets wrong.
The forge URL comes from GITHUB_SERVER_URL/GITHUB_API_URL rather than a
hardcoded hostname, so one lane serves github.com and a forge front door.
Run end to end against a stand-in release API before committing: three platforms
cross-compiled and published, a re-run of the same tag converging instead of
422ing, then a zip host installing the linux/arm64 artifact by URL+Sum and
serving its routes (200/201), refusing it when the digest is wrong, and
restarting from the digest cache with the release host down.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cd.hanzo.ai has been the delivery surface for a while; there was no build
surface. ci.hanzo.ai 404'd and this repo held only the reusable workflow, so
"how is the fleet building" had no answer outside per-repo pages.
This owns no run state. Hanzo Git schedules every job and holds every log; this
reads that and presents it. A CI service with its own run database would put two
answers to "did the build pass" in the fleet, and the one users look at would be
the one that drifts. git.hanzo.ai is the store, ci.hanzo.ai is the view.
Reads /v1/repos/{owner}/{repo}/actions/runs (our Gitea drops the /api prefix).
Scan strategy is repo-search sorted by activity, windowed: the instance mirrors
~1400 repos and almost none built recently, so walking all of them would spend
the whole refresh budget confirming silence. One poller, one cache, bounded
fan-out — N open dashboards cost the forge the same as one, and a status page
must never be what degrades the system it reports on.
A failed poll keeps the last good rows and says so, rather than blanking: an
empty page reads as "nothing is building", which is the opposite of the truth
during an outage. Same reason /healthz is liveness-only and does not gate on
having a snapshot.
⚠ The bug this caught in itself, before shipping: Hanzo Git reports every
finished run as status=completed regardless of outcome, and carries the verdict
in a separate `conclusion`. Bucketing on status alone drew 15 of 20 live runs
red — every success and every cancellation shown as failing. Both fields are now
required to decide a colour. `cancelled` is its own bucket, not a failure:
superseded pushes cancel in-flight runs and they are the largest category on
this fleet, so folding them into red makes a board nobody trusts.
Verified against the live instance: buckets match the raw API exactly —
18 completed/success, 4 completed/cancelled, 3 in_progress + 5 queued = 8
running, 0 failing.
Tenancy is the org slug, the same value Hanzo Git namespaces repos by, IAM
issues in the `owner` claim, and Hanzo CD fences projects with. Filtering here
is that boundary, not a parallel notion of who-sees-what.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Superseded: the Hanzo GitHub App pushes a webhook, so the forge tracks GitHub
without a per-repo workflow. This file called git.hanzo.ai/api/v1/.../mirror-sync
— a Gitea API for a system we no longer drive — and would sit inert in every repo.
One mechanism, in one place, instead of ~350 copies of a cron.
The forge resolves `uses:` only under WORKFLOW_DIRS, which is .hanzo/workflows.
This repo published its reusable from .github/workflows/build.yml, so every
caller -- cloud, commerce, console, git -- failed to resolve it:
resolve uses "hanzoai/ci/.github/workflows/build.yml@v1":
path ".github/workflows/build.yml" must be under a configured workflow directory
No build has run on any of those four repos since. Moving the file is the fix,
and it is also just the law: .hanzo/workflows is where CI lives, and GitHub gets
exactly one workflow that pushes nothing and only nudges canonical.
Callers move to hanzoai/ci/.hanzo/workflows/build.yml. Cutting a NEW tag rather
than force-moving v1 -- moving a floating tag is what put this repo's two heads
out of sync earlier today, and the same hazard is already on record from
luxfi/threshold@v1.9.4.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two things that made a release a manual job.
The semver tag was published v-stripped: `git tag v1.26.19` produced
ghcr.io/hanzoai/git:1.26.19, while every universe CR pins v1.26.19. Nothing in
between reconciled the two, so releases were finished by hand — `crane copy`
sha-<sha7>-amd64 onto the semver a human typed, then bump the CR. Publish the
ref name verbatim instead: identity in, identity out, nothing to remember at
the seam. The v-stripped alias stays for CRs already pinned that way (world
2.4.51) and is skipped when a repo tags without a v. `<ver>-amd64` is deleted —
no CR in the fleet pinned it, it was a third name for one digest.
The deploy step read .deploy.services on a caller that declares no `deploy:`;
`jq '.[]'` over null aborts the step. The deploy.on gate hid it on branch
pushes and is bypassed on a tag, so it would have surfaced only on the release
it was blocking. Guard on `.deploy` being a map, the same shape the build step
already uses for `images:`.
Two changes, both about getting images off GitHub:
Use the canonical registry name. oci.hanzo.ai and registry.hanzo.ai are
the same registry behind the same auth, so existing pulls by either name
keep resolving -- this only settles which name we build against.
Stop skipping quietly. Every path that gives up on publishing to our own
registry logged a ::notice:: and moved on, so a build that shipped to
GHCR alone looked identical to one that shipped to both. That silence is
why the fleet still pulls from GitHub. These are warnings now: the repos
missing a registry credential name themselves in their next run, which
is the list we need before any deploy can be moved over.
v1.0.7's `git pull --rebase` retry never worked: the universe clone is
`--depth 1` shallow, so rebase has no merge base and fails immediately →
push stayed rejected under concurrent rolls, hard-failing the Deploy step
with the expired kubeconfig (hanzoai/world v2.4.44 stuck at 2.4.43).
Replace rebase with a shallow-safe loop: on a non-fast-forward reject,
`git fetch --depth 1 origin main` + `reset --hard FETCH_HEAD`, re-apply the
one-file CR tag change, re-commit, retry (6×). A no-op after reset (remote
already carries our tag) counts as recorded. Never force-push. Verified
against a real shallow clone with a concurrent racing push: our CR change
lands, the racer's other-file commit is preserved.
Claude-Session: https://claude.ai/code/session_01NP4ehjWW2h98FkE4YjUKuJ
v1.0.7's `git pull --rebase` retry never worked: the universe clone is
`--depth 1` shallow, so rebase has no merge base and fails immediately →
push stayed rejected under concurrent rolls, hard-failing the Deploy step
with the expired kubeconfig (hanzoai/world v2.4.44 stuck at 2.4.43).
Replace rebase with a shallow-safe loop: on a non-fast-forward reject,
`git fetch --depth 1 origin main` + `reset --hard FETCH_HEAD`, re-apply the
one-file CR tag change, re-commit, retry (6×). A no-op after reset (remote
already carries our tag) counts as recorded. Never force-push. Verified
against a real shallow clone with a concurrent racing push: our CR change
lands, the racer's other-file commit is preserved.
Every service's deploy commits the desired image tag to universe/main, so
when several land at once a plain `git push` loses the race with a
non-fast-forward reject. Before, that set recorded=0 and (with an expired
runner kubeconfig) hard-failed the Deploy step, leaving the CR un-updated —
exactly why hanzoai/world stuck at 2.4.41 while 2.4.42 deploys "failed".
Wrap the push in a rebase-and-retry loop (up to 6): our one-file CR change
replays cleanly onto the moved remote (other services touch other CRs). A
genuine same-CR conflict (a concurrent roll of THIS service) aborts the
rebase and stays recorded=0 — never force, so no silent backward roll.
Verified against a live racing remote: clean race → rebased + pushed,
recorded=1; the racer's commit preserved.
Claude-Session: https://claude.ai/code/session_01NP4ehjWW2h98FkE4YjUKuJ
Every service's deploy commits the desired image tag to universe/main, so
when several land at once a plain `git push` loses the race with a
non-fast-forward reject. Before, that set recorded=0 and (with an expired
runner kubeconfig) hard-failed the Deploy step, leaving the CR un-updated —
exactly why hanzoai/world stuck at 2.4.41 while 2.4.42 deploys "failed".
Wrap the push in a rebase-and-retry loop (up to 6): our one-file CR change
replays cleanly onto the moved remote (other services touch other CRs). A
genuine same-CR conflict (a concurrent roll of THIS service) aborts the
rebase and stays recorded=0 — never force, so no silent backward roll.
Verified against a live racing remote: clean race → rebased + pushed,
recorded=1; the racer's commit preserved.
The Deploy step records the desired image tag durably in universe (the
in-cluster operator reconciles it every ~5min) and THEN runs runner-side
kubectl to accelerate + smoke-test the roll. Those kubectl calls were
unguarded, so when the short-lived DOKS kubeconfig from KMS expires (~7d)
every call fails "You must be logged in to the server" and the step reports
failure — even though the operator already rolled the image (hanzoai/world
v2.4.33/34/35 all showed a red Deploy while serving the new build live).
Track `recorded`: once the universe record is pushed (or already pins the
tag), runner kubectl is best-effort (warn, don't fail) because the operator
owns the rollout. With no durable record (bare-Deployment repos) kubectl is
the only path and stays fatal — the real deploy gate is preserved. Verified
under `set -euo pipefail`: recorded=1 → green on auth failure; recorded=0 → fatal.
Claude-Session: https://claude.ai/code/session_01NP4ehjWW2h98FkE4YjUKuJ
The Deploy step records the desired image tag durably in universe (the
in-cluster operator reconciles it every ~5min) and THEN runs runner-side
kubectl to accelerate + smoke-test the roll. Those kubectl calls were
unguarded, so when the short-lived DOKS kubeconfig from KMS expires (~7d)
every call fails "You must be logged in to the server" and the step reports
failure — even though the operator already rolled the image (hanzoai/world
v2.4.33/34/35 all showed a red Deploy while serving the new build live).
Track `recorded`: once the universe record is pushed (or already pins the
tag), runner kubectl is best-effort (warn, don't fail) because the operator
owns the rollout. With no durable record (bare-Deployment repos) kubectl is
the only path and stays fatal — the real deploy gate is preserved. Verified
under `set -euo pipefail`: recorded=1 → green on auth failure; recorded=0 → fatal.
The build loop read name/ctx/df/repo/sfx/plats but never passed images[].args,
so a repo's declared build args (e.g. sentry's required SENTRY_IMAGE pin) were
silently dropped. Now assemble --build-arg from BOTH static args and KMS-sourced
build_secrets: a repo declares 'build_secrets: [NAME]' per image; the KMS step
fetches each NAME from the same org/path/env and exports it (masked) for the
build step. Undeclared => empty => buildx line byte-for-byte unchanged.
Unblocks world Satellite/Terrain: VITE_MAPBOX_TOKEN now bakes into the Vite SPA
from KMS (hanzo/deploy, env=prod) at build. Key name IS the build-arg name.
The build loop read name/ctx/df/repo/sfx/plats but never passed images[].args,
so a repo's declared build args (e.g. sentry's required SENTRY_IMAGE pin) were
silently dropped. Now assemble --build-arg from BOTH static args and KMS-sourced
build_secrets: a repo declares 'build_secrets: [NAME]' per image; the KMS step
fetches each NAME from the same org/path/env and exports it (masked) for the
build step. Undeclared => empty => buildx line byte-for-byte unchanged.
Unblocks world Satellite/Terrain: VITE_MAPBOX_TOKEN now bakes into the Vite SPA
from KMS (hanzo/deploy, env=prod) at build. Key name IS the build-arg name.
A per-job GITHUB_TOKEN or package-less GH_PAT can't push a package linked to
another repo (ghcr.io/hanzoai/cms 403). New step reads the org ghcr push token
(buildx-ghcr-auth, admin write:packages) from the cluster via the KMS kubeconfig
and re-logs ghcr with it — pushes/creates ANY org package. Fail-safe: public
forks with no KMS keep their repo-linked GITHUB_TOKEN. Native creds from KMS.
A per-job GITHUB_TOKEN or package-less GH_PAT can't push a package linked to
another repo (ghcr.io/hanzoai/cms 403). New step reads the org ghcr push token
(buildx-ghcr-auth, admin write:packages) from the cluster via the KMS kubeconfig
and re-logs ghcr with it — pushes/creates ANY org package. Fail-safe: public
forks with no KMS keep their repo-linked GITHUB_TOKEN. Native creds from KMS.
The automatic per-job GITHUB_TOKEN only writes a package linked to THIS repo, so
it 403s on a package created/linked elsewhere (ghcr.io/hanzoai/cms). Use the
KMS-backed GH_PAT (admin:org + write:packages) when present — it pushes/creates
any <org> package — and fall back to the automatic token for public forks that
have no GH_PAT. Native creds from KMS, not GitHub's scoped token.
The automatic per-job GITHUB_TOKEN only writes a package linked to THIS repo, so
it 403s on a package created/linked elsewhere (ghcr.io/hanzoai/cms). Use the
KMS-backed GH_PAT (admin:org + write:packages) when present — it pushes/creates
any <org> package — and fall back to the automatic token for public forks that
have no GH_PAT. Native creds from KMS, not GitHub's scoped token.
Both `docker login registry.hanzo.ai` calls ran unguarded under bash -e, so a
login FAILURE (registry down / transient) aborted the step and SKIPPED the whole
build — the image never reached GHCR (the primary). A missing cred was already
fail-safe; a failed login now is too: warn + skip the mirror, push GHCR-only.
Same best-effort principle as the cloud release lane.
Both `docker login registry.hanzo.ai` calls ran unguarded under bash -e, so a
login FAILURE (registry down / transient) aborted the step and SKIPPED the whole
build — the image never reached GHCR (the primary). A missing cred was already
fail-safe; a failed login now is too: warn + skip the mirror, push GHCR-only.
Same best-effort principle as the cloud release lane.
Deploy roll now pins spec.image.tag to the BARE semver (VER = ref_name w/o v)
on a tagged release — matching world 2.4.10 / cloud v1.801.62 — instead of the
sha-<short>-amd64 it hardcoded on every build. Branch/main pushes keep their
per-commit sha tag (continuous dev path), now arch-matched (bare sha for
multi-arch, fixing a latent -amd64 mismatch).
Build step also publishes the bare semver tag on tag builds (kept the -amd64
alias for back-compat). New semver backward-clobber guard: a branch build never
overwrites a service already pinned to a semver release — complements the
sha-ancestry guard (which only sees sha- tags), so neither kind of pin rolls
backward.
Emitter of the 'deploy(<svc>): sha-...' hanzo-ci commits. Consumed as @v1.
Deploy roll now pins spec.image.tag to the BARE semver (VER = ref_name w/o v)
on a tagged release — matching world 2.4.10 / cloud v1.801.62 — instead of the
sha-<short>-amd64 it hardcoded on every build. Branch/main pushes keep their
per-commit sha tag (continuous dev path), now arch-matched (bare sha for
multi-arch, fixing a latent -amd64 mismatch).
Build step also publishes the bare semver tag on tag builds (kept the -amd64
alias for back-compat). New semver backward-clobber guard: a branch build never
overwrites a service already pinned to a semver release — complements the
sha-ancestry guard (which only sees sha- tags), so neither kind of pin rolls
backward.
Emitter of the 'deploy(<svc>): sha-...' hanzo-ci commits. Consumed as @v1.
Builds finish out of order — a slow build of an older commit overwrote newer
rolls (observed live: studio pinned back one merge). The roll now skips when
the CR's sha is a descendant of the builder's. And the tag field alone left
same-image sidecars on stale tags every roll (reconciled by hand four times
tonight); every same-repo image reference in the CR now moves together.
Builds finish out of order — a slow build of an older commit overwrote newer
rolls (observed live: studio pinned back one merge). The roll now skips when
the CR's sha is a descendant of the builder's. And the tag field alone left
same-image sidecars on stale tags every roll (reconciled by hand four times
tonight); every same-repo image reference in the CR now moves together.
The reusable workflow provisions Go and C toolchains for hanzo.yml test
gates but never Node — any JS caller's gate (pnpm install && pnpm lint)
died at 'corepack: command not found' before reading package.json.
Mirror the Go step: guarded to package.json callers, setup-node 22 +
corepack enable so the repo's pinned packageManager shims resolve.
The reusable workflow provisions Go and C toolchains for hanzo.yml test
gates but never Node — any JS caller's gate (pnpm install && pnpm lint)
died at 'corepack: command not found' before reading package.json.
Mirror the Go step: guarded to package.json callers, setup-node 22 +
corepack enable so the repo's pinned packageManager shims resolve.
Public identity stays ghcr.io (GitHub imports keep working); every built tag
is also mirrored server-side (imagetools create) to OUR fleet registry so the
cluster never depends on GHCR to deploy. Credential = the cluster-synced
registry-credentials dockerconfig read via the KMS-fetched kubeconfig;
gracefully skips (GHCR-only) when unavailable. No rebuild, no extra minutes.
Public identity stays ghcr.io (GitHub imports keep working); every built tag
is also mirrored server-side (imagetools create) to OUR fleet registry so the
cluster never depends on GHCR to deploy. Credential = the cluster-synced
registry-credentials dockerconfig read via the KMS-fetched kubeconfig;
gracefully skips (GHCR-only) when unavailable. No rebuild, no extra minutes.
gitops-reconcile re-applies universe CRs every ~5min, so a CR patch (or
set-image) alone is reverted on the next cycle. The deploy step now
bumps infra/k8s/operator/crs/<svc>.yaml in universe (no-op-safe commit,
same KMS git token) and keeps the CR patch/set-image only to make the
roll immediate.
Claude-Session: https://claude.ai/code/session_01SpMZ69ur3tjAXCiwaa7Wv2
gitops-reconcile re-applies universe CRs every ~5min, so a CR patch (or
set-image) alone is reverted on the next cycle. The deploy step now
bumps infra/k8s/operator/crs/<svc>.yaml in universe (no-op-safe commit,
same KMS git token) and keeps the CR patch/set-image only to make the
roll immediate.
The hanzo operator reconciles Deployments from the Service CR — a bare
'kubectl set image' gets reverted on the next reconcile (observed on
world: rolled, served, reverted minutes later). Patch the CR's
spec.image when one exists; keep the Deployment fallback for
non-operator services.
Claude-Session: https://claude.ai/code/session_01SpMZ69ur3tjAXCiwaa7Wv2
The hanzo operator reconciles Deployments from the Service CR — a bare
'kubectl set image' gets reverted on the next reconcile (observed on
world: rolled, served, reverted minutes later). Patch the CR's
spec.image when one exists; keep the Deployment fallback for
non-operator services.
arc snapshot images intermittently lose archive.ubuntu.com Release files
(apt-get update exit 100 → every Go repo's build dies before its gates).
Repoint to the DO mirror (sources.list + deb822) and retry once; still a
no-op when gcc is baked in.
Claude-Session: https://claude.ai/code/session_01SpMZ69ur3tjAXCiwaa7Wv2
arc snapshot images intermittently lose archive.ubuntu.com Release files
(apt-get update exit 100 → every Go repo's build dies before its gates).
Repoint to the DO mirror (sources.list + deb822) and retry once; still a
no-op when gcc is baked in.
PyYAML can't install on arc nodes (sudo blocked by no_new_privs, no pip). Parse
hanzo.yml with a curl-installed static yq binary + jq instead. Proven by the cms
build (past provision→GHCR→KMS). Robust on bare AND pre-baked nodes.
PyYAML can't install on arc nodes (sudo blocked by no_new_privs, no pip). Parse
hanzo.yml with a curl-installed static yq binary + jq instead. Proven by the cms
build (past provision→GHCR→KMS). Robust on bare AND pre-baked nodes.
A hanzo.yml `test:` gate for a JS/TS repo (e.g. `corepack … && pnpm lint`) runs
directly on the runner, but the minimal arc runner image ships no user-PATH Node
(its bundled node is for the runner's own action execution only), so the gate died
`corepack: command not found` (exit 127) — e.g. hanzo.ai's lint gate.
Add a `Provision Node toolchain` step (the JS twin of the existing Go-toolchain
provision): `actions/setup-node@v4` pinned to LTS 22, guarded to repos with a
package.json (`hashFiles('package.json') != ''`) so non-JS callers are unaffected
and harmless if a future runner image bakes Node in. corepack then activates the
exact pnpm/npm the gate requests.
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
Co-authored-by: hanzo-dev <dev@hanzo.ai>
A hanzo.yml `test:` gate for a JS/TS repo (e.g. `corepack … && pnpm lint`) runs
directly on the runner, but the minimal arc runner image ships no user-PATH Node
(its bundled node is for the runner's own action execution only), so the gate died
`corepack: command not found` (exit 127) — e.g. hanzo.ai's lint gate.
Add a `Provision Node toolchain` step (the JS twin of the existing Go-toolchain
provision): `actions/setup-node@v4` pinned to LTS 22, guarded to repos with a
package.json (`hashFiles('package.json') != ''`) so non-JS callers are unaffected
and harmless if a future runner image bakes Node in. corepack then activates the
exact pnpm/npm the gate requests.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The reusable hard-subscripted `hanzo.yml['images']` in the delegate, buildx, and
deploy steps, so a repo that ships no container image (e.g. a static site deployed
via its own Cloudflare Pages deploy.yml, importing this reusable only for the
`test:` lint gate) failed with a Python KeyError before any build ran.
Use `.get('images') or []` in all three places: absent → empty list → the build
loop runs zero times and no GHCR push is attempted. Backward-compatible (every
repo with `images:` is unchanged) and it stops a cross-org repo (e.g.
hanzo-apps/hanzo.ai) from hitting `denied: permission_denied` on a vestigial push.
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The reusable hard-subscripted `hanzo.yml['images']` in the delegate, buildx, and
deploy steps, so a repo that ships no container image (e.g. a static site deployed
via its own Cloudflare Pages deploy.yml, importing this reusable only for the
`test:` lint gate) failed with a Python KeyError before any build ran.
Use `.get('images') or []` in all three places: absent → empty list → the build
loop runs zero times and no GHCR push is attempted. Backward-compatible (every
repo with `images:` is unchanged) and it stops a cross-org repo (e.g.
hanzo-apps/hanzo.ai) from hitting `denied: permission_denied` on a vestigial push.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The Test step runs `go vet`/`go test` on the runner, so `go` fetches
private hanzoai/* modules (GOPRIVATE → direct) via the runner's git. Repos
that authenticate builds with the KMS `gh_token` (GIT_TOKEN) rather than an
org GH_PAT had no runner-git credential, so the gate failed with
`fatal: could not read Username for 'https://github.com'` on
hanzoai/dbx, hanzoai/tasks, hanzoai/pubsub-go, etc.
Add a guarded step (before Test) that configures git `insteadOf` with the
SAME token the image build uses — GIT_TOKEN (set by the KMS step), GH_PAT
fallback — so the runner's git can clone private modules. Gated on go.mod;
no-op when no token is present.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Test step runs `go vet`/`go test` on the runner, so `go` fetches
private hanzoai/* modules (GOPRIVATE → direct) via the runner's git. Repos
that authenticate builds with the KMS `gh_token` (GIT_TOKEN) rather than an
org GH_PAT had no runner-git credential, so the gate failed with
`fatal: could not read Username for 'https://github.com'` on
hanzoai/dbx, hanzoai/tasks, hanzoai/pubsub-go, etc.
Add a guarded step (before Test) that configures git `insteadOf` with the
SAME token the image build uses — GIT_TOKEN (set by the KMS step), GH_PAT
fallback — so the runner's git can clone private modules. Gated on go.mod;
no-op when no token is present.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
hanzo.yml `test:` gates run directly on the runner (not in a build
container), but the stock arc runner image ships no Go and no C compiler.
Any Go test gate therefore died with `go: command not found` (exit 127) —
and CGO_ENABLED=1 gates would next hit `cgo: gcc not found`. This was
latent because most callers never reached the Test step (image build failed
first); hanzoai/commerce is the first to build clean and reach a Go gate.
Add two guarded steps before Test, mirroring the existing jq/PyYAML
provisioning: `actions/setup-go@v5` pinned to the repo's own go.mod version,
and a guarded gcc install. Both gated on `hashFiles('go.mod')` so pure JS/TS
callers are unaffected, and both no-op when the toolchain is already present.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
hanzo.yml `test:` gates run directly on the runner (not in a build
container), but the stock arc runner image ships no Go and no C compiler.
Any Go test gate therefore died with `go: command not found` (exit 127) —
and CGO_ENABLED=1 gates would next hit `cgo: gcc not found`. This was
latent because most callers never reached the Test step (image build failed
first); hanzoai/commerce is the first to build clean and reach a Go gate.
Add two guarded steps before Test, mirroring the existing jq/PyYAML
provisioning: `actions/setup-go@v5` pinned to the repo's own go.mod version,
and a guarded gcc install. Both gated on `hashFiles('go.mod')` so pure JS/TS
callers are unaffected, and both no-op when the toolchain is already present.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The KMS deploy-cred step read the org only from hanzo.yml kms.org or the
KMS_ORG var. No repo sets either, so ORG was always empty and the step
short-circuited with 'KMS not configured' — /v1/kms/auth/login was never
called. Every private-dep build then fell back to the org GH_PAT, which
cannot read private hanzoai/cloud, so go mod tidy failed with git exit 128
(commerce/ai/chat images never built; #70 enforcement could not deploy).
Derive ORG from github.repository_owner (hanzoai->hanzo, luxfi->lux,
zooai->zoo; owner as-is otherwise) when unset. hanzo.yml kms.org and
KMS_ORG still override. One place, zero per-repo config. The full path
(login -> token -> GET deploy/GITHUB_TOKEN -> read hanzoai/cloud) is
verified live against kms.hanzo.ai.
The KMS deploy-cred step read the org only from hanzo.yml kms.org or the
KMS_ORG var. No repo sets either, so ORG was always empty and the step
short-circuited with 'KMS not configured' — /v1/kms/auth/login was never
called. Every private-dep build then fell back to the org GH_PAT, which
cannot read private hanzoai/cloud, so go mod tidy failed with git exit 128
(commerce/ai/chat images never built; #70 enforcement could not deploy).
Derive ORG from github.repository_owner (hanzoai->hanzo, luxfi->lux,
zooai->zoo; owner as-is otherwise) when unset. hanzo.yml kms.org and
KMS_ORG still override. One place, zero per-repo config. The full path
(login -> token -> GET deploy/GITHUB_TOKEN -> read hanzoai/cloud) is
verified live against kms.hanzo.ai.
The default `mode: buildx` path is unchanged: buildx → test → deploy ON the
arc runner. `mode: delegate` instead POSTs each hanzo.yml image to platform's
direct build webhook (POST /v1/arcd/enqueue, bearer PLATFORM_BUILD_CALLBACK_TOKEN,
body {repo, sha, image, ref, branch, dockerfile, context, os, arch}) and exits
in seconds — platform builds in-cluster with BuildKit on its own pool, pushes to
the registry, and rolls the operator Service CR. Same downstream as platform's
GitHub-App webhook (one build path, two front doors); no runner buildx, no KMS,
no runner-side deploy.
- new `mode` workflow_call input (default `buildx`) — delegation is opt-in, so
existing repos are untouched.
- new "Delegate build to platform" step (mode == delegate): parses hanzo.yml,
enqueues one build per (image, platform) with the buildx tag shape the deploy
path expects (sha-<short>-<arch>[-<suffix>]); fails loud on a non-202.
- buildx/GHCR/KMS/test/deploy steps gated `mode != 'delegate'`; checkout + parse
toolchain still run (needed to read hanzo.yml).
- endpoint override via the PLATFORM_ENQUEUE_URL repo/org var.
- README documents the delegate opt-in.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The default `mode: buildx` path is unchanged: buildx → test → deploy ON the
arc runner. `mode: delegate` instead POSTs each hanzo.yml image to platform's
direct build webhook (POST /v1/arcd/enqueue, bearer PLATFORM_BUILD_CALLBACK_TOKEN,
body {repo, sha, image, ref, branch, dockerfile, context, os, arch}) and exits
in seconds — platform builds in-cluster with BuildKit on its own pool, pushes to
the registry, and rolls the operator Service CR. Same downstream as platform's
GitHub-App webhook (one build path, two front doors); no runner buildx, no KMS,
no runner-side deploy.
- new `mode` workflow_call input (default `buildx`) — delegation is opt-in, so
existing repos are untouched.
- new "Delegate build to platform" step (mode == delegate): parses hanzo.yml,
enqueues one build per (image, platform) with the buildx tag shape the deploy
path expects (sha-<short>-<arch>[-<suffix>]); fails loud on a non-202.
- buildx/GHCR/KMS/test/deploy steps gated `mode != 'delegate'`; checkout + parse
toolchain still run (needed to read hanzo.yml).
- endpoint override via the PLATFORM_ENQUEUE_URL repo/org var.
- README documents the delegate opt-in.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The buildx `gh_token` secret (for reading private cross-org Go modules like
github.com/hanzoai/cloud during `go mod tidy`) was mounted ONLY from the
KMS-fetched GIT_TOKEN. When the KMS deploy-cred login fails (or KMS is
unconfigured), GIT_TOKEN is empty, the --secret is omitted, and the buildx
stage's `go mod tidy` hits the private repo unauthenticated -> git exit 128 ->
image build fails. This broke every repo with a private cross-org dep
(hanzoai/commerce, hanzoai/ai) while the KMS login was down.
Fall back to the existing valid GH_PAT org secret — the SAME BuildKit gh_token
cloud/release.yml already uses successfully. Guarded + exported; no-op for
public-only builds when both are empty. One credential path, proven working.
DOKS has no arm64 nodes. Add per-image opt-in `platforms:` in hanzo.yml:
default stays [linux/amd64] (every existing repo's -amd64 tag shape UNCHANGED,
zero behavior change). Set [linux/amd64, linux/arm64] → buildx emits a
multi-arch manifest list (one digest, both arches); binfmt/QEMU installed for
arm64 emulation, and pure-Go CGO_ENABLED=0 Dockerfiles honoring $TARGETARCH
cross-compile natively (fast). For native-speed arm64, register a bare-metal
arm64 host (spark/GB10) as the hanzo-build-linux-arm64 runner (values exist).
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The step is documented BEST-EFFORT (GHCR push uses the workflow token;
deploy creds are optional), and its 'set -uo pipefail' deliberately omits
-e. But GitHub runs 'run:' under 'bash -eo pipefail', so errexit is active
regardless — an unguarded curl (a KMS secret 404 at the caller's org/path)
aborted the step (exit 22) and failed the whole build. Explicitly 'set +e'
so KMS degrades gracefully (missing kubeconfig -> deploy simply skipped).
The stock arc runner image (ghcr.io/actions/actions-runner:latest) is
minimal and ships neither jq nor python3-yaml, but the build step parses
hanzo.yml with python3+PyYAML under 'set -euo pipefail' and fails:
ModuleNotFoundError: No module named 'yaml'
(The KMS step swallowed the same error via '|| true'.)
Add a guarded setup step (apt-get python3-yaml jq) right after checkout —
idempotent, a no-op once a runner image bakes them in. Keeps the reusable
self-contained so any org can import it onto a bare runner.
luxfi/hanzoai/zooai Go modules are public (verified: dex, precompile, authz, pq,
age, keys all resolve unauthenticated). Setting GOPRIVATE for the whole orgs
routed them 'direct' and bypassed sum.golang.org, so a force-moved tag silently
poisoned every downstream go.sum (the recurring 'checksum mismatch' build
breakage). Dropping it lets go resolve them via proxy.golang.org + sumdb —
canonical IMMUTABLE hashes a re-publish cannot break. The GH_PAT git credential
stays for authenticated direct fallback + any genuinely-private repo (re-add via
a NARROW GOPRIVATE, never whole-org).
The reusable build read ."tag-suffix" unconditionally, so any repo without it
(every repo today) got broken tags like sha-xxx-amd64-null. Default to empty in
both build and deploy so single-variant repos get clean tags; multi-variant
repos still qualify (…-ce, ce-latest). Unblocks adopting the reusable workflow.
* build: KMS secret fetch over canonical /v1/kms + buildx gh_token secret
The Infisical /api/* surface was removed when KMS migrated to luxfi/kms, so the
old /api/v1/auth/universal-auth/login + /api/v3/secrets/raw calls now 404 and
every repo's CI fails to fetch GHCR_TOKEN/KUBECONFIG. Rewrite to /v1/kms:
- auth: POST /v1/kms/auth/login {clientId,clientSecret} (IAM client_credentials
via the <org>-kms app) -> accessToken
- fetch: per-secret GET /v1/kms/orgs/<org>/secrets/<path>/<name>?env=<env>
- org from hanzo.yml kms.org or KMS_ORG var
Also fetch GIT_TOKEN and pass it to buildx as the gh_token BuildKit secret so
Dockerfiles can clone private Go modules (luxfi/dex, luxfi/precompile). Empty-safe.
* build: one GITHUB_TOKEN in KMS for GHCR push + private-module clone (DRY)
A single GitHub PAT with repo + write:packages does both jobs, so collapse the
split GHCR_TOKEN/GIT_TOKEN to one KMS key GITHUB_TOKEN. KUBECONFIG stays separate
(different concern). Shell var GHTOKEN avoids the reserved GITHUB_ env prefix.
* build: GHCR push via automatic workflow token; KMS only for cross-org clone + kubeconfig
GitHub injects a per-job GITHUB_TOKEN scoped to the running repo; with
permissions: packages: write it can push the repo's own GHCR package (same-org),
so the push path needs no stored credential. KMS is now best-effort, supplying
only what the automatic token can't: a cross-org PAT to clone other orgs' private
Go modules (luxfi/dex, luxfi/precompile) + KUBECONFIG for deploy. Public repos
with no cross-org deps build with zero KMS dependency.
* ci: configure private Go module access (luxfi/hanzoai/zooai)
Any go-based `test:` step in a consumer's hanzo.yml needs to fetch
private github.com/{luxfi,hanzoai,zooai}/* modules; without auth go
hits 'git ls-remote … exit 128'. Add a guarded step that wires GH_PAT
via git insteadOf + sets GOPRIVATE job-wide. No-op when GH_PAT is
absent, so non-Go consumers (bootnode etc.) are unaffected.
* ci: make private-module auth runner-state-immune (fresh GIT_CONFIG_GLOBAL)
Upgrade the naive 'git config --global' to the robust pattern proven in
hanzoai/iam: fresh per-job GIT_CONFIG_GLOBAL + GIT_CONFIG_NOSYSTEM=1, probe
from a neutral dir. A plain global config is overridden on shared arc
runners by stale ~/.gitconfig state and actions/checkout's persisted
extraheader. Still a no-op without GH_PAT.
2026-06-22 17:21:04 -07:00
21 changed files with 4829 additions and 47 deletions
`kotlin-sdk`, `cpp-sdk`, `cli`) stop carrying eight copies of the same eight
lines.
```yaml
client:
spec:{repo:hanzoai/cloud, path:openapi.yaml } # these are the defaults
generate:./scripts/generate.sh # $SPEC is the fetched document
version:'package.json:jq -r .version package.json'# optional; see below
```
There is deliberately **no `build:`**. The repo already declared how it proves
itself, in `test:`, and that block runs over the regenerated tree — which is
exactly the gate. A second declaration would be one assertion written twice.
It fires on `repository_dispatch: spec-update`, which **hanzoai/cloud sends once
per release**:
```yaml
on:
repository_dispatch:{types:[spec-update] }
workflow_dispatch:
```
The coupler is the document, **passed by value at a pinned ref**. The payload
carries `(version, sha, spec_sha256)`; the lane fetches `openapi.yaml` at that
sha and refuses if the bytes hash to anything else — every projection of one
release is generated from one digest. Reading a live host instead would be a lie
about which deploy the client describes.
Three gates, in order:
| gate | refuses |
|---|---|
| digest | a client generated from a different document than its siblings |
| `test:` | a spec change that produces a client which does not compile — **including its examples** |
| `.spec-lock` | is committed beside the code: `ref` + `sha256`, so anyone can ask a client repo *which document are you?* without running a generator |
On a delta — and only after `test:` has passed over exactly those bytes — the
lane commits the projection, bumps the **patch** (derived, never typed: a
projection never earns a minor or a major) and pushes the tag. The repo's own tag
lane publishes it, so the registry credential stays where the publish is.
`version:` says **where this client's version lives**, because that answer is
genuinely different per language:
| value | meaning |
|---|---|
| `"<file>:<command printing it>"` | it lives in a file — rewrite it, commit, tag |
| `tag` | the tag **is** the version (a Go module has nothing to rewrite) |
| absent | CI cannot derive one — the projection is committed and gated, nothing is cut |
The third state is not a gap to fill later. A repo whose version is not `x.y.z`
(a `-alpha.N` gradle build) has no patch for this lane to derive, and guessing
one would tag bytes under a number nobody chose.
Credential: **`SPEC_TOKEN`** — a fine-grained token with `contents:read` on the
spec repo.
## `binaries:` — publish a plugin once, install it everywhere
`images:` ships an OCI image a **cluster** runs. `binaries:` ships an
executable a **running host** installs: a [zip](https://github.com/zap-proto/zip)
plugin, fetched at run time by URL and verified against its SHA-256 before it is
ever made executable. Build it once per OS/arch here; every host picks up the
same bits, and nobody rebuilds the world to ship a plugin.
# `golang:alpine`, `golang:1-alpine`, `ARG GO_VERSION=INVALID` — no version to
# compare. Say so once; do not guess and do not fail.
[ -z "${iM:-}" ] && { echo "gover: $df: '$ref' names no Go version — cannot check it against $gosrc ($govers)"; continue; }
if [ "$(num "$iM" "$im" "$ip")" -lt "$(num "$mM" "$mm" "$mp")" ]; then
echo "::error file=$df::Go builder image is older than the module it builds: '$ref' provides Go $iM.$im${ip:+.$ip} but $gosrc requires go $govers. The golang images set GOTOOLCHAIN=local, so this build fails with 'go.mod requires go >= $govers'. Fix: pin the base to golang:$govers-<variant>, and add 'ENV GOTOOLCHAIN=auto' to the builder stage so a future go.mod bump downloads the toolchain instead of failing."
rc=1
elif [ "$ip" -lt 0 ] && [ "$mp" -ge 0 ]; then
echo "::warning file=$df::'$ref' floats to the newest patch of $iM.$im, while $gosrc pins go $govers. It builds today and fails the moment a registry mirror serves an older patch. Pin golang:$govers-<variant> to make it hermetic."
echo "imgver: no version for $repo. Declare one — a package.json/Cargo.toml/VERSION/pyproject.toml under '$ctx', or IMGVER_VERSION. We don't ship shas." >&2
# Hanzo CI — this repo's own build, driven by this repo's own reusable workflow.
#
# hanzoai/ci is two things that belong together: the reusable pipeline every
# other repo imports (.hanzo/workflows/build.yml), and ci.hanzo.ai, the
# dashboard that shows what that pipeline did. So the dashboard image is built
# by the pipeline it reports on — if the pipeline breaks, the thing that would
# tell you cannot ship, which is the correct and honest coupling.
#
# Until now there was no self-build at all: build.yml is workflow_call-only, so
# the v0.1.0 image was produced out of band and there was no repeatable way to
# cut a second one.
images:
- name:ci
context:.
dockerfile:Dockerfile
repo:ghcr.io/hanzoai/ci
test:
- name:go-vet
run:|
set -e
export GOWORK=off
go vet ./...
- name:go-unit
# The whole tree, not a named list — an allowlist stops covering whatever is
# added after it is written. Small repo; ./... costs nothing.
#
# scope_test.go is the load-bearing one: it asserts the surface refuses a
# request with no X-Org-Id and that `?org=` can only narrow. This service
# shipped once with those properties absent and disclosed every org's build
# metadata to the internet, so a red gate here must block the image.
#
# render_test.go is the other one that has to stay green: it pins the
# vendored @hanzo/brand sheet to the hash of the version it claims to be and
# rejects any colour the page names for itself. Both gates are offline —
# checking that we use one design system costs this pipeline no npm, no
# registry and no network.
run:|
set -e
export GOWORK=off
go test -count=1 ./...
- name:build-yml-is-one-file
# The reusable pipeline is published at TWO paths because two forges read two
# directories — github.com only `.github/workflows`, git.hanzo.ai only
# `.hanzo/workflows`. That is one artifact spelled twice, and nothing until
# now asserted it: the `.hanzo` copy had drifted nine lines (a truncated
# second `on:` block and a duplicate `name:` key) and no reader could see it,
# because the only consumer of that copy is a forge no push from here reaches.
#
# The ONLY legitimate difference is the path each names for itself, so
# normalise that one spelling and demand byte equality of the rest. A gate
# that allowed "the important parts match" would be a gate that cannot say
# what important means.
run:|
set -e
norm() { sed 's|\.hanzo/workflows/build\.yml|.github/workflows/build.yml|g' "$1"; }
if ! diff -u <(norm .github/workflows/build.yml) <(norm .hanzo/workflows/build.yml); then
echo "::error::the two published copies of the reusable have diverged. They are one file at two paths — edit both, or the forge runs a pipeline github.com has never seen."
exit 1
fi
echo "OK: .github and .hanzo copies are one file ($(wc -l < .github/workflows/build.yml) lines)"
- name:imgver
# bin/imgver decides the version EVERY image in the fleet publishes — this
# workflow's build lane calls it, and so does the imgver composite action the
# repos that hand-roll their own deploy.yml use. A wrong number here is one
# tag covering two digests, which a node running imagePullPolicy:
# IfNotPresent never picks up and no reader can see. Offline and
# deterministic: the registry floor is injected, so it needs no network.
run:bash bin/imgver_test.sh
- name:gover
# bin/gover is the gate the build lane runs against every Dockerfile before
# it builds: it refuses a Go builder image older than the go.mod it
# compiles. That failure is not hypothetical — the golang images set
# GOTOOLCHAIN=local, so the mismatch is a hard mid-build death, and
# hanzoai/visor v1.108.16 shipped it. A sweep of the orgs found 54 more
# Dockerfiles already below their own go.mod.
#
# The gate is only worth having if it is exact in BOTH directions: a false
# negative lets the next visor through, and a false positive blocks a build
# that would have worked, which is how gates get skipped. So the suite pins
# the refusals AND the allowances — a newer image than the floor is fine, an
# alpine suffix is not a Go patch, a floating tag warns instead of failing,
# and a multi-module repo is judged by its NEAREST go.mod. Offline and
# deterministic: temp dirs, no registry, no network.
run:bash bin/gover_test.sh
# No `deploy:` ON PURPOSE. Rollout is a reviewed tag pin in hanzoai/universe
# (infra/k8s/operator/crs/ci.yaml), the same rule cloud and git follow: a
# pipeline that both builds and rolls itself out can put an unreviewed image on
# a public host, and cd.hanzo.ai's selfHeal would undo a direct patch anyway.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
zeekayandhanzo-dev
zandhanzo-dev