The gate calls hanzoai/ci [email protected] with linux-amd64 as the only
architecture that gates, and linux-arm64 as an advisory leg that runs and
reports without holding anything. The o11y pod is scheduled on amd64, and the
one shared arm64 node kept the release queued for hours behind other repos.
hanzo.yml states version 1.5.89. The two images' series had drifted apart
(o11y at 1.5.88, o11y-site at 1.5.82) and far ahead of the git tags (v1.5.75), so
a derived number differed per image and a git tag would have overwritten an
August image of the same name. imgver publishes the declared version once and
the next patch thereafter, for both images alike.
Co-authored-by: Hanzo Dev <[email protected]>
hanzoai/ci keeps the reusable at .github/workflows/build.yml — the one path
both planes resolve, since github.com reads a reusable only from there and the
forge reads it too. The v1 tag moved with it, so a caller still naming
.hanzo/workflows/build.yml asks for a blob that is not in the tree:
InsertRun: read hanzoai/ci@v1:.hanzo/workflows/build.yml: object does not exist
The caller itself stays at .hanzo/workflows/cicd.yml on purpose: github.com has
no runner for these labels, so a caller placed there could never be scheduled.
Co-authored-by: Hanzo Dev <[email protected]>
Four workflows with no `needs:` between them cannot order themselves, so
the gate and the publish raced instead. A red gate published anyway:
v1.5.61, v1.5.62 and v1.5.63 were each built and pushed on a commit whose
gate had already failed on that same commit, and production ran v1.5.62.
v1.5.61 b217afa gate 35417 FAILURE image 35419 success
v1.5.62 ceeb972 gate 35624 FAILURE image 35625 success <- shipped
v1.5.63 fac2358 gate 46116 FAILURE image 46117 success
`images:` moves into hanzo.yml so the shared lane's single sequential job
runs `test:` and only then builds -- one job, one order, nothing to wire.
Adding `needs:` across four workflows would have been a second mechanism
to keep in step.
The blocker this file recorded was stale: the shared lane already mounts
the `gh_token` BuildKit secret the Dockerfile needs for the private
hanzoai/sqlite and hanzoai/datastore-go modules, from KMS with the org
GH_PAT as fallback. Version metadata keeps arriving too, via the lane's
`args:` placeholders, so the binary still names its own release rather
than reporting its ARG default.
ci.yaml goes with them. It ran the same two commands as `test:`, and
checked out hanzoai/cloud at a pinned SHA for a
`replace github.com/hanzoai/cloud => ../cloud` that go.mod no longer has
-- a second gate that can go red for a reason unrelated to this repo is
how a fleet learns to ignore red.
Co-authored-by: Hanzo Dev <[email protected]>
github.com has no runner for hanzo-build-linux-amd64, so a caller under
.github/workflows is a gate that can never be scheduled. This is the ~7-line
caller on the plane that can (git.hanzo.ai git-runner fleet), pinned @v1.
Co-authored-by: Hanzo Dev <[email protected]>
Mount(app *zip.App, deps cloud.Deps) fused two separate things — WHAT the 367
routes are, and WHICH host's dependencies serve them. It read exactly one field
of that second argument, deps.Logger, on exactly one line, to announce itself.
That one line cost three things, each worse than the last:
- github.com/hanzoai/o11y required github.com/hanzoai/cloud, which requires
github.com/hanzoai/o11y. A module CYCLE, for a log line.
- Because the cycle made a whole-module resolve impossible in an image with no
cloud checkout, the Dockerfile scoped itself to one package to route around
it — and wrote the workaround down as if it were a design decision.
- So cmd/community, the binary the image runs, could not import the route
declarations without dragging the embedding host in. It never imported them.
353 typed ops, 353 published operations and 353 MCP tools were true of the
source and absent from the process.
The last one is the P0 and it was invisible to every proof the conversion had:
the tests mount the table directly inside the test process, which is exactly the
thing that cannot fail. An uncalled package-level func is legal Go — the package
built, the route arithmetic added up, and the shipped server published nothing.
Status codes could not see it either. /version answers 200 either way, because
the service's own registration answers it. The tell was the body of an UNKNOWN
path: text/plain "404 page not found", which is pkg/web/noopweb's http.NotFound
behind the console's terminal catch-all.
A ROUTE TABLE IS A VALUE. It takes the router it registers into, and nothing
else. The router already carries the host's logger — app.Logger() IS the value
Deps.Logger was — so the second argument never carried information the first
did not already have.
Mount(app *zip.App) error. cloud is gone from go.mod, and ~20 transitive
requirements go with it (hanzoai/{iam,commerce,authz,tasks,vfs,orm,ha,s3-go,…}).
Standing proof, and it must stay this way:
go list -deps ./cmd/community | grep hanzoai/cloud # EMPTY
go list -deps ./cmd/community | grep -x github.com/hanzoai/o11y # PRESENT
The second line is the one that matters. pkg/query-service/app's new publish()
mounts the ONE table onto the router the server serves, then calls app.Prepare()
— which is what turns the registry into doors: /.well-known/openapi.json, /docs,
/mcp, the by-name call plane. This server serves through Fiber().Listener rather
than zip's own Listen, so without that call the document would exist in the
process and answer on no port.
Order is the contract, twice. AFTER the service's own routes: Fiber matches in
registration order, so every path is still answered by the handler that has
always answered it — no relay hop, no buffered round-trip, and therefore
livetail, the long-poll and the chunked export still stream on this listener. The
table NAMES the surface here; it does not stand in front of it. BEFORE the
console's terminal catch-all, or the SPA would answer for Prepare's routes.
Three tests read this out of the router instead of asserting it about the source:
the served document publishes 353 operations as application/json, every path in
it is o11y's own surface, and an earlier registration outranks the declaration.
hanzoai/cloud keeps working with a mechanical call-site change —
o11y.Mount(a, deps) -> o11y.Mount(a) at apps/o11y/o11y.go — and cmd/cloud builds
clean against this tree. That change lands in cloud with its next o11y pin bump;
cloud's current pin (v1.5.41) is unaffected.
Co-authored-by: Hanzo Dev <[email protected]>
The ghcr login was fixed and the build still failed, one step later:
::warning::Cannot parse GitHub Actions Runtime Token: missing part #2
ERROR: failed to solve: failed to configure gha cache exporter:
token is malformed: token contains an invalid number of segments
`type=gha` resolves its backend from an Actions runtime token, and git.hanzo.ai
does not issue one buildx can parse. Because the exporter is configured as part
of the solve, that does not degrade to an uncached build -- it fails the BUILD.
A cache meant to save time was costing the entire job. Same family as
upload-artifact v4+, which the forge also refuses.
Registry cache instead, at the image each workflow already pushes, so no new
credential and no new home. ignore-error=true on both ends: the first build has
no cache image yet, and a 404 there must not be fatal -- that is the second
failure visible in luxfi/dex today.
Co-authored-by: Hanzo Dev <[email protected]>
Both image workflows logged in with `secrets.GH_PAT || secrets.GITHUB_TOKEN`.
That resolves today only because orgs/hanzoai happens to provision GH_PAT -- an
org secret that is absent evaluates to "", so the chain silently swaps to the
git.hanzo.ai token, which ghcr.io rejects with `denied: denied`. The log never
says which credential was used, so the failure reads as a registry problem; that
is how the same defect in luxfi/trader stayed misdiagnosed until the token was
tested directly against the registry.
GHCR_USER + GHCR_TOKEN, named directly. The build secret below is untouched:
`gh_token=${{ secrets.GH_PAT }}` is the credential the Dockerfile mounts to
fetch private hanzoai modules, which is a different job from logging in.
Co-authored-by: Hanzo Dev <[email protected]>
o11y wrote the same server down twice. Emitter A is the typed op registry:
identity 45 + infra 44 + logs 9 + telemetry 5 = 103 ops registered on the
cloud.Router at their full public /v1/o11y/... paths, read from a real mount by
cloud's describe.go and woven into the one published document.
Emitter B was `o11y generate openapi`: a swaggest reflector walked the runtime's
gorilla/mux tree and wrote a committed 629 KB docs/api/openapi.yml — 120 paths,
174 ops, spelled /api/v1 x59, /api/v2 x57, /api/v3, /api/v4, /api/v5 x2, and
ZERO /v1/o11y. It described the fork's internal tree behind the delegation
wildcard's /v1/o11y/* -> /api/* rewrite: a surface no customer can address, in a
shape that breaks two house rules on its face (/api/, and v2 through v5). No CI
step regenerated or verified it, so it could only drift.
B dies whole: cmd/openapi.go, registerGenerateOpenAPI, pkg/o11y/openapi.go (the
reflector, unreachable once the command went) and the committed artifact.
pkg/http/handler's OpenAPIDef stays — it is per-handler prose the typed ops read,
not a second document.
Two scratch leaks from the same effort go with it:
- zzz_surface_measure_test.go, self-labelled "TEMPORARY … not for commit",
committed anyway and writing to a hardcoded /home/z/.cache path.
- replace github.com/hanzoai/cloud => ../cloud. No CI runner has that sibling,
so ci.yaml carried a second checkout of cloud at an exact commit to supply
it — the same fact in two places, which is what LLM.md's own section warned
about. The replace hid a broken pin, too: the required v1.801.218 no longer
compiles against current hanzoai/tasks, and nothing noticed because CI never
built the required version. go.mod now requires cloud v1.801.350 from the
proxy like every other private hanzoai module; the checkout step is gone.
go build ./... clean; go test ./ ./cmd/... ./pkg/http/handler/... ./pkg/apiserver/...
./pkg/o11y/... all ok.
Co-authored-by: Hanzo Dev <[email protected]>
The three workflows that build this repo were deleted from .github/workflows
without a replacement anywhere, so the promoted o11y-site App had nothing left
that could build its image — and no red run to show it, because a workflow that
does not exist cannot fail. Restored byte-for-byte from history at the path the
forge reads:
ci.yaml go build + go test, with the pinned cloud sibling
docker.yaml ghcr.io/hanzoai/o11y (headless server)
docker-site.yaml ghcr.io/hanzoai/o11y-site (SPA, pinned by crs/o11y-site.yaml)
runs-on is untouched — the forge's runner answers the same labels. The only edit
is docker-site.yaml's paths filter, which pointed at its own old location and
would never have matched again.
release-drafter.yml is not restored: it drafts GitHub releases off an upstream
SigNoz config, which means nothing once GitHub is a mirror.
.github/workflows now holds exactly one file, sync.yml, replaced with the
canonical copy — the variant here predated it and drifted.
Co-authored-by: Hanzo Dev <[email protected]>
zeekayandHanzo Dev