ghcr.io/hanzoai/static served only linux/amd64 from v0.5.1 through 0.5.14,
so an image built on it has no manifest an arm64 node can pull. 0.5.15 is
the same server source published as an index over both architectures.
Co-authored-by: Hanzo Dev <[email protected]>
Nothing on the host was compressed. Measured against the live site with
Accept-Encoding: gzip, deflate, br, zstd, every response came back identity:
HTML, CSS, JS, .txt and .xml alike. The index is 1,527,061 bytes raw; /docs is
1,527,061; /sitemap.xml 454,222; /llms.txt 205,265; one RSC prefetch
(/docs/games/__next.docs.txt) 1,029,280.
The cause is the base image. hanzoai/static had no content-encoding path until
900bccf, which shipped as v0.5.6; this Dockerfile pinned v0.5.1.
Verified by building the v0.5.6 binary and serving this site's own index
through it, rather than trusting the tag:
identity 1,527,061 bytes, no Content-Encoding
Accept gzip 213,911 bytes, Content-Encoding: gzip, Vary: Accept-Encoding
banner.png passed through, no Content-Encoding
gzip;q=0 1,527,061 bytes - the one spelling that means no
1.31 MB off every cold load of the front page.
Co-authored-by: Hanzo Dev <[email protected]>
The value is a pk- prefixed PUBLISHABLE key -- the Stripe vocabulary, and the
thing the Dockerfile gate already tests for. Naming it EVENT_INGEST_KEY hid
that: the old gate read 'EVENT_INGEST_KEY is not a publishable key', naming
one thing two ways in a single sentence.
One substitution covers all three spellings, since the framework prefixes
wrap the same token:
EVENT_INGEST_KEY -> PUBLISHABLE_KEY (KMS name + build-arg)
NEXT_PUBLIC_EVENT_INGEST_KEY -> NEXT_PUBLIC_PUBLISHABLE_KEY (Next inlines NEXT_PUBLIC_*)
VITE_EVENT_INGEST_KEY -> VITE_PUBLISHABLE_KEY (Vite inlines VITE_*)
KMS already carries deploy/PUBLISHABLE_KEY (env prod) with the same value,
written and read back before this landed, so no build can reach a name that
does not exist yet. The old KMS entry stays until every repo is renamed, then
it goes -- one name, no alias.
Co-authored-by: Hanzo Dev <[email protected]>
The build fails on disk, not on code: run 24173 completed all 21 turbo tasks and
all 3193 pages and was then evicted, EmptyDir "docker-storage" exceeds 38Gi with
MemoryPressure=False. Trimming the stored layer does not reach that — the quota
measures the PEAK during the RUN, where node_modules, .next, out and the turbo
cache necessarily coexist before any cleanup runs.
So reduce the working set instead of the output. This repo is 21 apps and 36
packages; apps/docs depends on 11 of them. `turbo prune docs --docker` emits a
workspace of docs plus its transitive internal deps, so the other 46 never land
on the runner at all. Same source, same lockfile — the image is unchanged; what
changes is how much of the monorepo exists on disk while it builds.
Root scripts are not part of a pruned workspace, so scripts/ is copied
explicitly — otherwise the export gate would vanish, which is exactly the kind of
silent hole the gate exists to prevent.
NOT verified by a local build: images are built by CI, not here. The failure mode
if this is wrong is the failure mode today — an evicted or failed run, no image.
Both gates survive the change (export + ingest key), so a broken artifact still
cannot reach the registry.
Five commits lived only on git.hanzo.ai. They carry what made the lane run --
the has_actions row, .dockerignore (context 16GB -> 323MB), in-layer cache
removal, a reclaim step, and folding install+build+clean into ONE layer so
node_modules never persists into a layer.
Kept here because they are independently correct at any volume size, and because
a pull mirror force-syncs: re-mirroring while the forge was ahead would have
destroyed them.
Dockerfile conflict resolved keeping BOTH. The gate runs before the build chain,
so a keyless build is refused before it spends 38GiB discovering it cannot fit.
Their ARG/ENV pair is replaced by the gated EVENT_INGEST_KEY form, which is the
name the KMS entry and both lanes use.
The disk work is NOT sufficient: run 24173 completed 21/21 turbo tasks and all
3193 pages and was then evicted (EmptyDir exceeds 38Gi, MemoryPressure=False).
The && chain shrinks the stored layer; the quota measures the peak during the
RUN. That is a sizing decision, not another edit.
A guard in deploy.yml protects one of six lanes. The image live right now is
tagged ae51c3fc — a SHA that exists only on the forge, whose forge run FAILED —
so something else built it, the fail-closed KMS step never ran, and the bundle
shipped with no key. Proven by streaming the 2.7GB layer: zero occurrences.
This is the shape the export gate already solved by living in the Dockerfile
rather than in a workflow. Same treatment: an empty or non-pk- key now refuses
the artifact instead of producing a hollow one, for every builder, without any
of them knowing this repo is special.
Both lanes are wired to supply it, or the gate would simply break docs:
- forge lane already fetches from KMS; its --build-arg is renamed to
EVENT_INGEST_KEY to match the ARG.
- ci lane now declares build_secrets: [EVENT_INGEST_KEY]; the reusable fetches
that name from KMS and passes it as a build-arg of the same name, and since
hanzoai/ci 5750426 it fails closed when a declared secret cannot be read.
The prefix stays a property of this build: EVENT_INGEST_KEY is the one name in
KMS, NEXT_PUBLIC_ is added here because that is what makes Next inline it.
Gate verified on all three branches: pk- exits 0, empty exits 1, wrong prefix
exits 1.
The previous pass fixed the right resource and not enough of it. The build now
finishes -- 21/21 tasks, 8m59s -- and the pod is still evicted immediately
after, during the export:
git-runner-8 Evicted: Usage of EmptyDir volume "docker-storage"
exceeds the limit "38Gi"
What was left is the biggest item: node_modules for a 21-app / 36-package
workspace, installed in its own RUN. A RUN is a layer and a layer stores what
existed when it ended, so deleting node_modules later frees nothing -- it stays
on disk for the whole build. Chaining install, build and cleanup with && means
it never outlives its own layer.
Also drops the pnpm store and ~/.cache, which the earlier cleanup missed.
Nothing downstream needs the toolchain: the export gate is pure sh over out/,
and the serving stage copies out/ alone.
Co-authored-by: Hanzo Dev <[email protected]>
I had this wrong in the previous commit and am reverting that part. The build
does not OOM. The kubelet evicts the runner pod:
git-runner-5 Evicted: Usage of EmptyDir volume "docker-storage"
exceeds the limit "38Gi"
git-runner-7 Evicted: node was low on resource: ephemeral-storage
MemoryPressure=False, node memory usage 7%
Eviction restarts the pod, which restarts dockerd, which is what I read as an
OOM kill. Both look identical from the job log -- it truncates mid-export with
no error and the run sits in_progress -- so the only way to tell them apart is
to ask the kubelet, which I should have done first.
Reverts the worker/heap bound: memory was never the constraint, and lowering
the heap only made the build slower and hold its disk for longer.
Fixes the actual constraint, in the order they matter:
- .dockerignore, which did not exist, so COPY . . shipped the entire tree.
A tree that has ever been built locally carries ~16G of artifacts that the
build immediately regenerates. Context: 16G -> 323M.
- rm -rf .next and the turbo cache in the SAME RUN that creates them, ~9G.
A later RUN would free nothing; the earlier layer still stores the files.
- a reclaim step that prunes the runner and prints df before and after, so
the next failure of this kind is legible in its own log.
The pool is still over-subscribed -- two runners per node, each with a 38Gi
emptyDir, on a ~98G disk -- and that is a universe change, not a docs one.
Co-authored-by: Hanzo Dev <[email protected]>
The docs build has been failing on the CI fleet for weeks in a way that reads as
a broken Dockerfile: the log stops mid-export with no error, no failure marker,
and the forge keeps reporting the job as running.
It is not the build. Nothing inside the container can see the pod's limits --
docker info reports the node's 33.6GB and nproc reports the node's 8 cores,
because the build container's cgroup carries neither. So Next forked 7 static
generation workers, each inheriting --max-old-space-size=24576, against a 26Gi
pod. That does not OOM the build; it OOMs the runner's own dockerd, which takes
the build container with it. Hence the silence.
Bound it: 3 workers plus the parent at 5120 MiB each is a 20GiB ceiling inside
26Gi. Same bound the fleet already puts on Go (-p=2) and Rust
(CARGO_BUILD_JOBS=4); Node was the toolchain nobody had capped.
Local builds are unchanged -- NEXT_BUILD_CPUS is unset outside the Dockerfile.
Co-authored-by: Hanzo Dev <[email protected]>
The build declared no ARG for it and the workflow passed none, so the bundle
shipped keyless. Cloud then files every pageview under the reserved $public
tenant -- pageview and error only, and our own org cannot read it. Ingest answers
200 either way, so the loss was silent: docs was the largest single contributor
to $public (21 of 30 events measured) and none of it was visible to us.
Next inlines NEXT_PUBLIC_* at build time, so this cannot be delivered as pod env.
The value is publishable and write-only by design; shipping it in a bundle is the
documented use.
docs.hanzo.ai could not be republished. This makes every path that could do it
real or explicitly named, and moves the one check that matters somewhere every
path passes through.
The export gate is now in the Dockerfile. This site fails by exporting NOTHING:
the build succeeds, the layer is valid, the push succeeds, the host answers 404,
and no builder notices because from a builder's point of view nothing went wrong.
It used to be gated between build and push in deploy.yml, which only works for
deploy.yml — buildx builds and pushes in a single invocation with no step in
between to hold, and hanzo.yml's `test:` block runs AFTER that push, so neither
could have held it back. scripts/check-export.sh states what a site is, once;
apps/<app>/export.require names the sections that vanish without breaking
anything else. A failed gate means no image exists, so no lane can push past it.
deploy.yml keeps its own call of the same script against the finished image,
which answers the one question the build stage cannot: did the export land in the
IMAGE. Tested against seven exports — healthy, empty, nav-less, under the page
floor, section missing, absent directory, default require-file resolution — under
both dash and busybox ash.
docs/studio/ is the case that motivates the require file. It is a submodule, and
hanzoai/ci checks out without recursing, so on that lane the section silently
disappears while page count and nav both stay green.
.github/workflows/cicd.yml is seven lines importing hanzoai/ci, configured by the
root hanzo.yml. It is workflow_dispatch-only: give this repo two push-triggered
builders for one image and one commit yields two images under two tag schemes,
and the pin names one of them by luck. It names our own runner pool, the same
label the forge lane answers to. It cannot schedule yet, and the reason is not
Actions — Actions is on and the repo is public; every runner we own is registered
to git.hanzo.ai, and GitHub-hosted runners are not something we build on.
RELEASE.md is the runbook: four builders, the credential each needs, and the
exact edit that makes an image live. The finding worth repeating here is that
universe pins a digest beside the tag and the chart renders repo:tag@digest — so
the digest is what gets pulled, and moving the tag alone is a silent no-op.
Verified against the live systems rather than the docs: the pin resolves to the
digest the values file names; no image exists for current main in ANY of the
three tag shapes, so all lanes are cold rather than one quietly working; the
forge repo answers 404 on /actions while another repo on the same forge answers
200; /v1/runner answers 401 with the token configured on the far side, so it is
armed and one credential away.
Co-authored-by: Hanzo Dev <[email protected]>
.github/workflows held 19 files; it holds sync.yml alone now, the canonical
push-refs workflow from universe infra/ci/sync.yml. What it replaced was a
mirror-sync nudge documenting GitHub as upstream and asking the forge to pull.
Thirteen workflows moved into .hanzo/workflows unchanged — byte-identical, so
git records them as pure renames. runs-on is untouched: the forge's runner
answers hanzo-docs-build-linux-amd64. That covers the only checks (lint, test),
the only publisher of the ~30 npm packages (release), the daily pricing commit
(sync-zen-pricing), and the nine sibling static sites, each of which is the
ONLY deploy of its host. Those nine still reach Cloudflare Pages; they move to
images one host at a time, because cf-zones declares only three of the nine and
an App CR with a guessed host is wrong even while inert. LLM.md carries the
list and the steps.
deploy-docs.yml becomes deploy.yml and only builds and pushes. It had patched
`app docs` with kubectl and waited on the rollout, and it built the image from
a heredoc Dockerfile pinned to static:0.4.1 while the repo's own Dockerfile —
the one the fabric builds — pinned v0.5.1. The recipe lives in the Dockerfile
alone now; the workflow runs it, gates the export by reading /public out of the
built image so what is checked is what serves, and pushes only if the gate
passes. ghcr auth is REGISTRY_TOKEN, since a forge-issued GITHUB_TOKEN
authenticates nothing there.
The Dockerfile takes APP, so every static site in the monorepo builds one way
and the nine siblings need no new Dockerfile when they flip.
Deleted four workflows that could not have worked: deploy-liquid-docs and
deploy-zap-docs build apps/liquid and apps/zap-docs, neither of which exists
(liquid's Cloudflare job was already `if: false`), and fix-cf-domain and
setup-gui-domain are dispatch-only one-shots calling the decommissioned
Infisical /api/v3/secrets/raw route.
Also corrects three changesets that named packages absent from the workspace,
which made `changeset version` fail and release.yml publish nothing:
@hanzo/zen-models -> @zenlm/models (sync-zen-pricing.yml wrote that name every
day), and fumadocs-mdx / @fumadocs/content -> @hanzo/docs-mdx /
@hanzo/docs-content per the rename rule in LLM.md.
Co-authored-by: Hanzo Dev <[email protected]>
Our own docs branded a Hanzo product as an 'AI Gateway' (/docs/llm title+H1,
skills index, and ~90 files across docs/bot-docs/cloud). Renamed to 'AI API'
to match hanzo.ai/app/bot. Third-party product names are facts about someone
else's product and are NOT rewritten: 'Vercel AI Gateway' restored verbatim
(same carve-out as LiteLLM); HanzoBot's own 'Gateway' transport untouched.
Pins ghcr.io/hanzoai/static 0.4.1 -> v0.5.1: only v0.5.1 carries 79ac4f9
(serve directory index in place), so docs was the last host 301ing /docs/x/
-> /docs/x/index.html, which poisons Next's router into prefetching
.../index.html/__next._tree.txt — 8 failed requests per page load.
The native docs pod CrashLoopBackOff'd: the 0.4.1 static binary has no -404 flag
(flags are -port/-root/-s3-*/-spa), so '/static ... -404 404.html' printed usage and
exited. Serve with '/static -port 3000 -root /public'. Image pulls + starts clean now.
The native docs image build failed at 'ghcr.io/hanzoai/static:v0.3.1: not found'.
hanzoai/static is public, but there is no v0.3.1 tag — the real tags are 0.3.0 /
0.4.0 / 0.4.1 / latest (no 'v' prefix). Pin the base to 0.4.1 (newest multi-arch
stable) in both the repo Dockerfile and the CI Dockerfile.serve. With this + the bun
fix + public bases, ghcr.io/hanzoai/docs finally builds with real content for the
docs-landing native serve.
The platform-native docs image (ghcr.io/hanzoai/docs, served by hanzoai/static behind
the docs-landing ingress) shipped with an EMPTY /public — every path 404'd. Cause: the
build stage runs 'pnpm build --filter=docs' → build:pre → 'bun ./scripts/*.ts', but
node:22-alpine has no bun, so the Fumadocs export silently produced nothing. Bring the
musl bun binary in from oven/bun:1-alpine (+ libstdc++/libgcc). Same root cause as the
CI bun fix — this is the native-PaaS path (no Cloudflare) for docs.hanzo.ai.
One root Dockerfile the platform fabric (/v1/runner buildkit) builds: pnpm
NEXT_EXPORT=1 static export → ghcr.io/hanzoai/static:v0.3.1 runtime
(clean-URL .html fallback, -404 404.html, :3000). Replaces the stale
apps/docs nginx wrapper (nginx is banned from the stack) and the retired
GitHub-Actions → Cloudflare Pages deploy path.
zeekayandHanzo Dev