Compare commits

...
Author SHA1 Message Date
zeekay ddd3df4675 Merge the GitHub line: one history, so a tag move survives the sync
Hanzo CI/CD / cicd (push) Failing after 9s
CI/CD / gate (push) Failing after 10s
hanzoai/ci had two diverged mains — five commits only on github.com (the
vendormark and site-zip-publish lines) and seven only on git.hanzo.ai (the
delegate gate, the publishable rule, the registry-credential fix). Their
merge-base was f098b39, which is exactly where v1 and v2 still pointed, so the
drift was invisible: both tags named the last commit the two lines agreed on.

That is what reverted the first attempt at this. Moving v1/v2 on the forge alone
held for three minutes and was then pushed back to f098b39 by the sync that
mirrors github.com into the forge — runs 101 and 102 are the tags reappearing at
the old commit. A tag is a ref like any other; if one side of a mirrored pair
does not have the commit, the sync restores what it does have. So the histories
have to converge before the tag can move anywhere.

The single conflict was two correct changes to the same lines, and both are
kept. The registry credential is fetched UNCONDITIONALLY (this line): it is not
an object-store credential, and a repo declaring only `images:` was reaching
docker login with two empty strings. The object-store gate narrows to `bucket:`
only (the github line): the site lane no longer writes to S3 — it hands the zip
to cloud — so it no longer needs a key that is not in KMS for any org. Taking
either side alone would have re-broken the other.

Under `mode: buildx` all 24 steps still have effective conditions identical to
f098b39. All ten gates pass, including build-yml-is-one-file, which is the gate
that caught 8a415e9 patching only the .github copy.
2026-08-06 14:16:39 -07:00
zeekay b047a7a569 build: the forge lane is the lane that runs
Hanzo CI/CD / cicd (push) Successful in 1m25s
CI/CD / gate (push) Successful in 1m26s
8a415e9 moved the REGISTRY_USER/REGISTRY_PASSWORD fetch out of the
`bucket:`/`site:` gate in .github/workflows/build.yml and left
.hanzo/workflows/build.yml untouched. Those two files are one artifact spelled
twice — github.com reads only `.github/workflows`, git.hanzo.ai reads only
`.hanzo/workflows` — and this repo's own `build-yml-is-one-file` gate exists to
refuse exactly this. It would have caught it: the gate passes at a830325 and
fails at 8a415e9.

The consequence is the inverse of the fix's intent. The copy that got the fix is
read by the forge nobody builds on; the copy the runners actually execute still
asks KMS for the registry credential only when a repo declares a bucket or a
site. So a repo declaring only `images:` — the common shape — still arrives at
`docker login registry.hanzo.ai` with two empty strings, and deleting the
REGISTRY_USER/REGISTRY_PASSWORD org secrets on the strength of 8a415e9 would
have broken every image build in three orgs rather than none.

They were byte-identical at a830325 (md5 1c027d00), and the only difference the
gate tolerates is a self-naming path that neither copy actually spells
differently, so the mirror is a literal copy.

No condition changes: under `mode: buildx` all 24 steps have effective
conditions identical to f098b39, which is what the v1 tag points at today.
2026-08-06 12:52:27 -07:00
hanzo-dev 8a415e9745 build: the registry credential is not an object-store credential
Hanzo CI/CD / cicd (push) Failing after 2m17s
CI/CD / gate (push) Failing after 2m18s
REGISTRY_USER/REGISTRY_PASSWORD were fetched from KMS only inside the
`bucket:`/`site:` gate — a gate about writing bytes to hanzoai/s3, which has
nothing to do with pushing an image. A repo declaring only `images:` therefore
asked KMS for nothing, arrived at `docker login registry.hanzo.ai` with two
empty strings, and failed at the push with no error pointing anywhere near this
step.

hanzoai/bot is the repo that found it, and it will not be the last: `images:`
without `bucket:`/`site:` is the common shape, not the exotic one.

The tempting fix is to paste the two names into the forge's org secrets, and
that is precisely the CI-only copy the comment four lines down warns against —
"a second thing to rotate and the one nobody remembers". The contract is that
the ONLY things living in the forge are KMS_CLIENT_ID/SECRET. So the fetch
moves out of the gate instead of the credential moving into GitHub.

Unconditional is safe: `get` returns empty when KMS has no such key, nothing is
exported, and the login step still falls back to the kubeconfig read for repos
provisioned before these names existed. The S3 keys stay gated, because those
really are specific to the two lanes that write to a bucket.
2026-08-06 12:35:47 -07:00
zandGitHub f7c2309483 Merge pull request #11 from hanzoai/ci/site-zip-publish
the site lane publishes with a credential that exists
2026-08-06 02:10:27 -07:00
zandGitHub 33ceb2c495 Merge pull request #10 from hanzoai/ci/vendormark
refuse a site-generator vendor's branding, in all four places it hides
2026-08-06 02:10:23 -07:00
zeekayandhanzo-dev 9a6647cd30 the site lane publishes with a credential that exists
`site:` has never worked. It staged the export with `mc mirror` behind
S3_ADMIN_ACCESS_KEY/S3_ADMIN_SECRET_KEY read from KMS, and those names are in
KMS for no org — so the lane's fail-closed check refused every caller that
declared one. Of the 155 hanzo.yml files across the three orgs, not one
declares a `site:`. It was dead code, and hanzoai/console grew its own
build -> zip -> publish workflow to ship at all, which is the per-repo build
logic this contract exists to prevent.

The fix is not to provision that credential. It is to stop needing it: the zip
goes to /v1/projects/<slug>/deploy and the landed prefix is promoted by
/v1/sites/<slug>/publish, both on the IAM bearer the KMS step already mints.
CI names no bucket and no org — sitePrefix(org, slug) is server-side from the
validated principal, which is what makes the prefix unforgeable.

The work is bin/sitepublish, not forty inlined lines nobody can run offline.
It verifies rather than assumes: publish returning 200 says the request was
accepted, not that the release is serving, so it re-reads the list and refuses
unless the release it just published is the one that is live.

Every assertion is written to fail when it finds nothing, because three real
defects had the opposite shape — a success case that included "I resolved
nothing":

  * an object-shaped read of a body that is not an object. `.releaseId // empty`
    yields "" for an array, a null or an error body, and every later test on ""
    passes vacuously. The shape is asserted first.
  * `.releases[]` against GET /v1/sites/<slug>/releases, which is a BARE ARRAY
    (release.go:518, `type projectsReleases []projectsRelease`). That filter
    resolves nothing, so the check could only ever refuse — it never once
    tested what it claimed. Hence `.[]`, behind a `type == "array"` guard so
    that if cloud ever wraps the list THAT goes red, by name.
  * `[ "$a" = "$b" ]` on two empty strings, which is TRUE. Non-emptiness is
    proven before the comparison, and actives are COUNTED, so "none active"
    cannot read as "yes".

Reintroducing those three turns 3, 9 and 1 of the suite's tests red.

The size boundary is the server's, not a preference: the edge caps a body at
16 MiB (GATEWAY_BODY_LIMIT) and fasthttp refuses a larger POST before any
handler runs, reporting only "Error when parsing request". So the zip is
measured and refused here, with the number and the alternative. Of the 24 built
exports in the estate 22 fit; hanzo.ai (27.9 MiB zipped, 8536 files — also past
the server's own 5000-entry cap) and trillerfest.com (76.7 MiB) go to
bin/sitedeploy, which streams per-file against a presigned grant.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-05 23:29:20 -07:00
hanzo-dev 0539ecf756 refuse a site-generator vendor's branding, in all four places it hides
Several sites in the estate began as output from a hosted "prompt to
React app" generator. It does not sign its work in a comment you would
notice; it signs it in the shape of the repo, four ways at once: an asset
folder named after the vendor, a build plugin in package.json, an
injected third-party <script>, and the commit messages, author identities
and Co-authored-by trailers of its bot.

A sweep of 1,065 repositories found the mark in five shipped sites. One
served the vendor's upload folder as its FAVICON, so the vendor's path
was in the <head> of every page and in every browser tab. Three have been
cleared; that took a history rewrite and a force-push of every branch.
This gate is so that is never needed a second time.

The fourth place is the one nothing else looked at and the one that
survives any amount of tidying: a repo can render a perfect page and
still say in `git log` who really wrote it. A commit-msg hook on one
workstation already strips these trailers, but that hook does not run on
a runner, on a teammate's laptop, or on a commit made through the web UI.
This does.

Content is judged narrowly and paths are not. The vendor's name is also
an ordinary English adjective, and the naive gate is unshippable: across
those same 1,065 repos the bare word appears innocently in tokenizer
vocabularies, word lists, ML corpora, packed public-suffix data, a
disposable-email blocklist and an upstream Clerk bundle listing other
people's dev hosts. A gate that reds a fifth of the estate is a gate
someone switches off. So the content rules are only strings that cannot
occur in prose, and two candidate rules were measured and dropped for
redding that Clerk bundle. Paths, identities and trailers are refused on
the bare word, because there no third party can put it and no tidying
reaches it.

Measured: red on exactly 4 of 1,219 checkouts, every one a true positive.
17/17 in the suite, which pins the allowances as well as the refusals.
.vendormark-allow exists for the one honest case — the history-scrubber
rule files in hanzoai/.github, which cannot do their work without
spelling the string they remove.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-05 16:56:10 -07:00
zeekayandhanzo-dev bd06e29b29 ci: @v1, not @v2 — v2 is an alias for the v1 line
hanzoai/ci is tagged v1.0.32..v1.0.35 plus a bare floating `v2` that points
at the SAME commit as v1.0.35, so v2 is not a second major — it is another
name for the line already in use. The fleet was split 13 @v1 / 7 @v2 over one
thing, which is two ways to say one version.

Runs on git.hanzo.ai's own act_runner pool (git-runner, 10 replicas). Not ARC.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-05 15:57:25 -07:00
9 changed files with 989 additions and 343 deletions
+113 -171
View File
@@ -177,11 +177,23 @@ jobs:
# on they are ordinary content: status clean, diff empty, both sides
# of the merge shipping. It hides in generated and vendored files,
# which are large, rarely opened and effectively unreviewed —
# hanzoai/base carried `<<<<<<< HEAD` / `>>>>>>> upstream/master` at
# hanzoai/base carried ` / `>>>>>>> upstream/master` at
# lines 23807 and 23850 of a committed .d.ts. No compiler catches it
# there: a .d.ts nothing imports is never parsed, and in YAML, JSON,
# Markdown and every config format we ship the markers are silently
# VALID content.
# vendormark — a site-generator vendor's branding left in the tree.
# Several sites began as output from a hosted "prompt to React app"
# generator, which signs its work in four places at once: an asset
# folder named after the vendor (so its name is in every image URL,
# and on one site in the FAVICON, so it was in every browser tab), a
# build plugin in package.json, an injected third-party <script>,
# and the commit messages, author identities and Co-authored-by
# trailers of its bot. The last survives any amount of tidying the
# working tree — a repo can render a perfect page and still say in
# `git log` who really wrote it. Removing it needed a history
# rewrite and a force-push of every branch; this is so that is never
# needed twice.
#
# Both proven RED against cloud@v1.801.424 (the real outage) and GREEN
# against the repaired tree before landing here. They read the checkout
@@ -193,6 +205,7 @@ jobs:
"$RUNNER_TEMP/ci/bin/ignoretracked" .
"$RUNNER_TEMP/ci/bin/modsize" .
"$RUNNER_TEMP/ci/bin/conflictmarkers" .
"$RUNNER_TEMP/ci/bin/vendormark" .
- name: Provision parse toolchain (jq + PyYAML)
# This reusable parses the caller's hanzo.yml with python3 + PyYAML and
@@ -413,28 +426,45 @@ jobs:
if [ -n "$GIT_TOKEN" ]; then echo "::add-mask::$GIT_TOKEN"; echo "GIT_TOKEN=$GIT_TOKEN" >> "$GITHUB_ENV"; fi
KUBECONFIG_B64=$(get KUBECONFIG)
if [ -n "$KUBECONFIG_B64" ]; then echo "$KUBECONFIG_B64" | base64 -d > "$RUNNER_TEMP/kubeconfig"; echo "kubeconfig=$RUNNER_TEMP/kubeconfig" >> "$GITHUB_OUTPUT"; fi
# Object-store credential, for the two lanes that write bytes to
# hanzoai/s3: a repo publishing `binaries:` to a bucket:, and a repo
# publishing a `site:` (whose build output has to land in the org's own
# storage space before /v1/sites can promote it). Same four names the
# services read at run time, from the same org/path/env — a CI-only copy
# of a credential is a second thing to rotate and the one nobody
# remembers. One fetch serves both lanes; a repo declaring both does not
# ask KMS twice.
# The REGISTRY credential, and it is fetched UNCONDITIONALLY because
# every repo that publishes an image needs it — which is most of them.
#
# It used to sit inside the `bucket:`/`site:` gate below, and that gate
# is about writing bytes to hanzoai/s3, a completely unrelated lane. So
# a repo declaring only `images:` — hanzoai/bot, and it is not alone —
# asked KMS for nothing and reached the docker login step with empty
# credentials. The build then failed at the push with no clue pointing
# here, and the obvious "fix" is to paste REGISTRY_USER/REGISTRY_PASSWORD
# into the forge's org secrets, which is exactly the CI-only copy the
# comment below warns about: a second thing to rotate, and the one
# nobody remembers. The contract is that the ONLY things living in the
# forge are KMS_CLIENT_ID/SECRET, so the fetch moves out of the gate
# rather than the credential moving into GitHub.
#
# Empty is harmless: nothing is exported when KMS has no such key, and
# the login step already falls back to the kubeconfig read for repos
# provisioned before these names existed.
for k in REGISTRY_USER REGISTRY_PASSWORD; do
v=$(get "$k")
if [ -n "$v" ]; then
[ "$k" = REGISTRY_PASSWORD ] && echo "::add-mask::$v"
echo "$k=$v" >> "$GITHUB_ENV"
fi
done
# Object-store credential, for the ONE lane that writes bytes to
# hanzoai/s3 itself: a repo publishing `binaries:` to a bucket:. Same
# names the services read at run time, from the same org/path/env — a
# CI-only copy of a credential is a second thing to rotate and the one
# nobody remembers.
#
# `site:` used to be fetched here too, and that is exactly what killed
# it: S3_ADMIN_ACCESS_KEY/S3_ADMIN_SECRET_KEY are not in KMS for any
# org, so the site lane's fail-closed check refused every caller that
# ever declared one. The site lane no longer touches S3 — it hands the
# zip to cloud and cloud writes the bytes — so the credential it cannot
# have is one it no longer needs.
SITE_DECL="$(yq -r '.site | type' hanzo.yml 2>/dev/null || echo '!!null')"
if [ -n "$(yq -r '.bucket // ""' hanzo.yml 2>/dev/null || true)" ] || [ "$SITE_DECL" = '!!map' ]; then
# The registry credential comes from KMS like every other secret —
# the ONLY things that live in GitHub are KMS_CLIENT_ID/SECRET.
# Named here so a repo with no kubeconfig (most of them) can still
# reach registry.hanzo.ai; the kubeconfig read below stays as the
# fallback for repos provisioned before these names existed.
for k in REGISTRY_USER REGISTRY_PASSWORD; do
v=$(get "$k")
if [ -n "$v" ]; then
[ "$k" = REGISTRY_PASSWORD ] && echo "::add-mask::$v"
echo "$k=$v" >> "$GITHUB_ENV"
fi
done
if [ -n "$(yq -r '.bucket // ""' hanzo.yml 2>/dev/null || true)" ]; then
for k in S3_ADMIN_ACCESS_KEY S3_ADMIN_SECRET_KEY; do
v=$(get "$k")
if [ -n "$v" ]; then echo "::add-mask::$v"; echo "$k=$v" >> "$GITHUB_ENV"; fi
@@ -446,38 +476,30 @@ jobs:
if [ -n "$v" ]; then echo "$k=$v" >> "$GITHUB_ENV"; fi
done
fi
# `site:` needs two things beyond the S3 credential above, and BOTH come
# from this step — there is one KMS login in this workflow and the site
# lane does not add a second.
# `site:` needs exactly ONE thing from this step, and it is a thing this
# step already has — there is one KMS login in this workflow and the
# site lane does not add a second.
#
# CLOUD_PROJECTS_BUCKET — the bucket /v1/sites lists the build output
# in. The SERVER owns it (it is never in a publish request, which is
# what makes the org prefix unforgeable), so CI cannot be told it by
# hanzo.yml; it reads the same KMS name the cloud binary reads, and
# falls back to the same default. One name, one place.
# HANZO_API_TOKEN — the bearer for /v1/sites. It is the token THIS STEP
# ALREADY MINTED: cloud's /v1/kms/auth/login is a broker that
# exchanges clientId/clientSecret at IAM's client_credentials
# endpoint and "returns IAM's own JWT verbatim" (cloud
# apps/kms/login.go), scoped by its `owner` claim to exactly this
# org. api.hanzo.ai validates that same issuer and mints X-Org-Id
# from it. So the credential that reads this org's secrets and the
# credential that publishes this org's site are ONE machine identity,
# with one lifetime and one thing to rotate. It is masked here,
# before GITHUB_ENV is the first place it could surface.
# HANZO_API_TOKEN — the bearer for /v1/projects and /v1/sites. It is
# the token THIS STEP ALREADY MINTED: cloud's /v1/kms/auth/login is
# a broker that performs the IAM client_credentials exchange and
# "returns IAM's own JWT verbatim" (cloud apps/kms/login.go), scoped
# by its `owner` claim to exactly this org. api.hanzo.ai validates
# that same issuer and mints X-Org-Id from it. So the credential
# that reads this org's secrets and the credential that publishes
# this org's site are ONE machine identity, with one lifetime and
# one thing to rotate. It is masked here, before GITHUB_ENV is the
# first place it could surface.
#
# HANZO_ORG is exported with them because the S3 upload has to name the
# org segment the server will prepend to `source`. That it AGREES with
# the publish identity is not an assumption: every `get` above read
# /v1/kms/orgs/$ORG/..., and cloud re-checks the token's `owner` against
# that :org on each read — so an ORG that disagreed would have fetched no
# credential at all, and this lane fails closed below rather than
# uploading into a prefix nobody will look in.
# Neither the bucket nor the org is exported any more, because neither
# is CI's to name. The org segment is prepended SERVER-side from the
# validated principal and the bucket never appears in a request at all
# (cloud apps/projects/blob.go:60, `sitePrefix(org, slug)`) — which is
# precisely what makes the prefix unforgeable. CI passing them was CI
# restating something it could only get wrong.
if [ "$SITE_DECL" = '!!map' ]; then
v=$(get CLOUD_PROJECTS_BUCKET)
if [ -n "$v" ]; then echo "CLOUD_PROJECTS_BUCKET=$v" >> "$GITHUB_ENV"; fi
echo "::add-mask::$TOKEN"
{ echo "HANZO_API_TOKEN=$TOKEN"; echo "HANZO_ORG=$ORG"; } >> "$GITHUB_ENV"
echo "HANZO_API_TOKEN=$TOKEN" >> "$GITHUB_ENV"
fi
# Build-time secrets → --build-arg. A repo declares per-image
# `build_secrets: [NAME, ...]` in hanzo.yml; each NAME is fetched from
@@ -1667,52 +1689,40 @@ jobs:
# that an EDGE serves, which is why this is a third verb and not a flag on
# one of those two.
#
# The plane it publishes onto is hanzoai/cloud's Projects/sites API
# (`/v1/sites`), whose model is VALUES PLUS A POINTER: a build output is
# promoted into an immutable Release whose id is a digest of its object
# manifest, and the site's pointer is flipped to it. `publish` is
# promote+activate in one call — the 99% path — and rollback is the same
# pointer aimed at an older release. Every first-party Hanzo site is moving
# onto this, so the publish step belongs here ONCE rather than as a
# hand-rolled sync in each site repo.
# The plane it publishes onto is hanzoai/cloud's Projects/sites API,
# whose model is VALUES PLUS A POINTER: a build output is promoted into an
# immutable Release whose id is a digest of its object manifest, and the
# site's pointer is flipped to it. `publish` is promote+activate in one
# call — the 99% path — and rollback is the same pointer aimed at an older
# release.
#
# HOW THE BYTES GET THERE. Not through the API. cloud's own artifact path
# says so in as many words — a tar(.gz) through `/v1/projects/:slug/deploy`
# is the small builder-one-click case, and "large sites use the git/CI path
# that syncs to S3 directly and never streams bytes through this handler"
# (apps/projects/blob.go). A first-party site is ~125 MiB across ~8,400
# files; streaming that through a gateway body limit is not a path. So CI
# syncs the export into the org's OWN storage space and then asks the
# server to promote it, which is also why no object body ever crosses the
# API and no browser ever holds an S3 credential.
# WHY THIS LANE CHANGED. It used to stage the export with `mc mirror`
# against hanzoai/s3, gated on S3_ADMIN_ACCESS_KEY + S3_ADMIN_SECRET_KEY
# read from KMS. Those names are in KMS for NO org, and the step fails
# closed without them — so `site:` refused every caller that declared one
# and was dead code no repo could use. Measured before this change: of
# the 155 hanzo.yml files across the three orgs, NOT ONE carried a
# `site:` block, and hanzoai/console had grown its own build→zip→publish
# workflow to ship at all. Per-repo build logic is the one thing this contract exists to
# prevent, so the fix is not to provision that credential — it is to stop
# needing it.
#
# The sync client is `mc`, provisioned the way this file provisions every
# other tool it needs on a bare runner (jq, yq, kubectl, crane): ONE curl
# of ONE pinned static binary into ~/.local/bin, no sudo, no archive, no
# package manager. It is also the CLI of the SAME client family the server
# uses — hanzoai/s3-go is built on the MinIO Go client — so CI and cloud
# talk to hanzoai/s3 through one lineage instead of two. `mirror
# --overwrite --remove` makes the staging prefix EXACTLY the export: a file
# deleted from the build cannot linger into a release, and unchanged
# objects are not re-sent.
# The bytes now travel the route console proved: the zip goes to
# /v1/projects/<slug>/deploy and the landed prefix is promoted by
# /v1/sites/<slug>/publish, both on the IAM bearer the KMS step above
# ALREADY minted. CI holds no bucket key, names no bucket and names no
# org — the org segment is prepended server-side from the validated
# principal, which is what makes the prefix unforgeable. Nothing is left
# for an org to be missing.
#
# IDEMPOTENCY IS BY CONSTRUCTION, not by a remembered request key. The
# release id is a SHA-256 over the sorted manifest of (key, size, etag), so
# the same bytes produce the same id; cloud finds that release row already
# present and short-circuits — no copy, no new release, no second billable
# deploy. Re-running this job on an unchanged export is therefore a no-op
# that still ends green and still prints the release it left live. Nothing
# in this step needs to remember that it ran.
#
# The staging prefix is ONE per site (`<org>/.ci/<slug>`), a sibling of the
# live site prefix and of the server's own `<org>/.releases` space — a
# project slug can never be ".ci" because slugs forbid '.', and neither a
# deploy purge nor a project delete reaches it. Bounded, not per-commit:
# the org's storage holds the last export, not every export ever made. Two
# publishes racing on one site is the one thing that costs: cloud copies
# each object conditional on the exact ETag it digested, so a source that
# moves mid-publish is a 409 and the release is abandoned rather than
# half-made. That fails this step loudly; the next push converges.
# THE WORK IS IN bin/sitepublish, NOT HERE. Forty lines of zip → upload →
# promote → verify inlined in a workflow is forty lines nobody can run
# offline, and this file already learned that with imgver and gover.
# bin/sitepublish_test.sh drives it through a curl shim with no network,
# and is weighted toward the REFUSALS — a publish step that reports
# success when it verified nothing is worse than one that never verified,
# because it is indistinguishable from a working one until a site quietly
# stops updating.
if: inputs.mode != 'delegate' && github.event_name != 'pull_request'
env:
API: ${{ vars.API_ENDPOINT || 'https://api.hanzo.ai' }}
@@ -1738,87 +1748,19 @@ jobs:
if [ -z "$SLUG" ] || [ -z "$DIR" ]; then
echo "::error::site: needs both slug: (the /v1/sites project) and dir: (the built export)"; exit 1
fi
# The slug is a URL path segment AND an S3 key segment here, so it is held
# to the SAME grammar cloud validates project slugs with. Checked before
# anything is uploaded: a bad slug is a typo to fix, not a 404 to read.
echo "$SLUG" | grep -qE '^[a-z0-9]([a-z0-9-]{0,38}[a-z0-9])?$' \
|| { echo "::error::site.slug '$SLUG' is not a project slug (^[a-z0-9]([a-z0-9-]{0,38}[a-z0-9])?\$)"; exit 1; }
if [ -n "$BUILD" ]; then
echo "::group::build $SLUG ($BUILD)"
sh -c "$BUILD"
echo "::endgroup::"
fi
# A site is a thing with an index — the one contract cloud enforces
# however content arrives. Checked HERE, before the mirror, because
# `--remove` on a failed build would otherwise empty the staging prefix
# and turn a broken build into a broken publish.
[ -f "$DIR/index.html" ] \
|| { echo "::error::site.dir '$DIR' has no index.html at its root — /v1/sites refuses a source without one"; exit 1; }
# Fail closed. Publishing with no credential is not a warning: it would
# leave the site on its previous release with a green run claiming
# otherwise. (All three come from the KMS step above — never from git.)
if [ -z "${S3_ADMIN_ACCESS_KEY:-}" ] || [ -z "${S3_ADMIN_SECRET_KEY:-}" ] \
|| [ -z "${HANZO_API_TOKEN:-}" ] || [ -z "${HANZO_ORG:-}" ]; then
echo "::error::hanzo.yml declares site: but KMS supplied no publish credential (S3_ADMIN_ACCESS_KEY/SECRET_KEY + an IAM token for $API) — refusing to publish"; exit 1
# Fail closed on the ONE thing that can be missing, and say which one.
# An empty token here means the KMS login did not happen — not that a
# site credential was never provisioned, which is what the old message
# said and what sent people looking for a secret that does not exist.
if [ -z "${HANZO_API_TOKEN:-}" ]; then
echo "::error::hanzo.yml declares site: but the KMS step minted no IAM token for $API — refusing to publish. That is a KMS_CLIENT_ID/KMS_CLIENT_SECRET problem on this repo, not a missing site credential."; exit 1
fi
command -v mc >/dev/null 2>&1 || {
mkdir -p "$HOME/.local/bin"; export PATH="$HOME/.local/bin:$PATH"
curl -fsSL https://dl.min.io/client/mc/release/linux-amd64/archive/mc.RELEASE.2025-08-13T08-35-41Z \
-o "$HOME/.local/bin/mc" && chmod +x "$HOME/.local/bin/mc"
}
# Same public endpoint + scheme the binaries lane publishes through, from
# the same KMS names, so CI reaches hanzoai/s3 by ONE address.
SCHEME=https
if [ "${S3_PUBLIC_SECURE:-true}" = "false" ]; then SCHEME=http; fi
HOST="${S3_PUBLIC_ENDPOINT:-s3.hanzo.ai}"
BUCKET="${CLOUD_PROJECTS_BUCKET:-hanzo-sites}"
# The credential reaches mc through MC_HOST_<alias> — never `mc alias set`,
# whose key and secret would sit in argv for anything reading /proc, and
# never a config file on the runner. URL-encoded with jq so a secret
# containing '/' or '+' cannot re-parse the alias URL, and masked as a
# whole because the composed string carries the secret verbatim.
MC_HOST_hanzosite="$SCHEME://$(jq -rn --arg v "$S3_ADMIN_ACCESS_KEY" '$v|@uri'):$(jq -rn --arg v "$S3_ADMIN_SECRET_KEY" '$v|@uri')@$HOST"
echo "::add-mask::$MC_HOST_hanzosite"
export MC_HOST_hanzosite
# `source` is RELATIVE to the org's own storage root. The org segment is
# prepended SERVER-side from the validated principal and the bucket never
# appears in the request at all (cloud apps/projects/release.go,
# sourcePrefix) — an absolute path or a URL is refused — so the worst a
# source can address is something this org already owns. CI names the
# same prefix on the S3 side, org included, because there it IS writing a
# key.
SRC=".ci/$SLUG"
echo "::group::sync $DIR → $BUCKET/$HANZO_ORG/$SRC"
mc --no-color mirror --overwrite --remove --summary "$DIR" "hanzosite/$BUCKET/$HANZO_ORG/$SRC"
echo "::endgroup::"
# Promote + activate. Content types and cache policy are NOT our problem:
# copyRelease replaces the metadata on every object it promotes, so a
# release serves identically however its bytes were staged.
code=$(curl -sS -o "$RUNNER_TEMP/publish.json" -w '%{http_code}' \
-X POST "$API/v1/sites/$SLUG/publish" \
-H "Authorization: Bearer $HANZO_API_TOKEN" \
-H 'Content-Type: application/json' \
-d "$(jq -nc --arg s "$SRC" '{source:$s}')") \
|| { echo "::error::POST $API/v1/sites/$SLUG/publish did not complete"; exit 1; }
# Non-2xx is fatal and PRINTS ITS BODY. cloud answers this route with a
# specific status per cause — 402 hosting gate, 403 wrong org, 404 no such
# site for this tenant, 409 the source moved mid-publish, 413 past the
# object/byte caps, 503 storage unconfigured — and every one of them is a
# different fix, so swallowing the body would be discarding the answer.
case "$code" in
2??) ;;
*) echo "::error::publish $SLUG → HTTP $code"; cat "$RUNNER_TEMP/publish.json"; echo; exit 1 ;;
esac
rid=$(jq -r '.releaseId // empty' "$RUNNER_TEMP/publish.json")
url=$(jq -r '.url // empty' "$RUNNER_TEMP/publish.json")
objs=$(jq -r '.objects // 0' "$RUNNER_TEMP/publish.json")
[ -n "$rid" ] \
|| { echo "::error::publish answered $code with no releaseId:"; cat "$RUNNER_TEMP/publish.json"; echo; exit 1; }
# An empty url is a real answer, not a failure: the bare <slug> host is
# first-come across every org, so a site whose slug is already claimed is
# live at its own prefix with no public subdomain and is told to rename.
echo "published $rid ($objs objects) → ${url:-no public host — that slug is claimed; rename the site or bind a domain}"
{ echo "### Site — \`$SLUG\`"; echo
echo "| release | objects | url |"; echo "|---|---|---|"
echo "| \`$rid\` | $objs | ${url:-—} |"
} >> "$GITHUB_STEP_SUMMARY"
# Slug grammar, index.html, the server's file/byte caps and the edge
# BodyLimit are all checked inside sitepublish, before anything is sent,
# so each refusal names its own number instead of arriving as a 4xx.
HANZO_API="$API" "$CI_HOME/bin/sitepublish" "$SLUG" "$DIR"
+113 -171
View File
@@ -177,11 +177,23 @@ jobs:
# on they are ordinary content: status clean, diff empty, both sides
# of the merge shipping. It hides in generated and vendored files,
# which are large, rarely opened and effectively unreviewed —
# hanzoai/base carried `<<<<<<< HEAD` / `>>>>>>> upstream/master` at
# hanzoai/base carried ` / `>>>>>>> upstream/master` at
# lines 23807 and 23850 of a committed .d.ts. No compiler catches it
# there: a .d.ts nothing imports is never parsed, and in YAML, JSON,
# Markdown and every config format we ship the markers are silently
# VALID content.
# vendormark — a site-generator vendor's branding left in the tree.
# Several sites began as output from a hosted "prompt to React app"
# generator, which signs its work in four places at once: an asset
# folder named after the vendor (so its name is in every image URL,
# and on one site in the FAVICON, so it was in every browser tab), a
# build plugin in package.json, an injected third-party <script>,
# and the commit messages, author identities and Co-authored-by
# trailers of its bot. The last survives any amount of tidying the
# working tree — a repo can render a perfect page and still say in
# `git log` who really wrote it. Removing it needed a history
# rewrite and a force-push of every branch; this is so that is never
# needed twice.
#
# Both proven RED against cloud@v1.801.424 (the real outage) and GREEN
# against the repaired tree before landing here. They read the checkout
@@ -193,6 +205,7 @@ jobs:
"$RUNNER_TEMP/ci/bin/ignoretracked" .
"$RUNNER_TEMP/ci/bin/modsize" .
"$RUNNER_TEMP/ci/bin/conflictmarkers" .
"$RUNNER_TEMP/ci/bin/vendormark" .
- name: Provision parse toolchain (jq + PyYAML)
# This reusable parses the caller's hanzo.yml with python3 + PyYAML and
@@ -413,28 +426,45 @@ jobs:
if [ -n "$GIT_TOKEN" ]; then echo "::add-mask::$GIT_TOKEN"; echo "GIT_TOKEN=$GIT_TOKEN" >> "$GITHUB_ENV"; fi
KUBECONFIG_B64=$(get KUBECONFIG)
if [ -n "$KUBECONFIG_B64" ]; then echo "$KUBECONFIG_B64" | base64 -d > "$RUNNER_TEMP/kubeconfig"; echo "kubeconfig=$RUNNER_TEMP/kubeconfig" >> "$GITHUB_OUTPUT"; fi
# Object-store credential, for the two lanes that write bytes to
# hanzoai/s3: a repo publishing `binaries:` to a bucket:, and a repo
# publishing a `site:` (whose build output has to land in the org's own
# storage space before /v1/sites can promote it). Same four names the
# services read at run time, from the same org/path/env — a CI-only copy
# of a credential is a second thing to rotate and the one nobody
# remembers. One fetch serves both lanes; a repo declaring both does not
# ask KMS twice.
# The REGISTRY credential, and it is fetched UNCONDITIONALLY because
# every repo that publishes an image needs it — which is most of them.
#
# It used to sit inside the `bucket:`/`site:` gate below, and that gate
# is about writing bytes to hanzoai/s3, a completely unrelated lane. So
# a repo declaring only `images:` — hanzoai/bot, and it is not alone —
# asked KMS for nothing and reached the docker login step with empty
# credentials. The build then failed at the push with no clue pointing
# here, and the obvious "fix" is to paste REGISTRY_USER/REGISTRY_PASSWORD
# into the forge's org secrets, which is exactly the CI-only copy the
# comment below warns about: a second thing to rotate, and the one
# nobody remembers. The contract is that the ONLY things living in the
# forge are KMS_CLIENT_ID/SECRET, so the fetch moves out of the gate
# rather than the credential moving into GitHub.
#
# Empty is harmless: nothing is exported when KMS has no such key, and
# the login step already falls back to the kubeconfig read for repos
# provisioned before these names existed.
for k in REGISTRY_USER REGISTRY_PASSWORD; do
v=$(get "$k")
if [ -n "$v" ]; then
[ "$k" = REGISTRY_PASSWORD ] && echo "::add-mask::$v"
echo "$k=$v" >> "$GITHUB_ENV"
fi
done
# Object-store credential, for the ONE lane that writes bytes to
# hanzoai/s3 itself: a repo publishing `binaries:` to a bucket:. Same
# names the services read at run time, from the same org/path/env — a
# CI-only copy of a credential is a second thing to rotate and the one
# nobody remembers.
#
# `site:` used to be fetched here too, and that is exactly what killed
# it: S3_ADMIN_ACCESS_KEY/S3_ADMIN_SECRET_KEY are not in KMS for any
# org, so the site lane's fail-closed check refused every caller that
# ever declared one. The site lane no longer touches S3 — it hands the
# zip to cloud and cloud writes the bytes — so the credential it cannot
# have is one it no longer needs.
SITE_DECL="$(yq -r '.site | type' hanzo.yml 2>/dev/null || echo '!!null')"
if [ -n "$(yq -r '.bucket // ""' hanzo.yml 2>/dev/null || true)" ] || [ "$SITE_DECL" = '!!map' ]; then
# The registry credential comes from KMS like every other secret —
# the ONLY things that live in GitHub are KMS_CLIENT_ID/SECRET.
# Named here so a repo with no kubeconfig (most of them) can still
# reach registry.hanzo.ai; the kubeconfig read below stays as the
# fallback for repos provisioned before these names existed.
for k in REGISTRY_USER REGISTRY_PASSWORD; do
v=$(get "$k")
if [ -n "$v" ]; then
[ "$k" = REGISTRY_PASSWORD ] && echo "::add-mask::$v"
echo "$k=$v" >> "$GITHUB_ENV"
fi
done
if [ -n "$(yq -r '.bucket // ""' hanzo.yml 2>/dev/null || true)" ]; then
for k in S3_ADMIN_ACCESS_KEY S3_ADMIN_SECRET_KEY; do
v=$(get "$k")
if [ -n "$v" ]; then echo "::add-mask::$v"; echo "$k=$v" >> "$GITHUB_ENV"; fi
@@ -446,38 +476,30 @@ jobs:
if [ -n "$v" ]; then echo "$k=$v" >> "$GITHUB_ENV"; fi
done
fi
# `site:` needs two things beyond the S3 credential above, and BOTH come
# from this step — there is one KMS login in this workflow and the site
# lane does not add a second.
# `site:` needs exactly ONE thing from this step, and it is a thing this
# step already has — there is one KMS login in this workflow and the
# site lane does not add a second.
#
# CLOUD_PROJECTS_BUCKET — the bucket /v1/sites lists the build output
# in. The SERVER owns it (it is never in a publish request, which is
# what makes the org prefix unforgeable), so CI cannot be told it by
# hanzo.yml; it reads the same KMS name the cloud binary reads, and
# falls back to the same default. One name, one place.
# HANZO_API_TOKEN — the bearer for /v1/sites. It is the token THIS STEP
# ALREADY MINTED: cloud's /v1/kms/auth/login is a broker that
# exchanges clientId/clientSecret at IAM's client_credentials
# endpoint and "returns IAM's own JWT verbatim" (cloud
# apps/kms/login.go), scoped by its `owner` claim to exactly this
# org. api.hanzo.ai validates that same issuer and mints X-Org-Id
# from it. So the credential that reads this org's secrets and the
# credential that publishes this org's site are ONE machine identity,
# with one lifetime and one thing to rotate. It is masked here,
# before GITHUB_ENV is the first place it could surface.
# HANZO_API_TOKEN — the bearer for /v1/projects and /v1/sites. It is
# the token THIS STEP ALREADY MINTED: cloud's /v1/kms/auth/login is
# a broker that performs the IAM client_credentials exchange and
# "returns IAM's own JWT verbatim" (cloud apps/kms/login.go), scoped
# by its `owner` claim to exactly this org. api.hanzo.ai validates
# that same issuer and mints X-Org-Id from it. So the credential
# that reads this org's secrets and the credential that publishes
# this org's site are ONE machine identity, with one lifetime and
# one thing to rotate. It is masked here, before GITHUB_ENV is the
# first place it could surface.
#
# HANZO_ORG is exported with them because the S3 upload has to name the
# org segment the server will prepend to `source`. That it AGREES with
# the publish identity is not an assumption: every `get` above read
# /v1/kms/orgs/$ORG/..., and cloud re-checks the token's `owner` against
# that :org on each read — so an ORG that disagreed would have fetched no
# credential at all, and this lane fails closed below rather than
# uploading into a prefix nobody will look in.
# Neither the bucket nor the org is exported any more, because neither
# is CI's to name. The org segment is prepended SERVER-side from the
# validated principal and the bucket never appears in a request at all
# (cloud apps/projects/blob.go:60, `sitePrefix(org, slug)`) — which is
# precisely what makes the prefix unforgeable. CI passing them was CI
# restating something it could only get wrong.
if [ "$SITE_DECL" = '!!map' ]; then
v=$(get CLOUD_PROJECTS_BUCKET)
if [ -n "$v" ]; then echo "CLOUD_PROJECTS_BUCKET=$v" >> "$GITHUB_ENV"; fi
echo "::add-mask::$TOKEN"
{ echo "HANZO_API_TOKEN=$TOKEN"; echo "HANZO_ORG=$ORG"; } >> "$GITHUB_ENV"
echo "HANZO_API_TOKEN=$TOKEN" >> "$GITHUB_ENV"
fi
# Build-time secrets → --build-arg. A repo declares per-image
# `build_secrets: [NAME, ...]` in hanzo.yml; each NAME is fetched from
@@ -1667,52 +1689,40 @@ jobs:
# that an EDGE serves, which is why this is a third verb and not a flag on
# one of those two.
#
# The plane it publishes onto is hanzoai/cloud's Projects/sites API
# (`/v1/sites`), whose model is VALUES PLUS A POINTER: a build output is
# promoted into an immutable Release whose id is a digest of its object
# manifest, and the site's pointer is flipped to it. `publish` is
# promote+activate in one call — the 99% path — and rollback is the same
# pointer aimed at an older release. Every first-party Hanzo site is moving
# onto this, so the publish step belongs here ONCE rather than as a
# hand-rolled sync in each site repo.
# The plane it publishes onto is hanzoai/cloud's Projects/sites API,
# whose model is VALUES PLUS A POINTER: a build output is promoted into an
# immutable Release whose id is a digest of its object manifest, and the
# site's pointer is flipped to it. `publish` is promote+activate in one
# call — the 99% path — and rollback is the same pointer aimed at an older
# release.
#
# HOW THE BYTES GET THERE. Not through the API. cloud's own artifact path
# says so in as many words — a tar(.gz) through `/v1/projects/:slug/deploy`
# is the small builder-one-click case, and "large sites use the git/CI path
# that syncs to S3 directly and never streams bytes through this handler"
# (apps/projects/blob.go). A first-party site is ~125 MiB across ~8,400
# files; streaming that through a gateway body limit is not a path. So CI
# syncs the export into the org's OWN storage space and then asks the
# server to promote it, which is also why no object body ever crosses the
# API and no browser ever holds an S3 credential.
# WHY THIS LANE CHANGED. It used to stage the export with `mc mirror`
# against hanzoai/s3, gated on S3_ADMIN_ACCESS_KEY + S3_ADMIN_SECRET_KEY
# read from KMS. Those names are in KMS for NO org, and the step fails
# closed without them — so `site:` refused every caller that declared one
# and was dead code no repo could use. Measured before this change: of
# the 155 hanzo.yml files across the three orgs, NOT ONE carried a
# `site:` block, and hanzoai/console had grown its own build→zip→publish
# workflow to ship at all. Per-repo build logic is the one thing this contract exists to
# prevent, so the fix is not to provision that credential — it is to stop
# needing it.
#
# The sync client is `mc`, provisioned the way this file provisions every
# other tool it needs on a bare runner (jq, yq, kubectl, crane): ONE curl
# of ONE pinned static binary into ~/.local/bin, no sudo, no archive, no
# package manager. It is also the CLI of the SAME client family the server
# uses — hanzoai/s3-go is built on the MinIO Go client — so CI and cloud
# talk to hanzoai/s3 through one lineage instead of two. `mirror
# --overwrite --remove` makes the staging prefix EXACTLY the export: a file
# deleted from the build cannot linger into a release, and unchanged
# objects are not re-sent.
# The bytes now travel the route console proved: the zip goes to
# /v1/projects/<slug>/deploy and the landed prefix is promoted by
# /v1/sites/<slug>/publish, both on the IAM bearer the KMS step above
# ALREADY minted. CI holds no bucket key, names no bucket and names no
# org — the org segment is prepended server-side from the validated
# principal, which is what makes the prefix unforgeable. Nothing is left
# for an org to be missing.
#
# IDEMPOTENCY IS BY CONSTRUCTION, not by a remembered request key. The
# release id is a SHA-256 over the sorted manifest of (key, size, etag), so
# the same bytes produce the same id; cloud finds that release row already
# present and short-circuits — no copy, no new release, no second billable
# deploy. Re-running this job on an unchanged export is therefore a no-op
# that still ends green and still prints the release it left live. Nothing
# in this step needs to remember that it ran.
#
# The staging prefix is ONE per site (`<org>/.ci/<slug>`), a sibling of the
# live site prefix and of the server's own `<org>/.releases` space — a
# project slug can never be ".ci" because slugs forbid '.', and neither a
# deploy purge nor a project delete reaches it. Bounded, not per-commit:
# the org's storage holds the last export, not every export ever made. Two
# publishes racing on one site is the one thing that costs: cloud copies
# each object conditional on the exact ETag it digested, so a source that
# moves mid-publish is a 409 and the release is abandoned rather than
# half-made. That fails this step loudly; the next push converges.
# THE WORK IS IN bin/sitepublish, NOT HERE. Forty lines of zip → upload →
# promote → verify inlined in a workflow is forty lines nobody can run
# offline, and this file already learned that with imgver and gover.
# bin/sitepublish_test.sh drives it through a curl shim with no network,
# and is weighted toward the REFUSALS — a publish step that reports
# success when it verified nothing is worse than one that never verified,
# because it is indistinguishable from a working one until a site quietly
# stops updating.
if: inputs.mode != 'delegate' && github.event_name != 'pull_request'
env:
API: ${{ vars.API_ENDPOINT || 'https://api.hanzo.ai' }}
@@ -1738,87 +1748,19 @@ jobs:
if [ -z "$SLUG" ] || [ -z "$DIR" ]; then
echo "::error::site: needs both slug: (the /v1/sites project) and dir: (the built export)"; exit 1
fi
# The slug is a URL path segment AND an S3 key segment here, so it is held
# to the SAME grammar cloud validates project slugs with. Checked before
# anything is uploaded: a bad slug is a typo to fix, not a 404 to read.
echo "$SLUG" | grep -qE '^[a-z0-9]([a-z0-9-]{0,38}[a-z0-9])?$' \
|| { echo "::error::site.slug '$SLUG' is not a project slug (^[a-z0-9]([a-z0-9-]{0,38}[a-z0-9])?\$)"; exit 1; }
if [ -n "$BUILD" ]; then
echo "::group::build $SLUG ($BUILD)"
sh -c "$BUILD"
echo "::endgroup::"
fi
# A site is a thing with an index — the one contract cloud enforces
# however content arrives. Checked HERE, before the mirror, because
# `--remove` on a failed build would otherwise empty the staging prefix
# and turn a broken build into a broken publish.
[ -f "$DIR/index.html" ] \
|| { echo "::error::site.dir '$DIR' has no index.html at its root — /v1/sites refuses a source without one"; exit 1; }
# Fail closed. Publishing with no credential is not a warning: it would
# leave the site on its previous release with a green run claiming
# otherwise. (All three come from the KMS step above — never from git.)
if [ -z "${S3_ADMIN_ACCESS_KEY:-}" ] || [ -z "${S3_ADMIN_SECRET_KEY:-}" ] \
|| [ -z "${HANZO_API_TOKEN:-}" ] || [ -z "${HANZO_ORG:-}" ]; then
echo "::error::hanzo.yml declares site: but KMS supplied no publish credential (S3_ADMIN_ACCESS_KEY/SECRET_KEY + an IAM token for $API) — refusing to publish"; exit 1
# Fail closed on the ONE thing that can be missing, and say which one.
# An empty token here means the KMS login did not happen — not that a
# site credential was never provisioned, which is what the old message
# said and what sent people looking for a secret that does not exist.
if [ -z "${HANZO_API_TOKEN:-}" ]; then
echo "::error::hanzo.yml declares site: but the KMS step minted no IAM token for $API — refusing to publish. That is a KMS_CLIENT_ID/KMS_CLIENT_SECRET problem on this repo, not a missing site credential."; exit 1
fi
command -v mc >/dev/null 2>&1 || {
mkdir -p "$HOME/.local/bin"; export PATH="$HOME/.local/bin:$PATH"
curl -fsSL https://dl.min.io/client/mc/release/linux-amd64/archive/mc.RELEASE.2025-08-13T08-35-41Z \
-o "$HOME/.local/bin/mc" && chmod +x "$HOME/.local/bin/mc"
}
# Same public endpoint + scheme the binaries lane publishes through, from
# the same KMS names, so CI reaches hanzoai/s3 by ONE address.
SCHEME=https
if [ "${S3_PUBLIC_SECURE:-true}" = "false" ]; then SCHEME=http; fi
HOST="${S3_PUBLIC_ENDPOINT:-s3.hanzo.ai}"
BUCKET="${CLOUD_PROJECTS_BUCKET:-hanzo-sites}"
# The credential reaches mc through MC_HOST_<alias> — never `mc alias set`,
# whose key and secret would sit in argv for anything reading /proc, and
# never a config file on the runner. URL-encoded with jq so a secret
# containing '/' or '+' cannot re-parse the alias URL, and masked as a
# whole because the composed string carries the secret verbatim.
MC_HOST_hanzosite="$SCHEME://$(jq -rn --arg v "$S3_ADMIN_ACCESS_KEY" '$v|@uri'):$(jq -rn --arg v "$S3_ADMIN_SECRET_KEY" '$v|@uri')@$HOST"
echo "::add-mask::$MC_HOST_hanzosite"
export MC_HOST_hanzosite
# `source` is RELATIVE to the org's own storage root. The org segment is
# prepended SERVER-side from the validated principal and the bucket never
# appears in the request at all (cloud apps/projects/release.go,
# sourcePrefix) — an absolute path or a URL is refused — so the worst a
# source can address is something this org already owns. CI names the
# same prefix on the S3 side, org included, because there it IS writing a
# key.
SRC=".ci/$SLUG"
echo "::group::sync $DIR → $BUCKET/$HANZO_ORG/$SRC"
mc --no-color mirror --overwrite --remove --summary "$DIR" "hanzosite/$BUCKET/$HANZO_ORG/$SRC"
echo "::endgroup::"
# Promote + activate. Content types and cache policy are NOT our problem:
# copyRelease replaces the metadata on every object it promotes, so a
# release serves identically however its bytes were staged.
code=$(curl -sS -o "$RUNNER_TEMP/publish.json" -w '%{http_code}' \
-X POST "$API/v1/sites/$SLUG/publish" \
-H "Authorization: Bearer $HANZO_API_TOKEN" \
-H 'Content-Type: application/json' \
-d "$(jq -nc --arg s "$SRC" '{source:$s}')") \
|| { echo "::error::POST $API/v1/sites/$SLUG/publish did not complete"; exit 1; }
# Non-2xx is fatal and PRINTS ITS BODY. cloud answers this route with a
# specific status per cause — 402 hosting gate, 403 wrong org, 404 no such
# site for this tenant, 409 the source moved mid-publish, 413 past the
# object/byte caps, 503 storage unconfigured — and every one of them is a
# different fix, so swallowing the body would be discarding the answer.
case "$code" in
2??) ;;
*) echo "::error::publish $SLUG → HTTP $code"; cat "$RUNNER_TEMP/publish.json"; echo; exit 1 ;;
esac
rid=$(jq -r '.releaseId // empty' "$RUNNER_TEMP/publish.json")
url=$(jq -r '.url // empty' "$RUNNER_TEMP/publish.json")
objs=$(jq -r '.objects // 0' "$RUNNER_TEMP/publish.json")
[ -n "$rid" ] \
|| { echo "::error::publish answered $code with no releaseId:"; cat "$RUNNER_TEMP/publish.json"; echo; exit 1; }
# An empty url is a real answer, not a failure: the bare <slug> host is
# first-come across every org, so a site whose slug is already claimed is
# live at its own prefix with no public subdomain and is told to rename.
echo "published $rid ($objs objects) → ${url:-no public host — that slug is claimed; rename the site or bind a domain}"
{ echo "### Site — \`$SLUG\`"; echo
echo "| release | objects | url |"; echo "|---|---|---|"
echo "| \`$rid\` | $objs | ${url:-—} |"
} >> "$GITHUB_STEP_SUMMARY"
# Slug grammar, index.html, the server's file/byte caps and the edge
# BodyLimit are all checked inside sitepublish, before anything is sent,
# so each refusal names its own number instead of arriving as a 4xx.
HANZO_API="$API" "$CI_HOME/bin/sitepublish" "$SLUG" "$DIR"
+1 -1
View File
@@ -23,5 +23,5 @@ concurrency:
cancel-in-progress: true
jobs:
gate:
uses: hanzoai/ci/.hanzo/workflows/build.yml@v2
uses: hanzoai/ci/.hanzo/workflows/build.yml@v1
secrets: inherit
+37
View File
@@ -171,6 +171,43 @@ Builds are `CGO_ENABLED=0 -trimpath`: the host that installs this runs it on
whatever base image the host is, and the digest must be a function of the
source, not of the checkout path.
## `site:` — a static export, promoted to an immutable release
`images:` ships an OCI image a **cluster** runs; `binaries:` ships an executable
a **host** installs. `site:` ships a static export an **edge** serves — no image,
no CR, no replicas. Building a container so a Go binary can serve `/public` is
the shape this retires.
```yaml
site:
slug: hanzo-console # the project on the Sites plane
dir: out # the built export; needs index.html at its root
build: npm ci && npm run build # optional; run first
on: [main] # same branch gate as deploy.on; tags always publish
```
That is the whole configuration. **There is no credential to provision**: the
bearer is the IAM JWT the workflow already mints from `KMS_CLIENT_ID` /
`KMS_CLIENT_SECRET`, so a repo that can build can publish. CI names no bucket and
no org — the org segment is prepended server-side from the validated principal,
which is what makes the prefix unforgeable.
The export is zipped and posted to `/v1/projects/<slug>/deploy`, then that prefix
is promoted by `/v1/sites/<slug>/publish` into an **immutable release** whose id
digests its object manifest. The site's pointer is flipped to it, and the step
then re-reads the release list and refuses unless the release it just published
is the one that is live. Rollback is the same pointer aimed at an older release.
**One size boundary, and it is the server's.** cloud's public edge caps a request
body at 16 MiB (`GATEWAY_BODY_LIMIT`) and refuses a larger POST before any
handler runs — reporting only `Error when parsing request`, which names neither
size nor cause. `bin/sitepublish` therefore measures the zip and refuses *with
the number* first. Of the 24 built exports in the estate, 22 fit; the two that do
not (`hanzo.ai` at 27.9 MiB zipped and 8,536 files, `trillerfest.com` at 76.7
MiB) use [`bin/sitedeploy`](bin/sitedeploy), which streams per-file against a
presigned grant and has no body limit. `hanzo.ai` is also past the server's own
5,000-entry cap, so no transport makes that export a release.
## Runners — our fleet or your own
By default the build runs on the **Hanzo `git-runner` fleet** on git.hanzo.ai
+220
View File
@@ -0,0 +1,220 @@
#!/usr/bin/env bash
# sitepublish — publish a built static export as an immutable Release on the
# Hanzo Sites plane. One implementation, every `site:` in the fleet.
#
# sitepublish <slug> <dir> # e.g. sitepublish hanzo-console out
#
# WHY THIS EXISTS: the reusable's `site:` lane used to stage the export with
# `mc mirror` against hanzoai/s3, which needed S3_ADMIN_ACCESS_KEY and
# S3_ADMIN_SECRET_KEY. Those names are not in KMS for any org, and the lane
# fails closed without them — so `site:` refused for every caller that ever
# declared it, and hanzoai/console had to grow its own build→zip→publish
# workflow to ship at all. Per-repo build logic is the one thing this repo
# exists to prevent. The bytes now travel the SAME route console proved, and
# the credential is the one CI already holds.
#
# THE TWO STEPS:
#
# POST /v1/projects/<slug>/deploy Content-Type: application/zip, zip as body
# POST /v1/sites/<slug>/publish {"source":"<slug>"}
#
# The first lands the export at the org's site prefix; the second promotes that
# prefix into an immutable Release and flips it live. `source` is ORG-RELATIVE
# and the org segment is prepended server-side from the validated principal
# (cloud apps/projects/blob.go:60, `sitePrefix(org, slug) = org + "/" + slug`),
# which is why "<slug>" is the whole of it and why a caller cannot address
# another tenant's bytes by writing a longer path.
#
# ONE CREDENTIAL, ALREADY MINTED. HANZO_API_TOKEN is the IAM JWT the workflow's
# KMS step already holds: cloud's /v1/kms/auth/login is a broker that performs
# the IAM client_credentials exchange and returns IAM's own JWT verbatim, so the
# token that reads this org's secrets is the token that publishes this org's
# site. api.hanzo.ai validates it and mints X-Org-Id from the `owner` claim —
# identity is never a header this script sends, so there is nothing here to
# spoof. Do NOT reintroduce S3_ADMIN_*; a standing bucket key was what this
# replaced.
#
# THE SIZE BOUNDARY IS THE SERVER'S, NOT A PREFERENCE. cloud's public edge sets
# BodyLimit from GATEWAY_BODY_LIMIT, default 16 MiB (cloud config.go:312), and
# fasthttp refuses an oversized POST BEFORE any handler runs — it surfaces as an
# opaque 400 "Error when parsing request" that reads like a malformed payload
# rather than a size cap (cloud apps/projects/grant.go:5 tells that story about
# a ~170 MB export). So the zip is measured HERE and refused HERE, with the
# number and the alternative, instead of being sent to fail unreadably. Measured
# across the estate's 24 built exports, 22 fit; hanzo.ai (27.9 MiB zipped, 8536
# files) and trillerfest.com (76.7 MiB) do not. Those go to bin/sitedeploy,
# which streams per-file against a presigned grant and has no body limit —
# hanzo.ai is over the server's own 5000-entry cap anyway, so no transport makes
# it a Release.
#
# ENV: HANZO_API_TOKEN (required) the IAM bearer; the workflow's KMS step mints it
# HANZO_API (https://api.hanzo.ai)
# SITEPUBLISH_MAX_ZIP (16777216) mirror of the server's GATEWAY_BODY_LIMIT
# SITEPUBLISH_PLAN=1 print what it would send and exit; no network. Test seam.
set -euo pipefail
slug="${1:?usage: sitepublish <slug> <dir>}"
dir="${2:?usage: sitepublish <slug> <dir>}"
api="${HANZO_API:-https://api.hanzo.ai}"
# Server-side caps, mirrored so a breach is named in CI instead of arriving as a
# 413 (or, for the body limit, as an unreadable 400). cloud apps/projects/blob.go
# :29-31. bin/sitepublish_test.sh pins these numbers; if cloud moves them the
# test is what catches the drift.
max_zip="${SITEPUBLISH_MAX_ZIP:-16777216}" # gateway BodyLimit, 16 MiB
max_files=5000 # maxFiles
max_file_bytes=$((64 << 20)) # maxFileBytes, 64 MiB
max_total_bytes=$((512 << 20)) # maxTotalBytes uncompressed, 512 MiB
die() { echo "::error::$*"; exit 1; }
# The slug is a URL path segment AND an S3 key segment, so it is held to the
# grammar cloud validates project slugs with — checked before anything is built
# or sent, because a bad slug is a typo to fix and not a 404 to interpret.
echo "$slug" | grep -qE '^[a-z0-9]([a-z0-9-]{0,38}[a-z0-9])?$' \
|| die "slug '$slug' is not a project slug (^[a-z0-9]([a-z0-9-]{0,38}[a-z0-9])?\$)"
[ -d "$dir" ] || die "'$dir' is not a directory — the build produced no export"
# A site is a thing with an index. cloud enforces it however the bytes arrive;
# checking here turns a failed build into a failed publish with a readable cause
# rather than a promoted release that 404s at its own root.
[ -f "$dir/index.html" ] || die "'$dir' has no index.html at its root — /v1/sites refuses a source without one"
# CNAME does not travel: a GitHub Pages artifact that means nothing to S3 and
# would ship a stale hostname claim into the bucket. Same exclusion bin/sitedeploy
# makes, for the same reason.
files=$(cd "$dir" && find . -type f ! -name CNAME | wc -l)
[ "$files" -gt 0 ] || die "'$dir' contains no files — refusing to publish an empty release"
[ "$files" -le "$max_files" ] \
|| die "'$dir' holds $files files; cloud caps one artifact at $max_files (apps/projects/blob.go maxFiles). Use bin/sitedeploy — but note the Releases plane has the same cap, so this export cannot become a Release."
bytes=$(cd "$dir" && find . -type f ! -name CNAME -exec wc -c {} + | awk '$2!="total"{s+=$1} END{print s+0}')
[ "$bytes" -le "$max_total_bytes" ] \
|| die "'$dir' is $bytes bytes uncompressed; cloud caps an artifact at $max_total_bytes (maxTotalBytes)"
# Per-file cap, checked with find rather than a loop so a 9000-file export costs
# one traversal. -size uses 512-byte blocks with `c` for bytes; +N c is "strictly
# greater than N bytes", which is the cap's own boundary.
big=$(cd "$dir" && find . -type f ! -name CNAME -size +${max_file_bytes}c -printf '%P (%s bytes)\n' | head -3)
[ -z "$big" ] || die "these files exceed cloud's ${max_file_bytes}-byte per-file cap (maxFileBytes):
$big"
if [ -n "${SITEPUBLISH_PLAN:-}" ]; then
printf 'slug=%s dir=%s files=%s bytes=%s api=%s source=%s\n' \
"$slug" "$dir" "$files" "$bytes" "$api" "$slug"
exit 0
fi
# No apostrophe in this message: inside "${VAR:?word}" a single quote opens a
# quoted section for the PARSER, and the script dies at EOF with "unexpected EOF
# while looking for matching quote" — a syntax error reported at the last line,
# nowhere near the one that caused it.
: "${HANZO_API_TOKEN:?HANZO_API_TOKEN is unset. It is the IAM bearer minted by the KMS step of the reusable workflow, from KMS_CLIENT_ID/KMS_CLIENT_SECRET; there is no second credential to seal.}"
tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
zipf="$tmp/site.zip"
# -X drops extra file attributes (uid/gid/timestamps beyond the DOS fields) so
# the same tree zips to the same bytes on any runner. Entries are stored
# RELATIVE to the export root — cloud strips a leading "./" and refuses absolute
# or ".."-escaping names (blob.go:362-370), and strips a common top-level prefix
# if one exists, so a flat archive is what it expects and what this sends.
( cd "$dir" && zip -qXr "$zipf" . -x './CNAME' ) || die "zip of '$dir' failed"
zbytes=$(wc -c < "$zipf")
# The refusal the whole size story is about. Above this the POST dies at the
# edge with a 400 that names nothing.
[ "$zbytes" -le "$max_zip" ] || die "the zipped export is $zbytes bytes, past cloud's ${max_zip}-byte edge BodyLimit (GATEWAY_BODY_LIMIT, cloud config.go). fasthttp refuses the POST before any handler runs and reports only 'Error when parsing request', so this is refused here where the number is visible. Publish this export with bin/sitedeploy, which streams per-file against a presigned grant and has no body limit."
echo "publishing $slug — $files files, $bytes bytes ($zbytes zipped)"
# ---- 1. the bytes -----------------------------------------------------------
# Every response body is KEPT and printed on failure. cloud answers these routes
# with a specific status per cause — 402 hosting gate, 403 wrong org, 404 no such
# site for this tenant, 409 the source moved mid-publish, 413 past the caps, 503
# storage unconfigured — and each is a different fix, so discarding the body
# discards the answer.
code=$(curl -sS -o "$tmp/deploy.json" -w '%{http_code}' \
-X POST "$api/v1/projects/$slug/deploy" \
-H "Authorization: Bearer $HANZO_API_TOKEN" \
-H 'Content-Type: application/zip' \
--data-binary "@$zipf") \
|| die "POST $api/v1/projects/$slug/deploy did not complete"
case "$code" in
2??) ;;
*) echo "::error::upload $slug → HTTP $code"; head -c 800 "$tmp/deploy.json"; echo; exit 1 ;;
esac
# ---- 2. promote + activate --------------------------------------------------
# `source` is the org-relative prefix the upload just landed at. Content types
# and cache policy are not ours to set: copyRelease rewrites the metadata on
# every object it promotes, so a release serves identically however its bytes
# were staged.
code=$(curl -sS -o "$tmp/publish.json" -w '%{http_code}' \
-X POST "$api/v1/sites/$slug/publish" \
-H "Authorization: Bearer $HANZO_API_TOKEN" \
-H 'Content-Type: application/json' \
-d "$(jq -nc --arg s "$slug" '{source:$s}')") \
|| die "POST $api/v1/sites/$slug/publish did not complete"
case "$code" in
2??) ;;
*) echo "::error::publish $slug → HTTP $code"; head -c 800 "$tmp/publish.json"; echo; exit 1 ;;
esac
# Guard the SHAPE before reading a field out of it. `jq -r '.releaseId // empty'`
# on an error body, an array, or a null yields "" and every later test on it
# passes vacuously — which is the defect class this whole script is written
# against. Assert it is an object carrying the field, then read it.
jq -e 'type == "object" and (.releaseId | type) == "string" and (.releaseId | length) > 0' \
"$tmp/publish.json" >/dev/null \
|| { echo "::error::publish answered $code but the body is not a release object with a releaseId:"; head -c 800 "$tmp/publish.json"; echo; exit 1; }
rid=$(jq -r '.releaseId' "$tmp/publish.json")
objs=$(jq -r '.objects // 0' "$tmp/publish.json")
url=$(jq -r '.url // ""' "$tmp/publish.json")
# ---- 3. verify the release we just made is the one that is live -------------
# A 200 from publish says the request was accepted, not that this release is
# serving. The list is the only thing that can say so.
#
# THE LIST IS A BARE JSON ARRAY — `type projectsReleases []projectsRelease`
# (cloud apps/projects/release.go:518). `.releases[]` against it resolves
# NOTHING, and an assertion that resolves nothing is an assertion that cannot
# fail for the reason it was written. Hence `.[]`, and hence the shape guard
# first: if cloud ever wraps this in an object, THAT is what goes red, loudly,
# instead of the check quietly matching zero releases forever.
code=$(curl -sS -o "$tmp/releases.json" -w '%{http_code}' \
"$api/v1/sites/$slug/releases" \
-H "Authorization: Bearer $HANZO_API_TOKEN") \
|| die "GET $api/v1/sites/$slug/releases did not complete"
case "$code" in
2??) ;;
*) echo "::error::list releases for $slug → HTTP $code"; head -c 800 "$tmp/releases.json"; echo; exit 1 ;;
esac
jq -e 'type == "array" and length > 0' "$tmp/releases.json" >/dev/null \
|| { echo "::error::GET /v1/sites/$slug/releases did not answer a non-empty JSON array (it is the bare-array shape from release.go:518). Body:"; head -c 800 "$tmp/releases.json"; echo; exit 1; }
# Exactly one release is active. Counting first is what makes "none active"
# distinguishable from "the wrong one is active" — a bare grep for '"active":true'
# matches ANY release in the list and would pass on both.
nactive=$(jq '[.[] | select(.active == true)] | length' "$tmp/releases.json")
[ "$nactive" = 1 ] \
|| { echo "::error::expected exactly 1 active release for $slug, found $nactive:"; jq -c '[.[]|{releaseId,active}]' "$tmp/releases.json"; exit 1; }
active=$(jq -r 'map(select(.active == true))[0].releaseId // ""' "$tmp/releases.json")
# Both sides proven non-empty BEFORE they are compared. `[ "$a" = "$b" ]` with
# two empty strings is TRUE, so an equality test on unguarded values reports
# success precisely when it learned nothing.
[ -n "$active" ] || { echo "::error::the active release for $slug carries no releaseId:"; jq -c '.' "$tmp/releases.json"; exit 1; }
[ -n "$rid" ] || die "publish returned no releaseId to verify against"
[ "$active" = "$rid" ] \
|| { echo "::error::published $rid but $active is live for $slug — the flip did not take"; exit 1; }
echo "live: $rid ($objs objects) → ${url:-no public host — that slug is claimed; rename the site or bind a domain}"
if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
{ echo "### Site — \`$slug\`"; echo
echo "| release | objects | files | zipped | url |"
echo "|---|---|---|---|---|"
echo "| \`$rid\` | $objs | $files | $zbytes B | ${url:-—} |"
} >> "$GITHUB_STEP_SUMMARY"
fi
+168
View File
@@ -0,0 +1,168 @@
#!/usr/bin/env bash
# Tests for bin/sitepublish. Runs OFFLINE and deterministically: a `curl` shim on
# PATH answers the three routes from fixtures, so every case here — including the
# ones that must FAIL — needs no token, no bucket and no cluster.
# Run: bash bin/sitepublish_test.sh
#
# The suite is weighted toward REFUSALS on purpose. A publish step that reports
# success when it verified nothing is worse than one that does not verify at all,
# because it is indistinguishable from a working one until a site silently stops
# updating. Three real defects of that exact shape are pinned below by name:
#
# • an object-shaped read (`.releaseId`) against a body that is not an object
# • `.releases[]` against the BARE ARRAY /v1/sites/<slug>/releases returns,
# which resolves nothing and therefore can never fail for its stated reason
# • `[ "$a" = "$b" ]` on two values that are both empty, which is TRUE
set -uo pipefail
cd "$(dirname "$0")/.."
SP="$PWD/bin/sitepublish"
tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
fail=0
t() { # t <name> <want_rc> <got_rc> [<must-contain> <output>]
local name="$1" want="$2" got="$3" needle="${4:-}" out="${5:-}"
if [ "$got" != "$want" ]; then
printf 'FAIL %-58s rc=%s (want %s)\n' "$name" "$got" "$want"; fail=1; return
fi
if [ -n "$needle" ] && ! printf '%s' "$out" | grep -qF -- "$needle"; then
printf 'FAIL %-58s rc=%s but missing %q\n' "$name" "$got" "$needle"; fail=1
printf ' got: %s\n' "$(printf '%s' "$out" | head -c 300)"; return
fi
printf 'ok %-58s rc=%s\n' "$name" "$got"
}
# ---- a curl shim -------------------------------------------------------------
# Dispatches on the URL and writes the fixture the scenario names to the file the
# real curl would have written, then prints the status the way `-w %{http_code}`
# does. Everything the script does with a response goes through this, so the
# tests exercise the REAL parsing, not a mock of it.
shim="$tmp/bin"; mkdir -p "$shim"
cat > "$shim/curl" <<'SHIM'
#!/usr/bin/env bash
out=/dev/null url=
while [ $# -gt 0 ]; do
case "$1" in
-o) out="$2"; shift 2 ;;
http*|https*) url="$1"; shift ;;
*) shift ;;
esac
done
case "$url" in
*/deploy) printf '%s' "${T_DEPLOY_BODY:-{\"ok\":true\}}" > "$out"; printf '%s' "${T_DEPLOY_CODE:-200}" ;;
*/publish) printf '%s' "${T_PUBLISH_BODY:-}" > "$out"; printf '%s' "${T_PUBLISH_CODE:-200}" ;;
*/releases) printf '%s' "${T_LIST_BODY:-}" > "$out"; printf '%s' "${T_LIST_CODE:-200}" ;;
*) echo "shim: unexpected url $url" >&2; exit 9 ;;
esac
SHIM
chmod +x "$shim/curl"
site="$tmp/site"; mkdir -p "$site/assets"
echo '<h1>hi</h1>' > "$site/index.html"
echo 'body{}' > "$site/assets/app.css"
echo 'hanzo.ai' > "$site/CNAME"
run() { # run <dir> — publish with the shim on PATH, current T_* scenario
PATH="$shim:$PATH" HANZO_API_TOKEN=tok HANZO_API=https://api.test \
bash "$SP" a-slug "$1" 2>&1
}
OK_PUB='{"releaseId":"rel-1","slug":"a-slug","objects":2,"active":true,"url":"https://a-slug.hanzo.page"}'
OK_LIST='[{"releaseId":"rel-1","active":true},{"releaseId":"rel-0","active":false}]'
# ---- the plan seam: no network, no token ------------------------------------
out=$(SITEPUBLISH_PLAN=1 bash "$SP" a-slug "$site" 2>&1); rc=$?
t "plan prints the manifest and exits" 0 $rc "files=2" "$out"
# CNAME is a GitHub Pages artifact: it means nothing to S3 and would ship a stale
# hostname claim. Two files, not three.
t "plan excludes CNAME from the count" 0 $rc "files=2 " "$out"
t "plan sends source=<slug>, org-relative" 0 $rc "source=a-slug" "$out"
# ---- refusals that need no network ------------------------------------------
out=$(SITEPUBLISH_PLAN=1 bash "$SP" 'Bad_Slug' "$site" 2>&1); rc=$?
t "an invalid slug is refused before anything" 1 $rc "is not a project slug" "$out"
out=$(SITEPUBLISH_PLAN=1 bash "$SP" a-slug "$tmp/nope" 2>&1); rc=$?
t "a missing export dir is refused" 1 $rc "is not a directory" "$out"
noidx="$tmp/noidx"; mkdir -p "$noidx"; echo x > "$noidx/page.html"
out=$(SITEPUBLISH_PLAN=1 bash "$SP" a-slug "$noidx" 2>&1); rc=$?
t "an export with no index.html is refused" 1 $rc "no index.html at its root" "$out"
empty="$tmp/empty"; mkdir -p "$empty"; echo x > "$empty/index.html"; rm "$empty/index.html"
out=$(SITEPUBLISH_PLAN=1 bash "$SP" a-slug "$empty" 2>&1); rc=$?
t "an empty export is refused" 1 $rc "no index.html" "$out"
# The size refusal is the whole reason this script measures before it sends:
# past the edge BodyLimit fasthttp rejects the POST before any handler runs and
# reports only "Error when parsing request", which names neither size nor cause.
out=$(SITEPUBLISH_MAX_ZIP=1 PATH="$shim:$PATH" HANZO_API_TOKEN=tok \
bash "$SP" a-slug "$site" 2>&1); rc=$?
t "a zip past the edge BodyLimit is refused here" 1 $rc "BodyLimit" "$out"
t " ...and the refusal names bin/sitedeploy" 1 $rc "bin/sitedeploy" "$out"
# The server caps one artifact at 5000 entries; a 5001-file export can never
# become a Release by ANY transport, so saying so here beats a 413 later.
many="$tmp/many"; mkdir -p "$many"; echo x > "$many/index.html"
( cd "$many" && touch f{1..5001} )
out=$(SITEPUBLISH_PLAN=1 bash "$SP" a-slug "$many" 2>&1); rc=$?
t "an export past maxFiles=5000 is refused" 1 $rc "cloud caps one artifact at 5000" "$out"
# ---- the happy path ----------------------------------------------------------
out=$(T_PUBLISH_BODY="$OK_PUB" T_LIST_BODY="$OK_LIST" run "$site"); rc=$?
t "publish + verified flip succeeds" 0 $rc "live: rel-1" "$out"
# ---- transport failures carry the server's answer ---------------------------
# Each status is a different fix, so the body is printed rather than swallowed.
out=$(T_DEPLOY_CODE=413 T_DEPLOY_BODY='{"error":"artifact exceeds"}' run "$site"); rc=$?
t "a non-2xx upload fails and prints the body" 1 $rc "artifact exceeds" "$out"
out=$(T_PUBLISH_CODE=402 T_PUBLISH_BODY='{"error":"hosting not enabled"}' run "$site"); rc=$?
t "a non-2xx publish fails and prints the body" 1 $rc "hosting not enabled" "$out"
out=$(T_PUBLISH_BODY="$OK_PUB" T_LIST_CODE=503 T_LIST_BODY='{"error":"storage"}' run "$site"); rc=$?
t "a non-2xx release list fails" 1 $rc "list releases" "$out"
# ---- defect 1: an object-shaped read against a non-object -------------------
# `jq -r '.releaseId // empty'` yields "" for an array, a null, or an error body,
# and every later test on "" passes vacuously. The shape is asserted first.
out=$(T_PUBLISH_BODY='[{"releaseId":"rel-1"}]' T_LIST_BODY="$OK_LIST" run "$site"); rc=$?
t "publish answering an ARRAY is refused" 1 $rc "not a release object" "$out"
out=$(T_PUBLISH_BODY='{"ok":true}' T_LIST_BODY="$OK_LIST" run "$site"); rc=$?
t "publish answering no releaseId is refused" 1 $rc "not a release object" "$out"
out=$(T_PUBLISH_BODY='{"releaseId":""}' T_LIST_BODY="$OK_LIST" run "$site"); rc=$?
t "publish answering an EMPTY releaseId is refused" 1 $rc "not a release object" "$out"
out=$(T_PUBLISH_BODY='not json' T_LIST_BODY="$OK_LIST" run "$site"); rc=$?
t "publish answering non-JSON is refused" 1 $rc "not a release object" "$out"
# ---- defect 2: the list is a BARE ARRAY -------------------------------------
# release.go:518 is `type projectsReleases []projectsRelease`. A `.releases[]`
# filter resolves NOTHING against that, so an assertion built on it can only ever
# refuse — it never once tested what it claimed to. The guard demands the array
# shape, so the day cloud wraps the list THAT is what goes red, by name.
out=$(T_PUBLISH_BODY="$OK_PUB" T_LIST_BODY='{"releases":[{"releaseId":"rel-1","active":true}]}' run "$site"); rc=$?
t "a WRAPPED release list is refused, loudly" 1 $rc "bare-array shape" "$out"
out=$(T_PUBLISH_BODY="$OK_PUB" T_LIST_BODY='[]' run "$site"); rc=$?
t "an EMPTY release list is refused" 1 $rc "non-empty JSON array" "$out"
out=$(T_PUBLISH_BODY="$OK_PUB" T_LIST_BODY='null' run "$site"); rc=$?
t "a null release list is refused" 1 $rc "non-empty JSON array" "$out"
# ---- defect 3: counting, so "none" cannot read as "yes" ---------------------
# A bare `grep '"active":true'` matches ANY release in the list, so it passes
# both when the wrong release is live and when the list merely mentions one.
out=$(T_PUBLISH_BODY="$OK_PUB" T_LIST_BODY='[{"releaseId":"rel-1","active":false}]' run "$site"); rc=$?
t "ZERO active releases is refused" 1 $rc "expected exactly 1 active release" "$out"
out=$(T_PUBLISH_BODY="$OK_PUB" \
T_LIST_BODY='[{"releaseId":"rel-1","active":true},{"releaseId":"rel-2","active":true}]' run "$site"); rc=$?
t "TWO active releases is refused" 1 $rc "expected exactly 1 active release" "$out"
out=$(T_PUBLISH_BODY="$OK_PUB" T_LIST_BODY='[{"releaseId":"rel-9","active":true}]' run "$site"); rc=$?
t "a DIFFERENT release being live is refused" 1 $rc "the flip did not take" "$out"
# The both-empty comparison, head on: an active entry whose releaseId is "".
# `[ "$active" = "$rid" ]` with both empty is TRUE and would report success on a
# release that does not exist. Non-emptiness is proven before the comparison.
out=$(T_PUBLISH_BODY="$OK_PUB" T_LIST_BODY='[{"releaseId":"","active":true}]' run "$site"); rc=$?
t "an active release with an EMPTY id is refused" 1 $rc "carries no releaseId" "$out"
out=$(T_PUBLISH_BODY="$OK_PUB" T_LIST_BODY='[{"active":true}]' run "$site"); rc=$?
t "an active release with NO id field is refused" 1 $rc "carries no releaseId" "$out"
# ---- the credential is required ---------------------------------------------
out=$(PATH="$shim:$PATH" HANZO_API_TOKEN= bash "$SP" a-slug "$site" 2>&1); rc=$?
t "a missing bearer refuses before any request" 1 $rc "HANZO_API_TOKEN is unset" "$out"
echo
[ "$fail" = 0 ] && echo "sitepublish: all tests passed" || echo "sitepublish: FAILURES"
exit "$fail"
Executable
+161
View File
@@ -0,0 +1,161 @@
#!/usr/bin/env bash
# vendormark — refuse a repo that carries a site-generator vendor's branding.
# One implementation, every caller.
#
# vendormark [dir] [git-range]
#
# WHAT THIS CATCHES
#
# Several sites in the estate began life as output from a hosted "prompt to
# React app" generator. The generator does not sign its work in a comment you
# would notice — it signs it in the SHAPE of the repo, in four places at once:
#
# 1. an asset folder named after the vendor, e.g. public/<vendor>-uploads/,
# so every image the browser requests carries the vendor's name in its URL
# 2. a build plugin dependency in package.json and the lockfiles
# 3. an injected third-party <script> tag in index.html
# 4. commit messages, author identities and Co-authored-by trailers written
# by the vendor's bot
#
# The fourth is the one nothing else looks at, and the one that survives every
# amount of tidying the working tree: a repo can render a perfect page and
# still say, in `git log`, who really wrote it.
#
# This was not a hypothetical. A sweep of 1,065 repositories across the three
# orgs found the mark in five shipped sites — one of them serving the vendor's
# upload folder as its FAVICON, so the vendor's path was in the <head> of every
# page and in every browser tab. Purging it needed a history rewrite and a
# force-push of every branch. This gate exists so that is never needed twice.
#
# WHY THE CONTENT RULES ARE SO SPECIFIC
#
# The vendor's name is also an ordinary English adjective, and the naive gate —
# case-insensitive substring of that word — is unshippable. Measured across all
# 1,065 repos, the bare word appears legitimately in 20+ of them and NONE of
# them are contaminated:
#
# tokenizer vocab.json (3 repos) "lovable</w>": 38565
# English word lists words_alpha.txt, faker adjectives
# ML training/eval corpora alpaca, kto, BAGEL eval prompts
# x/net publicsuffix packed data a substring of concatenated domains
# vendored prompt corpora third-party collections that quote it
# our own marketing copy a competitor named in a case study
#
# A gate that reds a fifth of the estate on day one is a gate someone switches
# off, and then we are worse off than before it existed. So the content rules
# are only the strings that CANNOT occur in prose: a path segment, an npm
# package name, an injected script URL, a generator meta tag, a bot's email.
#
# Two candidate rules were DROPPED after measuring them, and it is worth saying
# why: `lovableproject.com` and `lovable.app` red an upstream Clerk SDK bundle,
# which ships a list of dev-preview host suffixes naming several generators. A
# third party's honest list of other people's hosts is not our contamination.
# Those two strings are still refused in commit messages and git identities,
# where no third party can put them.
#
# PATHS AND IDENTITIES ARE ABSOLUTE
#
# Content is judged narrowly; paths and git identities are not. There is no
# legitimate reason for the vendor's name to appear in a filename we track or
# in an author, committer or Co-authored-by line we write. Those are refused on
# the bare word, no exceptions, because that is where the mark is load-bearing
# and where tidying the working tree never reaches.
#
# THE ESCAPE HATCH
#
# `.vendormark-allow` — one path glob per line, `#` comments ignored. It exists
# for exactly one honest case: a repo whose JOB is to name these vendors, such
# as the history-scrubber rule files in hanzoai/.github, which cannot do their
# work without spelling the string they remove. If you are reaching for it for
# any other reason, you are about to ship the mark.
#
# EXIT: 0 clean, 1 mark found.
set -uo pipefail
root=${1:-.}
range=${2:-}
cd "$root" 2>/dev/null || { echo "vendormark: no such directory: $root" >&2; exit 1; }
git rev-parse --git-dir >/dev/null 2>&1 || exit 0 # not a repo: nothing to check
# PATHS: the bare vendor name, no exceptions. Add a vendor here and its
# filenames are refused everywhere.
VENDORS='lovable'
# CONTENT: only strings that cannot occur in prose — an asset folder, an npm
# package, the injected script, a generator meta tag, the bot's address. Spelled
# out in full rather than composed from $VENDORS, because half of them do not
# contain the vendor's name at all. See the header for the two that were
# measured against the estate and dropped.
CONTENT_SIG='lovable-uploads/|lovable-tagger|@lovable\.dev|cdn\.gpteng\.co|gptengineer\.js|content="[Ll]ovable"'
# IDENTITIES AND MESSAGES: the bare word plus the vendor hosts, because nothing
# outside our own commits can write these.
IDENT_SIG='lovable|gptengineer|gpt-engineer'
fail=0
tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
# --- allowlist ------------------------------------------------------------
: > "$tmp/allow"
if [ -f .vendormark-allow ]; then
grep -vE '^\s*(#|$)' .vendormark-allow > "$tmp/allow" || true
fi
allowed() { # $1 = path
[ -s "$tmp/allow" ] || return 1
while IFS= read -r g; do
# shellcheck disable=SC2254
case "$1" in $g) return 0;; esac
done < "$tmp/allow"
return 1
}
# --- 1. tracked paths -----------------------------------------------------
git ls-files 2>/dev/null | grep -iE "$VENDORS" > "$tmp/paths" || true
if [ -s "$tmp/paths" ]; then
n=0
while IFS= read -r p; do allowed "$p" || { echo " path: $p"; n=$((n+1)); }; done < "$tmp/paths"
if [ "$n" -gt 0 ]; then
echo "::error::${n} tracked path(s) carry a site-generator vendor's name. Every one of them is a URL the browser requests, so the vendor's branding is in the page. Move the assets to a neutral folder (public/img/) and repoint the references in the same change — a favicon or logo that a <link> or <img> depends on must be REPLACED with the real brand mark, not merely deleted."
fail=1
fi
fi
# --- 2. tracked content ---------------------------------------------------
git grep -Iin -E "$CONTENT_SIG" HEAD 2>/dev/null | sed 's#^HEAD:##' > "$tmp/content" || true
if [ -s "$tmp/content" ]; then
n=0
while IFS= read -r line; do
p=${line%%:*}
allowed "$p" || { echo " content: $(echo "$line" | cut -c1-140)"; n=$((n+1)); }
done < "$tmp/content"
if [ "$n" -gt 0 ]; then
echo "::error::${n} tracked line(s) reference a site-generator vendor's asset folder, build plugin or injected script. Drop the dependency and the script tag; repoint the asset URLs."
fail=1
fi
fi
# --- 3. commit messages, identities, trailers -----------------------------
# Default scope is HEAD alone, which is what a fetch-depth:1 CI checkout has and
# is enough to refuse the commit being pushed. Pass a range (or set
# VENDORMARK_RANGE) to judge more, e.g. origin/main..HEAD for a whole PR.
range=${range:-${VENDORMARK_RANGE:-}}
if [ -n "$range" ]; then set -- "$range"; else set -- -1 HEAD; fi
git log "$@" --format='%H%x01%an <%ae>%x01%cn <%ce>%x01%B%x01%(trailers:unfold=true)%x02' 2>/dev/null \
| tr -d '\n' | tr '\002' '\n' > "$tmp/log" || true
if [ -s "$tmp/log" ]; then
n=0
while IFS= read -r c; do
echo "$c" | grep -qiE "$IDENT_SIG" || continue
echo " commit: $(echo "$c" | tr '\001' ' ' | cut -c1-150)"
n=$((n+1))
done < "$tmp/log"
if [ "$n" -gt 0 ]; then
echo "::error::${n} commit(s) name a site-generator vendor in the message, the author/committer identity or a Co-authored-by trailer. A working tree can be tidied; \`git log\` cannot be, short of a history rewrite and a force-push of every branch. Reword before pushing. Describe the work — \"generated placeholder assets\" — and never credit the generator."
fail=1
fi
fi
if [ "$fail" -eq 0 ]; then
echo "vendormark: OK — no site-generator branding in paths, tracked content, messages or identities"
fi
exit "$fail"
+142
View File
@@ -0,0 +1,142 @@
#!/usr/bin/env bash
# vendormark_test.sh — the refusals AND the allowances, both pinned.
#
# A gate is only worth having if it is exact in both directions. A false
# negative lets the vendor's mark back into a shipped site; a false positive
# reds a repo that was always fine, and a gate that reds honest repos is a gate
# someone switches off. So this suite asserts BOTH halves, and the allowance
# half is the larger one on purpose: the vendor's name is also an ordinary
# English adjective, and it occurs innocently in tokenizer vocabularies, word
# lists, ML corpora, packed public-suffix data and third-party host lists all
# over the estate.
#
# Offline and deterministic: temp repos, no network.
set -uo pipefail
BIN=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/vendormark
pass=0; fail=0
V=$(printf 'lo%s' 'vable') # not spelled literally, so this file does not
# itself trip the gate it is testing
newrepo() {
d=$(mktemp -d)
git -C "$d" init -q
git -C "$d" config user.email dev@hanzo.ai
git -C "$d" config user.name "Hanzo Dev"
printf 'x\n' > "$d/README.md"
git -C "$d" add -A
git -C "$d" commit -qm "initial"
echo "$d"
}
check() { # name expected_rc dir [range]
local name=$1 want=$2 dir=$3 range=${4:-}
"$BIN" "$dir" $range >/dev/null 2>&1; local got=$?
if [ "$got" = "$want" ]; then pass=$((pass+1)); echo " ok $name"
else fail=$((fail+1)); echo " FAIL $name (want rc=$want, got rc=$got)"; fi
}
echo "REFUSALS — the mark in each of the four places it hides"
d=$(newrepo)
check "clean repo is green" 0 "$d"
mkdir -p "$d/public/$V-uploads"
printf 'PNG\n' > "$d/public/$V-uploads/28d53ec4.png"
git -C "$d" add -A && git -C "$d" commit -qm "add an image"
check "MUTATION: tracked path named for the vendor" 1 "$d"
git -C "$d" rm -rq "public/$V-uploads" && git -C "$d" commit -qm "move the image"
check "MUTATION REVERTED: green again" 0 "$d"
rm -rf "$d"
d=$(newrepo)
printf '<img src="/%s-uploads/28d53ec4.png" />\n' "$V" > "$d/index.html"
git -C "$d" add -A && git -C "$d" commit -qm "a page"
check "content: asset folder referenced in markup" 1 "$d"
rm -rf "$d"
d=$(newrepo)
printf '{"devDependencies":{"%s-tagger":"^1.1.3"}}\n' "$V" > "$d/package.json"
git -C "$d" add -A && git -C "$d" commit -qm "deps"
check "content: the vendor's build plugin in package.json" 1 "$d"
rm -rf "$d"
d=$(newrepo)
printf '<script src="https://cdn.gpteng.co/gptengineer.js"></script>\n' > "$d/index.html"
git -C "$d" add -A && git -C "$d" commit -qm "a page"
check "content: the vendor's injected script tag" 1 "$d"
rm -rf "$d"
d=$(newrepo)
printf '<meta name="generator" content="Lovable" />\n' > "$d/index.html"
git -C "$d" add -A && git -C "$d" commit -qm "a page"
check "content: the generator meta tag" 1 "$d"
rm -rf "$d"
d=$(newrepo)
printf 'y\n' > "$d/a.txt"; git -C "$d" add -A
git -C "$d" commit -qm "Update $V project template"
check "MUTATION: the vendor named in a commit message" 1 "$d"
git -C "$d" commit -q --amend -m "update the project template"
check "MUTATION REVERTED: reworded message is green" 0 "$d"
rm -rf "$d"
d=$(newrepo)
printf 'y\n' > "$d/a.txt"; git -C "$d" add -A
# --no-verify ON PURPOSE. A commit-msg hook on the workstation
# (~/.githooks/commit-msg) already strips vendor co-author trailers, and with it
# enabled this case cannot be constructed — the hook silently rewrites the
# trailer to ours and the assertion passes for the wrong reason. That hook is
# per-machine: it does not run on a runner, on a teammate's laptop, or on a
# commit made through the GitHub web UI. This gate is the layer that does. So
# the test bypasses the hook to prove the GATE catches what the hook would have.
git -C "$d" commit -q --no-verify -m "a change
Co-authored-by: $V bot <bot@$V.dev>"
check "MUTATION: Co-authored-by credits the vendor" 1 "$d"
git -C "$d" commit -q --amend --no-verify -m "a change"
check "MUTATION REVERTED: trailer dropped is green" 0 "$d"
rm -rf "$d"
d=$(newrepo)
printf 'y\n' > "$d/a.txt"; git -C "$d" add -A
git -C "$d" -c user.name="$V" -c user.email="bot@$V.dev" commit -qm "a change"
check "MUTATION: the vendor as author identity" 1 "$d"
rm -rf "$d"
d=$(newrepo)
printf 'y\n' > "$d/a.txt"; git -C "$d" add -A; git -C "$d" commit -qm "clean subject"
printf 'z\n' > "$d/b.txt"; git -C "$d" add -A; git -C "$d" commit -qm "Visual edit in $V"
printf 'w\n' > "$d/c.txt"; git -C "$d" add -A; git -C "$d" commit -qm "clean again"
check "default scope (HEAD only) misses an older bad message" 0 "$d"
check "explicit range catches it" 1 "$d" "HEAD~3..HEAD"
rm -rf "$d"
echo
echo "ALLOWANCES — the same word, innocently, as it really occurs in the estate"
d=$(newrepo)
printf ' "%s</w>": 38565,\n' "$V" > "$d/vocab.json"
printf '%s\n' "$V" > "$d/words_alpha.txt"
printf 'Tell me a story about a %s character.\n' "$V" > "$d/harmless.txt"
printf 'caseStudy: %s and v0 activate users by turning a prompt into an app\n' "${V^}" > "$d/guide.yaml"
printf 'mail2%s.com\n' "$V" > "$d/generic_emails.txt"
printf 'let e=[".%s.app",".%sproject.com",".webcontainer-api.io"];\n' "$V" "$V" > "$d/clerk-bundle.js"
printf 'A collection of UI components. Integrate them in v0, %s, Bolt.\n' "${V^}" > "$d/registries.json"
git -C "$d" add -A && git -C "$d" commit -qm "corpora, word lists and third-party bundles"
check "tokenizer vocab / word list / ML corpus / marketing prose / blocklist / upstream host list" 0 "$d"
rm -rf "$d"
echo
echo "ESCAPE HATCH — a repo whose job is to name the vendor"
d=$(newrepo)
mkdir -p "$d/tools/rules"
printf 'Hanzo Dev <dev@hanzo.ai> <bot@%s.dev>\n' "$V" > "$d/tools/rules/mailmap.txt"
git -C "$d" add -A && git -C "$d" commit -qm "scrubber rules"
check "scrubber rule file is refused by default" 1 "$d"
printf 'tools/rules/*\n' > "$d/.vendormark-allow"
git -C "$d" add -A && git -C "$d" commit -qm "declare the rule files"
check "...and allowed once declared in .vendormark-allow" 0 "$d"
rm -rf "$d"
echo
echo "vendormark_test: $pass passed, $fail failed"
[ "$fail" -eq 0 ]
+34
View File
@@ -121,6 +121,24 @@ test:
# IfNotPresent never picks up and no reader can see. Offline and
# deterministic: the registry floor is injected, so it needs no network.
run: bash bin/imgver_test.sh
- name: vendormark
# bin/vendormark is the gate the build lane runs against every checkout: it
# refuses a site-generator vendor's branding in a tracked path, in tracked
# content, in a commit message, in an author identity or in a Co-authored-by
# trailer. Five shipped sites carried it — one serving the vendor's upload
# folder as its favicon — and clearing them took a history rewrite and a
# force-push of every branch.
#
# The suite pins the refusals AND the allowances, and the allowances are the
# larger half on purpose. The vendor's name is also an ordinary English
# adjective: across the 1,065 repos swept it appears innocently in tokenizer
# vocabularies, word lists, ML corpora, packed public-suffix data and an
# upstream Clerk bundle's list of other people's dev hosts. A gate that reds
# a fifth of the estate is a gate someone switches off, so the content rules
# are only strings that cannot occur in prose, and two candidate rules were
# measured and DROPPED for redding that Clerk bundle. Offline and
# deterministic: temp repos, no network.
run: bash bin/vendormark_test.sh
- name: gover
# bin/gover is the gate the build lane runs against every Dockerfile before
# it builds: it refuses a Go builder image older than the go.mod it
@@ -137,6 +155,22 @@ test:
# and a multi-module repo is judged by its NEAREST go.mod. Offline and
# deterministic: temp dirs, no registry, no network.
run: bash bin/gover_test.sh
- name: sitepublish
# bin/sitepublish is the whole of the `site:` lane. The lane it replaced was
# dead: it staged with `mc mirror` behind S3_ADMIN_ACCESS_KEY/SECRET_KEY,
# names KMS holds for no org, so it failed closed for every caller — and of
# the 155 hanzo.yml files across the three orgs, not one declared a `site:`.
#
# The suite is weighted toward REFUSALS because the failure that matters here
# is a green publish that verified nothing — indistinguishable from a working
# one until a site quietly stops updating. Three defects of exactly that
# shape are pinned by name: an object-shaped read of a body that is not an
# object, `.releases[]` against the BARE ARRAY the list route returns (which
# resolves nothing and so can never fail for its stated reason), and
# `[ "$a" = "$b" ]` on two empty strings, which is TRUE. Reintroducing them
# turns 3, 9 and 1 of these tests red respectively. Offline and
# deterministic: a curl shim answers the routes from fixtures, no network.
run: bash bin/sitepublish_test.sh
# No `deploy:` ON PURPOSE. Rollout is a reviewed tag pin in hanzoai/universe
# (infra/k8s/operator/crs/ci.yaml), the same rule cloud and git follow: a