mirror of
https://github.com/hanzoai/ci.git
synced 2026-10-20 15:28:05 +00:00
zeekay
28c0db22ea
The twenty sampled assets were asked one after another, each a round trip through the edge to the origin: 10.4 s from dgx, seconds of every publish from a runner. They go out together now, and one that does not answer still sends everything. Hashing runs one batch per core into a file of its own, sorted after, so the record is the same bytes in any order. A last line, `timing:`, names each phase's seconds.
696 lines
38 KiB
Bash
Executable File
696 lines
38 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# site — publish a built static export to the Hanzo Sites plane.
|
|
# One implementation, every static site.
|
|
#
|
|
# site <slug> <dir> # e.g. site hanzo-ai out
|
|
#
|
|
# WHY THIS IS A SCRIPT AND NOT INLINE SHELL: the same forty lines of enqueue →
|
|
# upload → complete were about to be pasted into hanzo.ai, hanzo.app, hips,
|
|
# computer and every static surface after them. Written N times it is right N-1
|
|
# times and then wrong once — which is exactly how `sha-<short>` became the only
|
|
# tag eleven repos ever published (see bin/imgver). The plane's contract lives
|
|
# here, once.
|
|
#
|
|
# THE THREE STEPS, and why the bytes never pass through the API:
|
|
#
|
|
# POST /v1/projects/<slug>/deployments -> 202 {id, bucket, prefix, upload}
|
|
# POST <upload.url> per file -> the bytes, straight to S3
|
|
# POST /v1/projects/<slug>/deployments/<id>/complete
|
|
#
|
|
# A real export is large — hanzo.ai is 128 MB across 8403 files — and cloud's
|
|
# BodyLimit is 16 MiB, so an artifact POST is refused by fasthttp BEFORE any
|
|
# handler runs, and it fails as an opaque 400 "Error when parsing request" that
|
|
# reads like a malformed payload rather than a size cap. The git source is the
|
|
# documented route for exactly this shape.
|
|
#
|
|
# THE ONE BODY THIS LANE STILL POSTS is the completion's `keys` manifest, and it
|
|
# grows with the object count — so the same edge limit still bounds it, and cloud
|
|
# caps nothing on that route (apps/projects/deploy.go:388 declares Keys with no
|
|
# length check). It is therefore built and MEASURED before the enqueue: past the
|
|
# limit this refuses with the number, having sent nothing. Measured at the point
|
|
# it is sent instead, the same refusal arrives as that opaque 400 after every
|
|
# object is already in the bucket and the deployment is already open.
|
|
#
|
|
# NO STANDING S3 CREDENTIAL. The 202 carries `upload`: a presigned POST policy
|
|
# that is prefix-scoped (starts-with $key "<org>/<slug>/", enforced by S3 itself),
|
|
# short-lived (30 min) and size-bounded. That replaced handing every repo the
|
|
# bucket's own long-lived access key — one key for a bucket whose only tenant
|
|
# separation is the key prefix, so every repo holding it could overwrite EVERY
|
|
# org's site (cloud apps/projects/grant.go). Do not reintroduce SITES_S3_* here.
|
|
# HANZO_DEPLOY_TOKEN is the ONE credential this needs.
|
|
#
|
|
# DELETION IS THE SERVER'S. The grant authorizes writes only, so CI cannot remove
|
|
# a file; `keys` in the completion is the manifest cloud reconciles the prefix
|
|
# against, and it deletes what the build no longer produces. That is where
|
|
# `aws s3 sync --delete` went. It fails CLOSED on an empty manifest, and so does
|
|
# this script — a build that enumerated nothing has failed, and honouring it
|
|
# literally would delete the live site.
|
|
#
|
|
# ENV: HANZO_DEPLOY_TOKEN (required) the org-scoped `sk-` key; mint at POST /v1/keys
|
|
# HANZO_API (https://api.hanzo.ai)
|
|
# GATEWAY_BODY_LIMIT (16777216) the edge's own name for its own limit
|
|
# SITE_JOBS (24) parallel uploads
|
|
# SITE_COMMIT / SITE_BRANCH recorded on the deployment
|
|
# SITE_URL the public host serving the prefix; the previous release's
|
|
# hash manifest is read from it (default: the project's liveUrl)
|
|
# SITE_COPY_FROM <slug>:<dir> of a site just published from the same build;
|
|
# its identical objects are copied by the plane, not sent
|
|
# SITE_PLAN=1 print the manifest and exit; no network. The test seam.
|
|
set -euo pipefail
|
|
|
|
slug="${1:?usage: sitedeploy <slug> <dir>}"
|
|
dir="${2:?usage: sitedeploy <slug> <dir>}"
|
|
api="${HANZO_API:-https://api.hanzo.ai}"
|
|
jobs="${SITE_JOBS:-24}"
|
|
|
|
# `16 << 20`, quoted from cloud internal/edge/edge.go:38 so a reader can check it
|
|
# against the source rather than trust it. That package is the one definition;
|
|
# config.go:312 consumes it and cmd/cloud/main.go:124 binds it to the public
|
|
# door's fasthttp socket, so it is the outermost thing a request meets. The env
|
|
# var carries the SERVER'S OWN NAME: one value, one name, wherever it travels —
|
|
# a runner told a different edge limit is told it once and both lanes obey.
|
|
#
|
|
# READ THE WAY THE SERVER READS IT. edge.envInt (edge.go:40) is TrimSpace, then
|
|
# strconv.Atoi, then `err != nil || n <= 0` falls back to the default — so the
|
|
# server SHRUGS at a malformed value and enforces 16 MiB. Bash does not shrug:
|
|
# `[ "$body" -le "$max_body" ]` with a non-numeric operand is an error, the `||`
|
|
# branch fires, and the publish refuses against a limit that was never in force,
|
|
# printing a number that does not exist. One typo in one variable would stop
|
|
# every publisher, fail-CLOSED. Mirrored here so the two cannot disagree about
|
|
# the value of the same variable.
|
|
max_body="${GATEWAY_BODY_LIMIT-}"
|
|
max_body="${max_body#"${max_body%%[![:space:]]*}"}" # TrimSpace
|
|
max_body="${max_body%"${max_body##*[![:space:]]}"}"
|
|
max_body="${max_body#+}" # Atoi accepts a leading sign
|
|
case "$max_body" in ''|*[!0-9]*) max_body=16777216 ;; esac
|
|
# 18 digits: past that bash's own `-gt` errors, exactly where Atoi returns an
|
|
# error and Go keeps its default. The two agree everywhere below 10^18 bytes.
|
|
{ [ "${#max_body}" -le 18 ] && [ "$max_body" -gt 0 ]; } || max_body=16777216
|
|
|
|
# LC_ALL=C for the same reason the manifest sort below carries it: under a
|
|
# comma-decimal locale awk renders "16,0 MiB". The number in a refusal is read
|
|
# by people and matched by tests, and neither wants it to depend on the runner.
|
|
human() { LC_ALL=C awk -v b="$1" 'BEGIN{ split("B KiB MiB GiB", u, " "); i = 1
|
|
while (b >= 1024 && i < 4) { b /= 1024; i++ }
|
|
printf (i == 1 ? "%d %s" : "%.1f %s"), b, u[i] }'; }
|
|
# A private scratch dir, not fixed /tmp names: two sites deploying at once on a
|
|
# runner that is not a container would otherwise share sd-enq.json, and the
|
|
# second grant would send the first site's files into the other's prefix.
|
|
work=$(mktemp -d); trap 'rm -rf "$work"' EXIT
|
|
|
|
# ---- where the time goes ------------------------------------------------------
|
|
# One line at the end names each phase's seconds, so a slow publish says which
|
|
# leg was slow on the run that was slow, rather than after a second run with a
|
|
# stopwatch. EPOCHREALTIME is bash 5's; an older bash counts whole seconds.
|
|
lap() {
|
|
local t=${EPOCHREALTIME:-$SECONDS}
|
|
printf '%s %s\n' "$1" "${t/,/.}" >> "$work/laps"
|
|
}
|
|
laps() {
|
|
[ -s "$work/laps" ] || return 0
|
|
LC_ALL=C awk 'NR > 1 { printf "%s%s %.1fs", sep, $1, $2 - prev; sep = ", " }
|
|
{ prev = $2 } END { if (NR > 1) print "" }' "$work/laps" | sed 's/^/timing: /'
|
|
}
|
|
lap start
|
|
|
|
[ -d "$dir" ] || { echo "::error::$dir is not a directory — the build produced no export"; exit 1; }
|
|
|
|
# ---- content type -----------------------------------------------------------
|
|
# What the object STORES, which is what a reader of the S3 origin gets. The
|
|
# serving path names the type from the key and falls back to the stored one only
|
|
# for an extension it does not know, so this is the fallback rather than the
|
|
# authority — and an object stored as application/octet-stream is one a browser
|
|
# DOWNLOADS instead of rendering.
|
|
ctype_of() {
|
|
case "${1##*.}" in
|
|
html|htm) REPLY='text/html; charset=utf-8' ;;
|
|
css) REPLY='text/css; charset=utf-8' ;;
|
|
js|mjs) REPLY='text/javascript; charset=utf-8' ;;
|
|
json) REPLY='application/json; charset=utf-8' ;;
|
|
xml) REPLY='application/xml; charset=utf-8' ;;
|
|
txt) REPLY='text/plain; charset=utf-8' ;;
|
|
md) REPLY='text/markdown; charset=utf-8' ;;
|
|
svg) REPLY='image/svg+xml' ;;
|
|
png) REPLY='image/png' ;;
|
|
jpg|jpeg) REPLY='image/jpeg' ;;
|
|
gif) REPLY='image/gif' ;;
|
|
webp) REPLY='image/webp' ;;
|
|
avif) REPLY='image/avif' ;;
|
|
ico) REPLY='image/x-icon' ;;
|
|
woff) REPLY='font/woff' ;;
|
|
woff2) REPLY='font/woff2' ;;
|
|
ttf) REPLY='font/ttf' ;;
|
|
otf) REPLY='font/otf' ;;
|
|
wasm) REPLY='application/wasm' ;;
|
|
pdf) REPLY='application/pdf' ;;
|
|
webmanifest) REPLY='application/manifest+json' ;;
|
|
map) REPLY='application/json' ;;
|
|
*) REPLY='application/octet-stream' ;;
|
|
esac
|
|
}
|
|
ctype() { ctype_of "$1"; printf '%s\n' "$REPLY"; }
|
|
|
|
# ---- cache control is the server's -------------------------------------------
|
|
# Nothing here computes one. The serving path composes CacheControlFor(key,
|
|
# site.CacheControl) on every request and deliberately does NOT read the stored
|
|
# object's header (cloud apps/sites/sites.go), so a Cache-Control stamped here
|
|
# reaches no reader.
|
|
#
|
|
# It was a copy of the server's policy, and it had drifted exactly as a second
|
|
# copy does: this script matched hashed BASENAMES only, while the server also
|
|
# treats the build-output PREFIXES (`_next/static/`, `assets/`) as immutable —
|
|
# so every unhashed-but-immutable asset in every Next export carried max-age=3600
|
|
# where the rule says a year. One rule, one place, and the place is what serves.
|
|
|
|
# ---- the manifest -----------------------------------------------------------
|
|
# Paths RELATIVE to the export root, which is exactly what the completion's
|
|
# `keys` must carry (cloud reconciles `keep[rel]` against them).
|
|
#
|
|
# CNAME does not travel. It is a GitHub Pages artifact that means nothing to S3
|
|
# and would ship a stale hostname claim into the bucket.
|
|
# _site/manifest.txt is the release's own record of its hashes (1b below). It is
|
|
# a key of the release like any other, so `keys` names it and reconciliation
|
|
# keeps it; it is written last, and its bytes are not part of `bytes`.
|
|
mkey=_site/manifest.txt
|
|
manifest=$( { (cd "$dir" && find . -type f ! -name CNAME ! -path "./$mkey" | sed 's|^\./||'); echo "$mkey"; } | LC_ALL=C sort)
|
|
count=$(printf '%s' "$manifest" | grep -c . || true)
|
|
[ "$count" -gt 1 ] || { echo "::error::$dir contains no files — refusing to deploy an empty manifest"; exit 1; }
|
|
printf '%s\n' "$manifest" | grep . > "$work/manifest"
|
|
|
|
# ---- the completion body, built BEFORE anything is sent ----------------------
|
|
# It is the last request of the three and the only one whose size grows with the
|
|
# release, so it is the only one the edge limit can refuse — and refusing it late
|
|
# is the expensive way to learn: the deployment is open, every object is already
|
|
# in the bucket, and the answer is an opaque 400.
|
|
#
|
|
# `wc -c`, not `stat`: the size flag is spelled -f%z on BSD and -c%s on GNU, and a
|
|
# `stat -f%z || stat -c%s` fallback does not work — on Linux the first arm fails
|
|
# INSIDE the pipeline, awk still exits 0, and the `||` never fires, so the byte
|
|
# count silently reports 0. wc is the one spelling both agree on.
|
|
bytes=$(cd "$dir" && find . -type f ! -name CNAME ! -path ./_site/manifest.txt -exec wc -c {} + | awk '$2!="total"{s+=$1} END{print s+0}')
|
|
|
|
# --rawfile, not --args: `jq --args ... $(cat manifest)` passes every key as an
|
|
# ARGV entry, and execve caps the argument block (~2 MiB on Linux). A wide export
|
|
# would die there first — "Argument list too long", from the tool that builds the
|
|
# body rather than from the size of it — which is the same unreadable failure one
|
|
# layer up, and it would fire before this check could name the real ceiling.
|
|
jq -n --arg c "${SITE_COMMIT:-}" --argjson f "$count" --argjson b "${bytes:-0}" \
|
|
--rawfile k "$work/manifest" \
|
|
'{status:"live", commit:$c, files:$f, bytes:$b,
|
|
keys:($k | rtrimstr("\n") | split("\n"))}' > "$work/sd-done.json"
|
|
body=$(wc -c < "$work/sd-done.json" | tr -d ' ') # BSD wc pads its count, GNU does not
|
|
|
|
[ "$body" -le "$max_body" ] || {
|
|
# Exact bytes beside the human number, because the band where a site FIRST
|
|
# crosses is the ~100 KiB just above the cap, and there both sides round to
|
|
# "16.0 MiB" — a refusal that reads as 16.0 over 16.0 explains nothing.
|
|
echo "::error::publish aborted: the completion manifest is $(human "$body") ($body bytes) for $count files, over the $(human "$max_body") ($max_body bytes) edge limit (GATEWAY_BODY_LIMIT, cloud internal/edge/edge.go:38). The bytes stream per file and are not the problem — the manifest cloud reconciles the prefix against is one body, and fasthttp refuses an oversized POST before any handler runs, answering only 'Error when parsing request', which names neither size nor cause. Reduce the object count: a full prerender is what makes an export this wide."
|
|
exit 1; }
|
|
|
|
lap scan
|
|
if [ -n "${SITE_PLAN:-}" ]; then
|
|
printf 'slug=%s dir=%s files=%s\n' "$slug" "$dir" "$count"
|
|
while IFS= read -r k; do [ -n "$k" ] && printf '%s\t%s\n' "$k" "$(ctype "$k")"; done <<< "$manifest"
|
|
exit 0
|
|
fi
|
|
|
|
# One line, every publish, whether or not it is anywhere near the limit. The size
|
|
# of a release explains most of what this lane then does, and a build step that
|
|
# reports it only when it refuses leaves the number unknown on the run that
|
|
# introduced it — which is the run where a growing object count is cheapest to
|
|
# see.
|
|
echo "release: $(human "$bytes") / $count files; manifest $(human "$body") of the $(human "$max_body") edge limit"
|
|
|
|
# Spelled out rather than ${VAR:?...}. That form reports through the shell, and
|
|
# the EXIT trap above resets the status on its way out, so the refusal printed
|
|
# and the script still exited 0 — a deploy that never happened, reported as a
|
|
# success. The brace in the mint example also closed the expansion early, cutting
|
|
# the message off mid-sentence.
|
|
if [ -z "${HANZO_DEPLOY_TOKEN:-}" ]; then
|
|
echo "::error::HANZO_DEPLOY_TOKEN is unset. It is the ONE credential this needs; set it as an Actions secret on git.hanzo.ai, which is what reads .hanzo/workflows. Mint one: POST /v1/keys {\"type\":\"secret\"}" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# ---- 0a. does the credential authenticate at all? ---------------------------
|
|
# Every org-scoped route answers a rejected credential and an ABSENT one with the
|
|
# same 403, `{"error":"X-Org-Id required"}`. That message names a HEADER, and the
|
|
# header is one CI cannot set — the gateway strips a client-supplied X-Org-Id and
|
|
# mints it from the validated bearer — so the true cause (the bearer resolved to
|
|
# no principal) is reported as the one thing the caller is forbidden to fix.
|
|
#
|
|
# It cost a full investigation on hanzo.ai: every gate green, then `ensure project
|
|
# hanzo-ai returned HTTP 403 X-Org-Id required` on the last step, reading as a
|
|
# server or header regression when the credential had simply stopped resolving.
|
|
# Verified against the live API — no auth, a bogus `sk-`, and a revoked key are
|
|
# byte-identical on /v1/projects.
|
|
#
|
|
# /v1/keys refuses in a DIFFERENT voice ("sign in to manage API keys"), so it
|
|
# still separates the two failures this script cannot otherwise tell apart: a
|
|
# bearer that does not authenticate, versus one that authenticates and has no
|
|
# org. Name which, and say what to do about it.
|
|
#
|
|
# ADVISORY, NOT FATAL — and that is the whole point of this block.
|
|
#
|
|
# `exit 1` on any non-200 would rest on /v1/keys "200s for a live key", and this
|
|
# script cannot rely on that: /v1/keys is the SESSION-scoped key-MANAGEMENT
|
|
# route, and a machine `sk-` key is not supposed to manage keys. Measured against
|
|
# the live API, an anonymous request and a bogus `sk-` are byte-identical there —
|
|
# both 403 `{"code":"forbidden","error":"sign in to manage API keys"}`. A probe
|
|
# that answers the same for a valid and an invalid credential cannot decide
|
|
# anything, least of all whether the site ships.
|
|
#
|
|
# It froze the Sites plane. hanzo.ai and hips both died here in ~1s, every run,
|
|
# with the whole build and every gate already green above it — and the message
|
|
# said the credential was revoked, which is a claim this probe is not entitled
|
|
# to make.
|
|
#
|
|
# So the preflight keeps its diagnosis and loses its veto. The operations below
|
|
# are the authority: they are the ones that actually need the credential, they
|
|
# already report their own status precisely, and unlike this one they cannot be
|
|
# wrong about what they measured. A credential that is genuinely dead still
|
|
# fails the deploy — one step later, from the call that means something.
|
|
#
|
|
# `|| true`, not `|| echo 000`: curl ALREADY writes 000 to stdout when it cannot
|
|
# connect, so appending another makes kcode `000000`, which matches no case and
|
|
# falls to the hard-fail branch — a preflight that reports an unreachable API as
|
|
# "your credential is revoked". Pinned by the unreachable-host test below.
|
|
kcode=$(curl -sS -o /dev/null -w '%{http_code}' \
|
|
-H "Authorization: Bearer $HANZO_DEPLOY_TOKEN" "$api/v1/keys" || true)
|
|
case "$kcode" in
|
|
200) : ;;
|
|
000|"") echo "::warning::could not reach $api/v1/keys to preflight the credential; continuing" ;;
|
|
*)
|
|
echo "::warning::the credential did not read as a live key at $api/v1/keys (HTTP $kcode)."
|
|
echo "That route is session-scoped key MANAGEMENT, which a machine key is not"
|
|
echo "expected to hold, so this is a hint and not a verdict — continuing, and the"
|
|
echo "calls below will say for certain."
|
|
echo "If one of them refuses the credential: the token resolves to no principal —"
|
|
echo "absent from this repo's scope, expired, or revoked. It is NOT a missing"
|
|
echo "X-Org-Id header; CI cannot set that header, cloud derives it from this"
|
|
echo "bearer. Mint a replacement with POST $api/v1/keys {\"type\":\"secret\"} as a"
|
|
echo "member of the owning org, then set it as the ORG-level Actions secret"
|
|
echo "HANZO_DEPLOY_TOKEN on git.hanzo.ai — that is what reads"
|
|
echo ".hanzo/workflows, not GitHub." ;;
|
|
esac
|
|
|
|
# ---- POST, with 5xx asked again ---------------------------------------------
|
|
# 5xx is the plane saying "not now"; 4xx is it saying "no". Only the first is
|
|
# worth repeating, and repeating it is worth a lot here: every gate has already
|
|
# passed by this point, so a publish lost to one transient 503 throws away a
|
|
# whole green build. hanzo.ai spent an hour alternating — 16:21 failed, 16:24
|
|
# and 16:28 published, 16:41 failed twice, 16:54 through 17:04 published, 17:21
|
|
# failed — on `ensure project ... 503`, with sign and insights flapping beside
|
|
# it. Nothing was wrong with any of those builds.
|
|
#
|
|
# A curl that cannot speak at all reports 000, which retries for the same reason
|
|
# and then falls through to the caller's own error path.
|
|
#
|
|
# Both callers below are idempotent, which is what makes a retry safe rather
|
|
# than merely convenient: creating a project answers 409 the second time, and an
|
|
# enqueue that never returned its 202 created no deployment to duplicate.
|
|
post() {
|
|
local out=$1 url=$2 body=$3 code attempt
|
|
for attempt in 1 2 3 4 5; do
|
|
code=$(curl -sS -o "$out" -w '%{http_code}' -X POST "$url" \
|
|
-H "Authorization: Bearer $HANZO_DEPLOY_TOKEN" -H 'Content-Type: application/json' \
|
|
-d "$body" || printf '000')
|
|
case "$code" in 5??|000) ;; *) break ;; esac
|
|
[ "$attempt" -lt 5 ] || break
|
|
sleep $((attempt * 4))
|
|
done
|
|
printf '%s' "$code"
|
|
}
|
|
|
|
# ---- 0b. ensure the project exists (self-provision) --------------------------
|
|
# A brand-new slug has no project, and the enqueue below 404s on a missing one —
|
|
# which is why every never-before-deployed site failed its first run. Create it
|
|
# idempotently here instead: 200/201 the first time, 409 every time after, both
|
|
# success. So a new site needs no out-of-band `POST /v1/projects`; committing the
|
|
# workflow is enough. The repo is linked so the deploy's lifecycle events name a
|
|
# repo and reach the notify subscriptions keyed on it — the enqueue itself no
|
|
# longer requires a link; SITE_REPO overrides the origin for a
|
|
# checkout whose remote is not the canonical repo.
|
|
repo="${SITE_REPO:-$(git remote get-url origin 2>/dev/null || true)}"
|
|
pcode=$(post "$work"/sd-proj.json "$api/v1/projects" \
|
|
"$(jq -nc --arg s "$slug" --arg u "$repo" --arg b "${SITE_BRANCH:-main}" \
|
|
'{slug:$s, name:$s} + (if $u=="" then {} else {repo:{url:$u, branch:$b}} end)')")
|
|
case "$pcode" in
|
|
200|201|409) : ;; # created now, or already there — either is the state we need
|
|
# Creating a project and publishing to one are separate rights, and a
|
|
# credential can hold the second without the first. Refused means only that
|
|
# this key may not open a project; whether the project is there is a
|
|
# different question, and asking it is one GET. A key that may publish to an
|
|
# existing project should publish to it.
|
|
401|403)
|
|
if [ "$(curl -sS -o /dev/null -w '%{http_code}' \
|
|
-H "Authorization: Bearer $HANZO_DEPLOY_TOKEN" \
|
|
"$api/v1/projects/$slug" || true)" = 200 ]; then
|
|
echo "::notice::$slug exists; this key may publish to it but not create projects"
|
|
else
|
|
echo "::error::ensure project $slug returned HTTP $pcode"; head -c 400 "$work"/sd-proj.json; echo; exit 1
|
|
fi ;;
|
|
*) echo "::error::ensure project $slug returned HTTP $pcode"; head -c 400 "$work"/sd-proj.json; echo; exit 1 ;;
|
|
esac
|
|
|
|
# ---- 1. enqueue -------------------------------------------------------------
|
|
# 202 Accepted is the success code: the deployment is queued, not live. bucket
|
|
# and prefix come FROM cloud (sitePrefix(org, slug) is server-side) — never guess
|
|
# them, or the upload lands where nothing is served the moment an org or slug
|
|
# changes.
|
|
#
|
|
# POST on the DEPLOYMENTS COLLECTION, which is the address that opens one.
|
|
# /deploy is the ARCHIVE address and reads any body as bytes, so JSON sent there
|
|
# is sniffed as a tar and refused — 400 "invalid artifact: tar: unexpected EOF",
|
|
# with the build and every gate already green above it. The collection is where a
|
|
# deployment is created; the verb is the method and the noun is the resource.
|
|
#
|
|
# `commit` is the whole body. `source` was the Content-Type discriminator the
|
|
# split removed, and `branch` was always discarded server-side — a deployment has
|
|
# no branch column, and the lifecycle event derives it from the project's own
|
|
# linked branch. jq builds it so a commit message can never break the JSON.
|
|
code=$(post "$work"/sd-enq.json "$api/v1/projects/$slug/deployments" \
|
|
"$(jq -nc --arg c "${SITE_COMMIT:-}" '{commit:$c}')")
|
|
if [ "$code" != "202" ]; then
|
|
echo "::error::enqueue $api/v1/projects/$slug/deployments returned HTTP $code"; head -c 600 "$work"/sd-enq.json; echo; exit 1
|
|
fi
|
|
dep=$(jq -r '.id' "$work"/sd-enq.json)
|
|
prefix=$(jq -r '.prefix' "$work"/sd-enq.json)
|
|
upload_url=$(jq -r '.upload.url // empty' "$work"/sd-enq.json)
|
|
if [ -z "$upload_url" ]; then
|
|
echo "::error::the 202 carried no upload grant, so there is no way to write the bytes."
|
|
echo " cloud mints one only when presigning is configured (S3_ADMIN_* on the cloud deployment)."
|
|
exit 1
|
|
fi
|
|
jq -c '{id,version,status,bucket,prefix}' "$work"/sd-enq.json
|
|
|
|
lap enqueue
|
|
# A build that dies after this point would leave the deployment "queued" and the
|
|
# project stuck "building" forever. Report the failure so cloud records an honest
|
|
# terminal state instead of a lie by omission.
|
|
fail() {
|
|
curl -sS -X POST "$api/v1/projects/$slug/deployments/$dep/complete" \
|
|
-H "Authorization: Bearer $HANZO_DEPLOY_TOKEN" -H 'Content-Type: application/json' \
|
|
-d '{"status":"error","message":"CI upload failed"}' >/dev/null 2>&1 || true
|
|
echo "::error::marked deployment $dep as error"
|
|
}
|
|
trap 'fail' ERR
|
|
|
|
# ---- 1b. what changed ---------------------------------------------------------
|
|
# Most of a release is what the last one already put there: content-hashed chunks
|
|
# never change under their name, and a one-page edit rewrites one document. So
|
|
# the publisher keeps a `<hash> <key>` manifest INSIDE the prefix it serves
|
|
# (_site/manifest.txt), reads the previous one back over the public host, and
|
|
# uploads only the keys whose hash differs or which it lacks. The grant authorizes
|
|
# writes only and the server needs to know nothing: `keys` still names the whole
|
|
# release, so reconciliation deletes exactly what it always did.
|
|
#
|
|
# The manifest is written LAST. A publish cut anywhere earlier leaves the old one
|
|
# in place, so the next run re-sends whatever the cut run may not have finished —
|
|
# an object is only ever skipped once a run that completed its uploads said so.
|
|
#
|
|
# It fails toward the full upload, never toward a partial site: no manifest, an
|
|
# unreadable or malformed one, or a sampled unchanged asset that does not answer
|
|
# 200 all mean "send everything". SITE_FULL=1 forces that.
|
|
mkdir -p "$dir/_site"
|
|
grep -vxF "$mkey" "$work/manifest" > "$work/real"
|
|
# Every core at once. Each batch writes a file of its own: hashers sharing one
|
|
# pipe flush in blocks that can end mid-line, and two of them interleaving there
|
|
# would corrupt a line. Sorted after, so the record reads the same whatever order
|
|
# they finished in; -k3 is the key, since two spaces follow the hash.
|
|
cores=$(getconf _NPROCESSORS_ONLN 2>/dev/null || echo 4)
|
|
mkdir -p "$work/h"
|
|
(cd "$dir" && tr '\n' '\0' < "$work/real" | H="$work/h" xargs -0 -n 256 -P "$cores" bash -c \
|
|
'if command -v sha256sum >/dev/null 2>&1; then sha256sum "$@"; else shasum -a 256 "$@"; fi > "$(mktemp "$H/b.XXXXXX")"' _)
|
|
cat "$work"/h/b.* | LC_ALL=C sort -t ' ' -k3 > "$work/hashes"
|
|
lap hash
|
|
[ "$(grep -c . "$work/hashes")" = "$((count - 1))" ] || { echo "::error::hashed $(grep -c . "$work/hashes") of $((count - 1)) files"; exit 1; }
|
|
cp "$work/real" "$work/upload"
|
|
: > "$work/prev"
|
|
if [ -z "${SITE_FULL:-}" ]; then
|
|
# SITE_URL is the host that serves this prefix to readers. The project's own
|
|
# liveUrl is the plane's preview host, which does not always route the prefix.
|
|
base="${SITE_URL:-}"
|
|
[ -n "$base" ] || base=$(curl -sS -m 20 -H "Authorization: Bearer $HANZO_DEPLOY_TOKEN" "$api/v1/projects/$slug" 2>/dev/null \
|
|
| jq -r '.liveUrl // empty' 2>/dev/null || true)
|
|
if [ -n "$base" ]; then
|
|
base=${base%/}
|
|
pcode2=$(curl -sS -m 30 -H 'Cache-Control: no-cache' -o "$work/prev.raw" -w '%{http_code}' \
|
|
"$base/_site/manifest.txt?t=$(date +%s)" 2>/dev/null || true)
|
|
if [ "$pcode2" = 200 ] && [ -s "$work/prev.raw" ] \
|
|
&& ! grep -qvE '^[0-9a-f]{64} [^ ]' "$work/prev.raw"; then
|
|
cp "$work/prev.raw" "$work/prev"
|
|
fi
|
|
fi
|
|
fi
|
|
if [ -s "$work/prev" ]; then
|
|
awk 'NR==FNR{p[substr($0,67)]=substr($0,1,64); next}
|
|
{f=substr($0,67); if (p[f] != substr($0,1,64)) print f}' "$work/prev" "$work/hashes" > "$work/changed"
|
|
# Ask the origin for a sample of what it is about to be trusted to still hold.
|
|
sample=$(grep -vxFf "$work/changed" "$work/real" | grep '^_next/static/' | LC_ALL=C sort -R | head -20 || true)
|
|
# All at once: each is a round trip through the edge to the origin, and twenty
|
|
# of them one after another were seconds of every publish.
|
|
bad= i=0
|
|
for rel in $sample; do
|
|
i=$((i + 1))
|
|
( c=$(curl -g -sS -m 20 -r 0-0 -o /dev/null -w '%{http_code}' "$base/$rel?t=$(date +%s)" 2>/dev/null || true)
|
|
printf '%s %s\n' "${c:-000}" "$rel" > "$work/probe.$i" ) &
|
|
done
|
|
wait
|
|
for f in "$work"/probe.*; do
|
|
[ -e "$f" ] || continue
|
|
read -r c rel < "$f"
|
|
case "$c" in 200|206) ;; *) bad="$rel ($c)"; break ;; esac
|
|
done
|
|
if [ -n "$bad" ]; then
|
|
echo "previous manifest is not trusted: $bad does not answer; sending everything"
|
|
else
|
|
cp "$work/changed" "$work/upload"
|
|
echo "incremental: $(grep -c . "$work/changed" || true) of $((count - 1)) files differ from the live release"
|
|
fi
|
|
else
|
|
echo "no previous manifest at the live host; sending everything"
|
|
fi
|
|
lap diff
|
|
# ---- 1c. what the plane can copy ----------------------------------------------
|
|
# A host that shares its build with a site published a moment ago (SITE_COPY_FROM
|
|
# = "<slug>:<dir>") differs from it by a page or two. The objects of this upload
|
|
# that are byte-identical to that site's are not sent: the completion names them
|
|
# and cloud copies them inside the object store. Only when the grant says the
|
|
# plane does so; one that does not would leave those keys missing.
|
|
: > "$work/copy"
|
|
if [ -n "${SITE_COPY_FROM:-}" ] && [ "$(jq -r '.upload.copy // false' "$work"/sd-enq.json)" = true ]; then
|
|
peer_slug=${SITE_COPY_FROM%%:*}; peer_dir=${SITE_COPY_FROM#*:}
|
|
while IFS= read -r k; do
|
|
[ -f "$peer_dir/$k" ] && cmp -s "$dir/$k" "$peer_dir/$k" && printf '%s\n' "$k"
|
|
done < "$work/upload" > "$work/copy" || true
|
|
if [ -s "$work/copy" ]; then
|
|
grep -vxFf "$work/copy" "$work/upload" > "$work/upload.rest" || true
|
|
mv "$work/upload.rest" "$work/upload"
|
|
jq --arg f "$peer_slug" --rawfile c "$work/copy" \
|
|
'. + {copyFrom:$f, copy:($c | rtrimstr("\n") | split("\n"))}' "$work/sd-done.json" > "$work/sd-done2.json"
|
|
mv "$work/sd-done2.json" "$work/sd-done.json"
|
|
echo "copy: $(grep -c . "$work/copy") objects from $peer_slug inside the store"
|
|
fi
|
|
fi
|
|
cp "$work/hashes" "$dir/$mkey"
|
|
|
|
# ---- 2. the bytes -----------------------------------------------------------
|
|
# `file` goes LAST: S3 ignores every field after the file part, so a grant field
|
|
# trailing the body is silently dropped and the signature check fails.
|
|
#
|
|
# `key` is DROPPED from the grant's fields and re-sent per object. The grant
|
|
# carries key="<org>/<slug>/" — the starts-with PLACEHOLDER, not a destination —
|
|
# and forwarding it verbatim alongside the real key posts `key` twice, which S3
|
|
# answers 400 for every object. That is the whole of the first end-to-end run:
|
|
# 8403 files, 8403 400s. Everything else in the map (bucket, policy, x-amz-*) is
|
|
# covered by the signature and must travel untouched.
|
|
# ONE curl for a whole batch, not one per object. Each object is its own POST, and
|
|
# a curl per object is a TCP and TLS handshake per object through the edge — that,
|
|
# not the bytes, is what made 7621 files take a quarter of an hour. `--parallel`
|
|
# multiplexes the transfers over a few HTTP/2 connections; the config file gives
|
|
# each its own section, and `write-out` names every answer so a failure is read
|
|
# from what came back rather than from an exit status a parallel run blurs.
|
|
#
|
|
# `file` goes LAST in each section, for the reason above.
|
|
#
|
|
# RETRIED, like the API calls above and for the same reason: a single object is
|
|
# idempotent — the same key with the same bytes — so a second attempt can only
|
|
# replace it with itself. One flaky connection out of eight thousand must not fail
|
|
# the whole release. What did not answer 2xx, or did not answer at all, goes round
|
|
# again.
|
|
fields=$(jq -r '.upload.fields | to_entries[] | select(.key != "key") | "\(.key)=\(.value)"' "$work"/sd-enq.json)
|
|
esc() { REPLY=${1//\\/\\\\}; REPLY=${REPLY//\"/\\\"}; }
|
|
put_all() {
|
|
local list="$1" pass rel line sep
|
|
cp "$list" "$work/todo"
|
|
for pass in 1 2 3 4 5; do
|
|
[ -s "$work/todo" ] || return 0
|
|
{
|
|
sep=
|
|
while IFS= read -r rel; do
|
|
printf '%surl = "%s"\n' "$sep" "$upload_url"; sep=$'next\n'
|
|
esc "key=$prefix/$rel"; printf 'form = "%s"\n' "$REPLY"
|
|
while IFS= read -r line; do esc "$line"; printf 'form = "%s"\n' "$REPLY"; done <<< "$fields"
|
|
ctype_of "$rel"; esc "Content-Type=$REPLY"; printf 'form = "%s"\n' "$REPLY"
|
|
esc "file=@$dir/$rel"; printf 'form = "%s"\n' "$REPLY"
|
|
esc "${rel//%/%%}"; printf 'write-out = "%%{http_code} %s\\n"\noutput = "/dev/null"\nmax-time = 120\n' "$REPLY"
|
|
done < "$work/todo"
|
|
} > "$work/curl.cfg"
|
|
curl -sS --parallel --parallel-max "$jobs" -K "$work/curl.cfg" > "$work/results" 2> "$work/curl.err" || true
|
|
# Not done: answered outside 2xx, or missing from the answers entirely.
|
|
{ awk '$1 !~ /^2/ { sub(/^[^ ]+ /, ""); print }' "$work/results"
|
|
comm -23 <(LC_ALL=C sort "$work/todo") <(awk '{ sub(/^[^ ]+ /, ""); print }' "$work/results" | LC_ALL=C sort)
|
|
} | LC_ALL=C sort -u > "$work/todo.next"
|
|
mv "$work/todo.next" "$work/todo"
|
|
[ -s "$work/todo" ] || return 0
|
|
echo "upload pass $pass: $(grep -c . "$work/todo") not accepted; again" >&2
|
|
sleep "$((pass * 2))"
|
|
done
|
|
while IFS= read -r rel; do echo "::error::upload failed after 5 attempts: $rel"; done < <(head -20 "$work/todo")
|
|
tail -3 "$work/curl.err" >&2
|
|
return 1
|
|
}
|
|
|
|
# ASSETS BEFORE DOCUMENTS, because the bytes are live where they land. This
|
|
# publisher writes into the prefix the edge already serves: there is no release to
|
|
# flip, so every object is public the instant S3 accepts it. A single flat pass
|
|
# therefore has no ordering between a document and the hashed assets it names, and
|
|
# a document is only servable once all of them are in the bucket.
|
|
#
|
|
# Splitting on `.html` gives that ordering with no coordination and no new
|
|
# machinery. An asset name carries a content hash, so writing one never replaces
|
|
# an object a live document is still naming — phase one is purely additive, and
|
|
# whatever is published stays servable throughout it. The document is the switch,
|
|
# thrown once everything it can name is already there.
|
|
#
|
|
# It is also what lets a caller cancel a superseded run safely: the run dies in
|
|
# phase one having only added unreferenced objects, and what is live is unchanged.
|
|
grep -v '\.html$' "$work/upload" > "$work/manifest.assets" || true
|
|
grep '\.html$' "$work/upload" > "$work/manifest.docs" || true
|
|
n_assets=$(grep -c . "$work/manifest.assets" || true)
|
|
n_docs=$(grep -c . "$work/manifest.docs" || true)
|
|
echo "uploading $(grep -c . "$work/upload" || true) of $count files to s3://$(jq -r .bucket "$work"/sd-enq.json)/$prefix ($jobs parallel: $n_assets assets, then $n_docs documents)"
|
|
if [ "$n_assets" -gt 0 ]; then
|
|
put_all "$work/manifest.assets"
|
|
fi
|
|
if [ "$n_docs" -gt 0 ]; then
|
|
put_all "$work/manifest.docs"
|
|
fi
|
|
|
|
# The manifest is what lets the next run skip these; it is only true once they
|
|
# are all in the bucket, so it goes last.
|
|
printf '%s\n' "$mkey" > "$work/manifest.last"
|
|
put_all "$work/manifest.last"
|
|
lap upload
|
|
|
|
# ---- 3. flip it live --------------------------------------------------------
|
|
# The body was built and measured at the top, before a single object moved, so
|
|
# there is nothing left to assemble here — only to send it.
|
|
# Asked again on a 5xx: the edge answered 502 for a completion whose 7621-key
|
|
# reconcile outlasted it, after every object was already in the bucket, and that
|
|
# marked a deployment whose bytes were live as an error.
|
|
for attempt in 1 2 3; do
|
|
code=$(curl -sS -m 300 -o "$work"/sd-resp.json -w '%{http_code}' \
|
|
-X POST "$api/v1/projects/$slug/deployments/$dep/complete" \
|
|
-H "Authorization: Bearer $HANZO_DEPLOY_TOKEN" -H 'Content-Type: application/json' \
|
|
--data-binary @"$work"/sd-done.json || printf '000')
|
|
case "$code" in 5??|000) [ "$attempt" -lt 3 ] && sleep $((attempt * 5)) ;; *) break ;; esac
|
|
done
|
|
trap - ERR
|
|
if [ "$code" != "200" ]; then
|
|
echo "::error::complete returned HTTP $code"; head -c 600 "$work"/sd-resp.json; echo; fail; exit 1
|
|
fi
|
|
jq -c '{status,liveUrl,version,files,bytes}' "$work"/sd-resp.json
|
|
lap complete
|
|
|
|
# ---- 4. read back what we just published ------------------------------------
|
|
# A publish that uploads and flips live has still proved nothing about whether
|
|
# the bytes come back. An object store can answer `200` with a correct
|
|
# `content-length` and a valid MD5 etag and then deliver ZERO body bytes —
|
|
# StatObject knowing a size GetObject cannot serve. It strikes per object, so
|
|
# sibling paths stay healthy and nothing that merely walks routes sees it.
|
|
#
|
|
# A HEAD is blind to this by construction: it never asks for a body, so `curl -I`
|
|
# reports a healthy 200 on a page that cannot load. The check has to DOWNLOAD and
|
|
# compare what arrived against what was promised. curl exits 18 on the short read
|
|
# ("transfer closed with N bytes remaining"), which is the signal we want.
|
|
#
|
|
# Retried, because a flip-to-live is not instantaneous and a single miss here
|
|
# would fail honest deploys. Three attempts is enough to separate propagation
|
|
# from a genuinely unreadable object.
|
|
live=$(jq -r '.liveUrl // empty' "$work"/sd-resp.json)
|
|
if [ -z "$live" ]; then
|
|
echo "::warning::complete returned no liveUrl — published bytes were not read back"
|
|
else
|
|
readback() {
|
|
local code got len
|
|
last=
|
|
code=$(curl -sS -L --max-time 30 \
|
|
-D "$work"/sd-vfy.hdr -o "$work"/sd-vfy.body \
|
|
-w '%{http_code}' "$live" 2>"$work"/sd-vfy.err) || {
|
|
echo " transfer failed: $(tr -d '\r' < "$work"/sd-vfy.err | tail -1)"; return 1; }
|
|
last=$code
|
|
[ "$code" = "200" ] || { echo " HTTP $code"; return 1; }
|
|
got=$(wc -c < "$work"/sd-vfy.body | tr -d ' ')
|
|
# -L writes every hop's headers to one file. Only the answer that carried
|
|
# the body may promise its length: a redirect hop's `content-length: 17`
|
|
# ("Moved Permanently") ahead of a chunked 200 is not a short read.
|
|
len=$(awk 'BEGIN{IGNORECASE=1} /^HTTP\//{len=""}
|
|
/^content-length:/{gsub(/\r/,""); len=$2} END{print len}' \
|
|
"$work"/sd-vfy.hdr)
|
|
[ "${got:-0}" -gt 0 ] || { echo " 200 with an EMPTY body"; return 1; }
|
|
if [ -n "$len" ] && [ "$got" -ne "$len" ]; then
|
|
echo " got $got bytes, content-length promised $len"; return 1
|
|
fi
|
|
echo " $got bytes"; return 0
|
|
}
|
|
|
|
# A client preview is not served before its client signs off: the edge answers
|
|
# every request for its host 404 (hanzoai/universe static-sites.yaml, route
|
|
# client-previews), so the 200 above never arrives. Its release is read back
|
|
# from cloud instead, with the credential that published it: the deployment
|
|
# this run completed must be live, hold exactly the files and bytes this run
|
|
# sent, and be the deployment the project serves. That proves cloud took this
|
|
# release as the project's own. It cannot prove the store returns each body;
|
|
# the first publish after the host is served reads its bytes back publicly.
|
|
#
|
|
# Only a 404 is answered this way. A 5xx, a dropped transfer, an empty or
|
|
# short body is a failure whatever cloud's record says.
|
|
recorded() {
|
|
local d p
|
|
d=$(curl -sS --max-time 30 -H "Authorization: Bearer $HANZO_DEPLOY_TOKEN" \
|
|
"$api/v1/projects/$slug/deployments/$dep" 2>/dev/null) || return 1
|
|
p=$(curl -sS --max-time 30 -H "Authorization: Bearer $HANZO_DEPLOY_TOKEN" \
|
|
"$api/v1/projects/$slug" 2>/dev/null) || return 1
|
|
jq -e --slurpfile sent "$work"/sd-done.json --arg id "$dep" \
|
|
'.id == $id and .status == "live"
|
|
and .files == $sent[0].files and .bytes == $sent[0].bytes' \
|
|
<<<"$d" >/dev/null 2>&1 || return 1
|
|
jq -e --arg id "$dep" '.status == "live" and .currentDeploymentId == $id' \
|
|
<<<"$p" >/dev/null 2>&1
|
|
}
|
|
|
|
ok= last=
|
|
for attempt in 1 2 3; do
|
|
printf 'read-back %s (attempt %d): ' "$live" "$attempt"
|
|
if readback; then ok=1; break; fi
|
|
# Named out loud: 404 is also what a name nothing routes answers, so a
|
|
# release counted from cloud's record is printed as exactly that.
|
|
if [ "$last" = 404 ] && recorded; then
|
|
echo " not served at its host: cloud holds deployment $dep as the project's live release, $(jq -r .files "$work"/sd-done.json) files"
|
|
echo "::notice::$live answers 404, so this release was read back from cloud's deployment record, not from the host. A client preview is not served until its client signs off; any other site answering 404 is not routed."
|
|
ok=1; break
|
|
fi
|
|
[ "$attempt" -lt 3 ] && sleep 5
|
|
done
|
|
|
|
if [ -z "$ok" ]; then
|
|
echo "::error::$live is live and does not serve its own bytes. The upload and the" >&2
|
|
echo "::error::flip both succeeded, so this is not a retry — the stored object is" >&2
|
|
echo "::error::unreadable. Republish or roll back; do not leave it serving." >&2
|
|
exit 1
|
|
fi
|
|
fi
|
|
lap readback
|
|
laps
|