Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Hanzo AI
|
2c5eeb6352 | ||
|
z
|
d07661c0cc | ||
|
z
|
c67f9fbfcd | ||
 hanzo-dev
|
c380563306 | ||
|
hanzo-dev
|
daacc3ca9b | ||
|
hanzo-devandGitHub
|
9231a46d2b | ||
|
hanzo-devandGitHub
|
be43b6f323 |
+7
-193
@@ -16,18 +16,6 @@ on:
|
||||
of your own self-hosted arc runners instead.
|
||||
type: string
|
||||
default: '["hanzo-build-linux-amd64"]'
|
||||
mode:
|
||||
description: >-
|
||||
Build execution mode. `buildx` (default) runs the full buildx →
|
||||
test → deploy pipeline ON the arc runner. `delegate` instead POSTs the
|
||||
build to platform.hanzo.ai (`/v1/arcd/enqueue`) — platform builds
|
||||
in-cluster with BuildKit and rolls the operator Service CR itself, so
|
||||
the GitHub job finishes in seconds with no runner buildx. A repo opts
|
||||
in by passing `with: { mode: delegate }`; everything else is unchanged.
|
||||
Requires the `PLATFORM_BUILD_CALLBACK_TOKEN` secret (via secrets:
|
||||
inherit).
|
||||
type: string
|
||||
default: buildx
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
@@ -39,76 +27,7 @@ jobs:
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Provision parse toolchain (jq + PyYAML)
|
||||
# This reusable parses the caller's hanzo.yml with python3 + PyYAML and
|
||||
# slices JSON with jq. The stock arc runner image
|
||||
# (ghcr.io/actions/actions-runner:latest) is minimal and ships NEITHER,
|
||||
# so provision them here. Guarded (a no-op the moment a runner image bakes
|
||||
# them in) — this keeps the reusable self-contained: any org can import it
|
||||
# onto a bare runner and it just works.
|
||||
run: |
|
||||
set -e
|
||||
need=0
|
||||
python3 -c 'import yaml' 2>/dev/null || need=1
|
||||
command -v jq >/dev/null 2>&1 || need=1
|
||||
if [ "$need" = 1 ]; then
|
||||
sudo apt-get update -qq
|
||||
sudo apt-get install -y -qq python3-yaml jq
|
||||
fi
|
||||
python3 -c 'import yaml; print("PyYAML", yaml.__version__)'
|
||||
jq --version
|
||||
|
||||
- name: Delegate build to platform (mode=delegate)
|
||||
# The GHA-escape fast path: instead of running buildx on this runner, POST
|
||||
# each image in hanzo.yml to platform.hanzo.ai's direct-enqueue webhook
|
||||
# (`/v1/arcd/enqueue`). Platform creates a build_job row, launches an
|
||||
# in-cluster BuildKit Job on its own pool, pushes to the registry, and —
|
||||
# for a system service — patches the operator Service CR to roll it. The
|
||||
# downstream is IDENTICAL to the platform GitHub-App webhook path (one
|
||||
# build path, two front doors), so a delegated build behaves exactly like
|
||||
# a platform-native one. This job then exits in seconds — no buildx, no
|
||||
# KMS, no runner-side deploy.
|
||||
if: inputs.mode == 'delegate'
|
||||
env:
|
||||
ENQUEUE_URL: ${{ vars.PLATFORM_ENQUEUE_URL || 'https://platform.hanzo.ai/v1/arcd/enqueue' }}
|
||||
ENQUEUE_TOKEN: ${{ secrets.PLATFORM_BUILD_CALLBACK_TOKEN }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ -z "${ENQUEUE_TOKEN:-}" ]; then
|
||||
echo "::error::mode=delegate needs the PLATFORM_BUILD_CALLBACK_TOKEN secret (secrets: inherit)"; exit 1
|
||||
fi
|
||||
REPO="${{ github.repository }}"
|
||||
SHA="${{ github.sha }}"
|
||||
SHORT=$(echo "$SHA" | cut -c1-7)
|
||||
REF="${{ github.ref }}"
|
||||
BRANCH="${{ github.ref_name }}"
|
||||
# One enqueue per (image, platform), mirroring the buildx tag shape the
|
||||
# deploy path expects (`sha-<short>-<arch>[-<suffix>]`). Default arch is
|
||||
# amd64 (single-arch), so an existing repo's tag shape is unchanged.
|
||||
python3 -c "import yaml,json;print(json.dumps(yaml.safe_load(open('hanzo.yml')).get('images') or []))" | jq -c '.[]' | while read -r img; do
|
||||
name=$(echo "$img"|jq -r .name); repo=$(echo "$img"|jq -r .repo)
|
||||
ctx=$(echo "$img"|jq -r .context); df=$(echo "$img"|jq -r '.dockerfile // (.context+"/Dockerfile")')
|
||||
sfx=$(echo "$img"|jq -r '."tag-suffix" // ""')
|
||||
echo "$img" | jq -r '(.platforms // ["linux/amd64"])[]' | while read -r plat; do
|
||||
arch="${plat##*/}"
|
||||
image="${repo}:sha-${SHORT}-${arch}${sfx:+-$sfx}"
|
||||
body=$(jq -nc \
|
||||
--arg repo "$REPO" --arg sha "$SHA" --arg image "$image" \
|
||||
--arg ref "$REF" --arg branch "$BRANCH" \
|
||||
--arg dockerfile "$df" --arg context "$ctx" --arg arch "$arch" \
|
||||
'{repo:$repo,sha:$sha,image:$image,ref:$ref,branch:$branch,dockerfile:$dockerfile,context:$context,os:"linux",arch:$arch}')
|
||||
echo "::group::delegate $name → $image"
|
||||
code=$(curl -sS -o /tmp/enqueue.out -w '%{http_code}' -X POST "$ENQUEUE_URL" \
|
||||
-H "Authorization: Bearer $ENQUEUE_TOKEN" -H 'Content-Type: application/json' -d "$body")
|
||||
cat /tmp/enqueue.out; echo
|
||||
# 202 Accepted = queued; 409 = no live runner for the pool (surface it loud).
|
||||
if [ "$code" != "202" ]; then echo "::error::enqueue $image failed (HTTP $code)"; exit 1; fi
|
||||
echo "::endgroup::"
|
||||
done
|
||||
done
|
||||
|
||||
- name: Authenticated git for go modules (rate-limit + any private repo)
|
||||
if: inputs.mode != 'delegate'
|
||||
# luxfi/hanzoai/zooai Go modules are PUBLIC, so `go` resolves them through
|
||||
# the default public proxy (proxy.golang.org) + checksum db (sum.golang.org)
|
||||
# — canonical, IMMUTABLE hashes that a force-moved tag can no longer break.
|
||||
@@ -145,7 +64,6 @@ jobs:
|
||||
|| echo "::warning::GH_PAT set but repo probe failed"
|
||||
|
||||
- name: Log in to GHCR (automatic workflow token)
|
||||
if: inputs.mode != 'delegate'
|
||||
# GHCR push needs no stored credential: GitHub injects a per-job
|
||||
# GITHUB_TOKEN scoped to THIS repo, and `permissions: packages: write`
|
||||
# (above) lets it push the repo's own package (e.g. zooai/node →
|
||||
@@ -155,19 +73,12 @@ jobs:
|
||||
|
||||
- name: Fetch deploy credentials from KMS
|
||||
id: kms
|
||||
if: inputs.mode != 'delegate'
|
||||
env:
|
||||
KMS_CLIENT_ID: ${{ secrets.KMS_CLIENT_ID }}
|
||||
KMS_CLIENT_SECRET: ${{ secrets.KMS_CLIENT_SECRET }}
|
||||
KMS_ENDPOINT: ${{ vars.KMS_ENDPOINT || 'https://kms.hanzo.ai' }}
|
||||
KMS_ORG: ${{ vars.KMS_ORG }}
|
||||
run: |
|
||||
# This step is BEST-EFFORT (see below): GHCR push already works via the
|
||||
# workflow token, and deploy creds are optional. GitHub wraps `run:` in
|
||||
# `bash -eo pipefail`, so we MUST explicitly `set +e` — otherwise an
|
||||
# unguarded curl (e.g. a KMS secret 404 at this org/path) aborts the
|
||||
# step and fails the whole build. Keep -u/-o pipefail; drop -e.
|
||||
set +e
|
||||
set -uo pipefail
|
||||
# Canonical luxfi/kms surface — /v1/kms (the Infisical /api/* surface was
|
||||
# removed when KMS migrated to luxfi/kms, MPC-rooted). Auth = an IAM
|
||||
@@ -184,22 +95,6 @@ jobs:
|
||||
# KUBECONFIG — cluster access for the deploy step.
|
||||
ORG="$(python3 -c "import yaml;print((yaml.safe_load(open('hanzo.yml')).get('kms') or {}).get('org','') )" 2>/dev/null || true)"
|
||||
: "${ORG:=${KMS_ORG:-}}"
|
||||
# Systemic default: when neither hanzo.yml `kms.org` nor the KMS_ORG var
|
||||
# is set, derive the KMS org from the GitHub owner. The org → KMS-org
|
||||
# relation is a small, stable first-party fact; keeping it here (ONE
|
||||
# place) is why every repo gets the canonical KMS deploy-cred path with
|
||||
# zero per-repo config — before this, ORG was always empty and this whole
|
||||
# step short-circuited, so no build ever fetched its private-dep git
|
||||
# credential from KMS (it silently fell back to GH_PAT and failed on
|
||||
# private hanzoai/cloud). hanzo.yml `kms.org` and KMS_ORG still override.
|
||||
if [ -z "$ORG" ]; then
|
||||
case "${{ github.repository_owner }}" in
|
||||
hanzoai) ORG=hanzo ;;
|
||||
luxfi) ORG=lux ;;
|
||||
zooai) ORG=zoo ;;
|
||||
*) ORG="${{ github.repository_owner }}" ;;
|
||||
esac
|
||||
fi
|
||||
ENV="$(python3 -c "import yaml;print((yaml.safe_load(open('hanzo.yml')).get('kms') or {}).get('environment','prod'))" 2>/dev/null || echo prod)"
|
||||
PATHQ="$(python3 -c "import yaml;print((yaml.safe_load(open('hanzo.yml')).get('kms') or {}).get('path','/deploy').strip('/'))" 2>/dev/null || echo deploy)"
|
||||
if [ -z "${KMS_CLIENT_ID:-}" ] || [ -z "$ORG" ]; then
|
||||
@@ -218,110 +113,29 @@ jobs:
|
||||
if [ -n "$KUBECONFIG_B64" ]; then echo "$KUBECONFIG_B64" | base64 -d > "$RUNNER_TEMP/kubeconfig"; echo "kubeconfig=$RUNNER_TEMP/kubeconfig" >> "$GITHUB_OUTPUT"; fi
|
||||
|
||||
- name: Build & push images (per hanzo.yml)
|
||||
if: inputs.mode != 'delegate'
|
||||
env:
|
||||
GH_PAT: ${{ secrets.GH_PAT }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# `images:` is OPTIONAL. A repo that ships no container image — e.g. a
|
||||
# static site deployed via Cloudflare Pages (its own deploy.yml), which
|
||||
# imports this reusable purely for the `test:` lint gate — omits the key
|
||||
# entirely. `.get('images') or []` yields an empty list, the loop below
|
||||
# runs zero times, and no cross-org GHCR push is attempted (so a repo in
|
||||
# a different GitHub org never hits `denied: permission_denied`).
|
||||
# Build-time private cross-org Go module read (the buildx `gh_token`
|
||||
# secret): prefer the KMS-fetched GIT_TOKEN, else fall back to the org
|
||||
# GH_PAT — the SAME BuildKit gh_token cloud's release.yml uses (proven
|
||||
# working). Keeps image builds green when the KMS deploy-cred fetch is
|
||||
# unavailable (a repo with a private cross-org dep like hanzoai/cloud
|
||||
# otherwise fails `go mod tidy` with git exit 128 in the buildx stage).
|
||||
# No-op for public-only builds when both are empty. Exported so the
|
||||
# `--secret id=gh_token,env=GIT_TOKEN` below reads it from the env.
|
||||
export GIT_TOKEN="${GIT_TOKEN:-${GH_PAT:-}}"
|
||||
SHORT=$(echo "${{ github.sha }}" | cut -c1-7)
|
||||
IS_TAG=$([ "${{ github.ref_type }}" = "tag" ] && echo 1 || echo 0)
|
||||
VER="${{ github.ref_name }}"; VER="${VER#v}"
|
||||
python3 -c "import yaml,json;print(json.dumps(yaml.safe_load(open('hanzo.yml')).get('images') or []))" | jq -c '.[]' | while read -r img; do
|
||||
python3 -c "import yaml,json;print(json.dumps(yaml.safe_load(open('hanzo.yml'))['images']))" | jq -c '.[]' | while read -r img; do
|
||||
name=$(echo "$img"|jq -r .name); ctx=$(echo "$img"|jq -r .context)
|
||||
df=$(echo "$img"|jq -r '.dockerfile // (.context+"/Dockerfile")'); repo=$(echo "$img"|jq -r .repo)
|
||||
# tag-suffix is OPTIONAL (most repos ship a single variant). When set
|
||||
# (e.g. "ce"/"ee") it qualifies every tag; when absent the tags are
|
||||
# clean (no trailing dash). Build + deploy must agree on this shape.
|
||||
sfx=$(echo "$img"|jq -r '."tag-suffix" // ""')
|
||||
# platforms is OPT-IN per image in hanzo.yml (default: amd64 only, so
|
||||
# every existing repo's tag shape "-amd64" is UNCHANGED). Set e.g.
|
||||
# platforms: [linux/amd64, linux/arm64]
|
||||
# to emit a multi-arch MANIFEST LIST — one digest serving both arches.
|
||||
# DOKS has no arm64 nodes, so arm64 builds via buildx QEMU emulation
|
||||
# (binfmt set up below); pure-Go (CGO_ENABLED=0) Dockerfiles that honor
|
||||
# $TARGETARCH cross-compile natively (fast, no emulation). For true
|
||||
# native-speed arm64, register a bare-metal arm64 host (spark/GB10) as
|
||||
# the hanzo-build-linux-arm64 self-hosted runner (values-build-arm64.yaml).
|
||||
plats=$(echo "$img"|jq -r '(.platforms // ["linux/amd64"]) | join(",")')
|
||||
if [ "$plats" = "linux/amd64" ]; then
|
||||
# single-arch: keep the exact legacy tag shape (-amd64) deploys expect.
|
||||
TAGS="-t $repo:sha-${SHORT}-amd64${sfx:+-$sfx} -t $repo:${sfx:+$sfx-}latest"
|
||||
[ "$IS_TAG" = 1 ] && TAGS="$TAGS -t $repo:${VER}-amd64${sfx:+-$sfx}"
|
||||
else
|
||||
# multi-arch: one arch-neutral manifest-list tag (no -amd64 suffix).
|
||||
docker run --privileged --rm tonistiigi/binfmt --install arm64 >/dev/null 2>&1 || true
|
||||
TAGS="-t $repo:sha-${SHORT}${sfx:+-$sfx} -t $repo:${sfx:+$sfx-}latest"
|
||||
[ "$IS_TAG" = 1 ] && TAGS="$TAGS -t $repo:${VER}${sfx:+-$sfx}"
|
||||
fi
|
||||
echo "::group::build $name → $repo (${sfx}) [$plats]"
|
||||
TAGS="-t $repo:sha-${SHORT}-amd64${sfx:+-$sfx} -t $repo:${sfx:+$sfx-}latest"
|
||||
[ "$IS_TAG" = 1 ] && TAGS="$TAGS -t $repo:${VER}-amd64${sfx:+-$sfx}"
|
||||
echo "::group::build $name → $repo (${sfx})"
|
||||
# GIT_TOKEN (from KMS, via GITHUB_ENV) is passed as the `gh_token`
|
||||
# BuildKit secret so Dockerfiles can clone private Go modules; omitted
|
||||
# cleanly when absent (public-only builds unaffected).
|
||||
docker buildx build --platform "$plats" ${GIT_TOKEN:+--secret id=gh_token,env=GIT_TOKEN} --push $TAGS -f "$df" "$ctx"
|
||||
docker buildx build --platform linux/amd64 ${GIT_TOKEN:+--secret id=gh_token,env=GIT_TOKEN} --push $TAGS -f "$df" "$ctx"
|
||||
echo "::endgroup::"
|
||||
done
|
||||
|
||||
- name: Provision Go toolchain (go test gates on bare runners)
|
||||
# hanzo.yml `test:` gates (e.g. `go vet ./...`, `go test ...`) run
|
||||
# DIRECTLY on the runner, NOT inside a build container — but the stock
|
||||
# arc runner image (ghcr.io/actions/actions-runner) ships no Go, so a Go
|
||||
# gate dies with `go: command not found` (exit 127). Provision the repo's
|
||||
# OWN Go version from go.mod so the toolchain matches the module exactly.
|
||||
# Guarded to Go repos (go.mod present) so pure-JS/TS callers are
|
||||
# unaffected; harmless if a future runner image bakes Go in.
|
||||
if: inputs.mode != 'delegate' && hashFiles('go.mod') != ''
|
||||
uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version-file: go.mod
|
||||
cache: false
|
||||
|
||||
- name: Provision C toolchain (cgo test gates)
|
||||
# CGO_ENABLED=1 gates (e.g. go-sqlite3, which bundles the sqlite
|
||||
# amalgamation) need a C compiler; the minimal arc runner ships none
|
||||
# (`cgo: gcc not found`). Same guarded provision the parse-toolchain step
|
||||
# above uses — a no-op when gcc is already present.
|
||||
if: inputs.mode != 'delegate' && hashFiles('go.mod') != ''
|
||||
run: command -v gcc >/dev/null 2>&1 || { sudo apt-get update -qq && sudo apt-get install -y -qq gcc; }
|
||||
|
||||
- name: Authenticate runner git for private Go modules (test gates)
|
||||
# The Test step runs `go vet`/`go test` ON the runner (not in buildx), so
|
||||
# `go` fetches private hanzoai/* modules (GOPRIVATE → direct) through the
|
||||
# runner's git, which needs a credential. The IMAGE build authenticates
|
||||
# via the KMS `gh_token` inside buildx (GIT_TOKEN); the earlier GH_PAT
|
||||
# git-auth step is a no-op for repos that rely on that KMS token (GH_PAT
|
||||
# unset) — so `go vet` dies with `could not read Username for github.com`.
|
||||
# Reuse the SAME token here for the runner's git (GIT_TOKEN, set by the
|
||||
# KMS step above; GH_PAT fallback). No-op when neither is present.
|
||||
if: inputs.mode != 'delegate' && hashFiles('go.mod') != ''
|
||||
env:
|
||||
GH_PAT: ${{ secrets.GH_PAT }}
|
||||
run: |
|
||||
set -uo pipefail
|
||||
TOKEN="${GIT_TOKEN:-${GH_PAT:-}}"
|
||||
if [ -z "$TOKEN" ]; then echo "no git token — public modules only"; exit 0; fi
|
||||
CFG="$RUNNER_TEMP/gitconfig-go-test"; : > "$CFG"
|
||||
GIT_CONFIG_GLOBAL="$CFG" git config --global \
|
||||
url."https://x-access-token:${TOKEN}@github.com/".insteadOf "https://github.com/"
|
||||
{ echo "GIT_CONFIG_GLOBAL=$CFG"; echo "GIT_CONFIG_NOSYSTEM=1"; } >> "$GITHUB_ENV"
|
||||
echo "runner git authenticated for private Go modules"
|
||||
|
||||
- name: Test (per hanzo.yml)
|
||||
if: inputs.mode != 'delegate'
|
||||
run: |
|
||||
set -euo pipefail
|
||||
python3 -c "import yaml,json;print(json.dumps(yaml.safe_load(open('hanzo.yml')).get('test') or []))" | jq -c '.[]' | while read -r t; do
|
||||
@@ -330,7 +144,7 @@ jobs:
|
||||
done
|
||||
|
||||
- name: Deploy (per hanzo.yml)
|
||||
if: inputs.mode != 'delegate' && github.event_name != 'pull_request' && steps.kms.outputs.kubeconfig != ''
|
||||
if: github.event_name != 'pull_request' && steps.kms.outputs.kubeconfig != ''
|
||||
env:
|
||||
KUBECONFIG: ${{ steps.kms.outputs.kubeconfig }}
|
||||
run: |
|
||||
@@ -343,7 +157,7 @@ jobs:
|
||||
fi
|
||||
SHORT=$(echo "${{ github.sha }}" | cut -c1-7)
|
||||
NS=$(python3 -c "import yaml;print(yaml.safe_load(open('hanzo.yml'))['deploy']['namespace'])")
|
||||
python3 -c "import yaml,json;print(json.dumps(yaml.safe_load(open('hanzo.yml')).get('images') or []))" > /tmp/imgs.json
|
||||
python3 -c "import yaml,json;print(json.dumps(yaml.safe_load(open('hanzo.yml'))['images']))" > /tmp/imgs.json
|
||||
python3 -c "import yaml,json;print(json.dumps(yaml.safe_load(open('hanzo.yml'))['deploy']['services']))" | jq -c '.[]' | while read -r s; do
|
||||
svc=$(echo "$s"|jq -r .name); imgname=$(echo "$s"|jq -r .image)
|
||||
repo=$(jq -r --arg n "$imgname" '.[]|select(.name==$n)|.repo' /tmp/imgs.json)
|
||||
|
||||
@@ -52,33 +52,6 @@ build minutes). To run on **your own** self-hosted arc runners, pass their label
|
||||
secrets: inherit
|
||||
```
|
||||
|
||||
## Delegate to platform (skip runner buildx)
|
||||
|
||||
By default the build runs buildx **on** the arc runner. To instead hand the build
|
||||
to **platform.hanzo.ai** — which builds in-cluster with BuildKit and rolls the
|
||||
service itself — pass `mode: delegate`:
|
||||
|
||||
```yaml
|
||||
uses: hanzoai/ci/.github/workflows/build.yml@v1
|
||||
with:
|
||||
mode: delegate
|
||||
secrets: inherit
|
||||
```
|
||||
|
||||
The GitHub job then just POSTs each image in `hanzo.yml` to platform's direct
|
||||
build webhook (`/v1/arcd/enqueue`) and exits in **seconds** — no runner buildx,
|
||||
no KMS, no runner-side deploy. Platform creates the build job, launches an
|
||||
in-cluster BuildKit Job on its own pool, pushes to the registry, and patches the
|
||||
operator `Service` CR to roll it. It's the same build path as the platform
|
||||
GitHub-App webhook — one build path, two front doors.
|
||||
|
||||
Requires one extra secret, `PLATFORM_BUILD_CALLBACK_TOKEN` (org- or repo-level,
|
||||
picked up via `secrets: inherit`). Override the endpoint with the
|
||||
`PLATFORM_ENQUEUE_URL` repo/org variable (default `https://platform.hanzo.ai/v1/arcd/enqueue`).
|
||||
|
||||
`mode: buildx` (the default) is unchanged — existing repos keep running buildx on
|
||||
arc, so delegation is strictly opt-in.
|
||||
|
||||
## Credentials
|
||||
|
||||
The only GitHub secrets a repo sets are `KMS_CLIENT_ID` / `KMS_CLIENT_SECRET`
|
||||
|
||||
Reference in New Issue
Block a user