zoo/universe
2
0
Fork 0
Code Issues Pull Requests Releases Activity

ci: a deploy is a commit, so stop applying to the cluster

Browse Source 
Two paths in this workflow wrote to zoo-k8s directly. `deploy-single` ran
kubectl apply -k over one namespace's overlay; `deploy-all` ran a script that
applied the CRDs, applied the whole overlay, and then hand-patched luxd flags
onto a StatefulSet the Rust operator renders -- a patch whose own comment
explains it persists because the operator does not re-render an existing pod
template.

That is a second writer for objects zoo-fleet declares from deploy/*/*.yaml,
and the disagreement is not quiet. Both stamp the pod template
(app.kubernetes.io/managed-by: hanzo-operator vs Helm), and a pod-template
label is not a field two controllers can differ about without a restart, so
each flip rolls the workload. It does not converge; it oscillates.

It also applied the WRONG THING. k8s/ is the hand-applied tree whose currency
deploy/README.md says outright is unknown -- some of it ahead of the cluster,
some behind, some describing objects that were deleted. A pipeline that applies
it on every push re-asserts that drift on a schedule.

What is left writes to git and tests: record the pin, open the PR, run E2E.
zoo-fleet reconciles from the merged file. The kubectl calls still in the E2E
job read pods, events and logs on failure, and write nothing.

The k8s/ tree stays, for what deploy/ deliberately does not cover -- CRDs,
namespaces, the operator's custom resources. `make k8s-apply env=<env>` is that
path, with a human behind it.

deploy-all and e2e-only meant the same thing once neither deployed, so the
modes are now the two acts that remain: pin and e2e.

Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
Hanzo AI
2026-09-03 21:32:59 -07:00
parent a9cf29e4c2
commit 19b7efd0e5
2 changed files with 330 additions and 123 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.hanzo/workflows/ci.yml
+330
View File
@@ -0,0 +1,330 @@
# Universe CI -- Layer 2 Orchestrator
#
# Runs from .hanzo/workflows, which is the one directory this forge collects.
# Gitea takes the FIRST of WORKFLOW_DIRS that exists in the pushed commit and
# ignores the rest (modules/actions/workflows.go, listWorkflowsInDirs breaks on
# the first hit), so this directory and .github/workflows were never going to
# coexist: creating one switches the other off silently, with no run, no queued
# job and no error. Everything this repo still wants moved here in one commit.
#
# THIS WORKFLOW DOES NOT DEPLOY. It writes to git and it tests; the cluster is
# reached by neither.
#
# It used to end in `kubectl apply -k k8s/<ns>`, on both paths: once per service
# and once over the whole overlay, the second also applying CRDs and hand-patching
# a StatefulSet the Rust operator owns. That is a second writer for objects
# cd.hanzo.ai now declares from deploy/*/*.yaml, and the two do not disagree
# quietly -- they disagree on the pod template, where every flip is a rolling
# restart. Not a race that settles: a loop.
#
# So the deploy is a COMMIT. Move the pin, and zoo-fleet reconciles it (declared
# in hanzo/universe/infra/k8s/hanzo-cd/project-zoo.yaml). What CI can still do
# wrong is then bounded by what review lets into the file.
#
# Two modes:
# 1. pin (repository_dispatch): a service was built and tested. Record it in
# the manifest and open the PR.
# 2. e2e (push/schedule/dispatch): run the suite against what is deployed.
#
# Layer 1 (build-service.yml): per-repo lint + unit test + build + push
# Layer 2 (this): pin tracking + integration E2E
#
# The k8s/ kustomize tree still holds what deploy/ does not -- CRDs, namespaces,
# the operator's own custom resources. Applying it stays a deliberate act with a
# human behind it (`make k8s-apply env=<env>`), never a step in a pipeline.
name: Universe CI
on:
repository_dispatch:
types: [service-built]
push:
branches: [main, test, dev]
paths:
- 'manifests/**'
- 'scripts/**'
- '.hanzo/workflows/ci.yml'
- 'e2e/**'
workflow_dispatch:
inputs:
environment:
description: "Target environment"
type: choice
options:
- auto
- main
- test
- dev
default: auto
schedule:
# Nightly full E2E at 03:00 UTC
- cron: '0 3 * * *'
concurrency:
group: universe-ci-${{ github.event.client_payload.branch || github.ref_name }}
cancel-in-progress: false
env:
CLUSTER: do-sfo3-zoo-k8s
permissions:
contents: write
packages: write
id-token: write
jobs:
# ============================================================
# Resolve environment from trigger context
# ============================================================
resolve:
name: Resolve Environment
runs-on: [zoo-build-linux-amd64]
outputs:
environment: ${{ steps.env.outputs.environment }}
namespace: ${{ steps.env.outputs.namespace }}
mode: ${{ steps.env.outputs.mode }}
service_key: ${{ steps.env.outputs.service_key }}
service_sha: ${{ steps.env.outputs.service_sha }}
service_image: ${{ steps.env.outputs.service_image }}
service_app_name: ${{ steps.env.outputs.service_app_name }}
steps:
- name: Resolve
id: env
run: |
EVENT="${GITHUB_EVENT_NAME}"
if [ "${EVENT}" = "repository_dispatch" ]; then
BRANCH='${{ github.event.client_payload.branch }}'
MODE="pin"
echo "service_key=${{ github.event.client_payload.service_key }}" >> "$GITHUB_OUTPUT"
echo "service_sha=${{ github.event.client_payload.sha }}" >> "$GITHUB_OUTPUT"
echo "service_image=${{ github.event.client_payload.image }}" >> "$GITHUB_OUTPUT"
echo "service_app_name=${{ github.event.client_payload.app_name }}" >> "$GITHUB_OUTPUT"
elif [ "${EVENT}" = "workflow_dispatch" ]; then
BRANCH="${GITHUB_REF_NAME}"
INPUT_ENV='${{ inputs.environment }}'
if [ "${INPUT_ENV}" != "auto" ] && [ -n "${INPUT_ENV}" ]; then
BRANCH="${INPUT_ENV}"
fi
MODE="e2e"
elif [ "${EVENT}" = "schedule" ]; then
BRANCH="dev"
MODE="e2e"
else
BRANCH="${GITHUB_REF_NAME}"
MODE="e2e"
fi
case "${BRANCH}" in
main) NS="zoo-mainnet" ;;
test) NS="zoo-testnet" ;;
dev) NS="zoo-devnet" ;;
*) NS="zoo-devnet" BRANCH="dev" ;;
esac
echo "environment=${BRANCH}" >> "$GITHUB_OUTPUT"
echo "namespace=${NS}" >> "$GITHUB_OUTPUT"
echo "mode=${MODE}" >> "$GITHUB_OUTPUT"
echo "Environment: ${BRANCH}"
echo "Namespace: ${NS}"
echo "Mode: ${MODE}"
# ============================================================
# Mode: pin -- record what was built, and open the PR that deploys it
# ============================================================
update-manifest:
name: Update Manifest
runs-on: [zoo-build-linux-amd64]
needs: resolve
if: needs.resolve.outputs.mode == 'pin'
steps:
- name: Fetch secrets from KMS
id: kms
env:
KMS_CLIENT_ID: ${{ secrets.KMS_CLIENT_ID }}
KMS_CLIENT_SECRET: ${{ secrets.KMS_CLIENT_SECRET }}
run: |
set -euo pipefail
KMS_URL="${KMS_URL:-https://kms.hanzo.ai}"
WORKSPACE="zoo-platform"
ACCESS_TOKEN="$(
curl -fsS --max-time 10 -X POST "${KMS_URL}/api/v1/auth/universal-auth/login" \
-H "Content-Type: application/json" \
-d "{\"clientId\":\"${KMS_CLIENT_ID}\",\"clientSecret\":\"${KMS_CLIENT_SECRET}\"}" \
| jq -r '.accessToken'
)"
[ -n "${ACCESS_TOKEN}" ] && [ "${ACCESS_TOKEN}" != "null" ] || { echo "::error::KMS auth failed"; exit 1; }
fetch_secret() {
curl -fsS "${KMS_URL}/api/v3/secrets/raw/${1}?workspaceSlug=${WORKSPACE}&environment=prod&secretPath=/ci&viewSecretValue=true&include_imports=true" \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
| jq -r '.secret.secretValue'
}
for name in GCP_CREDENTIAL TOKEN_GITHUB_CICD DIGITALOCEAN_ACCESS_TOKEN; do
val="$(fetch_secret "$name" 2>/dev/null || echo "")"
[ -n "$val" ] && [ "$val" != "null" ] || continue
echo "::add-mask::${val}"
{ echo "${name}<<ENDOFSECRET"; echo "${val}"; echo "ENDOFSECRET"; } >> "$GITHUB_OUTPUT"
done
- uses: actions/checkout@v4
with:
ref: ${{ needs.resolve.outputs.environment }}
token: ${{ steps.kms.outputs.TOKEN_GITHUB_CICD }}
- name: Update manifest
run: |
bash scripts/ci/update-manifest.sh \
"${{ needs.resolve.outputs.service_key }}" \
"${{ needs.resolve.outputs.service_sha }}" \
"${{ needs.resolve.outputs.service_image }}"
- name: Create deploy PR (triggers E2E before merge)
env:
GH_TOKEN: ${{ steps.kms.outputs.TOKEN_GITHUB_CICD }}
run: |
git config user.name "zoo-ci"
git config user.email "[email protected]"
git add manifests/
if git diff --cached --quiet; then
echo "No manifest changes"
exit 0
fi
SERVICE="${{ needs.resolve.outputs.service_key }}"
SHA="${{ needs.resolve.outputs.service_sha }}"
VERSION="${{ github.event.client_payload.version }}"
LABEL="${VERSION:-${SHA::7}}"
BRANCH="deploy/${SERVICE}-${LABEL}"
BASE="${{ needs.resolve.outputs.environment }}"
git checkout -b "${BRANCH}"
git commit -m "deploy: ${SERVICE} v${LABEL}"
git push origin "${BRANCH}"
gh pr create \
--base "${BASE}" \
--head "${BRANCH}" \
--title "deploy: ${SERVICE} v${LABEL}" \
--body "$(cat <<EOF
**Service**: \`${SERVICE}\`
**Version**: \`${LABEL}\`
**Image**: \`${{ needs.resolve.outputs.service_image }}\`
PR E2E runs automatically. Auto-merges on pass (minor/patch).
EOF
)"
gh pr merge "${BRANCH}" --auto --squash 2>/dev/null || true
# ============================================================
# E2E tests
# ============================================================
e2e:
name: E2E
runs-on: [zoo-build-linux-amd64]
needs: [resolve]
if: needs.resolve.outputs.mode == 'e2e'
timeout-minutes: 25
steps:
- uses: actions/checkout@v4
with:
ref: ${{ needs.resolve.outputs.environment }}
- uses: actions/setup-node@v4
with:
node-version: 20
- name: Install dependencies
run: npm ci 2>/dev/null || true
- name: Install Playwright
run: npx playwright install --with-deps chromium 2>/dev/null || true
- name: Fetch secrets from KMS
id: kms
env:
KMS_CLIENT_ID: ${{ secrets.KMS_CLIENT_ID }}
KMS_CLIENT_SECRET: ${{ secrets.KMS_CLIENT_SECRET }}
run: |
set -euo pipefail
KMS_URL="${KMS_URL:-https://kms.hanzo.ai}"
WORKSPACE="zoo-platform"
ACCESS_TOKEN="$(
curl -fsS --max-time 10 -X POST "${KMS_URL}/api/v1/auth/universal-auth/login" \
-H "Content-Type: application/json" \
-d "{\"clientId\":\"${KMS_CLIENT_ID}\",\"clientSecret\":\"${KMS_CLIENT_SECRET}\"}" \
| jq -r '.accessToken'
)"
[ -n "${ACCESS_TOKEN}" ] && [ "${ACCESS_TOKEN}" != "null" ] || { echo "::error::KMS auth failed"; exit 1; }
fetch_secret() {
curl -fsS "${KMS_URL}/api/v3/secrets/raw/${1}?workspaceSlug=${WORKSPACE}&environment=prod&secretPath=/ci&viewSecretValue=true&include_imports=true" \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
| jq -r '.secret.secretValue'
}
for name in GCP_CREDENTIAL TOKEN_GITHUB_CICD DIGITALOCEAN_ACCESS_TOKEN; do
val="$(fetch_secret "$name" 2>/dev/null || echo "")"
[ -n "$val" ] && [ "$val" != "null" ] || continue
echo "::add-mask::${val}"
{ echo "${name}<<ENDOFSECRET"; echo "${val}"; echo "ENDOFSECRET"; } >> "$GITHUB_OUTPUT"
done
- name: Install doctl
uses: digitalocean/action-doctl@v2
with:
token: ${{ steps.kms.outputs.DIGITALOCEAN_ACCESS_TOKEN }}
- name: Configure kubectl
run: doctl kubernetes cluster kubeconfig save "${{ env.CLUSTER }}"
- name: Run E2E
timeout-minutes: 15
env:
E2E_ENV: ${{ needs.resolve.outputs.environment }}
E2E_NAMESPACE: ${{ needs.resolve.outputs.namespace }}
CI: "true"
run: |
if [ -f e2e/run.sh ]; then
bash e2e/run.sh
elif [ -f playwright.config.ts ]; then
npx playwright test
else
echo "No E2E tests configured yet -- skipping"
fi
- name: Upload results
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-${{ needs.resolve.outputs.environment }}-${{ github.run_number }}
path: |
playwright-report/
e2e/test-results/
retention-days: 7
- name: Collect failure diagnostics
if: failure()
run: |
NS="${{ needs.resolve.outputs.namespace }}"
echo "=== Pod Status ==="
kubectl get pods -n "${NS}" -o wide 2>/dev/null || true
echo ""
echo "=== Events (last 20) ==="
kubectl get events -n "${NS}" --sort-by='.lastTimestamp' 2>/dev/null | tail -20 || true
echo ""
echo "=== CrashLooping / Error Pod Logs ==="
for pod in $(kubectl get pods -n "${NS}" --no-headers 2>/dev/null | grep -vE 'Running|Completed' | awk '{print $1}'); do
echo "--- ${pod} ---"
kubectl logs -n "${NS}" "${pod}" --tail=50 2>/dev/null || true
done
scripts/ci/deploy-all.sh
-123
View File
@@ -1,123 +0,0 @@
#!/usr/bin/env bash
# Deploy every service from manifests/services.yml to the target namespace.
#
# Usage:
# ./deploy-all.sh <env>
#
# <env> ∈ {dev,test,main} → namespace ∈ {zoo-devnet, zoo-testnet, zoo-mainnet}.
# Cluster is always do-sfo3-zoo-k8s (set in the manifest).
set -euo pipefail
ENV="${1:?Usage: deploy-all.sh <dev|test|main>}"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
ROOT_DIR="$(cd "${SCRIPT_DIR}/../.." && pwd)"
MANIFEST="${ROOT_DIR}/manifests/services.yml"
if [[ ! -f "${MANIFEST}" ]]; then
echo "::error::Manifest not found: ${MANIFEST}"
exit 1
fi
case "${ENV}" in
main) NS="zoo-mainnet" ;;
test) NS="zoo-testnet" ;;
dev) NS="zoo-devnet" ;;
*) echo "::error::Unknown env '${ENV}' (expect dev/test/main)"; exit 1 ;;
esac
OVERLAY_DIR="${ROOT_DIR}/k8s/${NS}"
echo "========================================================"
echo " Deploying Zoo universe: ${ENV}"
echo " Namespace: ${NS}"
echo " Overlay: ${OVERLAY_DIR}"
echo "========================================================"
if [[ -d "${ROOT_DIR}/k8s/crds" ]]; then
echo "Applying CRDs..."
kubectl apply -f "${ROOT_DIR}/k8s/crds/"
fi
if [[ ! -d "${OVERLAY_DIR}" ]]; then
echo "::error::Overlay directory not found: ${OVERLAY_DIR}"
exit 1
fi
echo "Applying overlay: ${NS}..."
kubectl apply -k "${OVERLAY_DIR}"
# Inject luxd CLI flags onto the operator-rendered zoo-chain StatefulSet.
# The Rust zoo-operator owns this StatefulSet (it is not in the kustomize set,
# so it can't be patched via the overlay) and passes config as ZOO_* env vars,
# which luxd ignores for http-host / bootstrap / track-chains. Without
# --http-host=0.0.0.0 luxd binds its HTTP/health API to 127.0.0.1 and the
# kubelet liveness probe (pod IP) can never reach it → ~90s SIGTERM restart loop.
# The operator does not re-render an existing StatefulSet's pod template, so this
# patch persists across reconciles; re-running this deploy re-asserts it.
STS_PATCH="${OVERLAY_DIR}/statefulset-patch.yaml"
if [[ -f "${STS_PATCH}" ]]; then
echo "Patching luxd flags onto statefulset/zoo-chain..."
kubectl -n "${NS}" patch statefulset zoo-chain --type=strategic --patch-file "${STS_PATCH}"
fi
# Patch each manifest image onto its deployment so service-built dispatches
# take effect without committing image pins into the base.
python3 - "${MANIFEST}" "${NS}" <<'PYEOF'
import sys, subprocess
manifest_path = sys.argv[1]
namespace = sys.argv[2]
in_services = False
current_key = ""
current_image = ""
def apply(svc, image):
if not svc or not image:
return
print(f" set image {svc} -> {image}")
subprocess.run([
"kubectl", "set", "image",
f"deployment/{svc}", f"{svc}={image}",
"-n", namespace
], capture_output=True)
with open(manifest_path) as f:
for line in f:
stripped = line.strip()
if stripped == "services:":
in_services = True
continue
if not in_services:
continue
if not stripped or stripped.startswith("#"):
continue
indent = len(line) - len(line.lstrip())
if indent == 2 and stripped.endswith(":"):
if current_key:
apply(current_key, current_image)
current_key = stripped.rstrip(":")
current_image = ""
elif indent == 4 and current_key:
key, _, val = stripped.partition(":")
val = val.strip().strip('"').strip("'")
if key == "image":
current_image = val
if current_key:
apply(current_key, current_image)
PYEOF
echo ""
echo "Waiting for rollouts in ${NS}..."
kubectl rollout status deployment --all -n "${NS}" --timeout=300s 2>/dev/null || true
echo ""
echo "=== Pod Status ==="
kubectl get pods -n "${NS}" -o wide 2>/dev/null || true
echo ""
echo "========================================================"
echo " Deploy complete: ${ENV} (${NS})"
echo "========================================================"