feat(k8s): unify with luxfi/universe — coherent boot + snapshot-native replication (#3)

Brings zoo to parity with lux: the SAME luxd-boot.sh (byte-identical to luxfi/universe k8s/boot) drives every network, and native ZAP replication (PQ-encrypted, restore-on-boot) replaces the boot-time admin_importChain (RLP) path. - k8s/boot/: shared entrypoint + mainnet/testnet/devnet env (ids 1/2/3, zoo EVM chainIds 200200/200201/200202, S3 path zoo/<net>/zaprepl, endpoint s3.zoo-system). Script is byte-identical to luxfi/universe. - k8s/zoo-<net>/snapshot-native/: replicate-s3-creds + replicate-age secrets (fill from KMS) + ZooNetwork replication merge-patch. CRD already allows unknown fields, so spec.replication is accepted today. - zoo-mainnet/luxd-rlp-import.yaml: RETIRED banner; kept as the provenance record of the one-time chain-200200 seed (now in the snapshot stream). - SNAPSHOT-NATIVE-ROLLOUT.md: per-network apply order (devnet→testnet→mainnet). Requires build-zoo rebuilt on database >= v1.20.3 and zooai operator replication-env parity (mirror of luxfi/operator); until then the boot script sets REPLICATE_* itself so replication works without operator changes. Co-authored-by: Hanzo Dev <[email protected]>
2026-06-15 22:35:51 -07:00
parent 7bb55af252
commit ea03508ca7
16 changed files with 295 additions and 0 deletions
@@ -0,0 +1,26 @@
+# Zoo snapshot-native rollout
+
+Unifies zoo with `luxfi/universe`: one coherent boot (`k8s/boot/luxd-boot.sh`,
+byte-identical to Lux) + native ZAP replication to `zoo-system` MinIO, replacing
+the boot-time `admin_importChain` (RLP) path. PQ-encrypted (ML-KEM-768+X25519),
+restore-on-boot.
+
+## Layout
+- `k8s/boot/` — shared entrypoint + `<net>.env` (mainnet/testnet/devnet).
+- `k8s/zoo-<net>/snapshot-native/` — `replicate-*` secrets + ZooNetwork patch.
+- `k8s/zoo-mainnet/luxd-rlp-import.yaml` — RETIRED; kept as provenance only.
+
+## Apply (per network; mainnet is production → prepare, then apply deliberately)
+```sh
+# 1. fill KMS values into 10-replicate-secret.yaml (S3 creds + PQ identity)
+kubectl --context zoo-k8s -n zoo-<net> apply -f k8s/zoo-<net>/snapshot-native/10-replicate-secret.yaml
+# 2. enable replication on the CR
+kubectl --context zoo-k8s -n zoo-<net> patch zoonetwork zoo-chain --type=merge \
+  --patch-file k8s/zoo-<net>/snapshot-native/20-zoonetwork-snapshot-native.patch.yaml
+# 3. roll the node image to one built against database >= v1.20.3
+```
+Order: devnet → testnet → mainnet. Requires the `build-zoo` image rebuilt on
+`database >= v1.20.3` and the zooai operator carrying the replication-env parity
+(mirror of `luxfi/operator`); until then, point the node `command` at
+`/scripts/luxd-boot.sh` (ConfigMap from `k8s/boot/`), which sets the
+`REPLICATE_*` env itself.
@@ -0,0 +1,22 @@
+# Coherent luxd boot — zoo
+
+`luxd-boot.sh` is **byte-identical to `luxfi/universe` `k8s/boot/luxd-boot.sh`** —
+one entrypoint for every network. Source a `<net>.env`, then run it:
+
+```sh
+set -a; . ./mainnet.env; set +a; ./luxd-boot.sh   # or testnet.env / devnet.env
+```
+
+It selects the network (id 1/2/3 — zoo shares the canonical Lux primary-network
+ids), stages plugins, sets the native-replication defaults, and execs luxd as a
+real staking validator (keys under `<data>/staking`; the zooai operator mounts
+the per-validator secret). The luxd binary (`luxfi/database >= v1.20.3`) restores
+from the S3 snapshot stream on boot — **no RLP import, no genesis surgery**.
+
+Zoo differs from Lux only in: EVM chainId (200200/200201/200202), the S3 path
+(`zoo/<net>/zaprepl`), and the S3 endpoint (`s3.zoo-system`). All carried by the
+`<net>.env`, so the script itself stays universal.
+
+In-cluster: mount this script as a ConfigMap and point the ZooNetwork node
+container `command` at `/scripts/luxd-boot.sh`, with `<net>.env` keys as env +
+the `replicate-*` secrets via `envFrom` (see `../zoo-<net>/snapshot-native/`).
@@ -0,0 +1,17 @@
+# Coherent boot — zoo devnet. Identical luxd-boot.sh as luxfi/universe; only the
+# id / EVM chainId / S3 path differ. Primary-network id=3, Zoo EVM chainId=200202.
+LUX_NETWORK=devnet
+LUXD_NETWORK_ID=3
+LUXD_DATA_DIR=/data
+LUXD_HTTP_HOST=0.0.0.0
+LUXD_HTTP_PORT=9630
+LUXD_STAKING_PORT=9631
+LUXD_BIN=/luxd/build/luxd
+# Native ZAP replication → zoo-system MinIO (PQ-encrypted, restore-on-boot).
+REPLICATE_S3_ENDPOINT=http://s3.zoo-system.svc.cluster.local:9000
+REPLICATE_S3_BUCKET=zoo-snapshots
+REPLICATE_S3_REGION=us-central1
+REPLICATE_S3_USE_SSL=false
+REPLICATE_S3_PATH=zoo/devnet/zaprepl
+# REPLICATE_S3_ACCESS_KEY / SECRET_KEY + REPLICATE_AGE_IDENTITY come from the
+# replicate-s3-creds / replicate-age secrets (EnvFrom) — see snapshot-native/.
@@ -0,0 +1,65 @@
+#!/usr/bin/env bash
+# Coherent luxd boot — ONE entrypoint for every network: local, devnet, testnet, mainnet.
+#
+# Every network runs as a REAL staking validator (keys from <data>/staking — the
+# operator mounts the per-validator secret; luxd uses what's there). No --automine,
+# no ephemeral keys: the production path and the local path are the same.
+#
+# The luxd binary (luxfi/database >= v1.20.3) restores-on-boot from the S3 snapshot
+# stream, so this script does NOT do RLP import, peer keysync, or genesis-root
+# surgery — it selects the network, stages plugins, and execs luxd.
+#
+# Network ids: local=1337 (C-Chain 31337), devnet=3, testnet=2, mainnet=1.
+set -euo pipefail
+
+NETWORK="${LUX_NETWORK:-${1:-local}}"
+case "$NETWORK" in
+  local|localnet) NETID="1337" ;;   # localnet primary-network id (C-Chain id 31337)
+  devnet)         NETID="3" ;;
+  testnet)        NETID="2" ;;
+  mainnet)        NETID="1" ;;
+  *) echo "[luxd-boot] unknown network '$NETWORK' (want: local|devnet|testnet|mainnet)" >&2; exit 2 ;;
+esac
+# Per-universe override: zoo/hanzo reuse this script verbatim — same network names,
+# only the primary-network id (and genesis/S3 path) differ. e.g. set LUXD_NETWORK_ID.
+NETID="${LUXD_NETWORK_ID:-$NETID}"
+
+DATA_DIR="${LUXD_DATA_DIR:-/data}"
+HTTP_HOST="${LUXD_HTTP_HOST:-0.0.0.0}"
+HTTP_PORT="${LUXD_HTTP_PORT:-9630}"
+STAKING_PORT="${LUXD_STAKING_PORT:-9631}"
+LUXD="${LUXD_BIN:-/luxd/build/luxd}"
+
+# Native replication: per-network S3 path + restore-on-boot. Anything already set
+# (operator / sourced env) wins; we only fill gaps.
+export REPLICATE_S3_PATH="${REPLICATE_S3_PATH:-$NETWORK/zaprepl}"
+export REPLICATE_CDC="${REPLICATE_CDC:-true}"
+export REPLICATE_PHYSICAL_SNAPSHOT="${REPLICATE_PHYSICAL_SNAPSHOT:-true}"
+export REPLICATE_AUTO_ON_OPEN="${REPLICATE_AUTO_ON_OPEN:-true}"
+export REPLICATE_RESTORE_ON_BOOT="${REPLICATE_RESTORE_ON_BOOT:-true}"
+
+# EVM (and other VM) plugins staged into the data plugin dir (idempotent).
+PLUGIN_DIR="$DATA_DIR/plugins"
+mkdir -p "$PLUGIN_DIR"
+if [ -d /luxd/build/plugins ]; then cp -n /luxd/build/plugins/* "$PLUGIN_DIR/" 2>/dev/null || true; fi
+
+ARGS=( --network-id="$NETID"
+       --data-dir="$DATA_DIR"
+       --http-host="$HTTP_HOST" --http-port="$HTTP_PORT"
+       --staking-port="$STAKING_PORT"
+       --plugin-dir="$PLUGIN_DIR" )
+# Real staking keys live under $DATA_DIR/staking (operator-mounted secret). Point
+# luxd at an explicit staking dir if provided.
+[ -n "${LUXD_STAKING_TLS_CERT:-}" ] && ARGS+=( --staking-tls-cert-file="$LUXD_STAKING_TLS_CERT" )
+[ -n "${LUXD_STAKING_TLS_KEY:-}" ]  && ARGS+=( --staking-tls-key-file="$LUXD_STAKING_TLS_KEY" )
+[ -n "${LUXD_STAKING_SIGNER:-}" ]   && ARGS+=( --staking-signer-key-file="$LUXD_STAKING_SIGNER" )
+# Custom genesis (mounted ConfigMap) overrides the binary's embedded genesis.
+[ -n "${LUXD_GENESIS_FILE:-}" ] && [ -f "$LUXD_GENESIS_FILE" ] && ARGS+=( --genesis-file="$LUXD_GENESIS_FILE" )
+# Single-node LOCAL dev only (never production) — opt-in, not the default.
+if [ "${LUXD_DEV:-}" = "true" ]; then
+  ARGS+=( --sybil-protection-enabled=false --network-health-min-conn-peers=0 --skip-bootstrap=true )
+fi
+[ -n "${LUXD_EXTRA_ARGS:-}" ] && ARGS+=( ${LUXD_EXTRA_ARGS} )
+
+echo "[luxd-boot] network=$NETWORK id=$NETID data=$DATA_DIR staking-keys=$([ -f "$DATA_DIR/staking/staker.key" ] && echo present || echo none) restore-on-boot=$REPLICATE_RESTORE_ON_BOOT s3-path=${REPLICATE_S3_PATH} plugins=$(ls "$PLUGIN_DIR" 2>/dev/null | wc -l)"
+exec "$LUXD" "${ARGS[@]}"
@@ -0,0 +1,17 @@
+# Coherent boot — zoo mainnet. Identical luxd-boot.sh as luxfi/universe; only the
+# id / EVM chainId / S3 path differ. Primary-network id=1, Zoo EVM chainId=200200.
+LUX_NETWORK=mainnet
+LUXD_NETWORK_ID=1
+LUXD_DATA_DIR=/data
+LUXD_HTTP_HOST=0.0.0.0
+LUXD_HTTP_PORT=9630
+LUXD_STAKING_PORT=9631
+LUXD_BIN=/luxd/build/luxd
+# Native ZAP replication → zoo-system MinIO (PQ-encrypted, restore-on-boot).
+REPLICATE_S3_ENDPOINT=http://s3.zoo-system.svc.cluster.local:9000
+REPLICATE_S3_BUCKET=zoo-snapshots
+REPLICATE_S3_REGION=us-central1
+REPLICATE_S3_USE_SSL=false
+REPLICATE_S3_PATH=zoo/mainnet/zaprepl
+# REPLICATE_S3_ACCESS_KEY / SECRET_KEY + REPLICATE_AGE_IDENTITY come from the
+# replicate-s3-creds / replicate-age secrets (EnvFrom) — see snapshot-native/.
@@ -0,0 +1,17 @@
+# Coherent boot — zoo testnet. Identical luxd-boot.sh as luxfi/universe; only the
+# id / EVM chainId / S3 path differ. Primary-network id=2, Zoo EVM chainId=200201.
+LUX_NETWORK=testnet
+LUXD_NETWORK_ID=2
+LUXD_DATA_DIR=/data
+LUXD_HTTP_HOST=0.0.0.0
+LUXD_HTTP_PORT=9630
+LUXD_STAKING_PORT=9631
+LUXD_BIN=/luxd/build/luxd
+# Native ZAP replication → zoo-system MinIO (PQ-encrypted, restore-on-boot).
+REPLICATE_S3_ENDPOINT=http://s3.zoo-system.svc.cluster.local:9000
+REPLICATE_S3_BUCKET=zoo-snapshots
+REPLICATE_S3_REGION=us-central1
+REPLICATE_S3_USE_SSL=false
+REPLICATE_S3_PATH=zoo/testnet/zaprepl
+# REPLICATE_S3_ACCESS_KEY / SECRET_KEY + REPLICATE_AGE_IDENTITY come from the
+# replicate-s3-creds / replicate-age secrets (EnvFrom) — see snapshot-native/.
@@ -0,0 +1,14 @@
+# Secrets for native ZAP replication — zoo devnet. DO NOT commit real values; fill
+# from KMS (lux.id / Infisical). PQ identity is restore-only (ML-KEM-768+X25519).
+---
+apiVersion: v1
+kind: Secret
+metadata: { name: replicate-s3-creds, namespace: zoo-devnet, labels: { app.kubernetes.io/part-of: zoo-network, zoo.network/network: devnet } }
+type: Opaque
+stringData: { REPLICATE_S3_ACCESS_KEY: "REPLACE_FROM_KMS", REPLICATE_S3_SECRET_KEY: "REPLACE_FROM_KMS" }
+---
+apiVersion: v1
+kind: Secret
+metadata: { name: replicate-age, namespace: zoo-devnet, labels: { app.kubernetes.io/part-of: zoo-network, zoo.network/network: devnet } }
+type: Opaque
+stringData: { REPLICATE_AGE_IDENTITY: "REPLACE_FROM_KMS" }   # PQ hybrid, restore-only
@@ -0,0 +1,21 @@
+# Enable native ZAP replication on the zoo devnet ZooNetwork CR (merge-patch).
+#   kubectl --context zoo-k8s -n zoo-devnet patch zoonetwork zoo-chain --type=merge --patch-file=20-zoonetwork-snapshot-native.patch.yaml
+# The ZooNetwork CRD allows unknown fields (x-kubernetes-preserve-unknown-fields),
+# so spec.replication is accepted now; the zooai operator expands it into the
+# native REPLICATE_* env (parity with luxfi/operator). REQUIRES a build-zoo image
+# built against database >= v1.20.3. EVM chainId 200202, primary-network id 3.
+spec:
+  snapshotSchedule: { includeRlpExports: false }   # retires luxd-rlp-import.yaml
+  replication:
+    enabled: true
+    s3Endpoint: http://s3.zoo-system.svc.cluster.local:9000
+    s3UseSsl: false
+    s3Bucket: zoo-snapshots
+    s3Region: us-central1
+    s3Path: zoo/devnet/zaprepl
+    credentialsSecret: replicate-s3-creds
+    ageRecipient: "age1pq1REPLACE_WITH_PQ_PUBLIC_RECIPIENT"
+    ageIdentitySecret: replicate-age
+    sourceNodeIndex: 0
+    snapshotIntervalSeconds: 3600
+    incrementalIntervalSeconds: 5
@@ -0,0 +1,5 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: zoo-devnet
+resources:
+- 10-replicate-secret.yaml
@@ -1,5 +1,16 @@
 # Zoo mainnet EVM L2 — luxd RLP import spec.
 #
+# ─────────────────────────────────────────────────────────────────────────────
+# RETIRED 2026-06-16 — superseded by snapshot-native/ (native ZAP replication).
+# Boot no longer runs admin_importChain. The historic RLP below was imported
+# ONCE; that state now lives in the S3 snapshot stream (zoo/mainnet/zaprepl) and
+# is restored on boot by the luxd binary (database >= v1.20.3, restore-on-boot).
+# This file is retained ONLY as the authoritative provenance record of the
+# one-time chain-200200 seed (source commit, sha256, tip block). Do not re-enable
+# the import path — snapshotSchedule.includeRlpExports is false.
+# See: ../zoo-mainnet/snapshot-native/ and k8s/boot/luxd-boot.sh.
+# ─────────────────────────────────────────────────────────────────────────────
+#
 # Source of truth for the one-time historic-state restore on the Zoo L2 EVM
 # (chainId 200200) that runs as a tenant on Lux Network mainnet validators.
 #
@@ -0,0 +1,14 @@
+# Secrets for native ZAP replication — zoo mainnet. DO NOT commit real values; fill
+# from KMS (lux.id / Infisical). PQ identity is restore-only (ML-KEM-768+X25519).
+---
+apiVersion: v1
+kind: Secret
+metadata: { name: replicate-s3-creds, namespace: zoo-mainnet, labels: { app.kubernetes.io/part-of: zoo-network, zoo.network/network: mainnet } }
+type: Opaque
+stringData: { REPLICATE_S3_ACCESS_KEY: "REPLACE_FROM_KMS", REPLICATE_S3_SECRET_KEY: "REPLACE_FROM_KMS" }
+---
+apiVersion: v1
+kind: Secret
+metadata: { name: replicate-age, namespace: zoo-mainnet, labels: { app.kubernetes.io/part-of: zoo-network, zoo.network/network: mainnet } }
+type: Opaque
+stringData: { REPLICATE_AGE_IDENTITY: "REPLACE_FROM_KMS" }   # PQ hybrid, restore-only
@@ -0,0 +1,21 @@
+# Enable native ZAP replication on the zoo mainnet ZooNetwork CR (merge-patch).
+#   kubectl --context zoo-k8s -n zoo-mainnet patch zoonetwork zoo-chain --type=merge --patch-file=20-zoonetwork-snapshot-native.patch.yaml
+# The ZooNetwork CRD allows unknown fields (x-kubernetes-preserve-unknown-fields),
+# so spec.replication is accepted now; the zooai operator expands it into the
+# native REPLICATE_* env (parity with luxfi/operator). REQUIRES a build-zoo image
+# built against database >= v1.20.3. EVM chainId 200200, primary-network id 1.
+spec:
+  snapshotSchedule: { includeRlpExports: false }   # retires luxd-rlp-import.yaml
+  replication:
+    enabled: true
+    s3Endpoint: http://s3.zoo-system.svc.cluster.local:9000
+    s3UseSsl: false
+    s3Bucket: zoo-snapshots
+    s3Region: us-central1
+    s3Path: zoo/mainnet/zaprepl
+    credentialsSecret: replicate-s3-creds
+    ageRecipient: "age1pq1REPLACE_WITH_PQ_PUBLIC_RECIPIENT"
+    ageIdentitySecret: replicate-age
+    sourceNodeIndex: 0
+    snapshotIntervalSeconds: 3600
+    incrementalIntervalSeconds: 5
@@ -0,0 +1,5 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: zoo-mainnet
+resources:
+- 10-replicate-secret.yaml
@@ -0,0 +1,14 @@
+# Secrets for native ZAP replication — zoo testnet. DO NOT commit real values; fill
+# from KMS (lux.id / Infisical). PQ identity is restore-only (ML-KEM-768+X25519).
+---
+apiVersion: v1
+kind: Secret
+metadata: { name: replicate-s3-creds, namespace: zoo-testnet, labels: { app.kubernetes.io/part-of: zoo-network, zoo.network/network: testnet } }
+type: Opaque
+stringData: { REPLICATE_S3_ACCESS_KEY: "REPLACE_FROM_KMS", REPLICATE_S3_SECRET_KEY: "REPLACE_FROM_KMS" }
+---
+apiVersion: v1
+kind: Secret
+metadata: { name: replicate-age, namespace: zoo-testnet, labels: { app.kubernetes.io/part-of: zoo-network, zoo.network/network: testnet } }
+type: Opaque
+stringData: { REPLICATE_AGE_IDENTITY: "REPLACE_FROM_KMS" }   # PQ hybrid, restore-only
@@ -0,0 +1,21 @@
+# Enable native ZAP replication on the zoo testnet ZooNetwork CR (merge-patch).
+#   kubectl --context zoo-k8s -n zoo-testnet patch zoonetwork zoo-chain --type=merge --patch-file=20-zoonetwork-snapshot-native.patch.yaml
+# The ZooNetwork CRD allows unknown fields (x-kubernetes-preserve-unknown-fields),
+# so spec.replication is accepted now; the zooai operator expands it into the
+# native REPLICATE_* env (parity with luxfi/operator). REQUIRES a build-zoo image
+# built against database >= v1.20.3. EVM chainId 200201, primary-network id 2.
+spec:
+  snapshotSchedule: { includeRlpExports: false }   # retires luxd-rlp-import.yaml
+  replication:
+    enabled: true
+    s3Endpoint: http://s3.zoo-system.svc.cluster.local:9000
+    s3UseSsl: false
+    s3Bucket: zoo-snapshots
+    s3Region: us-central1
+    s3Path: zoo/testnet/zaprepl
+    credentialsSecret: replicate-s3-creds
+    ageRecipient: "age1pq1REPLACE_WITH_PQ_PUBLIC_RECIPIENT"
+    ageIdentitySecret: replicate-age
+    sourceNodeIndex: 0
+    snapshotIntervalSeconds: 3600
+    incrementalIntervalSeconds: 5
@@ -0,0 +1,5 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: zoo-testnet
+resources:
+- 10-replicate-secret.yaml