universe/LLM.md

LLM.md — Zoo Universe

Canonical guide for AI tooling working in ~/work/zoo/universe.

What this repo is

The Zoo brand's declaration of its sovereign-L1 surface. One source of truth — chain.yaml — drives every operational verb. K8s manifests under k8s/ deploy the brand services to do-sfo3-zoo-k8s. A 5-line Makefile is the optional convention; the underlying interface is the lux CLI.

Sovereign L1 model (2026-06-16)

Zoo runs its own primary network with its own validator set. C-chain is Lux Network's primary EVM — Zoo has Zoo EVM. Per-env primary networkID equals per-env EVM chainID (sovereign-L1 rule).

Env	networkID	httpPort	RPC
mainnet	200200	9700	https://api.zoo.network
testnet	200201	9710	https://api.zoo-test.network
devnet	200202	9720	https://api.zoo-dev.network
localnet	200203	9730	http://127.0.0.1:9730/v1/chain/c/rpc

Ports are explicit (httpPort + stakingPort in chain.yaml). No port+1 implicit rule. No collisions across envs.

CLI grammar — lux <verb> <brand>/<env>

The lux CLI reads chain.yaml files discovered under $LUX_BRAND_PATH (default ~/work/{lux,zoo,hanzo,pars,osage,adnexus}/universe) and resolves each <brand>/<env> to a RuntimeProfile (networkID, httpPort, stakingPort, dataDir, logDir, genesisFile, snapshotDir, snapshotName, serviceLabel). Every verb consumes the same record.

lux ctx                       # list discoverable brand/env tuples
lux ps                        # for each tuple: probe port → up/down/foreign
lux up    zoo/devnet          # boot luxd in K=1 PoA mode (foreground)
lux down  zoo/devnet          # identify by port probe + networkID match, SIGTERM
lux snap  zoo/devnet          # tar.zst the data-dir into snapshotDir
lux cycle zoo/devnet          # boot → deploy → stop → snap atomically

Identity for down is port + networkID handshake, never a PID file. If port 9720 responds with networkID 200202, that's our node. If it responds with a different networkID, we refuse to signal — better to leave a stranger alone than kill the wrong process.

Birth path (against upstream Lux primary)

One-time per env:

1. CreateChainTx on Lux P-chain (networkID 1/2/3/1337) — registers Zoo EVM blockchain with VMID + genesis blob.
2. AddValidatorTx on Lux P-chain — seeds initial validator set.
3. ConvertNetworkToL1Tx on Lux P-chain — flips Zoo to sovereign.

Post-conversion: Zoo's primary network runs fully sovereign on its networkID with its own validator set managed by an on-chain Solidity PoS staking contract. Validators stop talking to upstream Lux at runtime.

Makefile (optional convention)

5 lines, no per-env data. Reads chain.yaml indirectly via lux.

make up    env=devnet     # ≡ lux up    zoo/devnet
make cycle env=mainnet    # ≡ lux cycle zoo/mainnet
make cycle-all            # cycle every env in sequence

Per-env constants (networkID, ports, genesis path) live in chain.yaml. No NID_MAIN := 200200 duplication anywhere.

Per-brand DOKS cluster

Every brand owns its own DOKS cluster — no shared k8s across brands.

Brand	Cluster	Validator clusters
lux	do-sfo3-lux-k8s	+ do-sfo3-lux-test-k8s, -dev-k8s, -bootnode-k8s
hanzo	do-sfo3-hanzo-k8s	+ do-sfo3-hanzo-dev-k8s
zoo	do-sfo3-zoo-k8s	single cluster, 4 envs as namespaces
pars	do-sfo3-pars-k8s	single
osage	do-sfo3-osage-k8s	single
adnexus	do-sfo3-adnexus-k8s	single
liquidity	gke_liquidity-{main,test,dev}_us-central1	GKE per env

Zoo's k8s namespaces match chain.yaml#networks.<env>.namespace. The Makefile k8s targets (make k8s-apply env=mainnet) wrap kubectl --context do-sfo3-zoo-k8s apply -k k8s/zoo-<env>/.

Private universe

~/work/zoo-private/universe/ mirrors this repo with operational secrets that cannot live in the public universe:

zoo-private/universe/
├── validator-keys/     # KMS pointers for staking signers
├── secrets/            # SOPS-encrypted env blobs
├── bootstrap/          # pre-baked genesis with private allocations
├── snapshots/          # cold backups (RLP + zapdb tarballs)
├── runbooks/           # incident response (private)
└── backup/             # KMS dumps, recovery procedures

Nothing in the public ~/work/zoo/universe/ references KMS paths, NodeIDs, or proprietary code. Container images are in ghcr.io/zooai/* only.

chain.yaml — the contract

Schema (typed in pkg/brand/brand.go, parsed once, drives everything):

version: "1"
network:
  slug: zoo
  type: l1
  parent: null
networks:
  <env>:
    networkID: <uint32>          # sovereign-L1: == primaryEvmChainID
    primaryEvmChainID: <uint64>
    httpPort: <int>              # explicit
    stakingPort: <int>           # explicit, no port+1 rule
    genesisFile: <path>          # relative to chain.yaml dir
    rpcUrl: <url>                # remote
    cluster: <kubectl-context>
    namespace: <k8s-ns>
    imageTag: vX.Y.Z
runtime:
  dataDirTemplate: "~/.lux/{brand}-{env}"
  snapshotDir: "~/work/lux/snapshots"
  snapshotNameTemplate: "{networkID}-{brand}-{env}-with-contracts"
  serviceLabelTemplate: "ai.lux.{brand}.{env}"
  logLevel: info

Validation is enforced on load:

• networkID required, non-zero
• httpPort/stakingPort required, non-zero, distinct, no cross-env collisions
• networkID == primaryEvmChainID when both set (sovereign-L1 rule)
• version == "1" (future bumps will need a migrator)

chain.lock — the runtime invariant

<dataDir>/chain.lock is written on first boot:

{
  "version": "1",
  "brand": "zoo",
  "env": "devnet",
  "networkID": 200202,
  "httpPort": 9720,
  "stakingPort": 9721,
  "genesisHash": "0x...",
  "createdAt": "2026-06-16T22:00:00Z"
}

Subsequent lux up invocations refuse to mount the dir if the networkID/httpPort/genesisHash don't match. Prevents silent state corruption from booting the wrong genesis into an existing dir.

Storage (no Postgres)

Stateful services use Base/SQLite at /app/data/data.db. Litestream sidecar replicates WAL to MinIO (s3:9000) at 1s sync.

• k8s/base/s3.yaml — MinIO StatefulSet, 20Gi DO block storage
• k8s/base/replicate.yaml — Litestream ConfigMap, S3 endpoint
• k8s/base/bridge.yaml — example consumer (init restore, sidecar replicate, single writer)

Legacy sql.yaml (Postgres) was removed 2026-06-02.

Container registry

ghcr.io/zooai/* only. Banned: registry.digitalocean.com/hanzo/*, us-docker.pkg.dev/*, mutable tags (:latest, :main, :dev, :edge, :nightly, :master). Use semver vX.Y.Z or immutable per-arch branch tags.

Open deployment issues

1. zoo-operator main-amd64 — CrashLoopBackOff waits for zooai/operator CI to publish semver. Bump to vX.Y.Z then drop main-amd64 across all base/ manifests.
2. kms-operator + CRDs not installed in do-sfo3-zoo-k8s. Install from ~/work/hanzo/kms, replace manual K8s Secrets with KMSSecret CRs.
3. Orphan StatefulSets — mpc-node (pre-T-Chain) and sql (Postgres) lingering. Delete with kubectl.
4. consul Deployment in zoo-mainnet — not in any manifest.
5. zoo-chain ZooNetwork CR — unreconciled while operator is down.
6. Genesis for zoo-localnet — ~/work/zoo/genesis/configs/localnet/ is empty. Populate to unblock lux up zoo/localnet.

Rules for AI Assistants

1. Update THIS file with significant discoveries. Never create new summary files.
2. Never commit .env, .env.local, or any plaintext token. Use KMS at kms.hanzo.ai.
3. Never push Zoo workloads to any registry other than ghcr.io/zooai/*.
4. Never introduce Postgres — SQLite + Litestream is canonical.
5. All Zoo apps use the shared hanzo-k8s IAM, but Zoo's issuer is https://zoolabs.id, never hanzo.id and never zoo.id (which has no DNS). id.zoo.network is an ALIAS of that one issuer, not a second one. Zoo's OAuth clients are declared in infra/k8s/iam/provision.yaml and converged with make iam-provision — that document is the only supported way to change them; hand-editing an application in a console is silently reverted by the next converge. See the header of that file for the issuer decision and the per-app evidence.
6. Never reference Lux's C-chain as Zoo's — Zoo has Zoo EVM, sovereign. Lux C-chain is Lux's primary EVM and nothing else uses it.
7. Never use "subnet" / "snowman" / "snowball" language — Zoo consensus is Quasar, chains are chains, networks are L1s.
8. No shell scripts in this universe. Verbs live in the lux CLI; the Makefile only delegates.

Zoo mainnet → real 5-validator set (state-preserving zapdb graft) — 2026-07-14

STATUS: ✅ CUT OVER 2026-07-14 (organic, owner-approved). api.zoo.ngo now serves canonical state from the 5-validator zood-mv set (chainId 200200, tip 13,347, hash 0x891a539c — verified stable). Cutover = flipped cm zoo-system/zoo-dynamic routes.yaml service rpc-zoo backend zood-rpc→zood-mv + restarted zoo-ingress. zood-0 KEPT RUNNING as hot-standby (unrouted, PVC + VolumeSnapshot intact). ZERO regression — byte-identical state, 729 settlements. Block PRODUCTION on the 5-set confirms on the next organic/bridge tx (Zoo idle ~22h; no safe funded key to force one — owner chose organic over a treasury self-send). ROLLBACK if ever needed: flip rpc-zoo backend back to zood-rpc + restart zoo-ingress (zood-0 still at 13,347). RETIRE zood-0 (scale sts/zood → 0, KEEP PVC) only AFTER a block >13,347 is observed on zood-mv. Backup routes.yaml: scratchpad/zoo-routes-backup.yaml.

zood-mv (STS, ns zoo-mainnet) = 5 validators on ghcr.io/luxfi/node:v1.34.9 carrying the canonical Zoo C-Chain (chainId 200200) grafted by direct zapdb copy from zood-0. NEVER admin_importChain the RLP (diverges @ block 800 on both v1.34.9 & v1.36.2). Direct coreth-db copy is the faithful path — PROVEN.

Canonical reference (zood-0 == public api.zoo.ngo):

• tip 13347 = 0x891a539c9a82ef872acf608090818c2f08a856f15624e7f2b10f689615f55cc4
• 13346 = 0xdc9b647e… · genesis(0) = 0x7c548af4… · C-blockchainID 23hNkQEfXaGfVVJjDPEPGgvHNfGW5BXWx47kxe67fn1qgL2fsP (sovereign; NOT the old subnet-on-lux 4HBM17… in network-patch.yaml).
• settlement fingerprints: blk808=0x0b8c0b84… 2000=0x210984d2… 3599=0x5e7f7749…
• (task cited 0x3a173563@13346; live public+zood-0 both serve 0xdc9b647e@13346 → THAT is canonical; 0x3a173563 was stale.)

Graft recipe (reproducible): coreth C-db = /data/chainData/network-200200/ <Cbid>/db (50M, nested outer db/+inner db/badgerdb/ w/ ssts) = COMPLETE C state (proposervm+coreth). Shared /data/db (48K)=P/X only. CSI VolumeSnapshot data-zood-0 (non-disruptive) → restore PVC → tar -S coreth db → per mv PVC: rm -rf /data/db /data/chainData + untar coreth db + rm LOCKs + chown 1000. Fresh 5-val P-Chain rebuilds from zood-mv-genesis (5 stakers, same cChainGenesis

• upgrade.json as zood-0, sha-verified).

PROVEN (all 5): block 13347, tip=0x891a539c(==canonical), gen=0x7c548af4, settlement fingerprints match, P-height 0 / 5 validators, C-bootstrapped, peers=4. Fault-tolerance: killed mv-2 → 4 stayed converged @13347 (quorum 4/5) → mv-2 rejoined @canonical.

BLOCKER (cutover gate): "producing new blocks" unproven — needs a funded tx. Funded accts: treasury 0x9011 (KMS prod secret — DO NOT pull for a test); bridge signer 0xe016 / settler 0x202 (Lux-side, live-nonce → don't break bridge); no faucet deployed. Chain idle ~22h → no organic block in-session.

RESUME (once liveness authorized):

1. Prove finalization with a funded op key (owner throwaway, or authorized 0-value self-send from 0x9011 signed in-pod from KMS): 1 tx → all 5 advance to 13348 same hash; kill 1 → 4/5 finalize 13349 → restore → rejoin.
2. Cutover: cm zoo-system/zoo-dynamic routes.yaml api-zoo-ngo backend http://zood-rpc…:9630 → http://zood-mv.zoo-mainnet.svc.cluster.local:9630; restart zoo-ingress (ns zoo-system). Confirm external api.zoo.ngo ≥13348 canonical.
3. Retire zood-0: scale sts/zood → 0 (KEEP data-zood-0 PVC + snapshot). ALT organic: cutover first, keep zood-0 hot-standby; next bridge round-trip finalizes on 5-set (rollback = flip route back + scale zood-0 up).

BACKUPS (keep until cutover verified): VolumeSnapshot zoo-mainnet/zood0-cchain-snap (READY, 100Gi, crash-consistent). PVC data-zood-0 intact. NEVER regenesis Zoo.

Zoo crypto-sovereignty: own KMS + MPC + Safes — 2026-07-14

STATUS: ✅ zoo-mpc LIVE (DKG + 3-of-5 sign PROVEN) · ✅ zoo-kms LIVE (auth PROVEN) · ✅ kms/mpc.zoo.network REPOINTED to the zoo cluster (owner-approved 2026-07-14) · ✅ Zoo DAO + Z Safes DEPLOYED on 200200. Zoo is fully crypto-sovereign — own MPC ring + own KMS (Zoo-branded, do-sfo3-zoo-k8s) + own governance Safes. Manifests: k8s/zoo-mpc/, k8s/zoo-kms/, manifests/safes.yaml.

Secret enumeration (INVESTIGATE — proven, non-destructive)

kms.zoo.network → shared Hanzo KMS (luxfi/kms:1.11.9, ns hanzo, svc kms). PROVEN zero Zoo secrets stored to migrate → repoint strands nothing:

• native /v1/kms/orgs/zoo/secrets/* probes (paths ci/bridge/zoo-bridge, env prod+default) → all 404. Token org-scoping calibrated: 403 on org hanzo/zooai, 404 on zoo = authenticated+authorized, genuine not-found (no HTTP list endpoint exists by design).
• Infisical /api/v3 NOT implemented on 1.11.9 (SPA fallthrough). Zoo CI (workspace zoo-platform, path /ci: GCP_CREDENTIAL, TOKEN_GITHUB_CICD, DIGITALOCEAN_ACCESS_TOKEN) reads via kms.hanzo.ai and is currently failing at 0s (not populated/consumed).
• No zoo KMSSecret CRs, no zoo canonical CRs, no legacy Infisical/casibase store anywhere.
• IAM machine identities (zoo-{kms,mpc,app,explorer,foundation,mainnet,web3}-iam-creds, ns hanzo) + zoo-* IAM apps exist in the SHARED IAM — unaffected by any repoint.

zoo-mpc — LIVE, DKG PROVEN (5-node, 3-of-5)

ghcr.io/zooai/mpc:1.17.9 (wraps luxfi/mpc:v1.17.9; built via zooai/mpc docker.yml). All 5 pods "ALL PEERS ARE READY". node-id=zoo-mpc-node-{0..4} and peers MUST use the nodeID@host:port form — bare host:port yields synthetic peer-N ids that never match filterSelf(), so self is double-counted (expected=N+1) and /keygen (gated on ArePeersReady) stalls forever. DKG: POST :9800/keygen {org_id:zoo,wallet_id:zoo-treasury-v1} → EVM 0x8756621734fe274fdc426381c9a4f9dec8656243 (ecdsa 502a9a8e…), key persisted cluster-wide, 3-of-5 threshold sign PROVEN (status:signed). Internal API bearer = MPC_INTERNAL_API_KEY. Retain PVCs. Stale per-net mpc-node STS (zoo-mainnet/system/ testnet/devnet, all 0/0) are the OLD attempt — LEFT UNTOUCHED (preserve zoo-mpc-secrets w/ prior signing-key); new ring is the dedicated zoo-mpc ns the bridge tenant targets.

zoo-kms — LIVE, auth PROVEN

kms-0, ns zoo-kms, ghcr.io/zooai/kms:1.11.8 (wraps luxfi/kms:1.11.8 = EXACT lux-kms-go parity). Secrets-only (ZAP_PORT=0, MPC_VAULT_ID=""). White-labelled: KMS_IAM_OWNER=zoo → OIDC client_id=zoo-kms owner=zoo (NEVER the luxfi default lux-kms); IAM issuer hanzo.id. In-cluster health=200, zoo-kms machine login=200. PVC do-block-storage-retain. Do NOT use luxfi/kms:1.11.10 — it mandates a consensus-authorizer that fail-closes ("consensus validator authority is empty") without KMS_CONSENSUS_VALIDATORS/OPERATORS whenever a master key is set → crash-loop. That upgrade is owned by the KMS-issuer/consensus workstream; the zoo-mpc ring is ready to back it (MPC_VAULT_ID → zoo-treasury-v1).

Bootstrap (one-time, IMPERATIVE — never commit plaintext; create-if-absent, never regen)

kubectl -n zoo-mpc create secret generic mpc-secrets      --from-literal=jwt-secret=$(openssl rand -hex 32)          --from-literal=mpc-password=$(openssl rand -hex 32)
kubectl -n zoo-mpc create secret generic mpc-internal-api  --from-literal=MPC_INTERNAL_API_KEY=$(openssl rand -hex 32) --from-literal=MPC_THRESHOLD_AUTH_TOKEN=$(openssl rand -hex 32)
kubectl -n zoo-mpc apply -k k8s/zoo-mpc   # + copy ghcr-zooai pull secret from zoo-system
kubectl -n zoo-kms create secret generic kms-secrets      --from-literal=master-key-b64=$(openssl rand -base64 32)   --from-literal=encryption-key-b64=$(openssl rand -base64 32)
kubectl -n zoo-kms create secret generic zoo-kms-oidc     --from-literal=client-secret=<zoo-kms IAM clientSecret>    --from-literal=state-secret=$(openssl rand -hex 32)
kubectl -n zoo-kms apply -k k8s/zoo-kms

BACK UP encryption-key-b64 + mpc-password (their stores are unreadable without them).

DAO/Z Safes (manifests/safes.yaml) — ✅ DEPLOYED 2026-07-14

Deployed via the PROVEN ~/work/lux/standard/script/deploy_org_safes.sh (default salts, owner-approved). KEY CORRECTION: 0x9011 was at LIVE nonce 91 on Zoo (NOT nonce 0), so the earlier nonce-0 predictions (0x5232/0x3Be3) were UNACHIEVABLE (0x413e…factory = CREATE(0x9011,1), unreachable). The tool adapts to the live nonce → the ACTUAL deployed set:

• DAO Safe 0x229599f227231d8C90fcF1a78589F5DC4b7A6962 (salt zoo.dao.safe.v1) — 1/1 owner 0x9011 ✓
• Z/Team Safe 0xb8BbCdde5E8713f9849D1520D71Bc08Af94E6F00 (salt zoo.team.safe.v1) — 1/1 owner 0x9011 ✓
• Foundation Timelock 0x31C89A0509F7783b742B01Be9103c05BbF4B9510 — minDelay 86400, DAO=proposer ✓
• infra: singleton 0xF034942c / factory 0x2cd30691 / handler 0xcEfF061c / multisend 0x4Dc0A7aC (nonce 91-94)
• Block-production PROOF: the 7 deploy txs were the Zoo 5-set's first blocks — tip 13347→13354, all 5 zood-mv converged (13348 hash 0xf50dcdf3…, 13353 hash 0xe90cee14…). Record: ~/work/lux/standard/deployments/org-safes/zoo-200200.json.
• Salts are chain-specific (nonce-based); for cross-org-uniform addresses use Arachnid (future).

Repoint — ✅ DONE + VERIFIED 2026-07-14 (owner-approved)

kms.zoo.network + mpc.zoo.network (+mpc-api) cut over from the Hanzo cluster to the ZOO cluster. The zoo cluster routes via the Traefik file provider (cm zoo-system/zoo-dynamic routes.yaml — NOT k8s Ingress; class-annotation Ingresses are NOT picked up), TLS via Traefik's letsencrypt DNS-01 Cloudflare resolver. Cutover done:

1. added kms-zoo/mpc-zoo routers+services to zoo-dynamic (backend kms.zoo-kms.svc:80 / mpc-api-svc.zoo-mpc.svc:8081, passHostHeader for KMS white-labelling);
2. flipped the 3 Cloudflare A records → zoo LB 209.38.118.103 (grey-cloud; via the cloudflare-dns01-key Global API Key, X-Auth-Email [email protected]);
3. removed kms.zoo.network from the hanzo-domains CR + deleted the generated ingresses + hanzo-mpc/mpc-zoo-ingress. VERIFIED public: kms.zoo.network/v1/kms/health 200 + zoo-kms login 200 (Zoo cluster's own KMS). REVERSIBLE: re-add the hanzo-domains kms.zoo.network entry + flip DNS back. Zero secrets lost (proven empty). Backup: scratchpad/zoo-dynamic-BACKUP.yaml (this session). NOTE: Zoo CI (ci.yml) reads kms.hanzo.ai (default KMS_URL), NOT kms.zoo.network — UNAFFECTED by the repoint; its Infisical /api/v3 fetch was already non-functional (unimplemented on luxfi/kms + CI failing at 0s). Repointing CI to kms.zoo.network is a FUTURE step (populate the Zoo KMS with the zoo-platform/ci secrets + switch CI to the native /v1/kms API).

Lux KMS reality (step 6 answer)

lux-kms-go/kms-0 (v1.11.8) runs SECRETS-ONLY: MPC_VAULT_ID="" + ZAP_PORT=0 — it does NOT do MPC-backed signing; lux-mpc (5 nodes, healthy) exists but is NOT wired to the KMS. Zoo is deployed at exact parity (KMS secrets-only + independent, proven MPC ring).

---

Last updated 2026-07-14 — Zoo crypto-sovereignty COMPLETE: own KMS+MPC LIVE/proven, DAO+Z Safes DEPLOYED (200200), kms/mpc.zoo.network REPOINTED to zoo cluster. Prior: 5-validator graft; 2026-06-16 Sovereign L1 + lux CLI.