universe/LLM.md

# LLM.md — Zoo Universe

Canonical guide for AI tooling working in `~/work/zoo/universe`.

## What this repo is

The Zoo brand's declaration of its sovereign-L1 surface. One source of
truth — `chain.yaml` — drives every operational verb. K8s manifests
under `k8s/` deploy the brand services to `do-sfo3-zoo-k8s`. A 5-line
Makefile is the optional convention; the underlying interface is the
`lux` CLI.

## Sovereign L1 model (2026-06-16)

Zoo runs its own primary network with its own validator set. C-chain is
Lux Network's primary EVM — Zoo has Zoo EVM. Per-env primary networkID
equals per-env EVM chainID (sovereign-L1 rule).

| Env      | networkID | httpPort | RPC                                |
|----------|-----------|----------|------------------------------------|
| mainnet  | 200200    | 9700     | https://api.zoo.network            |
| testnet  | 200201    | 9710     | https://api.zoo-test.network       |
| devnet   | 200202    | 9720     | https://api.zoo-dev.network        |
| localnet | 200203    | 9730     | http://127.0.0.1:9730/v1/chain/c/rpc |

Ports are explicit (`httpPort` + `stakingPort` in chain.yaml). No
port+1 implicit rule. No collisions across envs.

## CLI grammar — `lux <verb> <brand>/<env>`

The `lux` CLI reads `chain.yaml` files discovered under `$LUX_BRAND_PATH`
(default `~/work/{lux,zoo,hanzo,pars,osage,adnexus}/universe`) and
resolves each `<brand>/<env>` to a `RuntimeProfile` (networkID, httpPort,
stakingPort, dataDir, logDir, genesisFile, snapshotDir, snapshotName,
serviceLabel). Every verb consumes the same record.

```
lux ctx                       # list discoverable brand/env tuples
lux ps                        # for each tuple: probe port → up/down/foreign
lux up    zoo/devnet          # boot luxd in K=1 PoA mode (foreground)
lux down  zoo/devnet          # identify by port probe + networkID match, SIGTERM
lux snap  zoo/devnet          # tar.zst the data-dir into snapshotDir
lux cycle zoo/devnet          # boot → deploy → stop → snap atomically
```

Identity for `down` is **port + networkID handshake**, never a PID
file. If port 9720 responds with networkID 200202, that's our node.
If it responds with a different networkID, we refuse to signal — better
to leave a stranger alone than kill the wrong process.

## Birth path (against upstream Lux primary)

One-time per env:
1. `CreateChainTx` on Lux P-chain (networkID 1/2/3/1337) — registers Zoo
   EVM blockchain with VMID + genesis blob.
2. `AddValidatorTx` on Lux P-chain — seeds initial validator set.
3. `ConvertNetworkToL1Tx` on Lux P-chain — flips Zoo to sovereign.

Post-conversion: Zoo's primary network runs fully sovereign on its
networkID with its own validator set managed by an on-chain Solidity
PoS staking contract. Validators stop talking to upstream Lux at runtime.

## Makefile (optional convention)

5 lines, no per-env data. Reads chain.yaml indirectly via `lux`.

```make
make up    env=devnet     # ≡ lux up    zoo/devnet
make cycle env=mainnet    # ≡ lux cycle zoo/mainnet
make cycle-all            # cycle every env in sequence
```

Per-env constants (networkID, ports, genesis path) live in chain.yaml.
No `NID_MAIN := 200200` duplication anywhere.

## Per-brand DOKS cluster

Every brand owns its own DOKS cluster — no shared k8s across brands.

| Brand     | Cluster                                   | Validator clusters                                       |
|-----------|-------------------------------------------|----------------------------------------------------------|
| lux       | do-sfo3-lux-k8s                           | + do-sfo3-lux-test-k8s, -dev-k8s, -bootnode-k8s          |
| hanzo     | do-sfo3-hanzo-k8s                         | + do-sfo3-hanzo-dev-k8s                                  |
| **zoo**   | **do-sfo3-zoo-k8s**                       | **single cluster, 4 envs as namespaces**                 |
| pars      | do-sfo3-pars-k8s                          | single                                                   |
| osage     | do-sfo3-osage-k8s                         | single                                                   |
| adnexus   | do-sfo3-adnexus-k8s                       | single                                                   |
| liquidity | gke_liquidity-{main,test,dev}_us-central1 | GKE per env                                              |

Zoo's k8s namespaces match `chain.yaml#networks.<env>.namespace`. The
Makefile k8s targets (`make k8s-apply env=mainnet`) wrap
`kubectl --context do-sfo3-zoo-k8s apply -k k8s/zoo-<env>/`.

## Private universe

`~/work/zoo-private/universe/` mirrors this repo with operational
secrets that cannot live in the public universe:

```
zoo-private/universe/
├── validator-keys/     # KMS pointers for staking signers
├── secrets/            # SOPS-encrypted env blobs
├── bootstrap/          # pre-baked genesis with private allocations
├── snapshots/          # cold backups (RLP + zapdb tarballs)
├── runbooks/           # incident response (private)
└── backup/             # KMS dumps, recovery procedures
```

Nothing in the public `~/work/zoo/universe/` references KMS paths,
NodeIDs, or proprietary code. Container images are in `ghcr.io/zooai/*`
only.

## chain.yaml — the contract

Schema (typed in `pkg/brand/brand.go`, parsed once, drives everything):

```yaml
version: "1"
network:
  slug: zoo
  type: l1
  parent: null
networks:
  <env>:
    networkID: <uint32>          # sovereign-L1: == primaryEvmChainID
    primaryEvmChainID: <uint64>
    httpPort: <int>              # explicit
    stakingPort: <int>           # explicit, no port+1 rule
    genesisFile: <path>          # relative to chain.yaml dir
    rpcUrl: <url>                # remote
    cluster: <kubectl-context>
    namespace: <k8s-ns>
    imageTag: vX.Y.Z
runtime:
  dataDirTemplate: "~/.lux/{brand}-{env}"
  snapshotDir: "~/work/lux/snapshots"
  snapshotNameTemplate: "{networkID}-{brand}-{env}-with-contracts"
  serviceLabelTemplate: "ai.lux.{brand}.{env}"
  logLevel: info
```

Validation is enforced on load:
- networkID required, non-zero
- httpPort/stakingPort required, non-zero, distinct, no cross-env collisions
- networkID == primaryEvmChainID when both set (sovereign-L1 rule)
- version == "1" (future bumps will need a migrator)

## chain.lock — the runtime invariant

`<dataDir>/chain.lock` is written on first boot:

```json
{
  "version": "1",
  "brand": "zoo",
  "env": "devnet",
  "networkID": 200202,
  "httpPort": 9720,
  "stakingPort": 9721,
  "genesisHash": "0x...",
  "createdAt": "2026-06-16T22:00:00Z"
}
```

Subsequent `lux up` invocations refuse to mount the dir if the
networkID/httpPort/genesisHash don't match. Prevents silent state
corruption from booting the wrong genesis into an existing dir.

## Storage (no Postgres)

Stateful services use Base/SQLite at `/app/data/data.db`. Litestream
sidecar replicates WAL to MinIO (`s3:9000`) at 1s sync.

- `k8s/base/s3.yaml` — MinIO StatefulSet, 20Gi DO block storage
- `k8s/base/replicate.yaml` — Litestream ConfigMap, S3 endpoint
- `k8s/base/bridge.yaml` — example consumer (init restore, sidecar replicate, single writer)

Legacy `sql.yaml` (Postgres) was removed 2026-06-02.

## Container registry

`ghcr.io/zooai/*` only. Banned: `registry.digitalocean.com/hanzo/*`,
`us-docker.pkg.dev/*`, mutable tags (`:latest`, `:main`, `:dev`,
`:edge`, `:nightly`, `:master`). Use semver `vX.Y.Z` or immutable
per-arch branch tags.

## Open deployment issues

1. **zoo-operator `main-amd64`** — CrashLoopBackOff waits for
   `zooai/operator` CI to publish semver. Bump to `vX.Y.Z` then drop
   `main-amd64` across all base/ manifests.
2. **kms-operator + CRDs not installed** in `do-sfo3-zoo-k8s`. Install
   from `~/work/hanzo/kms`, replace manual K8s Secrets with KMSSecret CRs.
3. **Orphan StatefulSets** — `mpc-node` (pre-T-Chain) and `sql`
   (Postgres) lingering. Delete with kubectl.
4. **consul Deployment in zoo-mainnet** — not in any manifest.
5. **`zoo-chain` ZooNetwork CR** — unreconciled while operator is down.
6. **Genesis for zoo-localnet** — `~/work/zoo/genesis/configs/localnet/`
   is empty. Populate to unblock `lux up zoo/localnet`.

## Rules for AI Assistants

1. **Update THIS file** with significant discoveries. Never create new
   summary files.
2. **Never** commit `.env`, `.env.local`, or any plaintext token.
   Use KMS at `kms.hanzo.ai`.
3. **Never** push Zoo workloads to any registry other than `ghcr.io/zooai/*`.
4. **Never** introduce Postgres — SQLite + Litestream is canonical.
5. **All Zoo apps use the shared hanzo-k8s IAM**, but Zoo's issuer is
   **`https://zoolabs.id`**, never `hanzo.id` and never `zoo.id` (which has no
   DNS). `id.zoo.network` is an ALIAS of that one issuer, not a second one.
   Zoo's OAuth clients are declared in `infra/k8s/iam/provision.yaml` and
   converged with `make iam-provision` — that document is the only supported
   way to change them; hand-editing an application in a console is silently
   reverted by the next converge. See the header of that file for the issuer
   decision and the per-app evidence.
6. **Never reference Lux's C-chain as Zoo's** — Zoo has Zoo EVM,
   sovereign. Lux C-chain is Lux's primary EVM and nothing else uses it.
7. **Never use "subnet" / "snowman" / "snowball" language** — Zoo
   consensus is **Quasar**, chains are chains, networks are L1s.
8. **No shell scripts** in this universe. Verbs live in the `lux` CLI;
   the Makefile only delegates.

## Zoo mainnet → real 5-validator set (state-preserving zapdb graft) — 2026-07-14

**STATUS: ✅ CUT OVER 2026-07-14 (organic, owner-approved). api.zoo.ngo now serves
canonical state from the 5-validator `zood-mv` set** (chainId 200200, tip 13,347,
hash 0x891a539c — verified stable). Cutover = flipped `cm zoo-system/zoo-dynamic`
`routes.yaml` service `rpc-zoo` backend `zood-rpc`→`zood-mv` + restarted zoo-ingress.
zood-0 KEPT RUNNING as hot-standby (unrouted, PVC + VolumeSnapshot intact). **ZERO
regression** — byte-identical state, 729 settlements. Block PRODUCTION on the 5-set
confirms on the next organic/bridge tx (Zoo idle ~22h; no safe funded key to force
one — owner chose organic over a treasury self-send). ROLLBACK if ever needed: flip
`rpc-zoo` backend back to `zood-rpc` + restart zoo-ingress (zood-0 still at 13,347).
RETIRE zood-0 (`scale sts/zood → 0`, KEEP PVC) only AFTER a block >13,347 is observed
on zood-mv. Backup routes.yaml: scratchpad/zoo-routes-backup.yaml.

`zood-mv` (STS, ns `zoo-mainnet`) = 5 validators on `ghcr.io/luxfi/node:v1.34.9`
carrying the canonical Zoo C-Chain (chainId 200200) grafted by **direct zapdb
copy** from `zood-0`. NEVER `admin_importChain` the RLP (diverges @ block 800 on
both v1.34.9 & v1.36.2). Direct coreth-db copy is the faithful path — PROVEN.

**Canonical reference (zood-0 == public api.zoo.ngo):**
- tip **13347 = 0x891a539c9a82ef872acf608090818c2f08a856f15624e7f2b10f689615f55cc4**
- 13346 = 0xdc9b647e… · genesis(0) = 0x7c548af4… · C-blockchainID
  `23hNkQEfXaGfVVJjDPEPGgvHNfGW5BXWx47kxe67fn1qgL2fsP` (sovereign; NOT the old
  subnet-on-lux `4HBM17…` in `network-patch.yaml`).
- settlement fingerprints: blk808=0x0b8c0b84… 2000=0x210984d2… 3599=0x5e7f7749…
- (task cited `0x3a173563@13346`; live public+zood-0 both serve `0xdc9b647e@13346`
  → THAT is canonical; `0x3a173563` was stale.)

**Graft recipe (reproducible):** coreth C-db = `/data/chainData/network-200200/
<Cbid>/db` (50M, nested outer `db/`+inner `db/badgerdb/` w/ ssts) = COMPLETE C
state (proposervm+coreth). Shared `/data/db` (48K)=P/X only. CSI VolumeSnapshot
`data-zood-0` (non-disruptive) → restore PVC → `tar -S` coreth db → per mv PVC:
`rm -rf /data/db /data/chainData` + untar coreth db + rm LOCKs + chown 1000.
Fresh 5-val P-Chain rebuilds from `zood-mv-genesis` (5 stakers, same cChainGenesis
+ upgrade.json as zood-0, sha-verified).

**PROVEN (all 5):** block 13347, tip=0x891a539c(==canonical), gen=0x7c548af4,
settlement fingerprints match, P-height 0 / 5 validators, C-bootstrapped, peers=4.
Fault-tolerance: killed mv-2 → 4 stayed converged @13347 (quorum 4/5) → mv-2
rejoined @canonical.

**BLOCKER (cutover gate):** "producing new blocks" unproven — needs a funded tx.
Funded accts: treasury 0x9011 (KMS prod secret — DO NOT pull for a test); bridge
signer 0xe016 / settler 0x202 (Lux-side, live-nonce → don't break bridge); no
faucet deployed. Chain idle ~22h → no organic block in-session.

**RESUME (once liveness authorized):**
1. Prove finalization with a funded op key (owner throwaway, or authorized 0-value
   self-send from 0x9011 signed in-pod from KMS): 1 tx → all 5 advance to 13348
   same hash; kill 1 → 4/5 finalize 13349 → restore → rejoin.
2. Cutover: `cm zoo-system/zoo-dynamic` `routes.yaml` `api-zoo-ngo` backend
   `http://zood-rpc…:9630` → `http://zood-mv.zoo-mainnet.svc.cluster.local:9630`;
   restart `zoo-ingress` (ns zoo-system). Confirm external api.zoo.ngo ≥13348 canonical.
3. Retire zood-0: `scale sts/zood → 0` (KEEP `data-zood-0` PVC + snapshot).
   ALT organic: cutover first, keep zood-0 hot-standby; next bridge round-trip
   finalizes on 5-set (rollback = flip route back + scale zood-0 up).

**BACKUPS (keep until cutover verified):** VolumeSnapshot `zoo-mainnet/zood0-cchain-snap`
(READY, 100Gi, crash-consistent). PVC `data-zood-0` intact. **NEVER regenesis Zoo.**

## Zoo crypto-sovereignty: own KMS + MPC + Safes — 2026-07-14

**STATUS: ✅ zoo-mpc LIVE (DKG + 3-of-5 sign PROVEN) · ✅ zoo-kms LIVE (auth PROVEN) ·
✅ kms/mpc.zoo.network REPOINTED to the zoo cluster (owner-approved 2026-07-14) · ✅ Zoo
DAO + Z Safes DEPLOYED on 200200.** Zoo is fully crypto-sovereign — own MPC ring + own
KMS (Zoo-branded, `do-sfo3-zoo-k8s`) + own governance Safes. Manifests: `k8s/zoo-mpc/`,
`k8s/zoo-kms/`, `manifests/safes.yaml`.

### Secret enumeration (INVESTIGATE — proven, non-destructive)
`kms.zoo.network` → shared **Hanzo** KMS (`luxfi/kms:1.11.9`, ns `hanzo`, svc `kms`).
PROVEN **zero Zoo secrets stored to migrate** → repoint strands nothing:
- native `/v1/kms/orgs/zoo/secrets/*` probes (paths ci/bridge/zoo-bridge, env prod+default)
  → all **404**. Token org-scoping calibrated: **403** on org `hanzo`/`zooai`, **404** on `zoo`
  = authenticated+authorized, genuine not-found (no HTTP list endpoint exists by design).
- Infisical `/api/v3` NOT implemented on 1.11.9 (SPA fallthrough). Zoo CI (workspace
  `zoo-platform`, path `/ci`: GCP_CREDENTIAL, TOKEN_GITHUB_CICD, DIGITALOCEAN_ACCESS_TOKEN)
  reads via `kms.hanzo.ai` and is currently failing at 0s (not populated/consumed).
- No zoo KMSSecret CRs, no zoo canonical CRs, no legacy Infisical/casibase store anywhere.
- IAM machine identities (`zoo-{kms,mpc,app,explorer,foundation,mainnet,web3}-iam-creds`,
  ns hanzo) + `zoo-*` IAM apps exist in the SHARED IAM — unaffected by any repoint.

### zoo-mpc — LIVE, DKG PROVEN (5-node, 3-of-5)
`ghcr.io/zooai/mpc:1.17.9` (wraps `luxfi/mpc:v1.17.9`; built via zooai/mpc `docker.yml`).
All 5 pods "ALL PEERS ARE READY". **node-id=`zoo-mpc-node-{0..4}` and peers MUST use the
`nodeID@host:port` form** — bare host:port yields synthetic `peer-N` ids that never match
`filterSelf()`, so self is double-counted (`expected=N+1`) and `/keygen` (gated on
`ArePeersReady`) stalls forever. DKG: `POST :9800/keygen {org_id:zoo,wallet_id:zoo-treasury-v1}`
→ **EVM `0x8756621734fe274fdc426381c9a4f9dec8656243`** (ecdsa 502a9a8e…), key persisted
cluster-wide, **3-of-5 threshold sign PROVEN** (`status:signed`). Internal API bearer =
`MPC_INTERNAL_API_KEY`. Retain PVCs. Stale per-net `mpc-node` STS (zoo-mainnet/system/
testnet/devnet, all 0/0) are the OLD attempt — LEFT UNTOUCHED (preserve `zoo-mpc-secrets`
w/ prior signing-key); new ring is the dedicated `zoo-mpc` ns the bridge tenant targets.

### zoo-kms — LIVE, auth PROVEN
`kms-0`, ns `zoo-kms`, `ghcr.io/zooai/kms:1.11.8` (wraps `luxfi/kms:1.11.8` = EXACT
lux-kms-go parity). Secrets-only (`ZAP_PORT=0`, `MPC_VAULT_ID=""`). White-labelled:
`KMS_IAM_OWNER=zoo` → OIDC `client_id=zoo-kms owner=zoo` (NEVER the luxfi default lux-kms);
IAM issuer `hanzo.id`. In-cluster `health=200`, `zoo-kms` machine `login=200`. PVC
`do-block-storage-retain`. **Do NOT use `luxfi/kms:1.11.10`** — it mandates a
consensus-authorizer that fail-closes ("consensus validator authority is empty") without
`KMS_CONSENSUS_VALIDATORS/OPERATORS` whenever a master key is set → crash-loop. That
upgrade is owned by the KMS-issuer/consensus workstream; the zoo-mpc ring is ready to back
it (`MPC_VAULT_ID` → `zoo-treasury-v1`).

### Bootstrap (one-time, IMPERATIVE — never commit plaintext; create-if-absent, never regen)
```
kubectl -n zoo-mpc create secret generic mpc-secrets      --from-literal=jwt-secret=$(openssl rand -hex 32)          --from-literal=mpc-password=$(openssl rand -hex 32)
kubectl -n zoo-mpc create secret generic mpc-internal-api  --from-literal=MPC_INTERNAL_API_KEY=$(openssl rand -hex 32) --from-literal=MPC_THRESHOLD_AUTH_TOKEN=$(openssl rand -hex 32)
kubectl -n zoo-mpc apply -k k8s/zoo-mpc   # + copy ghcr-zooai pull secret from zoo-system
kubectl -n zoo-kms create secret generic kms-secrets      --from-literal=master-key-b64=$(openssl rand -base64 32)   --from-literal=encryption-key-b64=$(openssl rand -base64 32)
kubectl -n zoo-kms create secret generic zoo-kms-oidc     --from-literal=client-secret=<zoo-kms IAM clientSecret>    --from-literal=state-secret=$(openssl rand -hex 32)
kubectl -n zoo-kms apply -k k8s/zoo-kms
```
BACK UP `encryption-key-b64` + `mpc-password` (their stores are unreadable without them).

### DAO/Z Safes (`manifests/safes.yaml`) — ✅ DEPLOYED 2026-07-14
Deployed via the PROVEN `~/work/lux/standard/script/deploy_org_safes.sh` (default salts,
owner-approved). KEY CORRECTION: `0x9011` was at LIVE nonce **91** on Zoo (NOT nonce 0), so
the earlier nonce-0 predictions (0x5232/0x3Be3) were UNACHIEVABLE (`0x413e…`factory =
CREATE(0x9011,1), unreachable). The tool adapts to the live nonce → the ACTUAL deployed set:
- DAO Safe **0x229599f227231d8C90fcF1a78589F5DC4b7A6962** (salt zoo.dao.safe.v1) — 1/1 owner 0x9011 ✓
- Z/Team Safe **0xb8BbCdde5E8713f9849D1520D71Bc08Af94E6F00** (salt zoo.team.safe.v1) — 1/1 owner 0x9011 ✓
- Foundation Timelock **0x31C89A0509F7783b742B01Be9103c05BbF4B9510** — minDelay 86400, DAO=proposer ✓
- infra: singleton 0xF034942c / factory 0x2cd30691 / handler 0xcEfF061c / multisend 0x4Dc0A7aC (nonce 91-94)
- **Block-production PROOF**: the 7 deploy txs were the Zoo 5-set's first blocks — tip 13347→13354,
  all 5 zood-mv converged (13348 hash 0xf50dcdf3…, 13353 hash 0xe90cee14…). Record:
  `~/work/lux/standard/deployments/org-safes/zoo-200200.json`.
- Salts are chain-specific (nonce-based); for cross-org-uniform addresses use Arachnid (future).

### Repoint — ✅ DONE + VERIFIED 2026-07-14 (owner-approved)
`kms.zoo.network` + `mpc.zoo.network` (+`mpc-api`) cut over from the Hanzo cluster to the ZOO
cluster. The zoo cluster routes via the **Traefik file provider** (`cm zoo-system/zoo-dynamic`
routes.yaml — NOT k8s Ingress; class-annotation Ingresses are NOT picked up), TLS via Traefik's
`letsencrypt` **DNS-01 Cloudflare** resolver. Cutover done:
  1. added `kms-zoo`/`mpc-zoo` routers+services to `zoo-dynamic` (backend `kms.zoo-kms.svc:80` /
     `mpc-api-svc.zoo-mpc.svc:8081`, passHostHeader for KMS white-labelling);
  2. flipped the 3 Cloudflare A records → zoo LB `209.38.118.103` (grey-cloud; via the
     `cloudflare-dns01-key` Global API Key, X-Auth-Email z@zeekay.io);
  3. removed `kms.zoo.network` from the `hanzo-domains` CR + deleted the generated ingresses +
     `hanzo-mpc/mpc-zoo-ingress`.
VERIFIED public: `kms.zoo.network/v1/kms/health` 200 + zoo-kms login 200 (Zoo cluster's own KMS).
REVERSIBLE: re-add the `hanzo-domains` kms.zoo.network entry + flip DNS back. Zero secrets lost
(proven empty). Backup: `scratchpad/zoo-dynamic-BACKUP.yaml` (this session).
NOTE: Zoo CI (`ci.yml`) reads `kms.hanzo.ai` (default `KMS_URL`), NOT kms.zoo.network — UNAFFECTED
by the repoint; its Infisical `/api/v3` fetch was already non-functional (unimplemented on
luxfi/kms + CI failing at 0s). Repointing CI to kms.zoo.network is a FUTURE step (populate the
Zoo KMS with the `zoo-platform/ci` secrets + switch CI to the native `/v1/kms` API).

### Lux KMS reality (step 6 answer)
`lux-kms-go/kms-0` (v1.11.8) runs **SECRETS-ONLY**: `MPC_VAULT_ID=""` + `ZAP_PORT=0` — it
does **NOT** do MPC-backed signing; `lux-mpc` (5 nodes, healthy) exists but is NOT wired to
the KMS. Zoo is deployed at exact parity (KMS secrets-only + independent, proven MPC ring).

---
*Last updated 2026-07-14 — Zoo crypto-sovereignty COMPLETE: own KMS+MPC LIVE/proven, DAO+Z
Safes DEPLOYED (200200), kms/mpc.zoo.network REPOINTED to zoo cluster. Prior: 5-validator
graft; 2026-06-16 Sovereign L1 + lux CLI.*