k8s: collapse zoo-operator to zoo-system, migrate to CR-driven bring-up

Operator topology fix: - base/operator.yaml: pin ghcr.io/zooai/operator:v0.1.1 (was :main floating). Add OPERATOR_NAMESPACE env via fieldRef: metadata.namespace so the leader-election Lease lock is namespace-scoped, not racing in a single hardcoded 'zoo-system' literal. imagePullPolicy: Always → IfNotPresent (semver tag is immutable). - zoo-system/patches.yaml: zoo-operator replicas → 2 (HA, leader-election picks one). Drop the stale zoo-explorer raw-Deployment patch (operator emits it now via ZooExplorer CR). - zoo-{mainnet,testnet,devnet}/network-patch.yaml: scale per-env zoo-operator → 0. Single zoo-system operator watches every namespace (--namespace '' default); per-env operators were thrashing the same Lease and never becoming leader. CR migration (away from raw manifests): - base/network.yaml: full rewrite as ZooNetwork + ZooChain + ZooExplorer + ZooGateway base CRs (CRDs already installed: zoonetworks / zoochains / zooexplorers / zoogateways). - base/explorer.yaml: emptied — ZooExplorer CR replaces the raw Deployment; operator renders zoo-explorer-explorer Deployment + Svc. - base/gateway.yaml: Service-only — ZooGateway moved to network.yaml. - base/ingress.yaml: kind: Ingress → 4 kind: IngressRoute (hanzo.ai/v1alpha1). Traefik proxy Deployment kept (it IS the hanzoai/ingress L7 proxy on this cluster). - zoo-{mainnet,testnet}/network-patch.yaml: env-specific patches — validators (mainnet=5, testnet=3), bootstrap IPs+IDs from live Lux mainnet+testnet svcs, TRACK_CHAINS = Zoo blockchainID per env. - zoo-{testnet,devnet}/ingressroute.yaml: new — api.test.zoo.network, explore.test.zoo.network, api-explore.test.zoo.network (and dev.* variants). - zoo-{system,testnet,devnet}/kustomization.yaml: include new ingressroute.yaml resource + strip mainnet-only IngressRoutes via $patch: delete pattern. - zoo-mainnet/zoo-industries.yaml: kind: Ingress → IngressRoute. Blockers surfaced (separate followups): 1. Operator does NOT translate `spec.config` env keys to lqd CLI flags. Rendered StatefulSet sets ZOO_BOOTSTRAP_IPS etc., but luxd reads LUXD_*-prefixed env. Operator controller.rs needs to render as CLI args. BLOCKS validators joining Lux primary. 2. hanzo.ai/v1alpha1 IngressRoute CRDs not registered on do-sfo3-zoo-k8s — current ingress is traefik:v3.3 with traefik.io/v1alpha1 native. Either install hanzoai/ingress CRDs OR migrate IngressRoutes to traefik.io group. 3. ZooNetwork operator doesn't emit per-pod LoadBalancer Services needed for validators to be reachable as external bootstrap peers. 4. Validator staking keys: operator only takes secret_prefix; doesn't derive from a KMS mnemonic. Need 5 Secrets per env OR operator extension.
2026-06-10 09:13:57 -07:00
parent a0d270a651
commit 320f061bda
15 changed files with 573 additions and 523 deletions
@@ -1,73 +1,6 @@
-# Explorer — Lux explorer (single Go binary) for Zoo EVM chain.
-# Indexer + GraphQL + SPA in one process, no Postgres, on :8090.
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: zoo-explorer
-  labels:
-    app: zoo-explorer
-    app.kubernetes.io/name: zoo-explorer
-    app.kubernetes.io/part-of: zoo
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app: zoo-explorer
-  template:
-    metadata:
-      labels:
-        app: zoo-explorer
-    spec:
-      imagePullSecrets:
-        - name: ghcr-secret
-        - name: hanzo
-      containers:
-        - name: explorer
-          image: ghcr.io/luxfi/explorer:1.2.1
-          ports:
-            - containerPort: 8090
-              name: http
-          env:
-            - name: CHAINS
-              value: zoo:http://zoo-rpc.zoo.svc.cluster.local:9650/ext/bc/zoo/rpc
-          volumeMounts:
-            - name: data
-              mountPath: /data
-          resources:
-            requests:
-              cpu: 250m
-              memory: 512Mi
-            limits:
-              cpu: "1"
-              memory: 2Gi
-          livenessProbe:
-            httpGet:
-              path: /health
-              port: 8090
-            initialDelaySeconds: 30
-            periodSeconds: 15
-          readinessProbe:
-            httpGet:
-              path: /health
-              port: 8090
-            initialDelaySeconds: 10
-            periodSeconds: 10
-      volumes:
-        - name: data
-          emptyDir: {}
---
-apiVersion: v1
-kind: Service
-metadata:
-  name: zoo-explorer
-  labels:
-    app: zoo-explorer
-    app.kubernetes.io/name: zoo-explorer
-    app.kubernetes.io/part-of: zoo
-spec:
-  selector:
-    app: zoo-explorer
-  ports:
-    - port: 8090
-      targetPort: 8090
-      name: http
+# Zoo explorer is now owned by the ZooExplorer CR in network.yaml.
+# The operator (src/controller.rs::reconcile_explorer) creates a Deployment
+# named "zoo-explorer-explorer" + matching Service.
+#
+# This file kept as a placeholder so any cross-overlay reference still
+# resolves; the kustomization no longer pulls in any raw resource here.
@@ -1,90 +1,9 @@
-# ZooGateway — API gateway routing to all 15 chain RPCs + Beluga L3
-apiVersion: zoo.network/v1alpha1
-kind: ZooGateway
-metadata:
-  name: zoo-gateway
-  labels:
-    app: zoo-gateway
-    app.kubernetes.io/name: zoo-gateway
-    app.kubernetes.io/part-of: zoo
-spec:
-  networkRef: zoo-chain
-  host: api.zoo.ngo
-  replicas: 1
-  port: 8080
-  autoRoutes: true
-  routes:
-    # Core application chains
-    - path: /v1/rpc
-      backend: zoo-rpc:9650
-      rewrite: /ext/bc/zoo-evm/rpc
-    - path: /v1/ws
-      backend: zoo-rpc:9650
-      rewrite: /ext/bc/zoo-evm/ws
-    - path: /v1/dex
-      backend: zoo-rpc:9650
-      rewrite: /ext/bc/zoo-dex/rpc
-    - path: /v1/fhe
-      backend: zoo-rpc:9650
-      rewrite: /ext/bc/zoo-fhe/rpc
-    # Security chains
-    - path: /v1/identity
-      backend: zoo-rpc:9650
-      rewrite: /ext/bc/zoo-identity/rpc
-    - path: /v1/keys
-      backend: zoo-rpc:9650
-      rewrite: /ext/bc/zoo-keys/rpc
-    - path: /v1/threshold
-      backend: zoo-rpc:9650
-      rewrite: /ext/bc/zoo-threshold/rpc
-    - path: /v1/mpc
-      backend: zoo-rpc:9650
-      rewrite: /ext/bc/zoo-mpc/rpc
-    # Privacy & cross-chain
-    - path: /v1/bridge
-      backend: zoo-bridge:3000
-      rewrite: /
-    - path: /v1/zk
-      backend: zoo-rpc:9650
-      rewrite: /ext/bc/zoo-zk/rpc
-    # Consensus & infrastructure
-    - path: /v1/quantum
-      backend: zoo-rpc:9650
-      rewrite: /ext/bc/zoo-quantum/rpc
-    - path: /v1/ai
-      backend: zoo-rpc:9650
-      rewrite: /ext/bc/zoo-ai/rpc
-    - path: /v1/graph
-      backend: zoo-rpc:9650
-      rewrite: /ext/bc/zoo-graph/rpc
-    - path: /v1/oracle
-      backend: zoo-rpc:9650
-      rewrite: /ext/bc/zoo-oracle/rpc
-    - path: /v1/relay
-      backend: zoo-rpc:9650
-      rewrite: /ext/bc/zoo-relay/rpc
-    # Beluga L3
-    - path: /v1/beluga
-      backend: zoo-rpc:9650
-      rewrite: /ext/bc/beluga/rpc
-    # Explorer
-    - path: /v1/explorer
-      backend: zoo-explorer:4000
-      rewrite: /
-  cors:
-    allowedOrigins:
-      - "*"
-    allowedMethods:
-      - "GET"
-      - "POST"
-      - "OPTIONS"
-    allowedHeaders:
-      - "Content-Type"
-      - "Authorization"
-  rateLimit:
-    requestsPerSecond: 100
-    burst: 200
---
+# Zoo gateway Service — the actual ZooGateway CR lives in network.yaml.
+# This Service is the in-cluster target used by IngressRoutes (ingress.yaml)
+# to reach the validator's RPC port via the operator-owned headless+rpc Service.
+#
+# Mainnet/testnet/devnet overlays patch the ZooGateway hosts field via
+# network-patch.yaml.
 apiVersion: v1
 kind: Service
 metadata:
@@ -94,9 +13,14 @@ metadata:
    app.kubernetes.io/name: zoo-gateway
    app.kubernetes.io/part-of: zoo
 spec:
+  type: ClusterIP
  selector:
-    app: zoo-gateway
+    app.kubernetes.io/instance: zoo-chain
+    app.kubernetes.io/name: zoo-node
  ports:
-    - port: 8080
-      targetPort: 8080
-      name: http
+    - port: 9630
+      targetPort: 9630
+      name: rpc
+    - port: 9631
+      targetPort: 9631
+      name: staking
@@ -182,62 +182,101 @@ spec:
            path: /var/lib/zoo-ingress
            type: DirectoryOrCreate
 ---
-# Ingress routes for Zoo services
-apiVersion: networking.k8s.io/v1
-kind: Ingress
+# IngressRoutes for Zoo services — hanzoai/ingress native CRD shape
+# (kind: IngressRoute, apiVersion: hanzo.ai/v1alpha1). NO `kind: Ingress`
+# with annotation-driven middleware — that was the Traefik compat path.
+#
+# Mainnet/testnet host names come from each overlay's ZooGateway CR;
+# the routes themselves are static here. Per-env overlays patch the
+# hosts in their network-patch.yaml ZooGateway and add explore-specific
+# IngressRoutes if needed.
+apiVersion: hanzo.ai/v1alpha1
+kind: IngressRoute
 metadata:
-  name: zoo-routes
+  name: zoo-rpc
  labels:
    app.kubernetes.io/part-of: zoo
-  annotations:
-    kubernetes.io/ingress.class: ingress
-    cert-manager.io/cluster-issuer: letsencrypt-prod
 spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`api.zoo.network`)
+      kind: Rule
+      services:
+        - name: zoo-chain-rpc
+          port: 9630
+          passHostHeader: false
  tls:
-    - hosts:
-        - api.zoo.network
-        - api.zoo.ngo
-        - explore.zoo.network
-        - bridge.zoo.network
-      secretName: zoo-network-tls
-  rules:
-    - host: api.zoo.network
-      http:
-        paths:
-          - path: /
-            pathType: Prefix
-            backend:
-              service:
-                name: zoo-rpc
-                port:
-                  number: 9650
-    - host: api.zoo.ngo
-      http:
-        paths:
-          - path: /
-            pathType: Prefix
-            backend:
-              service:
-                name: zoo-gateway
-                port:
-                  number: 8080
-    - host: explore.zoo.network
-      http:
-        paths:
-          - path: /
-            pathType: Prefix
-            backend:
-              service:
-                name: zoo-explorer
-                port:
-                  number: 4000
-    - host: bridge.zoo.network
-      http:
-        paths:
-          - path: /
-            pathType: Prefix
-            backend:
-              service:
-                name: zoo-bridge
-                port:
-                  number: 3000
+    certResolver: letsencrypt
+    domains:
+      - main: api.zoo.network
+---
+apiVersion: hanzo.ai/v1alpha1
+kind: IngressRoute
+metadata:
+  name: zoo-gateway
+  labels:
+    app.kubernetes.io/part-of: zoo
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`api.zoo.ngo`)
+      kind: Rule
+      services:
+        - name: zoo-gateway
+          port: 9630
+          passHostHeader: false
+  tls:
+    certResolver: letsencrypt
+    domains:
+      - main: api.zoo.ngo
+---
+apiVersion: hanzo.ai/v1alpha1
+kind: IngressRoute
+metadata:
+  name: zoo-explorer
+  labels:
+    app.kubernetes.io/part-of: zoo
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`explore.zoo.network`)
+      kind: Rule
+      services:
+        - name: zoo-explorer-explorer
+          port: 8090
+          passHostHeader: false
+    - match: Host(`api-explore.zoo.network`)
+      kind: Rule
+      services:
+        - name: zoo-explorer-explorer
+          port: 8090
+          passHostHeader: false
+  tls:
+    certResolver: letsencrypt
+    domains:
+      - main: explore.zoo.network
+      - main: api-explore.zoo.network
+---
+apiVersion: hanzo.ai/v1alpha1
+kind: IngressRoute
+metadata:
+  name: zoo-bridge
+  labels:
+    app.kubernetes.io/part-of: zoo
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`bridge.zoo.network`)
+      kind: Rule
+      services:
+        - name: zoo-bridge
+          port: 3000
+          passHostHeader: false
+  tls:
+    certResolver: letsencrypt
+    domains:
+      - main: bridge.zoo.network
@@ -1,6 +1,14 @@
-# ZooNetwork CR — full-chain L2 on Lux with all 15 VMs
-# Validators run every chain: consensus + compute + identity + keys + privacy
-# GPU-accelerated: EVM (cevm), FHE (TFHE), consensus (BLS12-381), PQ crypto (NTT)
+# ZooNetwork CR — Zoo validator cluster on Lux primary network.
+#
+# Each env overlay (zoo-mainnet, zoo-testnet, zoo-devnet) patches:
+#   spec.networkId       — Lux primary networkID (1/2/3)
+#   spec.validators      — replica count
+#   spec.image.tag       — semver pin (NEVER floating)
+#   spec.config          — flat map → ZOO_* env vars on the StatefulSet
+#                          including BOOTSTRAP_IPS / BOOTSTRAP_IDS / TRACK_CHAINS
+#
+# Operator controller (Rust): src/controller.rs::create_network.
+# CRD struct: src/crd.rs::ZooNetworkSpec.
 apiVersion: zoo.network/v1alpha1
 kind: ZooNetwork
 metadata:
@@ -10,187 +18,75 @@ metadata:
    app.kubernetes.io/name: zoo-chain
    app.kubernetes.io/part-of: zoo
 spec:
-  image:
-    repository: ghcr.io/zooai/zoo-node
-    tag: latest
-  validators: 5
  networkId: 1
-  subnet:
-    name: zoo
-    chainId: 200200
-    vmType: evm
-    coin: ZOO
-  # All chains on the same validator set — no separate deployments
-  chains:
-    # -- Core application chains
-    - name: zoo-evm
-      vmType: evm
-      chainId: 200200
-      coin: ZOO
-    - name: zoo-dex
-      vmType: dex
-      coin: ZOO
-    - name: zoo-fhe
-      vmType: fhe
-      chainId: 200203
-    # -- Security chains (replace external IAM/KMS/MPC)
-    - name: zoo-identity
-      vmType: identity
-      config:
-        credentialTTL: 8760h
-        allowSelfIssue: false
-        requireZKProofs: true
-        trustedIssuers: []
-    - name: zoo-keys
-      vmType: key
-      config:
-        # Default envelope: AES-256-GCM data key, ML-KEM-768 key encapsulation
-        # Maps to luxfi/kms AEAD+ML-KEM scheme
-        defaultScheme: aead+mlkem
-        mlkemEnabled: true
-        mldsaEnabled: true
-        defaultThreshold: 2
-        defaultTotalShares: 3
-        shareCacheSize: 1024
-        # Threshold reveal delegated to T-Chain (TFHE-gated)
-        thresholdRevealChain: zoo-threshold
-    - name: zoo-threshold
-      vmType: threshold
-      config:
-        threshold: 2
-        totalParties: 3
-        sessionTimeout: 300s
-        maxActiveSessions: 100
-        dailySigningQuota: 10000
-        # TFHE for threshold-gated key reveal from K-Chain
-        fheEnabled: true
-        fheScheme: tfhe
-        fheAccelerator: auto
-        # CKKS for encrypted ML inference (Beluga L3 confidential compute)
-        ckksEnabled: true
-        ckksPolyModulusDegree: 8192
-        ckksCoeffModBits: [60, 40, 40, 60]
-        allowedKeyTypes:
-          - secp256k1
-          - ed25519
-          - bls
-    - name: zoo-mpc
-      vmType: mpc
-      config:
-        # MPC signing chain — maps to luxfi/mpc CGGMP21 protocol
-        protocol: cggmp21
-        threshold: 2
-        totalParties: 3
-        sessionTimeout: 300s
-        maxActiveSessions: 200
-        # Keygen produces shares stored encrypted on K-Chain (ML-KEM)
-        keyShareStore: zoo-keys
-        # Signing requests authenticated via I-Chain DIDs
-        authChain: zoo-identity
-        supportedCurves:
-          - secp256k1
-          - ed25519
-        enablePresigning: true
-    # -- Privacy & cross-chain
-    - name: zoo-bridge-chain
-      vmType: bridge
-      config:
-        requireValidatorBond: true
-        maxBridgeAmount: "1000000000000000000000000"
-        dailyBridgeLimit: "10000000000000000000000000"
-    - name: zoo-zk
-      vmType: zk
-      config:
-        proofSystem: groth16
-        enableFHE: true
-        securityLevel: 128
-    # -- Consensus & infrastructure
-    - name: zoo-quantum
-      vmType: quantum
-      config:
-        coronaEnabled: true
-        hybridFinality: true
-    - name: zoo-ai
-      vmType: ai
-      config:
-        maxModelSize: 4Gi
-        inferenceTimeout: 30s
-    - name: zoo-graph
-      vmType: graph
-      config:
-        maxQueryDepth: 10
-        cacheSize: 512Mi
-    - name: zoo-oracle
-      vmType: oracle
-      config:
-        updateInterval: 30s
-        maxDataAge: 300s
-    - name: zoo-relay
-      vmType: relay
-      config:
-        maxMessageSize: 1Mi
-        targetChains:
-          - lux-mainnet
-          - lux-testnet
-  genesis:
-    secretRef:
-      name: zoo-genesis
-      key: genesis.json
-  staking:
-    secretPrefix: zoo-staking
-  node:
-    httpPort: 9650
-    stakingPort: 9651
-    args:
-      - --data-dir=/data
-      - --http-host=0.0.0.0
-      - --sybil-protection-enabled=true
-      - --track-chains=zoo
-    resources:
-      requests:
-        cpu: "2"
-        memory: 8Gi
-        nvidia.com/gpu: "1"
-      limits:
-        cpu: "8"
-        memory: 16Gi
-        nvidia.com/gpu: "1"
-    gpu:
-      enabled: true
-      backend: auto
-      evmAccelerator: cevm
-      fheAccelerator: auto
-      consensusAccelerator: true
-      pqCryptoAccelerator: true
-    storage:
-      size: 200Gi
-      storageClass: do-block-storage
-    readinessProbe:
-      httpGet:
-        path: /ext/health
-        port: 9650
-      initialDelaySeconds: 30
-      periodSeconds: 10
-    livenessProbe:
-      httpGet:
-        path: /ext/health
-        port: 9650
-      initialDelaySeconds: 60
-      periodSeconds: 30
+  validators: 0
+  image:
+    repository: ghcr.io/luxfi/node
+    tag: v1.27.4
+    pullPolicy: IfNotPresent
+  storage:
+    storageClass: do-block-storage
+    size: 100Gi
+  resources:
+    cpuRequest: "1"
+    cpuLimit: "4"
+    memoryRequest: 2Gi
+    memoryLimit: 8Gi
+  ports:
+    http: 9630
+    staking: 9631
+  service:
+    serviceType: ClusterIP
+  metrics:
+    enabled: true
+    port: 9090
+  # Flat key/value map → ZOO_<KEY_UPPERCASED> env vars on each validator pod.
+  # The image's entrypoint translates these into luxd CLI flags.
+  config: {}
 ---
-apiVersion: v1
-kind: Service
+apiVersion: zoo.network/v1alpha1
+kind: ZooChain
 metadata:
-  name: zoo-rpc
+  name: zoo-evm
  labels:
    app: zoo-chain
-    app.kubernetes.io/name: zoo-rpc
    app.kubernetes.io/part-of: zoo
 spec:
-  type: ClusterIP
-  selector:
-    app: zoo-chain
-  ports:
-    - port: 9650
-      targetPort: 9650
-      name: http
+  chainId: 200200
+  networkRef: zoo-chain
+---
+apiVersion: zoo.network/v1alpha1
+kind: ZooExplorer
+metadata:
+  name: zoo-explorer
+  labels:
+    app: zoo-explorer
+    app.kubernetes.io/part-of: zoo
+spec:
+  chainRef: zoo-evm
+  image: ghcr.io/luxfi/explorer:1.2.18
+  rpcEndpoint: http://zoo-chain-rpc.zoo.svc.cluster.local:9630/ext/bc/zoo/rpc
+  chainName: zoo
+  coinSymbol: ZOO
+  service:
+    serviceType: ClusterIP
+    port: 8090
+---
+apiVersion: zoo.network/v1alpha1
+kind: ZooGateway
+metadata:
+  name: zoo-gateway
+  labels:
+    app: zoo-gateway
+    app.kubernetes.io/part-of: zoo
+spec:
+  networkRef: zoo-chain
+  hosts:
+    - api.zoo.network
+  tls:
+    enabled: true
+    secretName: zoo-network-tls
+  service:
+    serviceType: ClusterIP
+    rpcPort: 9630
+    wsPort: 9631
@@ -1,4 +1,11 @@
-# zoo-operator — manages ZooNetwork CRs for Zoo subnet validators
+# zoo-operator — reconciles ZooNetwork / ZooChain / ZooExplorer / ZooGateway.
+#
+# Topology: ONE operator runs in zoo-system, watches every namespace
+# (--namespace "").  zoo-mainnet/zoo-testnet/zoo-devnet overlays scale
+# this Deployment to 0 so only zoo-system holds the lease.
+#
+# OPERATOR_NAMESPACE comes from the pod's own namespace (fieldRef) so
+# the lease lives in zoo-system — see src/main.rs line 106 + src/leader.rs.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -23,12 +30,14 @@ spec:
        - name: hanzo
      containers:
        - name: operator
-          # TODO: pin semver once zooai/operator CI publishes vX.Y.Z tags.
-          # main-amd64 is the immutable per-arch tag from the shared
-          # hanzoai/.github docker-build workflow.
-          image: ghcr.io/zooai/operator:main-amd64
-          imagePullPolicy: Always
+          image: ghcr.io/zooai/operator:v0.1.1
+          imagePullPolicy: IfNotPresent
          args: ["--namespace", ""]
+          env:
+            - name: OPERATOR_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
          ports:
            - containerPort: 8081
              name: health
@@ -0,0 +1,51 @@
+# Zoo devnet IngressRoutes — dev.zoo.network host space, separate from
+# mainnet's api.zoo.network / explore.zoo.network. base IngressRoutes
+# are stripped via the kustomization patch.
+apiVersion: hanzo.ai/v1alpha1
+kind: IngressRoute
+metadata:
+  name: zoo-rpc-devnet
+  labels:
+    app.kubernetes.io/part-of: zoo
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`api.dev.zoo.network`)
+      kind: Rule
+      services:
+        - name: zoo-chain-rpc
+          port: 9630
+          passHostHeader: false
+  tls:
+    certResolver: letsencrypt
+    domains:
+      - main: api.dev.zoo.network
+---
+apiVersion: hanzo.ai/v1alpha1
+kind: IngressRoute
+metadata:
+  name: zoo-explorer-devnet
+  labels:
+    app.kubernetes.io/part-of: zoo
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`explore.dev.zoo.network`)
+      kind: Rule
+      services:
+        - name: zoo-explorer-explorer
+          port: 8090
+          passHostHeader: false
+    - match: Host(`api-explore.dev.zoo.network`)
+      kind: Rule
+      services:
+        - name: zoo-explorer-explorer
+          port: 8090
+          passHostHeader: false
+  tls:
+    certResolver: letsencrypt
+    domains:
+      - main: explore.dev.zoo.network
+      - main: api-explore.dev.zoo.network
@@ -4,6 +4,7 @@ namespace: zoo-devnet

 resources:
  - ../base
+  - ingressroute.yaml

 labels:
  - pairs:
@@ -20,8 +21,46 @@ configMapGenerator:
      - ZOO_EVM_CHAIN_ID=200202
      - lux-rpc-url=https://api.lux-dev.network/ext/bc/C/rpc

+# Mainnet hosts in base/ingress.yaml don't apply to devnet — remove
+# them so the devnet-specific ingressroute.yaml is canonical.
 patches:
  - path: network-patch.yaml
+  - target:
+      kind: IngressRoute
+      name: zoo-rpc
+    patch: |
+      $patch: delete
+      apiVersion: hanzo.ai/v1alpha1
+      kind: IngressRoute
+      metadata:
+        name: zoo-rpc
+  - target:
+      kind: IngressRoute
+      name: zoo-gateway
+    patch: |
+      $patch: delete
+      apiVersion: hanzo.ai/v1alpha1
+      kind: IngressRoute
+      metadata:
+        name: zoo-gateway
+  - target:
+      kind: IngressRoute
+      name: zoo-explorer
+    patch: |
+      $patch: delete
+      apiVersion: hanzo.ai/v1alpha1
+      kind: IngressRoute
+      metadata:
+        name: zoo-explorer
+  - target:
+      kind: IngressRoute
+      name: zoo-bridge
+    patch: |
+      $patch: delete
+      apiVersion: hanzo.ai/v1alpha1
+      kind: IngressRoute
+      metadata:
+        name: zoo-bridge
  # Fix namespace name in base Namespace resource
  - target:
      kind: Namespace
@@ -25,12 +25,13 @@ spec:
      size: 20Gi
      storageClass: do-block-storage
 ---
+# Single zoo-operator lives in zoo-system; per-env Deployments scale to 0.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: zoo-operator
 spec:
-  replicas: 1
+  replicas: 0
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -41,24 +42,6 @@ spec:
 ---
 apiVersion: apps/v1
 kind: Deployment
-metadata:
-  name: zoo-explorer
-spec:
-  replicas: 1
-  template:
-    spec:
-      containers:
-        - name: explorer
-          resources:
-            requests:
-              cpu: 100m
-              memory: 256Mi
-            limits:
-              cpu: 500m
-              memory: 1Gi
---
-apiVersion: apps/v1
-kind: Deployment
 metadata:
  name: zoo-ingress
 spec:
@@ -1,36 +1,82 @@
-# Mainnet: 5 GPU validators, chain 200200, no separate MPC (chain-native T-Chain)
+# Zoo mainnet — 5 validators that join Lux mainnet primary network (networkId 1)
+# and track the Zoo EVM chain (chainID 200200, blockchainID
+# 4HBM17czc7zFQMPrd9CVZ6YbABm1jecf99iwvFw8Gbo6gFaU3 — looked up live
+# 2026-06-10 from api.lux.network/ext/bc/P platform.getBlockchains).
+#
+# Bootstrap IPs and NodeIDs are the public addresses of the Lux mainnet
+# validators (do-sfo3-lux-k8s, namespace lux-mainnet, LB IPs 2026-06-10).
+# RLP seed for chain 200200 is loaded by luxd-rlp-import.yaml below.
 apiVersion: zoo.network/v1alpha1
 kind: ZooNetwork
 metadata:
  name: zoo-chain
 spec:
-  validators: 5
  networkId: 1
-  subnet:
-    chainId: 200200
+  validators: 5
  image:
-    repository: ghcr.io/zooai/node
-    tag: main
-  node:
-    resources:
-      requests:
-        cpu: "2"
-        memory: 8Gi
-        nvidia.com/gpu: "1"
-      limits:
-        cpu: "8"
-        memory: 16Gi
-        nvidia.com/gpu: "1"
-    storage:
-      size: 200Gi
-      storageClass: do-block-storage
+    repository: ghcr.io/luxfi/node
+    tag: v1.27.4
+    pullPolicy: IfNotPresent
+  resources:
+    cpuRequest: "2"
+    cpuLimit: "8"
+    memoryRequest: 8Gi
+    memoryLimit: 16Gi
+  storage:
+    storageClass: do-block-storage
+    size: 200Gi
+  config:
+    BOOTSTRAP_IPS: "129.212.164.46:9631,137.184.246.133:9631,143.244.210.43:9631,143.244.210.71:9631,24.144.70.85:9631"
+    BOOTSTRAP_IDS: "NodeID-Mf3JfSY91oDwfBqf7rCLmhg4NDtDghw1f,NodeID-2TwSZ2oyeBK2mv7JiseEQ8m74rotDj4QR,NodeID-Ld9VFBQ9zGbd79z2vzaAqkQ3jHuqbtRpo,NodeID-8mY2fhUehN27v3LCU84BnnKEoeRfd2weC,NodeID-DwsrqSkPoE3pXWrUt9nkJ5yBycwRQ246X"
+    TRACK_CHAINS: "4HBM17czc7zFQMPrd9CVZ6YbABm1jecf99iwvFw8Gbo6gFaU3"
+    SYBIL_PROTECTION_ENABLED: "true"
 ---
+# Zoo EVM chain status — operator just stamps status; tracking happens
+# via TRACK_CHAINS env on the validator.
+apiVersion: zoo.network/v1alpha1
+kind: ZooChain
+metadata:
+  name: zoo-evm
+spec:
+  chainId: 200200
+  blockchainId: 4HBM17czc7zFQMPrd9CVZ6YbABm1jecf99iwvFw8Gbo6gFaU3
+  networkRef: zoo-chain
+---
+# Mainnet explorer — Lux explorer Go binary v1.2.18.
+apiVersion: zoo.network/v1alpha1
+kind: ZooExplorer
+metadata:
+  name: zoo-explorer
+spec:
+  chainRef: zoo-evm
+  image: ghcr.io/luxfi/explorer:1.2.18
+  rpcEndpoint: http://zoo-chain-rpc.zoo-mainnet.svc.cluster.local:9630/ext/bc/4HBM17czc7zFQMPrd9CVZ6YbABm1jecf99iwvFw8Gbo6gFaU3/rpc
+  chainName: zoo
+  coinSymbol: ZOO
+  service:
+    serviceType: ClusterIP
+    port: 8090
+---
+# Mainnet gateway — RPC entry point.
+apiVersion: zoo.network/v1alpha1
+kind: ZooGateway
+metadata:
+  name: zoo-gateway
+spec:
+  networkRef: zoo-chain
+  hosts:
+    - api.zoo.network
+  tls:
+    enabled: true
+    secretName: zoo-network-tls
+---
+# Single zoo-operator lives in zoo-system; per-env Deployments scale to 0.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: zoo-operator
 spec:
-  replicas: 1
+  replicas: 0
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -41,13 +87,6 @@ spec:
 ---
 apiVersion: apps/v1
 kind: Deployment
-metadata:
-  name: zoo-explorer
-spec:
-  replicas: 2
---
-apiVersion: apps/v1
-kind: Deployment
 metadata:
  name: zoo-ingress
 spec:
@@ -89,40 +89,24 @@ spec:
      port: 80
      targetPort: 3000
 ---
-apiVersion: networking.k8s.io/v1
-kind: Ingress
+apiVersion: hanzo.ai/v1alpha1
+kind: IngressRoute
 metadata:
  name: zoo-industries
  namespace: zoo-mainnet
  labels:
    app.kubernetes.io/name: zoo-industries
-  annotations:
-    cert-manager.io/cluster-issuer: letsencrypt-prod
-    kubernetes.io/ingress.class: ingress
 spec:
-  rules:
-    - host: zoo.industries
-      http:
-        paths:
-          - backend:
-              service:
-                name: zoo-industries
-                port:
-                  number: 80
-            path: /
-            pathType: Prefix
-    - host: www.zoo.industries
-      http:
-        paths:
-          - backend:
-              service:
-                name: zoo-industries
-                port:
-                  number: 80
-            path: /
-            pathType: Prefix
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`zoo.industries`) || Host(`www.zoo.industries`)
+      kind: Rule
+      services:
+        - name: zoo-industries
+          port: 80
  tls:
-    - hosts:
-        - zoo.industries
-        - www.zoo.industries
-      secretName: zoo-industries-tls
+    certResolver: letsencrypt
+    domains:
+      - main: zoo.industries
+      - main: www.zoo.industries
@@ -13,6 +13,44 @@ labels:

 patches:
  - path: patches.yaml
+  # zoo-system hosts the operator only — strip mainnet IngressRoutes
+  # inherited from base/ingress.yaml so they don't collide.
+  - target:
+      kind: IngressRoute
+      name: zoo-rpc
+    patch: |
+      $patch: delete
+      apiVersion: hanzo.ai/v1alpha1
+      kind: IngressRoute
+      metadata:
+        name: zoo-rpc
+  - target:
+      kind: IngressRoute
+      name: zoo-gateway
+    patch: |
+      $patch: delete
+      apiVersion: hanzo.ai/v1alpha1
+      kind: IngressRoute
+      metadata:
+        name: zoo-gateway
+  - target:
+      kind: IngressRoute
+      name: zoo-explorer
+    patch: |
+      $patch: delete
+      apiVersion: hanzo.ai/v1alpha1
+      kind: IngressRoute
+      metadata:
+        name: zoo-explorer
+  - target:
+      kind: IngressRoute
+      name: zoo-bridge
+    patch: |
+      $patch: delete
+      apiVersion: hanzo.ai/v1alpha1
+      kind: IngressRoute
+      metadata:
+        name: zoo-bridge
  # Fix namespace name in base Namespace resource
  - target:
      kind: Namespace
@@ -1,5 +1,6 @@
-# zoo-system: operator only. Scale everything else to 0 so the base
-# manifests can co-tenant the namespace without spawning real workloads.
+# zoo-system: hosts the single zoo-operator that watches every namespace.
+# Everything else is scaled to 0 so the base manifests can co-tenant the
+# namespace without spawning real workloads.
 apiVersion: zoo.network/v1alpha1
 kind: ZooNetwork
 metadata:
@@ -7,17 +8,18 @@ metadata:
 spec:
  validators: 0
 ---
+# Two-replica HA — only one is leader at a time (see src/leader.rs).
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  name: zoo-bridge
+  name: zoo-operator
 spec:
-  replicas: 0
+  replicas: 2
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  name: zoo-explorer
+  name: zoo-bridge
 spec:
  replicas: 0
 ---
@@ -0,0 +1,51 @@
+# Zoo testnet IngressRoutes — distinct hosts from mainnet so both can
+# coexist on the same hanzoai/ingress proxy. Mainnet routes from
+# base/ingress.yaml are emptied by the kustomization deletion below.
+apiVersion: hanzo.ai/v1alpha1
+kind: IngressRoute
+metadata:
+  name: zoo-rpc-testnet
+  labels:
+    app.kubernetes.io/part-of: zoo
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`api.test.zoo.network`)
+      kind: Rule
+      services:
+        - name: zoo-chain-rpc
+          port: 9630
+          passHostHeader: false
+  tls:
+    certResolver: letsencrypt
+    domains:
+      - main: api.test.zoo.network
+---
+apiVersion: hanzo.ai/v1alpha1
+kind: IngressRoute
+metadata:
+  name: zoo-explorer-testnet
+  labels:
+    app.kubernetes.io/part-of: zoo
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    - match: Host(`explore.test.zoo.network`)
+      kind: Rule
+      services:
+        - name: zoo-explorer-explorer
+          port: 8090
+          passHostHeader: false
+    - match: Host(`api-explore.test.zoo.network`)
+      kind: Rule
+      services:
+        - name: zoo-explorer-explorer
+          port: 8090
+          passHostHeader: false
+  tls:
+    certResolver: letsencrypt
+    domains:
+      - main: explore.test.zoo.network
+      - main: api-explore.test.zoo.network
@@ -4,6 +4,7 @@ namespace: zoo-testnet

 resources:
  - ../base
+  - ingressroute.yaml

 labels:
  - pairs:
@@ -20,8 +21,47 @@ configMapGenerator:
      - ZOO_EVM_CHAIN_ID=200201
      - lux-rpc-url=https://api.lux-test.network/ext/bc/C/rpc

+# Mainnet hosts in base/ingress.yaml don't apply to testnet — remove
+# them via kustomize so only the testnet-specific ingressroute.yaml
+# stays in the overlay.
 patches:
  - path: network-patch.yaml
+  - target:
+      kind: IngressRoute
+      name: zoo-rpc
+    patch: |
+      $patch: delete
+      apiVersion: hanzo.ai/v1alpha1
+      kind: IngressRoute
+      metadata:
+        name: zoo-rpc
+  - target:
+      kind: IngressRoute
+      name: zoo-gateway
+    patch: |
+      $patch: delete
+      apiVersion: hanzo.ai/v1alpha1
+      kind: IngressRoute
+      metadata:
+        name: zoo-gateway
+  - target:
+      kind: IngressRoute
+      name: zoo-explorer
+    patch: |
+      $patch: delete
+      apiVersion: hanzo.ai/v1alpha1
+      kind: IngressRoute
+      metadata:
+        name: zoo-explorer
+  - target:
+      kind: IngressRoute
+      name: zoo-bridge
+    patch: |
+      $patch: delete
+      apiVersion: hanzo.ai/v1alpha1
+      kind: IngressRoute
+      metadata:
+        name: zoo-bridge
  # Fix namespace name in base Namespace resource
  - target:
      kind: Namespace
@@ -1,36 +1,76 @@
-# Testnet: 3 GPU validators, chain 200201, no separate MPC (chain-native T-Chain)
+# Zoo testnet — 3 validators joining Lux testnet primary network
+# (networkId 2). Tracks Zoo testnet EVM chain (chainID 200201,
+# blockchainID c3PrgGAAoJXz8RgNAw3j7dSUjUiHVk3mtTneYfr7GvZRxmTPH —
+# looked up live 2026-06-10 from api.lux-test.network).
+#
+# Bootstrap IPs are the public LB IPs of lux-mainnet's luxd-0..4 in
+# do-sfo3-lux-k8s namespace lux-testnet (read live 2026-06-10).
 apiVersion: zoo.network/v1alpha1
 kind: ZooNetwork
 metadata:
  name: zoo-chain
 spec:
-  validators: 3
  networkId: 2
-  subnet:
-    chainId: 200201
+  validators: 3
  image:
-    repository: ghcr.io/zooai/node
-    tag: test
-  node:
-    resources:
-      requests:
-        cpu: "1"
-        memory: 4Gi
-        nvidia.com/gpu: "1"
-      limits:
-        cpu: "4"
-        memory: 8Gi
-        nvidia.com/gpu: "1"
-    storage:
-      size: 100Gi
-      storageClass: do-block-storage
+    repository: ghcr.io/luxfi/node
+    tag: v1.27.4
+    pullPolicy: IfNotPresent
+  resources:
+    cpuRequest: "1"
+    cpuLimit: "4"
+    memoryRequest: 4Gi
+    memoryLimit: 8Gi
+  storage:
+    storageClass: do-block-storage
+    size: 100Gi
+  config:
+    BOOTSTRAP_IPS: "134.199.187.16:9641,209.38.174.84:9641,209.38.173.80:9641,134.199.187.62:9641,164.90.247.204:9641"
+    BOOTSTRAP_IDS: "NodeID-Mf3JfSY91oDwfBqf7rCLmhg4NDtDghw1f,NodeID-2TwSZ2oyeBK2mv7JiseEQ8m74rotDj4QR,NodeID-Ld9VFBQ9zGbd79z2vzaAqkQ3jHuqbtRpo,NodeID-8mY2fhUehN27v3LCU84BnnKEoeRfd2weC,NodeID-4smuFz5Z9cm8BEyQU1oPQ9nwgXyMgaNqu"
+    TRACK_CHAINS: "c3PrgGAAoJXz8RgNAw3j7dSUjUiHVk3mtTneYfr7GvZRxmTPH"
+    SYBIL_PROTECTION_ENABLED: "true"
+---
+apiVersion: zoo.network/v1alpha1
+kind: ZooChain
+metadata:
+  name: zoo-evm
+spec:
+  chainId: 200201
+  blockchainId: c3PrgGAAoJXz8RgNAw3j7dSUjUiHVk3mtTneYfr7GvZRxmTPH
+  networkRef: zoo-chain
+---
+apiVersion: zoo.network/v1alpha1
+kind: ZooExplorer
+metadata:
+  name: zoo-explorer
+spec:
+  chainRef: zoo-evm
+  image: ghcr.io/luxfi/explorer:1.2.18
+  rpcEndpoint: http://zoo-chain-rpc.zoo-testnet.svc.cluster.local:9630/ext/bc/c3PrgGAAoJXz8RgNAw3j7dSUjUiHVk3mtTneYfr7GvZRxmTPH/rpc
+  chainName: zoo
+  coinSymbol: tZOO
+  service:
+    serviceType: ClusterIP
+    port: 8090
+---
+apiVersion: zoo.network/v1alpha1
+kind: ZooGateway
+metadata:
+  name: zoo-gateway
+spec:
+  networkRef: zoo-chain
+  hosts:
+    - api.test.zoo.network
+  tls:
+    enabled: true
+    secretName: zoo-testnet-tls
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: zoo-operator
 spec:
-  replicas: 1
+  replicas: 0
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -41,24 +81,6 @@ spec:
 ---
 apiVersion: apps/v1
 kind: Deployment
-metadata:
-  name: zoo-explorer
-spec:
-  replicas: 1
-  template:
-    spec:
-      containers:
-        - name: explorer
-          resources:
-            requests:
-              cpu: 100m
-              memory: 256Mi
-            limits:
-              cpu: 500m
-              memory: 1Gi
---
-apiVersion: apps/v1
-kind: Deployment
 metadata:
  name: zoo-ingress
 spec: