The zoo-operator readiness probe (GET :8081/readyz) returned 503 for ~9 days
(88k failures) on every replica and the beluga/zoo-chain CRs never reconciled
(empty PHASE/CHAINID).
Root cause — RBAC subject collision:
ClusterRoleBinding/zoo-operator is cluster-scoped, but each env overlay
rendered the SAME binding name and JSON6902-patched its single subject's
namespace to its own env. ClusterRoleBindings are not namespaced, so the four
overlays collided on apply and last-writer-won — only zoo-mainnet's SA ended
up bound. The operator runs with --namespace "" (all namespaces) behind ONE
cluster-global leader-election lease (zoo-operator-leader in zoo-system); the
lease winner was the zoo-devnet pod, whose SA was NOT a subject → 403 on every
zoo.network list/watch → controllers' InitialListFailed → leader never healthy
→ readyz 503 on all pods → 0/1 Deployments.
Fix: define the binding once in base/operator.yaml with ALL four per-env
operator SAs as subjects (one ClusterRole, one binding, every identity —
order-independent), and drop the per-overlay subject-rewrite patches that
caused the collision. The operator's /readyz gates on leadership, so once the
lease winner can list/watch cluster-wide it goes 1/1 and reconciles every
namespace.
Also align zoo-devnet/zoo-system ZooNetwork CRs to the v0.1.1 operator's
ZooNetworkSpec schema (validators:<int> + image{} + resources{} + config{}),
consistent with the mainnet/testnet patches, so the whole tree reconciles
under the pinned ghcr.io/zooai/operator:v0.1.1.
hanzo-dev
8eb8c88bea