hanzo-dev
8eb8c88bea
The zoo-operator readiness probe (GET :8081/readyz) returned 503 for ~9 days
(88k failures) on every replica and the beluga/zoo-chain CRs never reconciled
(empty PHASE/CHAINID).
Root cause — RBAC subject collision:
ClusterRoleBinding/zoo-operator is cluster-scoped, but each env overlay
rendered the SAME binding name and JSON6902-patched its single subject's
namespace to its own env. ClusterRoleBindings are not namespaced, so the four
overlays collided on apply and last-writer-won — only zoo-mainnet's SA ended
up bound. The operator runs with --namespace "" (all namespaces) behind ONE
cluster-global leader-election lease (zoo-operator-leader in zoo-system); the
lease winner was the zoo-devnet pod, whose SA was NOT a subject → 403 on every
zoo.network list/watch → controllers' InitialListFailed → leader never healthy
→ readyz 503 on all pods → 0/1 Deployments.
Fix: define the binding once in base/operator.yaml with ALL four per-env
operator SAs as subjects (one ClusterRole, one binding, every identity —
order-independent), and drop the per-overlay subject-rewrite patches that
caused the collision. The operator's /readyz gates on leadership, so once the
lease winner can list/watch cluster-wide it goes 1/1 and reconciles every
namespace.
Also align zoo-devnet/zoo-system ZooNetwork CRs to the v0.1.1 operator's
ZooNetworkSpec schema (validators:<int> + image{} + resources{} + config{}),
consistent with the mainnet/testnet patches, so the whole tree reconciles
under the pinned ghcr.io/zooai/operator:v0.1.1.
Zoo Universe
Infrastructure orchestration for Zoo Labs Foundation.
Architecture
zoo.ngo — Marketing site (Vite SPA)
zoo.exchange — DEX (Vite + Tamagui)
zoo.network — Zoo Network validators
zoo.wallet — Browser extension + mobile + web wallet
docs.zoo.ngo — Documentation (fumadocs)
zips.zoo.ngo — Zoo Improvement Proposals
Local Development
docker compose up # All services
docker compose up ngo # Just the website
Production (DOKS)
Three clusters:
do-sfo3-zoo-k8s— mainnetdo-sfo3-zoo-test-k8s— testnetdo-sfo3-zoo-dev-k8s— devnet
Deploy:
./deploy.sh mainnet
./deploy.sh testnet
./deploy.sh devnet
Repos
| Repo | Package | Purpose |
|---|---|---|
| zoo/ngo | @zooai/ngo-web | Marketing site |
| zoo/exchange | @zooai/exchange | DEX frontend |
| zoo/wallet | @zooai/wallet | Wallet (ext + mobile + web) |
| zoo/ui | @zooai/ui + @zooai/gui | Design system |
| zoo/node | — | Rust blockchain node |
| zoo/docs | @zoo/docs | Documentation |
| zoo/ZIPs | — | Improvement Proposals |
| zoo/papers | — | Research papers (47) |
| zoo/formal | — | Formal proofs (20) |