hanzo-dev
8eb8c88bea
The zoo-operator readiness probe (GET :8081/readyz) returned 503 for ~9 days
(88k failures) on every replica and the beluga/zoo-chain CRs never reconciled
(empty PHASE/CHAINID).
Root cause — RBAC subject collision:
ClusterRoleBinding/zoo-operator is cluster-scoped, but each env overlay
rendered the SAME binding name and JSON6902-patched its single subject's
namespace to its own env. ClusterRoleBindings are not namespaced, so the four
overlays collided on apply and last-writer-won — only zoo-mainnet's SA ended
up bound. The operator runs with --namespace "" (all namespaces) behind ONE
cluster-global leader-election lease (zoo-operator-leader in zoo-system); the
lease winner was the zoo-devnet pod, whose SA was NOT a subject → 403 on every
zoo.network list/watch → controllers' InitialListFailed → leader never healthy
→ readyz 503 on all pods → 0/1 Deployments.
Fix: define the binding once in base/operator.yaml with ALL four per-env
operator SAs as subjects (one ClusterRole, one binding, every identity —
order-independent), and drop the per-overlay subject-rewrite patches that
caused the collision. The operator's /readyz gates on leadership, so once the
lease winner can list/watch cluster-wide it goes 1/1 and reconciles every
namespace.
Also align zoo-devnet/zoo-system ZooNetwork CRs to the v0.1.1 operator's
ZooNetworkSpec schema (validators:<int> + image{} + resources{} + config{}),
consistent with the mainnet/testnet patches, so the whole tree reconciles
under the pinned ghcr.io/zooai/operator:v0.1.1.
74 lines
1.7 KiB
YAML
74 lines
1.7 KiB
YAML
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
namespace: zoo-system
|
|
|
|
resources:
|
|
- ../base
|
|
|
|
labels:
|
|
- pairs:
|
|
app.kubernetes.io/cluster: zoo-k8s
|
|
app.kubernetes.io/env: system
|
|
includeSelectors: false
|
|
|
|
patches:
|
|
- path: patches.yaml
|
|
# zoo-system hosts the operator only — strip mainnet IngressRoutes
|
|
# inherited from base/ingress.yaml so they don't collide.
|
|
- target:
|
|
kind: IngressRoute
|
|
name: zoo-rpc
|
|
patch: |
|
|
$patch: delete
|
|
apiVersion: hanzo.ai/v1alpha1
|
|
kind: IngressRoute
|
|
metadata:
|
|
name: zoo-rpc
|
|
- target:
|
|
kind: IngressRoute
|
|
name: zoo-gateway
|
|
patch: |
|
|
$patch: delete
|
|
apiVersion: hanzo.ai/v1alpha1
|
|
kind: IngressRoute
|
|
metadata:
|
|
name: zoo-gateway
|
|
- target:
|
|
kind: IngressRoute
|
|
name: zoo-explorer
|
|
patch: |
|
|
$patch: delete
|
|
apiVersion: hanzo.ai/v1alpha1
|
|
kind: IngressRoute
|
|
metadata:
|
|
name: zoo-explorer
|
|
- target:
|
|
kind: IngressRoute
|
|
name: zoo-bridge
|
|
patch: |
|
|
$patch: delete
|
|
apiVersion: hanzo.ai/v1alpha1
|
|
kind: IngressRoute
|
|
metadata:
|
|
name: zoo-bridge
|
|
# Fix namespace name in base Namespace resource
|
|
- target:
|
|
kind: Namespace
|
|
name: zoo
|
|
patch: |
|
|
- op: replace
|
|
path: /metadata/name
|
|
value: zoo-system
|
|
- op: add
|
|
path: /metadata/labels/app.kubernetes.io~1env
|
|
value: system
|
|
# NOTE: zoo-operator ClusterRoleBinding is NOT patched here — see base
|
|
# operator.yaml. It is cluster scoped and binds every per-env SA at once.
|
|
- target:
|
|
kind: ClusterRoleBinding
|
|
name: zoo-ingress
|
|
patch: |
|
|
- op: replace
|
|
path: /subjects/0/namespace
|
|
value: zoo-system
|