fix(zoo-operator): pin 0.1.4 — readiness is process health, not leadership

zoo-system/zoo-operator sat 0/1 for two days. The probe was not wrong:
path /readyz and port 8081 both match the server main.rs binds. The
endpoint itself was.

Two faults compounded.

1. 0.1.3's /readyz returned 503 for any replica that did not hold the
   lease. Readiness answers "can this pod serve?"; leadership answers
   "who reconciles?". Braiding them means a healthy standby is NotReady
   forever, so the Deployment never goes available. Measured on the live
   pod before the fix — /healthz=200 while /readyz=503, process fine.

2. Something else was holding zoo-system's lease. A stale zoo-testnet
   replica, floating on :main with no OPERATOR_NAMESPACE, had survived
   since the last apply of this file. main.rs falls back to the literal
   "zoo-system" when that env is unset, so it wrote zoo-system's lease
   from zoo-testnet and renewed it indefinitely. git has said replicas: 0
   for that overlay all along (zoo-testnet/network-patch.yaml) — the
   cluster had simply drifted, and the floating tag hid it.

So: pin the immutable semver, and record that the fieldRef and the
scale-to-0 overlays are load-bearing invariants rather than tidiness.
0.1.4 (zooai/operator 98be7d7) carries the /readyz change.

This file still tracked :main because the 0.1.3 pin was only ever made on
a feature branch; trunk kept floating and would have reverted the cluster
on the next clean apply.

Proof, zoo-system after rollout — two replicas, one leader, both Ready,
which 0.1.3 could not do by construction:

  NAME           READY   UP-TO-DATE   AVAILABLE
  zoo-operator   2/2     2            2

  lease holder = zoo-operator-7dd698654-t6v4q   (10.160.16.31)
  10.160.17.253  /readyz=200  /healthz=200      <- NOT the leader
  10.160.16.31   /readyz=200  /healthz=200

Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
Hanzo Dev
2026-07-27 22:38:13 -07:00
parent 0ae481730a
commit 62864bdba8
+16 -6
View File
@@ -6,6 +6,12 @@
#
# OPERATOR_NAMESPACE comes from the pod's own namespace (fieldRef) so
# the lease lives in zoo-system — see src/main.rs line 106 + src/leader.rs.
# That fieldRef is load-bearing: main.rs falls back to the literal
# "zoo-system", so a replica running WITHOUT this env writes zoo-system's
# lease from whatever namespace it is in. That is exactly how a stale
# zoo-testnet pod (no env, floating :main) held zoo-system's lease and
# starved the real operator. Scaling the overlays to 0 is the invariant,
# not a tidiness preference — apply them or a stray replica survives.
apiVersion: apps/v1
kind: Deployment
metadata:
@@ -30,12 +36,16 @@ spec:
- name: hanzo
containers:
- name: operator
# :main is the only tag zooai/operator CI has published — :v0.1.1 and
# :main-amd64 are NotFound in GHCR (verified in-cluster), so pinning
# them makes the Deployment ImagePullBackOff. Track the floating tag
# until CI cuts an immutable semver, then pin x.y.z+.
image: ghcr.io/zooai/operator:main
imagePullPolicy: Always
# Pinned to the immutable semver — the floating :main this used to
# track is what let a stale replica outlive its manifest (see above).
# 0.1.3 renders the zoo-chain StatefulSet with luxd CLI flags derived
# from the ZooNetwork CR (earlier images emitted only ZOO_* env, which
# luxd's viper EnvPrefix ignores). 0.1.4 unbraids readiness from
# leadership: /readyz used to 503 on every non-leader, so a healthy
# standby was NotReady forever and its Deployment never went available
# — zoo-system sat 0/1 for two days. Bump x.y.z+; never float.
image: ghcr.io/zooai/operator:0.1.4
imagePullPolicy: IfNotPresent
args: ["--namespace", ""]
env:
- name: OPERATOR_NAMESPACE