fix(zoo-operator): pin 0.1.4 — readiness is process health, not leadership
zoo-system/zoo-operator sat 0/1 for two days. The probe was not wrong: path /readyz and port 8081 both match the server main.rs binds. The endpoint itself was. Two faults compounded. 1. 0.1.3's /readyz returned 503 for any replica that did not hold the lease. Readiness answers "can this pod serve?"; leadership answers "who reconciles?". Braiding them means a healthy standby is NotReady forever, so the Deployment never goes available. Measured on the live pod before the fix — /healthz=200 while /readyz=503, process fine. 2. Something else was holding zoo-system's lease. A stale zoo-testnet replica, floating on :main with no OPERATOR_NAMESPACE, had survived since the last apply of this file. main.rs falls back to the literal "zoo-system" when that env is unset, so it wrote zoo-system's lease from zoo-testnet and renewed it indefinitely. git has said replicas: 0 for that overlay all along (zoo-testnet/network-patch.yaml) — the cluster had simply drifted, and the floating tag hid it. So: pin the immutable semver, and record that the fieldRef and the scale-to-0 overlays are load-bearing invariants rather than tidiness. 0.1.4 (zooai/operator 98be7d7) carries the /readyz change. This file still tracked :main because the 0.1.3 pin was only ever made on a feature branch; trunk kept floating and would have reverted the cluster on the next clean apply. Proof, zoo-system after rollout — two replicas, one leader, both Ready, which 0.1.3 could not do by construction: NAME READY UP-TO-DATE AVAILABLE zoo-operator 2/2 2 2 lease holder = zoo-operator-7dd698654-t6v4q (10.160.16.31) 10.160.17.253 /readyz=200 /healthz=200 <- NOT the leader 10.160.16.31 /readyz=200 /healthz=200 Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
+16
-6
@@ -6,6 +6,12 @@
|
||||
#
|
||||
# OPERATOR_NAMESPACE comes from the pod's own namespace (fieldRef) so
|
||||
# the lease lives in zoo-system — see src/main.rs line 106 + src/leader.rs.
|
||||
# That fieldRef is load-bearing: main.rs falls back to the literal
|
||||
# "zoo-system", so a replica running WITHOUT this env writes zoo-system's
|
||||
# lease from whatever namespace it is in. That is exactly how a stale
|
||||
# zoo-testnet pod (no env, floating :main) held zoo-system's lease and
|
||||
# starved the real operator. Scaling the overlays to 0 is the invariant,
|
||||
# not a tidiness preference — apply them or a stray replica survives.
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
@@ -30,12 +36,16 @@ spec:
|
||||
- name: hanzo
|
||||
containers:
|
||||
- name: operator
|
||||
# :main is the only tag zooai/operator CI has published — :v0.1.1 and
|
||||
# :main-amd64 are NotFound in GHCR (verified in-cluster), so pinning
|
||||
# them makes the Deployment ImagePullBackOff. Track the floating tag
|
||||
# until CI cuts an immutable semver, then pin x.y.z+.
|
||||
image: ghcr.io/zooai/operator:main
|
||||
imagePullPolicy: Always
|
||||
# Pinned to the immutable semver — the floating :main this used to
|
||||
# track is what let a stale replica outlive its manifest (see above).
|
||||
# 0.1.3 renders the zoo-chain StatefulSet with luxd CLI flags derived
|
||||
# from the ZooNetwork CR (earlier images emitted only ZOO_* env, which
|
||||
# luxd's viper EnvPrefix ignores). 0.1.4 unbraids readiness from
|
||||
# leadership: /readyz used to 503 on every non-leader, so a healthy
|
||||
# standby was NotReady forever and its Deployment never went available
|
||||
# — zoo-system sat 0/1 for two days. Bump x.y.z+; never float.
|
||||
image: ghcr.io/zooai/operator:0.1.4
|
||||
imagePullPolicy: IfNotPresent
|
||||
args: ["--namespace", ""]
|
||||
env:
|
||||
- name: OPERATOR_NAMESPACE
|
||||
|
||||
Reference in New Issue
Block a user