The image is one build serving every label and this file is the only
thing that distinguishes them, so without it the pod would present Lux on
a Zoo surface. It was mounted here and declared nowhere; four days in
ContainerCreating. Content is the wallet repo's own zoo overlay.
Claude-Session: https://claude.ai/code/session_018U2e2U4PPc8UaXYyeT9iNu
The pin rested on an arm64 pull error, "no match for platform in
manifest". On the amd64 node the same digest reports plainly: not found.
zoo-mainnet has no ghcr-luxfi secret, the one the values file names, so
the private repository answers every request as a miss on both nodes and
the earlier message says nothing about which platforms are published.
Provision the pull secret first; the manifest can then be read and the
architecture, if it needs constraining, constrained against evidence.
Co-authored-by: Hanzo Dev <[email protected]>
ghcr.io/zooai/computer publishes an amd64-only manifest. Scheduled on the
arm64 node the pull ends in "no match for platform in manifest" and the
Deployment never becomes ready.
Co-authored-by: Hanzo Dev <[email protected]>
Every Zoo workload is declared here, where zoo-fleet reads; these five had been
declared only in lux/universe's zoo-mainnet directory. Pins carried as they were
running, digest-pinned where the registry answered.
Co-authored-by: Hanzo Dev <[email protected]>
zooai/universe is zoo-build's one home; the duplicate declaration in
lux/universe had moved ahead to 0.1.9 while this one stayed at 0.1.7. Carry the
newer pin here before the duplicate is removed there, so nothing regresses.
Co-authored-by: Hanzo Dev <[email protected]>
zoo.vote was declared in lux/universe deploy/zoo-mainnet/zoo-vote.yaml. zoo-fleet's
generator reads THIS repository (hanzo/universe infra/k8s/hanzo-cd/project-zoo.yaml,
path deploy/*/*.yaml), so the workload had a declaration in a fleet that does not
serve it and none in the one that does. Declared here, deleted there.
The image is ghcr.io/zooai/vote, built by luxfi/vote's Image workflow: one source
builds lux.vote, zoo.vote and hanzo.vote, each into its own org's registry, so
this pod pulls with Zoo's credential and no other org's access can strand it. The
tenant is a build-arg and the host settles it again at runtime, which is why the
ingress names zoo.vote and www.zoo.vote — the app reads the hostname to know
which DAO it is, so what is routed here is part of its behaviour.
The tag is what the live Deployment runs, with a TODO to pin the first tag the
workflow prints. It could not be verified from a workstation, so it stays a tag
rather than becoming a digest that was guessed.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_015g5a1mCWuuSQTBZZywNnET
One A record at the hanzo-k8s ingress load balancer replaces the four GitHub
Pages records; the site is served from the Zoo prefix in object storage.
Co-authored-by: Hanzo Dev <[email protected]>
Two paths in this workflow wrote to zoo-k8s directly. `deploy-single` ran
kubectl apply -k over one namespace's overlay; `deploy-all` ran a script that
applied the CRDs, applied the whole overlay, and then hand-patched luxd flags
onto a StatefulSet the Rust operator renders -- a patch whose own comment
explains it persists because the operator does not re-render an existing pod
template.
That is a second writer for objects zoo-fleet declares from deploy/*/*.yaml,
and the disagreement is not quiet. Both stamp the pod template
(app.kubernetes.io/managed-by: hanzo-operator vs Helm), and a pod-template
label is not a field two controllers can differ about without a restart, so
each flip rolls the workload. It does not converge; it oscillates.
It also applied the WRONG THING. k8s/ is the hand-applied tree whose currency
deploy/README.md says outright is unknown -- some of it ahead of the cluster,
some behind, some describing objects that were deleted. A pipeline that applies
it on every push re-asserts that drift on a schedule.
What is left writes to git and tests: record the pin, open the PR, run E2E.
zoo-fleet reconciles from the merged file. The kubectl calls still in the E2E
job read pods, events and logs on failure, and write nothing.
The k8s/ tree stays, for what deploy/ deliberately does not cover -- CRDs,
namespaces, the operator's custom resources. `make k8s-apply env=<env>` is that
path, with a human behind it.
deploy-all and e2e-only meant the same thing once neither deployed, so the
modes are now the two acts that remain: pin and e2e.
Co-authored-by: Hanzo Dev <[email protected]>
26e7154 moved the tag from v1.36.58 to v1.36.68 to stop this file declaring a
version behind the fleet, and left the digest on v1.36.58's bytes. A reference
carrying both resolves BY DIGEST, so the declaration it left behind was the one
it set out to remove -- run .58, report .68 -- and now with the tag agreeing
with the running fleet, nothing was left to notice it. That is the failure mode
worth naming: a drifted pin does not fail, it succeeds at doing nothing.
Both halves are now v1.36.68 (ebd16ed5…), read from the registry in one answer.
The setup initContainer moves with them; it ran .58 by the same accident, and a
node whose init and main containers are different builds is a question nobody
should have to ask.
Consequence, on this file specifically: zood-mv is the only luxd StatefulSet on
RollingUpdate with podManagementPolicy Parallel, N=5 and alpha=4. A sync would
have restarted all five Zoo validators together onto ten-releases-old bytes,
and luxd waits for peers before it listens -- every node waiting for a node
that is also waiting.
Co-authored-by: Hanzo Dev <[email protected]>
A values file named a repository and a tag, and a tag is mutable. Under
pullPolicy: Always that means the fleet pulls whatever the name points at on the
morning it restarts; under IfNotPresent it means a node that pulled `:latest`
once keeps those bytes forever while its neighbour gets a newer build. Both
spellings of the same fault: the file does not say what runs.
41 of 50 image blocks now carry a digest beside the tag, resolved from the
registry, with pullPolicy: IfNotPresent throughout. The chart renders
repo:tag@digest and the reference resolves BY DIGEST -- the tag stays as the
name a human reads. 20 of those were floating (`latest`, `main`, `main-amd64`)
and now name specific bytes.
Four floating tags had a release naming the same digest, so they say it:
consul 1.20 -> 1.20.6, traefik v3.3 -> v3.3.7, zooai/docs latest -> 0.1.4. Each
was verified to be the same manifest, so this renames what is running and moves
nothing. gatus stays on `latest`: no v-tag names its bytes, so a version there
would have been a silent upgrade rather than a rename.
This is also what chart 0.1.13 requires. Its claimed list covers
ghcr.io/hanzoai/cloud:v*, and it refuses to RENDER one of those without a
digest -- so hanzo/cloud-api would have failed the render, which stops the sync
and leaves the running pods alone. Better than the alternative, but the pin is
the point.
Eight files name an image the registry does not have and are left alone rather
than guessed at; deploy/README.md lists them with what the registry does have.
Co-authored-by: Hanzo Dev <[email protected]>
deploy/applicationset.yaml contradicted the live zoo-fleet on every field that
decides where a sync lands: apiVersion apps.hanzo.ai/v1alpha1 against v1,
namespace cd against hanzo-cd, a github.com/zooai/universe mirror against
git.hanzo.ai/zoo/universe, and chart 0.1.0 against the version the fleet pins.
Applying it would not have adopted anything -- it would have stood up a second
generator over the same deploy/*/*.yaml, racing the first, with neither one
wrong on its own terms.
zoo-fleet is declared in hanzo/universe/infra/k8s/hanzo-cd/project-zoo.yaml,
behind an AppProject that fences Zoo's Applications to the Zoo cluster. One CD,
one standard, three orgs. A repo that also ships a generator is a second
opinion about a question that has one answer, so this copy goes.
The README's chart reference moves to 0.1.13 and says which file actually
decides it, so a helm template on a workstation renders what the cluster would
and a disagreement points at the ApplicationSet rather than at itself.
Co-authored-by: Hanzo Dev <[email protected]>
The zoo manifests, the compose file and the e2e helpers all composed
/v1/bc addresses against luxd, which answers under /v1/chain and no
longer serves the old segment.
A line that records an observation keeps its original spelling.
Co-authored-by: Hanzo Dev <[email protected]>
luxd serves chain APIs under one segment from constants.ChainAliasPrefix
(constants v1.6.4, node v1.36.181):
/v1/bc/<alias>/... -> /v1/chain/<alias>/...
/ext/bc/<alias>/... -> /v1/chain/<alias>/...
Observations keep their original addresses: a dated note or a recorded status
code is a fact about when it was written, not an instruction to follow.
The three chain-environment status pages ran an upstream build on a floating
`latest` tag, so what shipped changed whenever someone else cut a release, and
nothing here pinned it. That build also has no notion of a brand: the `BRAND=zoo`
env beside it was read by no code, which is why these pages carried no Zoo mark
at all.
ghcr.io/hanzoai/status v1.1.17 is the one status binary this estate runs. It
derives every mark from `brand` in status-config and proves each asset is present
at startup. GATUS_CONFIG_PATH is named explicitly because the mount is absolute
while the binary's default path is relative.
Chart 0.1.11 derives the pod's ServiceAccount from the rbac block, so stating
it a second time at the top level says nothing the declaration did not already
say -- and two places to write one name is how they came to disagree in the
first place.
The chart now fails the render if they ever do disagree, so this is safe to
carry as one line instead of two.
It reads from `prometheus.monitoring.svc:9090`, which does not exist on this
cluster -- so the panel has been a second lens onto nothing. Zoo's telemetry
goes to o11y over OTLZ, and one lens is the point.
It also carried `GF_SECURITY_ADMIN_PASSWORD: admin` beside
`GF_AUTH_ANONYMOUS_ENABLED: true`, in the clear, in a file in the repo. No
ingress pointed at it, so the exposure was in-cluster only, but "no route today"
is not a control.
Its 256Mi is what the otel-agent rollout on zoo-nodes-3mm2zq has been waiting
for: that node is at 99% of allocatable, the agent asks 192Mi, and a DaemonSet
rolling update stalls the whole set on one unschedulable pod.
rbac.serviceAccountName CREATES the ServiceAccount and binds nodes/stats and
nodes/proxy to it. The pod's identity is a SEPARATE key that the chart reads
from the top level, so setting only the rbac one produced exactly what it looks
like it would not: the RBAC was correct and unused, and the pods ran as
`default`.
kubeletstats therefore got 403 from every kubelet on this cluster since the
agent was introduced. hostmetrics kept working -- it reads /hostfs and needs no
API identity -- so the agent looked healthy and simply never produced any
k8s.node.* series.
Zoo had no observability at all. Not a competing stack to retire, an
absent one: four nodes with no host metrics, no logs, no traces, and
nothing reaching event.* — the cluster was simply dark, and the only
telemetry component in it was a Grafana with nothing of its own to read.
This is the agent hanzo and lux already run, and it differs from lux's by
the cluster identity and nothing else — same receivers, same pipelines,
same 300s intervals, same door. That sameness is the point: a third
cluster should cost a copy, not a design.
Reachability measured before writing it: a zoo pod dials 10.124.0.71:4319
open, so the shared DigitalOcean VPC carries this the same way it carries
lux.
imagePullSecrets is EMPTY on purpose. charts/app defaults it to
ghcr-secret, and on lux that credential answers 403 and turned a public
image into ImagePullBackOff across the whole fleet.
ghcr.io/hanzoai/otel-collector is public; sending a rejected credential
is worse than sending none.
No gateway here. The gateway exists to scrape kube-state-metrics and
chain exporters, and zoo runs neither — object-state metrics need
kube-state-metrics deployed first, which is a separate decision.
0.1.0 is the version the chart workflow's own comment names as never having
matched its source, so every Application claimed a chart nobody could
reproduce. 0.1.11 is what lux pins and what the registry actually holds.
ci.yml applied manifests with kubectl on every push, beside an ApplicationSet
that reconciles the same namespaces. Two delivery paths let the cluster
disagree with git while both report success; CD is the one that stays.
Claude-Session: https://claude.ai/code/session_01WAriF9qmzQJnBHfumZMxnd
Same check as the other universes: every values file renders against the
chart CD will use, at the version CD will use, so a file the chart can no
longer template fails on the push rather than at sync time.
CI does not deploy here either. The kustomize apply this repo's ci.yml still
carries is a second delivery path beside the ApplicationSet, and two paths
mean the cluster can disagree with git while both report success.
pkg.hanzo.ai is the estate's package host; a repo that resolves against
the public default is the one place a dependency can arrive from somewhere
nobody chose.
Read-only from here. Canonical: https://git.hanzo.ai/zoo/universe
This tip is an ancestor of canonical main, so no commit reachable from this
branch is unique to it.
Two ways origin/main disagreed with the live cluster, both of which only bite
on the next clean `apply -k`:
NETWORK_HEALTH_MIN_CONN_PEERS was absent. A single-validator devnet has no
peers, so luxd's default health gate (min 1 connected peer) holds the pod at
0/1 forever. The LIVE StatefulSet already runs
--network-health-min-conn-peers=0; trunk just had no way to reproduce it. The
key went missing when HTTP_ALLOWED_HOSTS was added to this same `config:`
block — the edit landed beside it and took its sibling with it.
tag was pinned to 0.2.0 while the live node runs ghcr.io/zooai/node:0.2.2
(the plugin-dir fix). Applying trunk downgraded the validator.
Neither showed up as an outage because nothing re-applies this overlay on a
schedule — no flux, no argo, no cronjob on zoo-k8s. It would have detonated
under whoever ran the next apply, which is the worst time to find it.
`kubectl kustomize k8s/zoo-devnet` now renders NETWORK_HEALTH_MIN_CONN_PEERS
"0" and tag 0.2.2, matching production.
Co-authored-by: Hanzo Dev <[email protected]>
The previous commit fixed the namespace half of the rpcEndpoint and then
overreached: it also swapped the path to /ext/bc/C/rpc because that is
what answered eth_chainId on the devnet validator. It answers 0x17872
(96370) — the Lux devnet C-Chain. C-Chain is Lux Network's primary EVM
and nothing else uses it, so pointing Zoo's explorer at it would have
made the Zoo devnet explorer index Lux blocks and call them Zoo's.
platform.getBlockchains on zoo-chain-0 lists only the Lux primary
chains (K/G/A/Z/Q/T/B/D/C/X) — there is no Zoo EVM there, and ZooChain
zoo-evm on devnet carries no blockchainId. So the honest state is: the
devnet Zoo EVM was never created, the explorer has nothing to index, and
no RPC path can change that. Keep the Zoo-EVM path and say so.
Co-authored-by: Hanzo Dev <[email protected]>
zoo-explorer-explorer had restarted 600+ times in both zoo-system and
zoo-devnet. The operator (<=0.1.3) rendered the ZooExplorer CR as
container args --rpc/--chain/--port, but ghcr.io/luxfi/explorer is a Go
binary whose stdlib `flag` set defines only -config/-data/-http/-mdns/
-version. Every start therefore ended in:
flag provided but not defined: -rpc
Usage of explorer: ...
i.e. exit 2 before the listener ever bound. operator 0.1.4
(src/controller.rs::reconcile_explorer) renders the CR into a chains.yaml
ConfigMap and passes --config=/etc/explorer/chains.yaml instead. 62864bd
already pinned 0.1.4 for the readiness fix, so only the rationale is
added here — but that pin is what stops the crash, and the comment should
say so or the next person re-floats the tag.
The crash was hiding two further defects. Both pods now come up Running
and are still useless without these:
* base rpcEndpoint pointed at zoo-chain-rpc.zoo.svc.cluster.local. There
is no `zoo` namespace on do-sfo3-zoo-k8s (`kubectl -n zoo get svc` ->
"No resources found"), so the indexer logs `no such host` every 30s and
has indexed zero blocks since it stopped crashing. `zoo` is not a
blockchain alias either — the EVM answers on /ext/bc/C/rpc. The CR is
applied into zoo-<env>, so the address is namespace-relative;
zoo-testnet/zoo-mainnet keep their own tracked-blockchainID overrides.
* the devnet validator ran without --http-allowed-hosts, so luxd's
localhost-only default 403s every in-cluster caller — the same
eth_chainId POST returns 0x17872 with `Host: localhost` and 403
Forbidden without it. spec.config renders as --kebab-case flags
(operator 0.1.3+), so HTTP_ALLOWED_HOSTS there is the one source of
truth; the unreferenced zoo-devnet/statefulset-patch.yaml is not.
Co-authored-by: Hanzo Dev <[email protected]>
zoo-system/zoo-operator sat 0/1 for two days. The probe was not wrong:
path /readyz and port 8081 both match the server main.rs binds. The
endpoint itself was.
Two faults compounded.
1. 0.1.3's /readyz returned 503 for any replica that did not hold the
lease. Readiness answers "can this pod serve?"; leadership answers
"who reconciles?". Braiding them means a healthy standby is NotReady
forever, so the Deployment never goes available. Measured on the live
pod before the fix — /healthz=200 while /readyz=503, process fine.
2. Something else was holding zoo-system's lease. A stale zoo-testnet
replica, floating on :main with no OPERATOR_NAMESPACE, had survived
since the last apply of this file. main.rs falls back to the literal
"zoo-system" when that env is unset, so it wrote zoo-system's lease
from zoo-testnet and renewed it indefinitely. git has said replicas: 0
for that overlay all along (zoo-testnet/network-patch.yaml) — the
cluster had simply drifted, and the floating tag hid it.
So: pin the immutable semver, and record that the fieldRef and the
scale-to-0 overlays are load-bearing invariants rather than tidiness.
0.1.4 (zooai/operator 98be7d7) carries the /readyz change.
This file still tracked :main because the 0.1.3 pin was only ever made on
a feature branch; trunk kept floating and would have reverted the cluster
on the next clean apply.
Proof, zoo-system after rollout — two replicas, one leader, both Ready,
which 0.1.3 could not do by construction:
NAME READY UP-TO-DATE AVAILABLE
zoo-operator 2/2 2 2
lease holder = zoo-operator-7dd698654-t6v4q (10.160.16.31)
10.160.17.253 /readyz=200 /healthz=200 <- NOT the leader
10.160.16.31 /readyz=200 /healthz=200
Co-authored-by: Hanzo Dev <[email protected]>
* beluga: wire L3 to run zooai/node (replace luxd stopgap)
Make the Beluga L3 (chainId 420420) a real thinking-chain instead of an
inert luxd subnet stub. The zoo-chain validators tracked beluga under
ghcr.io/luxfi/node (luxd) — a bare subnet runtime with no Zoo EVM and no
AI precompiles, so beluga could never instantiate (blockchainID empty).
Switch the zoo-chain ZooNetwork image to ghcr.io/zooai/node (zood), which
self-installs the Zoo EVM (AI/FHE/PQ/ZK precompiles, incl. the 0x0303
deterministic in-consensus inference precompile and the 0x0304 aivmbridge
that settles Tier-2 Proof-of-Thought to the Lux A-Chain), Zoo DEX and Zoo
FHE VM plugins at startup.
- base/network.yaml + zoo-{devnet,testnet,mainnet}/network-patch.yaml:
image ghcr.io/luxfi/node -> ghcr.io/zooai/node.
- zoo-{devnet,testnet,mainnet}/statefulset-patch.yaml: command ./luxd ->
zood (zood wraps luxfi/node, identical config.BuildFlagSet, so all flags
apply verbatim). Beluga's blockchainID is appended to --track-chains
once the CreateChainTx mints it on the Zoo primary network.
- base/beluga.yaml: enrich the ZooChain CR with vmId (Zoo EVM
2n2njofjYvece8gZWCNnc1mqkcqfW6kbrhPRZPVzwxrSQrQ4gE) and
genesisConfigMap (beluga-genesis), per the operator ZooChainSpec.
- base/kustomization.yaml: configMapGenerator beluga-genesis from
k8s/base/beluga/{genesis.json,config.json} (stable name, no hash).
- k8s/base/beluga/: deployed copy of the canonical genesis+config
(evmChainId 420420, BLG gas, precompile-upgrade activations at ts 0).
Image tags are floating `latest` only on devnet; testnet/mainnet pin the
immutable CI semver. The live rollout (restarts the L2 validators) is a
gated prod step performed after the ghcr.io/zooai/node image is verified.
* beluga: pin zood image to immutable sha-371ea17-amd64-node
ghcr.io/zooai/node built CGO-free (hermetic vendored build, digest
sha256:12e8244d…) and published. Pin the operator ZooChain CR image off the
floating 'latest' to the immutable build SHA per prod policy.
* beluga: pin devnet+testnet zood image to sha-371ea17-amd64-node (consistency)
* beluga(devnet): image is operator-derived from the ZooNetwork CR, not the patch
Per 'one and only one way, operator-derived': the zood image comes solely from
k8s/base/network.yaml (the ZooNetwork CR); the operator renders it. The patch
asserts only the luxd CLI flags the operator wrongly passes as ZOO_* env.
* beluga: pin zood image to clean semver 0.2.0 (not the sha hash)
Proper semver for all image pins — ghcr.io/zooai/node:0.2.0 (digest 12e8244d,
same build), not sha-371ea17-amd64-node. The ZooNetwork CR is the single source;
the operator renders it onto the StatefulSet.
- chain.yaml: type l1->l2, parent null->lux, validators 21->0. Zoo's
blocks are validated by the existing Lux Primary Network validator set
(platform.validatedBy(zoo bc 4HBM17cz...) -> netID 1111...LpoYY); the
networkID field is Zoo's OWN EVM chainID (200200), not a sovereign
primary networkID. Zoo still has its own EVM, distinct from C-Chain.
- configs/genesis/localnet/genesis.json: chainId 200202 -> 200203
(was a duplicate of devnet; localnet is 200203 per chain.yaml).
Co-authored-by: zeekay <[email protected]>
The Rust zoo-operator configures each validator by passing the ZooNetwork CR
config map as ZOO_* env vars. luxd (ghcr.io/luxfi/node, CMD ["./luxd"]) does
NOT read http-host / bootstrap / track-chains from ZOO_* env — only from real
CLI flags. So luxd bound its HTTP/health API to the default 127.0.0.1: the
kubelet liveness probe targets the pod IP, never reaches /ext/health, and
SIGTERMs the node ~90s after boot — an endless restart loop that kept testnet
and devnet validators from ever stabilizing (and would have hit mainnet too).
Inject the flags directly onto the operator-rendered StatefulSet via a
per-overlay statefulset-patch.yaml, applied by deploy-all.sh right after the
kustomize apply (the operator owns the StatefulSet, so it is not in the
kustomize set and cannot be patched through the overlay; the operator does not
re-render an existing pod template, so the patch persists).
- --http-host=0.0.0.0 + --http-allowed-hosts=* so the probe can reach health
- --bootstrap-ips/--bootstrap-ids/--network-id/--sybil-protection-enabled as
real flags (the ZOO_* env equivalents were silently ignored: nodes booted
with 'Adding bootstrap nodes ... count=0')
- devnet: drop the bogus --track-chains=zoo (and TRACK_CHAINS env) — luxd
rejects the bare 'zoo' alias; the devnet Zoo EVM chain isn't created yet
zoo-mainnet zoo-chain-0 was stuck Pending (FailedScheduling: Insufficient
memory on 0/5 nodes). Its validator memoryRequest was an anomalous 8Gi while
the zoo-nodes pool (s-4vcpu-8gb-amd) has only ~6.26Gi allocatable per node, so
a single 8Gi-request pod could never schedule. cpuRequest of 2 also tripped
Insufficient cpu on a packed node.
Lower requests to match zoo-testnet (memoryRequest 4Gi, cpuRequest 1) so all 5
replicas can co-schedule. Limits stay generous (memoryLimit 16Gi, cpuLimit 8)
for mainnet headroom.
The ghcr.io/zooai/operator:main binary (crate zoo-operator v0.1.2) watches
ZooChain via Api::all and deserializes each object into ZooChainSpec
(src/crd.rs): chainId (required u64) + optional blockchainId/vmId/
genesisConfigMap/networkRef. The installed CRD uses
x-kubernetes-preserve-unknown-fields, so enforcement is client-side in the
operator.
The old beluga CR carried a rich legacy schema (type/parent/vmType/
evmChainId/coin/coinName/genesis.secretRef/config/precompiles/resources)
with NO chainId. serde failed to decode it → the ZooChain list-watch's
InitialList failed → the whole ZooChain controller stalled, blocking chain
provisioning across all namespaces.
Migrate beluga to the canonical shape, mirroring the working zoo-evm CR:
evmChainId: 420420 -> chainId: 420420
parent: zoo-chain -> networkRef: zoo-chain
Drop fields the operator does not consume (it only stamps status; the L3 is
tracked via the validators' TRACK_CHAINS env, and EVM/precompile/genesis
params live in genesis.json + the VM build, not this tracking CR).
The operator Deployment pinned ghcr.io/zooai/operator:v0.1.1, but that tag (and
:main-amd64) return NotFound from GHCR — verified in-cluster: the v0.1.1
ReplicaSet went straight to ImagePullBackOff. The only tag zooai/operator CI
has published is :main. Pin to :main (imagePullPolicy: Always) so a fresh
apply actually starts the operator; switch to an immutable semver once CI cuts
one.
ghcr.io/luxfi/node:v1.27.4 is NotFound in GHCR (would ImagePullBackOff).
v1.30.40 is the published, proven tag the Lux production cluster
(do-sfo3-lux-k8s, lux-mainnet/testnet) currently runs 5/5 validators on.
Zoo is a Lux L2 and uses the upstream OSS node image directly.
The zoo-operator readiness probe (GET :8081/readyz) returned 503 for ~9 days
(88k failures) on every replica and the beluga/zoo-chain CRs never reconciled
(empty PHASE/CHAINID).
Root cause — RBAC subject collision:
ClusterRoleBinding/zoo-operator is cluster-scoped, but each env overlay
rendered the SAME binding name and JSON6902-patched its single subject's
namespace to its own env. ClusterRoleBindings are not namespaced, so the four
overlays collided on apply and last-writer-won — only zoo-mainnet's SA ended
up bound. The operator runs with --namespace "" (all namespaces) behind ONE
cluster-global leader-election lease (zoo-operator-leader in zoo-system); the
lease winner was the zoo-devnet pod, whose SA was NOT a subject → 403 on every
zoo.network list/watch → controllers' InitialListFailed → leader never healthy
→ readyz 503 on all pods → 0/1 Deployments.
Fix: define the binding once in base/operator.yaml with ALL four per-env
operator SAs as subjects (one ClusterRole, one binding, every identity —
order-independent), and drop the per-overlay subject-rewrite patches that
caused the collision. The operator's /readyz gates on leadership, so once the
lease winner can list/watch cluster-wide it goes 1/1 and reconciles every
namespace.
Also align zoo-devnet/zoo-system ZooNetwork CRs to the v0.1.1 operator's
ZooNetworkSpec schema (validators:<int> + image{} + resources{} + config{}),
consistent with the mainnet/testnet patches, so the whole tree reconciles
under the pinned ghcr.io/zooai/operator:v0.1.1.
* dns(zoo.ngo): proxy web3.zoo.ngo surfaces through CloudFlare
web3.zoo.ngo, api.web3.zoo.ngo and ws.web3.zoo.ngo are the API and web
hosts behind the zoo.cloud brand surface (zoo.cloud's www CNAME targets
web3.zoo.ngo). They were unproxied A records pointing straight at the
hanzo-k8s LB (134.199.136.205), exposing the origin and skipping
CloudFlare's edge TLS, caching and DDoS protection.
Set proxied: true to match the now-live CloudFlare state. sync.sh skips
proxied records, so this records source-of-truth; the flip was applied
via the CF API. Zone SSL mode is full and the origin serves a valid
Let's Encrypt cert — verified api/v1/chains=200 and api/v1/auth/me=401
through the proxy.
* dns(zoo.ngo): keep api.web3/ws.web3 unproxied — Universal SSL gap
CloudFlare Universal SSL on zoo.ngo covers zoo.ngo + *.zoo.ngo but NOT
the second-level *.web3.zoo.ngo. Proxying api.web3.zoo.ngo and
ws.web3.zoo.ngo breaks the edge TLS handshake because CF has no edge
cert for those names.
Only the apex web3.zoo.ngo (matched by *.zoo.ngo) stays proxied. api.web3
and ws.web3 serve direct from the hanzo-k8s LB with their valid Let's
Encrypt origin cert. Re-proxying requires an Advanced Certificate (ACM)
for *.web3.zoo.ngo, not entitled on this zone (CF error 1450).
---------
Co-authored-by: zeekay <[email protected]>
Brings zoo to parity with lux: the SAME luxd-boot.sh (byte-identical to
luxfi/universe k8s/boot) drives every network, and native ZAP replication
(PQ-encrypted, restore-on-boot) replaces the boot-time admin_importChain (RLP)
path.
- k8s/boot/: shared entrypoint + mainnet/testnet/devnet env (ids 1/2/3,
zoo EVM chainIds 200200/200201/200202, S3 path zoo/<net>/zaprepl,
endpoint s3.zoo-system). Script is byte-identical to luxfi/universe.
- k8s/zoo-<net>/snapshot-native/: replicate-s3-creds + replicate-age secrets
(fill from KMS) + ZooNetwork replication merge-patch. CRD already allows
unknown fields, so spec.replication is accepted today.
- zoo-mainnet/luxd-rlp-import.yaml: RETIRED banner; kept as the provenance
record of the one-time chain-200200 seed (now in the snapshot stream).
- SNAPSHOT-NATIVE-ROLLOUT.md: per-network apply order (devnet→testnet→mainnet).
Requires build-zoo rebuilt on database >= v1.20.3 and zooai operator
replication-env parity (mirror of luxfi/operator); until then the boot script
sets REPLICATE_* itself so replication works without operator changes.
Co-authored-by: Hanzo Dev <[email protected]>
Mirrors Lux's api.lux-test.network / api.lux-dev.network convention —
hyphenated env in the TLD, not a subdomain of zoo.network. All org
tooling reads the same shape.
- zoo-testnet/ingressroute.yaml: api.test.zoo.network →
api.zoo-test.network (+ explore + api-explore variants).
- zoo-testnet/network-patch.yaml: ZooGateway.spec.hosts updated.
- zoo-devnet/ingressroute.yaml: api.dev.zoo.network →
api.zoo-dev.network (+ explore variants).
CF zones for zoo-test.network + zoo-dev.network already exist (DNS
resolves through Cloudflare). DNS A records still need to be created
inside those zones pointing at the Zoo cluster ingress LB — separate
followup.
Operator topology fix:
- base/operator.yaml: pin ghcr.io/zooai/operator:v0.1.1 (was :main floating).
Add OPERATOR_NAMESPACE env via fieldRef: metadata.namespace so the
leader-election Lease lock is namespace-scoped, not racing in a single
hardcoded 'zoo-system' literal. imagePullPolicy: Always → IfNotPresent
(semver tag is immutable).
- zoo-system/patches.yaml: zoo-operator replicas → 2 (HA, leader-election
picks one). Drop the stale zoo-explorer raw-Deployment patch (operator
emits it now via ZooExplorer CR).
- zoo-{mainnet,testnet,devnet}/network-patch.yaml: scale per-env
zoo-operator → 0. Single zoo-system operator watches every namespace
(--namespace '' default); per-env operators were thrashing the same
Lease and never becoming leader.
CR migration (away from raw manifests):
- base/network.yaml: full rewrite as ZooNetwork + ZooChain + ZooExplorer
+ ZooGateway base CRs (CRDs already installed: zoonetworks /
zoochains / zooexplorers / zoogateways).
- base/explorer.yaml: emptied — ZooExplorer CR replaces the raw
Deployment; operator renders zoo-explorer-explorer Deployment + Svc.
- base/gateway.yaml: Service-only — ZooGateway moved to network.yaml.
- base/ingress.yaml: kind: Ingress → 4 kind: IngressRoute
(hanzo.ai/v1alpha1). Traefik proxy Deployment kept (it IS the
hanzoai/ingress L7 proxy on this cluster).
- zoo-{mainnet,testnet}/network-patch.yaml: env-specific patches —
validators (mainnet=5, testnet=3), bootstrap IPs+IDs from live Lux
mainnet+testnet svcs, TRACK_CHAINS = Zoo blockchainID per env.
- zoo-{testnet,devnet}/ingressroute.yaml: new — api.test.zoo.network,
explore.test.zoo.network, api-explore.test.zoo.network (and dev.*
variants).
- zoo-{system,testnet,devnet}/kustomization.yaml: include new
ingressroute.yaml resource + strip mainnet-only IngressRoutes via
$patch: delete pattern.
- zoo-mainnet/zoo-industries.yaml: kind: Ingress → IngressRoute.
Blockers surfaced (separate followups):
1. Operator does NOT translate `spec.config` env keys to lqd CLI
flags. Rendered StatefulSet sets ZOO_BOOTSTRAP_IPS etc., but
luxd reads LUXD_*-prefixed env. Operator controller.rs needs to
render as CLI args. BLOCKS validators joining Lux primary.
2. hanzo.ai/v1alpha1 IngressRoute CRDs not registered on
do-sfo3-zoo-k8s — current ingress is traefik:v3.3 with
traefik.io/v1alpha1 native. Either install hanzoai/ingress CRDs
OR migrate IngressRoutes to traefik.io group.
3. ZooNetwork operator doesn't emit per-pod LoadBalancer Services
needed for validators to be reachable as external bootstrap peers.
4. Validator staking keys: operator only takes secret_prefix; doesn't
derive from a KMS mnemonic. Need 5 Secrets per env OR operator
extension.
Mirrors ~/work/lux/state/chains/zoo-testnet/genesis.json into the
deployment mirror configs/genesis/testnet/genesis.json. Same pattern
as 59f1220 (zoo mainnet sync) — fixes deployment-mirror drift so that
zoo-testnet's 85-block RLP archive imports cleanly.
Source RLP archive at
~/work/lux/state/rlp/zoo-testnet/zoo-testnet-200201.rlp
(sha256 739c96ba8e8173e5b9133802928cb81c7c4b30f9ee8e50bbae734f57eabc4055,
159,648 bytes, blocks 0..84) was exported 2026-06-05 from the preserved
pebbledb at ~/work/lux/state/pebbledb/zoo-testnet via the namespace-
aware scripts/export-rlp.go.
Files now byte-identical (verified by cmp).
Today's bootstrap-chain CreateChainTx (#168, 2026-06-04) produced a
zoo mainnet chain with block-0 hash `0x88dd3a53…`, which orphans the
Zoo mainnet RLP archive (799 blocks, recorded parent hash
`0x7c548af4…`). admin_importChain returns "block 1 parent mismatch".
Root cause: bootstrap-chain consumed the cc7d3d7 form of zoo-mainnet/
genesis.json (969 bytes), which is a SUBSET of the empirical canonical.
The cc7d3d7 commit was labeled "RLP-matching pristine form" but
omitted the explicit block-0 header fields (parentHash, mixHash,
nonce, number, gasUsed, extraData, coinbase) and the
precompile-defer config fields (quasarTimestamp, fortunaTimestamp,
graniteTimestamp = 253399622400). luxd's defaults for these fields
produce a DIFFERENT block-0 hash than the empirical form.
Fix: sync this deployment-side genesis.json to the empirical canonical
at ~/work/lux/state/chains/zoo-mainnet/genesis.json (1449 bytes, sha256
f8482568e7a10e882c6bd0248549ad75257edec79a83f659f27e1a34e71b59fa).
Now bootstrap-chain will produce a chain with block-0 hash
0x7c548af47de27560779ccc67dda32a540944accc71dac3343da3b9cd18f14933
which matches the RLP archive's recorded parent.
Add LLM.md sibling at this path documenting:
- state-repo as source-of-truth (canonical) vs zoo/universe as
deployment mirror
- the regression history (cc7d3d7 incomplete form vs empirical)
- the trade-off when re-firing CreateChainTx after the wrong-genesis
chain has already committed (new chain ID required; aliases must
re-sync)
This unblocks the zoo mainnet RLP import — a fresh CreateChainTx will
spawn a chain whose block-0 matches the RLP, and the script at
~/work/lux/universe/k8s/lux-mainnet/scripts/zoo-rlp-reimport.sh can
fire cleanly. The orphaned oxDk6jU7… chain stays in P-state but is
effectively unused; the chain-aliases map updates to the new ID.
Followup: testnet+devnet zoo genesis files and hanzo/pars/spc mainnet
genesis files may have the same regression. Audit + sync in a sibling
commit.
Zoo L1 genesis blobs now live in this repo, moved out of
luxfi/genesis. Files are byte-identical to the luxfi/genesis source
at the time of the move (verified via cmp).
configs/genesis/mainnet/genesis.json — Zoo L1 mainnet (chainId 200200)
configs/genesis/mainnet/upgrade.json — Zoo L1 mainnet activation
configs/genesis/testnet/genesis.json — Zoo L1 testnet (chainId 200201)
configs/genesis/devnet/genesis.json — Zoo L1 devnet
luxfi/genesis carries MOVED.txt tombstones at the old
configs/zoo-{mainnet,testnet,devnet}/ paths and a no-shadow test
guard against accidental re-introduction. Each brand owns its own L1
genesis blob; luxfi/genesis only owns Lux primary-network genesis.
Coordination with the Lux universe DX cleanup — Cloudflare zone YAMLs
are pure brand assets and belong with their owning universe. Moved
2026-06-03 from lux/universe/k8s/dns/{zoo-exchange,zoo-network,zoo-ngo}.yaml.
k8s/cf-zones/sync.sh is the per-brand drift-detect-and-PATCH script
(same logic as the legacy lux/universe scripts/dns-sync.sh, scoped to
this directory). Dry-run by default; set CF_API_TOKEN and pass --apply
to push changes.
Single-zone usage:
./k8s/cf-zones/sync.sh --zone zoo-ngo
Coordination with the Lux universe DX cleanup — Zoo brand sites belong
in Zoo universe, not Lux universe. The corresponding files in
~/work/lux/universe/k8s/zoo-mainnet/ will be removed in the matched
Lux universe commit.
zoo-bot.yaml and zoo-build.yaml each declare a Deployment + Service +
hanzo.ai/v1alpha1 IngressRoute. The previous lux/universe/k8s/zoo-mainnet/
directory had no kustomization.yaml, so these were applied imperatively
to lux-k8s; folding them into the zoo/universe kustomization brings them
under the same brand-bounded reconciliation path as zoo-industries.
Live deployments preserved:
lux-k8s zoo-mainnet/zoo-bot 1/1 33d
lux-k8s zoo-mainnet/zoo-build 1/1 34d
lux-k8s zoo-mainnet/zoo-industries 1/1 33d (already in repo)
zoo-build image still on ghcr.io/luxfi/build-zoo:0.1.7 — follow-up to
migrate the build pipeline to ghcr.io/zooai/* per the canonical
container registry rule.
zooai now served by host-level arcd daemons (dbc Mac + evo WSL +
evo Windows) alongside hanzoai, luxfi, parsdao, zenlm. See
hanzoai/universe/infra/runners/README.md for the fleet topology.
Refactor plan in the docstring is done — luxfi/operator v0.7.0
content-hash-addressed tenantImports lives in
lux/universe/k8s/lux-mainnet/network.yaml. This file remains as the
authoritative record of Zoo's RLP source spec (URL/SHA256/blockchainID).
On future spec edits: update both this file and the matching
tenantImports[] entry; the operator's content hash flips and re-fires
the import automatically.
No more inline-shell-block coordination with the lux team. No more
sentinel version-bumping.
Zoo is an L2 on the Lux primary network — rides Lux's validator
set via primary-network staking and does not own a sovereign
networkID. Zoo's identity surfaces through its EVM chainID + the
brand fields.
Model:
- network.parent = lux → inherits Lux's networkID per env
- chains[*].evmChainIds.{mainnet,testnet,devnet} = {200200, 200201, 200202}
- Each chain gets its own chainID via CreateChainTx on Lux's P-chain
Per-env chainIDs sourced from
~/work/lux/genesis/configs/zoo-{env}/genesis.json — single source
of truth, no duplication in this manifest.
Lux primary L1 declares Zoo as an L2 child in its own chain.yaml.
Zoo is an L2 on the Lux primary network — rides Lux's validator
set via primary-network staking and does not own a sovereign
networkID. Zoo's identity surfaces through its EVM chainID + the
brand fields.
Model:
- network.parent = lux → inherits Lux's networkID per env
- chains[*].evmChainIds.{mainnet,testnet,devnet} = {200200, 200201, 200202}
- Each chain gets its own chainID via CreateChainTx on Lux's P-chain
Per-env chainIDs sourced from
~/work/lux/genesis/configs/zoo-{env}/genesis.json — single source
of truth, no duplication in this manifest.
Lux primary L1 declares Zoo as an L2 child in its own chain.yaml.
Until luxfi/operator grows LuxNetwork.spec.tenantImports[], the actual
RLP import block lives inline in lux/universe luxd-startup.yaml. This
file is the canonical record of Zoo's contribution to that file:
800 blocks from git df8c246 (sha256 0850841a...), imported to chainID
200200 on all 5 lux-mainnet luxd validators 2026-05-25.
Tip: block 799, hash 0xd6f92941bb...; verifiable at
https://api.lux.network/ext/bc/zoo/rpc.
Deploys via hanzoai/spa:1.2.0 runtime + zooai/zoo-industries content
image. Operator (kind: SPA, v0.3.2+) reconciles to Deployment +
Service + Ingress on zoo.industries. Replaces the prior Cloudflare
Pages deploy path (cf-pages.yml dropped from zoo.industries repo).
After ghcr.io/zooai/zoo-industries publishes its first image:
kubectl --context=do-sfo3-zoo-k8s apply -k k8s/zoo-mainnet/
Documents how zoo/app shims hanzo/desktop, the BrandConfig contract,
chain IDs (zoo 200200/200201/200202, hanzo 36900/36901), the sidecar
list (Ollama dropped), and the read-order for new contributors.
Cleans up registry drift across the live cluster:
- base/operator.yaml: 0.1.0 (non-existent in GHCR) -> :main + TODO semver
- base/bridge.yaml: 0.1.0 (non-existent in GHCR) -> :main + TODO semver
- base/explorer.yaml: blockscout :latest -> :7.0.2.commit.900ef697 (release SHA)
- zoo-system/patches.yaml: remove redundant operator image override
Adds .github/workflows/service-released.yml for auto-bump on
repository_dispatch from zoo service repos. Validates payload pins
semver (rejects :main/:latest) — forcing function for service CIs to
ship semver tags before auto-bump can land.
LLM.md: registry policy, image pin status table, auto-bump flow,
mpc-node orphan documentation, live cluster drift snapshot.
- zoo-build-linux-amd64: DOKS (hanzo-k8s), max 10
- zoo-build-linux-arm64: GKE (apps-gke), max 10
- Same clusters as hanzoai runners, separate GitHub App
zoo.exchange, zoo.ngo, zoo.fund, zoo.vote, zoo.foundation,
zoo.lab, zoo.network, zoo.voyage are all standalone repos.
They live at ~/work/zoo/{name} not embedded in universe.
Universe is the deployment control plane (K8s, CI/CD, E2E),
not a monorepo for application code.
hanzo-dev
