Each of these declares replicas: 0 -- no pod, no endpoint, no caller, and no
record of when it was parked. A declaration that runs nothing is not a
deployment, it is a note nobody can date, and it keeps a slot in every fleet
listing and every sync. Git history holds them if one was paused rather than
retired.
Co-authored-by: Hanzo Dev <[email protected]>
Both were scraped off do-sfo3-zoo-k8s, the DigitalOcean cluster that no longer
exists. zoo-zips runs the only nginx left in the fleet and answers 669 bytes of
meta-refresh, while the real zips.zoo.ngo serves 96 KB from elsewhere.
work-board duplicates the one behind work.hanzo.network; work.zoo.network does
not resolve.
Co-authored-by: Hanzo Dev <[email protected]>
status.zoo.network is served by zoo/status, which is healthy. This copy in
zoo-mainnet has no route, and it has been unready since the migration waiting
on a status-config ConfigMap that no file produces.
Co-authored-by: Hanzo Dev <[email protected]>
precompile v0.19.22 renamed the eleven keys that used Verify, Threshold or
Encapsulate to <name>Config, so all fifty-six answer to one convention - including
the LP-4200 unified PQCrypto block, which had been carrying four across its eight
addresses.
This config names eleven of them and moves with the release. An upgrade.json
holding an old key names a precompile that does not exist, and luxd refuses the
activation rather than ignoring it.
Co-authored-by: Hanzo Dev <[email protected]>
Zoo mainnet and testnet activated pqcryptoConfig. No precompile registers that
key - not in the suite, not in the EVM - and luxd refuses an activation for a
configKey it has never seen, so this line alone would stop a Zoo node at genesis.
It is the old unified LP-4200 PQCrypto key from before that block was split into
mlkem, mldsa, slhdsa, p3q, pulsar, corona, hqc and magnetar. Zoo already
activates all eight, so nothing is lost by removing the name they replaced.
Co-authored-by: Hanzo Dev <[email protected]>
The six precompiles that were linked and never activated now take effect at
1766708400 with everything else rather than at a second date of their own. This
chainconf is a single activation timestamp.
Backdating means a chain that already has blocks under the old set has to be
reimported: swap, v3 and modelRegistry write state when they activate, so the
state root at activation is not what the existing history recorded. That is the
cost of one date instead of two, and it is paid by reimport.
The node has to carry the keys before a chain may name them - luxd refuses an
activation for a configKey it has never seen - so this config needs a luxd built
after luxfi/evm started importing the suite registry.
Co-authored-by: Hanzo Dev <[email protected]>
Neither had one. api.zoo.ngo reached the ingress with nothing routing
/v1/chain; api.zoo-test.network was an A record at a DigitalOcean address that
no longer answers (now a proxied CNAME to the estate tunnel). Both route to
archive nodes on evo holding the full imported chains — 799 and 84 blocks, with
historical state at every height. Admin and debug are off on both.
Co-authored-by: Hanzo Dev <[email protected]>
Six precompiles were dispatchable and activated nowhere in the estate:
aiInference, aivmBridge, modelRegistry, starkfri, swap and v3. The cause was
upstream - luxfi/evm named its suite packages one by one and had drifted from the
suite, so those configKeys did not exist to be named and luxd would have rejected
an activation for them. The registry now imports the suite wholesale, so they do.
They take effect at 1790812800 (2026-10-01), not the existing 1766708400: that
date is December 2025 and already past, and swap, v3 and modelRegistry write
state when they activate, so backdating them onto a chain with accepted blocks
changes state roots under history.
Entries stay sorted by timestamp; precompileUpgrades must be monotonic.
Co-authored-by: Hanzo Dev <[email protected]>
zood pinned v1.36.68 and the wallet still offered /mainnet/ext/bc/C/rpc for both
Lux and Hanzo. luxd removed those spellings in v1.36.179, so the wallet was
handing users chain endpoints that no current node answers.
zood and the wallet now name the chain one way, on v1.36.181.
Co-authored-by: Hanzo Dev <[email protected]>
zoo-wallet.yaml mounts zoo-wallet-brand by name and nothing declared it, so the
only copy was the live object. Captured, and the one address that is now wrong
is corrected: Lux mainnet serves /v1/chain/C/rpc and the wallet still asked for
/mainnet/ext/bc/C/rpc, which returns nothing. Hanzo's and Zoo's entries are
left alone — neither chain has a public endpoint to point at yet.
Co-authored-by: Hanzo Dev <[email protected]>
Its 40Gi chain volume is a local-path PV on spark, so spark is the only node
that can mount it. The amd64 pin named evo and the pod went Pending on a
PersistentVolume node-affinity mismatch. The image being amd64-only argues for
evo, but spark runs it under its registered qemu-x86_64 binfmt handler, the same
way the object store does there. Placement follows the disk; the disk does not
move.
Co-authored-by: Hanzo Dev <[email protected]>
Its 40Gi chain volume is a local-path PV on spark, so spark is the only node
that can mount it. The amd64 pin named evo and the pod went Pending on a
PersistentVolume node-affinity mismatch. The image being amd64-only argues for
evo, but spark runs it under its registered qemu-x86_64 binfmt handler, the same
way the object store does there. Placement follows the disk; the disk does not
move.
Co-authored-by: Hanzo Dev <[email protected]>
Every image in zoo-mainnet except consul and the ingress is published for
linux/amd64 alone, and none of these files said so. Unpinned, the scheduler is
free to place them on spark, where the kubelet refuses the pull with `no match
for platform in manifest` -- which reads as a broken image and is a misplaced
pod. zoo-ui, zoo-gym, zoo-bot and zoo-build had been sitting in
ImagePullBackOff there.
The pin follows the artifact: an image that exists for one architecture may
only be scheduled where that architecture is. consul and the ingress are
multi-arch and stay free to land anywhere.
Co-authored-by: Hanzo Dev <[email protected]>
Every image in zoo-mainnet except consul and the ingress is published for
linux/amd64 alone, and none of these files said so. Unpinned, the scheduler is
free to place them on spark, where the kubelet refuses the pull with `no match
for platform in manifest` -- which reads as a broken image and is a misplaced
pod. zoo-ui, zoo-gym, zoo-bot and zoo-build had been sitting in
ImagePullBackOff there.
The pin follows the artifact: an image that exists for one architecture may
only be scheduled where that architecture is. consul and the ingress are
multi-arch and stay free to land anywhere.
Co-authored-by: Hanzo Dev <[email protected]>
Ten workloads named `ghcr-luxfi`, and zoo-mainnet has no secret by that name.
A pod naming a secret that does not exist gets no credential at all: the kubelet
falls back to anonymous, and GHCR answers a private package 403 -- which the pod
reports as ImagePullBackOff / NotFound, so it reads as a missing image and is a
missing secret. zoo-ui, zoo-gym, zoo-bot, zoo-build and zoo-wallet have been
sitting in that state.
ghcr-zooai is the name the namespace has, and it is the right one by ownership
as well: zoo's own credential for zoo's own images. Verified it reads every repo
these files reference -- zooai/gym, zoo-bot, ui, foundation, computer and the
luxfi/build-zoo, bounty-bot, work-board, wallet-web, explorer they share.
Co-authored-by: Hanzo Dev <[email protected]>
Ten workloads named `ghcr-luxfi`, and zoo-mainnet has no secret by that name.
A pod naming a secret that does not exist gets no credential at all: the kubelet
falls back to anonymous, and GHCR answers a private package 403 -- which the pod
reports as ImagePullBackOff / NotFound, so it reads as a missing image and is a
missing secret. zoo-ui, zoo-gym, zoo-bot, zoo-build and zoo-wallet have been
sitting in that state.
ghcr-zooai is the name the namespace has, and it is the right one by ownership
as well: zoo's own credential for zoo's own images. Verified it reads every repo
these files reference -- zooai/gym, zoo-bot, ui, foundation, computer and the
luxfi/build-zoo, bounty-bot, work-board, wallet-web, explorer they share.
Co-authored-by: Hanzo Dev <[email protected]>
The image is one build serving every label and this file is the only
thing that distinguishes them, so without it the pod would present Lux on
a Zoo surface. It was mounted here and declared nowhere; four days in
ContainerCreating. Content is the wallet repo's own zoo overlay.
Claude-Session: https://claude.ai/code/session_018U2e2U4PPc8UaXYyeT9iNu
Universe CI / Resolve Environment (push) Waiting to run
Universe CI / Update Manifest (push) Blocked by required conditions
Universe CI / E2E (push) Blocked by required conditions
CI/CD / render (push) Waiting to run
The primary-network chains answer to either case; one spelling beats two.
/v1/chain/c/rpc, /v1/chain/p, /v1/chain/x across the estate.
Co-authored-by: Hanzo AI <[email protected]>
Both declarations pinned 1.2.18; the release is 1.5.49. It is stateless
— DISABLE_INDEXER, no database, an emptyDir — so there is no migration
between them, only 30 releases of drift.
zoo-mainnet goes to zero replicas. Nothing routes to it: no Ingress or
route in any namespace names zoo-explorer, and the chain it reads,
https://api.zoo.ngo/, answers 404. The chain is down at the source —
zood-mv-0 sits Init:0/1 with its staking secret and its startup and
genesis ConfigMaps all missing — so two replicas were rendering a chain
that does not answer, for nobody.
The declaration stays. When zood has its keys the explorer comes back on
one line, at the current release rather than a year behind it.
Co-authored-by: Hanzo AI <[email protected]>
zoo.vote and zoo.industries are static bundles a pod was re-serving on every
request. Both move to the site plane: the `public/` tree of the image digest
each Deployment ran is synced to s3://hanzo-sites/zoo/vote and
s3://hanzo-sites/zoo/industries, and each file now declares the Middleware that
reads the prefix together with the route that names it. zoo-industries' server
generated /public/config.json at boot from three SPA_* variables; the plane has
no boot, so that file travels in the prefix carrying the values the pod served.
zoo-ngo is deleted rather than converted. zoo.ngo and www.zoo.ngo already route
to s3://hanzo-sites/zoo/ngo, and that prefix holds a newer build than the pod
was serving to nobody -- no route in this namespace named it. A site declared
twice is a site that goes stale in one of them.
Co-authored-by: Hanzo AI <[email protected]>
Universe CI / Resolve Environment (push) Successful in 0s
Universe CI / Update Manifest (push) Skipped
Universe CI / E2E (push) Failing after 10m20s
hanzo-build-linux-amd64, lux-build-linux-amd64, zoo-build-linux-amd64 and
every other org-prefixed spelling all meant the same machine. Which org a
job belongs to does not change which host can build it, so the org bought
one pool name per org per arch and each could go stale on its own -- the
evo host alone answered to fourteen names for one label.
One vocabulary now: linux-amd64, linux-arm64, macos-arm64, win-amd64. The
generic labels every plain workflow uses (ubuntu-latest, self-hosted) are
untouched, so naive CI keeps working.
Co-authored-by: Hanzo AI <[email protected]>
Its chain is 200203 and it declared itself network 200202 — the
devnet's — so the two halves of one document disagreed about which
network it describes, and a node reading it would announce itself on a
network whose chain it does not carry.
Co-authored-by: Hanzo AI <[email protected]>
The KMS fetch steps called `/api/v1/auth/universal-auth/login` and
`/api/v3/secrets/raw?workspaceSlug=...` — the Infisical surface. We do not run
Infisical; kms.hanzo.ai serves the Hanzo-native door and 404s those paths, so
every secret fetch failed and the build died at "KMS auth failed".
Rewritten to the canonical contract the working repos use:
`/v1/kms/auth/login` returning `.accessToken`, then
`/v1/kms/secrets/ci/<name>?env=prod` returning `.value`. The org is read from
the credential, so the `zoo-platform` workspace slug is gone.
Co-authored-by: Hanzo AI <[email protected]>
All five lived in .hanzo/workflows, a path github.com does not read, so with
github.com canonical they dispatched nothing.
cicd.yml also asked for hanzo-build-linux-amd64 — another org's pool. A
cross-org label does not fail a job, it queues it until the 24h timeout, which
reads as a slow build rather than a misrouted one. It asks for
zoo-build-linux-amd64 now, which every other workflow here already used.
The pin rested on an arm64 pull error, "no match for platform in
manifest". On the amd64 node the same digest reports plainly: not found.
zoo-mainnet has no ghcr-luxfi secret, the one the values file names, so
the private repository answers every request as a miss on both nodes and
the earlier message says nothing about which platforms are published.
Provision the pull secret first; the manifest can then be read and the
architecture, if it needs constraining, constrained against evidence.
Co-authored-by: Hanzo Dev <[email protected]>
ghcr.io/zooai/computer publishes an amd64-only manifest. Scheduled on the
arm64 node the pull ends in "no match for platform in manifest" and the
Deployment never becomes ready.
Co-authored-by: Hanzo Dev <[email protected]>
The step embedded multi-line Python at column zero — a heredoc terminator in
one repo, an inline -c script in the other — and both dedented straight out of
the run: block scalar, so the workflow did not parse as YAML at all and the
job it belongs to could never start.
It is one indented line now. The lookup is unchanged: read the ApplicationSet,
find the source whose chart is app, print its repository and target revision.
A missing source raises instead of returning nothing, which is the point of
parsing the document rather than grepping it — the comment above already said
a silent miss renders against whatever the registry calls latest.
The fleet runs both architectures and this lane published linux/amd64 only,
so an image deployed on half of it.
Both platforms come out of one build on the amd64 pool under binfmt rather
than a second native job: the arm64 host advertises `zoo-linux-arm64` and NOT
`zoo-build-linux-arm64` (hanzo/universe infra/runners/spark/config.yaml), and
a job asking for a label nothing serves queues forever — this reusable is
called by every zoo service repo, so that would stall the org. A pure-Go
Dockerfile honouring $TARGETARCH cross-compiles and pays nothing for the
emulation.
The single-entry arch matrix goes with it, since a two-platform build has no
one arch to put in a tag. The three `-amd64` names are published unchanged —
deploy/ and k8s/base pin `main-amd64` — and now carry both platforms. Beside
them the semver name is published without an arch: that is the tag
notify-universe already dispatches to universe, and nothing created it, so
the dispatched image did not resolve. A step then proves the pushed tag
carries both platforms, so multi-arch is a fact rather than an intention.
notify-universe also regains a runs-on. Its runner label had been reduced to
a whitespace-only line, which leaves the job unschedulable.
deploy/zoo carries the four services that had no declaration here at all --
exchange-api, status, and both exchanges -- so Zoo's own repo is where they are
declared. zoo.exchange among them serves its published bundle with no pod.
deploy/hanzo declared Hanzo's iam, billing, cloud, cloud-api and login. Hanzo
declares those too, so two Applications from two repos claimed the same objects
and CD said so: Deployment/iam is part of applications hanzo-iam and
zoo-hanzo-iam, and the same for billing. Whichever synced last decided what ran.
cert-manager was declared here, in Lux and in Hanzo. It is one cluster-scoped
install; Hanzo declares it.
otel-agent's ClusterRole and ClusterRoleBinding are cluster-scoped and carry one
name, so two brands declaring them is a collision by construction. One
declaration remains.
Every Zoo workload is declared here, where zoo-fleet reads; these five had been
declared only in lux/universe's zoo-mainnet directory. Pins carried as they were
running, digest-pinned where the registry answered.
Co-authored-by: Hanzo Dev <[email protected]>
Three records added alongside the sweep still named a retired address. Same
treatment as the rest of the zone: the origin the edge forwards to, proxied.
Co-authored-by: Hanzo Dev <[email protected]>
Records across these zones named an address this estate does not answer on.
Each now names the origin the edge forwards to.
They are proxied deliberately: Cloudflare answers, the origin is reached only
from Cloudflare, and the address never appears in a public answer. Unproxied, a
record both publishes the origin and makes a reassigned address a live takeover
of the name.
Co-authored-by: Hanzo Dev <[email protected]>
zooai/universe is zoo-build's one home; the duplicate declaration in
lux/universe had moved ahead to 0.1.9 while this one stayed at 0.1.7. Carry the
newer pin here before the duplicate is removed there, so nothing regresses.
Co-authored-by: Hanzo Dev <[email protected]>
zoo.vote was declared in lux/universe deploy/zoo-mainnet/zoo-vote.yaml. zoo-fleet's
generator reads THIS repository (hanzo/universe infra/k8s/hanzo-cd/project-zoo.yaml,
path deploy/*/*.yaml), so the workload had a declaration in a fleet that does not
serve it and none in the one that does. Declared here, deleted there.
The image is ghcr.io/zooai/vote, built by luxfi/vote's Image workflow: one source
builds lux.vote, zoo.vote and hanzo.vote, each into its own org's registry, so
this pod pulls with Zoo's credential and no other org's access can strand it. The
tenant is a build-arg and the host settles it again at runtime, which is why the
ingress names zoo.vote and www.zoo.vote — the app reads the hostname to know
which DAO it is, so what is routed here is part of its behaviour.
The tag is what the live Deployment runs, with a TODO to pin the first tag the
workflow prints. It could not be verified from a workstation, so it stays a tag
rather than becoming a digest that was guessed.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_015g5a1mCWuuSQTBZZywNnET
One A record at the hanzo-k8s ingress load balancer replaces the four GitHub
Pages records; the site is served from the Zoo prefix in object storage.
Co-authored-by: Hanzo Dev <[email protected]>
Two paths in this workflow wrote to zoo-k8s directly. `deploy-single` ran
kubectl apply -k over one namespace's overlay; `deploy-all` ran a script that
applied the CRDs, applied the whole overlay, and then hand-patched luxd flags
onto a StatefulSet the Rust operator renders -- a patch whose own comment
explains it persists because the operator does not re-render an existing pod
template.
That is a second writer for objects zoo-fleet declares from deploy/*/*.yaml,
and the disagreement is not quiet. Both stamp the pod template
(app.kubernetes.io/managed-by: hanzo-operator vs Helm), and a pod-template
label is not a field two controllers can differ about without a restart, so
each flip rolls the workload. It does not converge; it oscillates.
It also applied the WRONG THING. k8s/ is the hand-applied tree whose currency
deploy/README.md says outright is unknown -- some of it ahead of the cluster,
some behind, some describing objects that were deleted. A pipeline that applies
it on every push re-asserts that drift on a schedule.
What is left writes to git and tests: record the pin, open the PR, run E2E.
zoo-fleet reconciles from the merged file. The kubectl calls still in the E2E
job read pods, events and logs on failure, and write nothing.
The k8s/ tree stays, for what deploy/ deliberately does not cover -- CRDs,
namespaces, the operator's custom resources. `make k8s-apply env=<env>` is that
path, with a human behind it.
deploy-all and e2e-only meant the same thing once neither deployed, so the
modes are now the two acts that remain: pin and e2e.
Co-authored-by: Hanzo Dev <[email protected]>
26e7154 moved the tag from v1.36.58 to v1.36.68 to stop this file declaring a
version behind the fleet, and left the digest on v1.36.58's bytes. A reference
carrying both resolves BY DIGEST, so the declaration it left behind was the one
it set out to remove -- run .58, report .68 -- and now with the tag agreeing
with the running fleet, nothing was left to notice it. That is the failure mode
worth naming: a drifted pin does not fail, it succeeds at doing nothing.
Both halves are now v1.36.68 (ebd16ed5…), read from the registry in one answer.
The setup initContainer moves with them; it ran .58 by the same accident, and a
node whose init and main containers are different builds is a question nobody
should have to ask.
Consequence, on this file specifically: zood-mv is the only luxd StatefulSet on
RollingUpdate with podManagementPolicy Parallel, N=5 and alpha=4. A sync would
have restarted all five Zoo validators together onto ten-releases-old bytes,
and luxd waits for peers before it listens -- every node waiting for a node
that is also waiting.
Co-authored-by: Hanzo Dev <[email protected]>
A values file named a repository and a tag, and a tag is mutable. Under
pullPolicy: Always that means the fleet pulls whatever the name points at on the
morning it restarts; under IfNotPresent it means a node that pulled `:latest`
once keeps those bytes forever while its neighbour gets a newer build. Both
spellings of the same fault: the file does not say what runs.
41 of 50 image blocks now carry a digest beside the tag, resolved from the
registry, with pullPolicy: IfNotPresent throughout. The chart renders
repo:tag@digest and the reference resolves BY DIGEST -- the tag stays as the
name a human reads. 20 of those were floating (`latest`, `main`, `main-amd64`)
and now name specific bytes.
Four floating tags had a release naming the same digest, so they say it:
consul 1.20 -> 1.20.6, traefik v3.3 -> v3.3.7, zooai/docs latest -> 0.1.4. Each
was verified to be the same manifest, so this renames what is running and moves
nothing. gatus stays on `latest`: no v-tag names its bytes, so a version there
would have been a silent upgrade rather than a rename.
This is also what chart 0.1.13 requires. Its claimed list covers
ghcr.io/hanzoai/cloud:v*, and it refuses to RENDER one of those without a
digest -- so hanzo/cloud-api would have failed the render, which stops the sync
and leaves the running pods alone. Better than the alternative, but the pin is
the point.
Eight files name an image the registry does not have and are left alone rather
than guessed at; deploy/README.md lists them with what the registry does have.
Co-authored-by: Hanzo Dev <[email protected]>
deploy/applicationset.yaml contradicted the live zoo-fleet on every field that
decides where a sync lands: apiVersion apps.hanzo.ai/v1alpha1 against v1,
namespace cd against hanzo-cd, a github.com/zooai/universe mirror against
git.hanzo.ai/zoo/universe, and chart 0.1.0 against the version the fleet pins.
Applying it would not have adopted anything -- it would have stood up a second
generator over the same deploy/*/*.yaml, racing the first, with neither one
wrong on its own terms.
zoo-fleet is declared in hanzo/universe/infra/k8s/hanzo-cd/project-zoo.yaml,
behind an AppProject that fences Zoo's Applications to the Zoo cluster. One CD,
one standard, three orgs. A repo that also ships a generator is a second
opinion about a question that has one answer, so this copy goes.
The README's chart reference moves to 0.1.13 and says which file actually
decides it, so a helm template on a workstation renders what the cluster would
and a disagreement points at the ApplicationSet rather than at itself.
Co-authored-by: Hanzo Dev <[email protected]>
The zoo manifests, the compose file and the e2e helpers all composed
/v1/bc addresses against luxd, which answers under /v1/chain and no
longer serves the old segment.
A line that records an observation keeps its original spelling.
Co-authored-by: Hanzo Dev <[email protected]>
zeekayandHanzo Dev