Each of these declares replicas: 0 -- no pod, no endpoint, no caller, and no
record of when it was parked. A declaration that runs nothing is not a
deployment, it is a note nobody can date, and it keeps a slot in every fleet
listing and every sync. Git history holds them if one was paused rather than
retired.
Co-authored-by: Hanzo Dev <[email protected]>
Both were scraped off do-sfo3-zoo-k8s, the DigitalOcean cluster that no longer
exists. zoo-zips runs the only nginx left in the fleet and answers 669 bytes of
meta-refresh, while the real zips.zoo.ngo serves 96 KB from elsewhere.
work-board duplicates the one behind work.hanzo.network; work.zoo.network does
not resolve.
Co-authored-by: Hanzo Dev <[email protected]>
status.zoo.network is served by zoo/status, which is healthy. This copy in
zoo-mainnet has no route, and it has been unready since the migration waiting
on a status-config ConfigMap that no file produces.
Co-authored-by: Hanzo Dev <[email protected]>
precompile v0.19.22 renamed the eleven keys that used Verify, Threshold or
Encapsulate to <name>Config, so all fifty-six answer to one convention - including
the LP-4200 unified PQCrypto block, which had been carrying four across its eight
addresses.
This config names eleven of them and moves with the release. An upgrade.json
holding an old key names a precompile that does not exist, and luxd refuses the
activation rather than ignoring it.
Co-authored-by: Hanzo Dev <[email protected]>
Zoo mainnet and testnet activated pqcryptoConfig. No precompile registers that
key - not in the suite, not in the EVM - and luxd refuses an activation for a
configKey it has never seen, so this line alone would stop a Zoo node at genesis.
It is the old unified LP-4200 PQCrypto key from before that block was split into
mlkem, mldsa, slhdsa, p3q, pulsar, corona, hqc and magnetar. Zoo already
activates all eight, so nothing is lost by removing the name they replaced.
Co-authored-by: Hanzo Dev <[email protected]>
The six precompiles that were linked and never activated now take effect at
1766708400 with everything else rather than at a second date of their own. This
chainconf is a single activation timestamp.
Backdating means a chain that already has blocks under the old set has to be
reimported: swap, v3 and modelRegistry write state when they activate, so the
state root at activation is not what the existing history recorded. That is the
cost of one date instead of two, and it is paid by reimport.
The node has to carry the keys before a chain may name them - luxd refuses an
activation for a configKey it has never seen - so this config needs a luxd built
after luxfi/evm started importing the suite registry.
Co-authored-by: Hanzo Dev <[email protected]>
Neither had one. api.zoo.ngo reached the ingress with nothing routing
/v1/chain; api.zoo-test.network was an A record at a DigitalOcean address that
no longer answers (now a proxied CNAME to the estate tunnel). Both route to
archive nodes on evo holding the full imported chains — 799 and 84 blocks, with
historical state at every height. Admin and debug are off on both.
Co-authored-by: Hanzo Dev <[email protected]>
Six precompiles were dispatchable and activated nowhere in the estate:
aiInference, aivmBridge, modelRegistry, starkfri, swap and v3. The cause was
upstream - luxfi/evm named its suite packages one by one and had drifted from the
suite, so those configKeys did not exist to be named and luxd would have rejected
an activation for them. The registry now imports the suite wholesale, so they do.
They take effect at 1790812800 (2026-10-01), not the existing 1766708400: that
date is December 2025 and already past, and swap, v3 and modelRegistry write
state when they activate, so backdating them onto a chain with accepted blocks
changes state roots under history.
Entries stay sorted by timestamp; precompileUpgrades must be monotonic.
Co-authored-by: Hanzo Dev <[email protected]>
zood pinned v1.36.68 and the wallet still offered /mainnet/ext/bc/C/rpc for both
Lux and Hanzo. luxd removed those spellings in v1.36.179, so the wallet was
handing users chain endpoints that no current node answers.
zood and the wallet now name the chain one way, on v1.36.181.
Co-authored-by: Hanzo Dev <[email protected]>
zoo-wallet.yaml mounts zoo-wallet-brand by name and nothing declared it, so the
only copy was the live object. Captured, and the one address that is now wrong
is corrected: Lux mainnet serves /v1/chain/C/rpc and the wallet still asked for
/mainnet/ext/bc/C/rpc, which returns nothing. Hanzo's and Zoo's entries are
left alone — neither chain has a public endpoint to point at yet.
Co-authored-by: Hanzo Dev <[email protected]>
Its 40Gi chain volume is a local-path PV on spark, so spark is the only node
that can mount it. The amd64 pin named evo and the pod went Pending on a
PersistentVolume node-affinity mismatch. The image being amd64-only argues for
evo, but spark runs it under its registered qemu-x86_64 binfmt handler, the same
way the object store does there. Placement follows the disk; the disk does not
move.
Co-authored-by: Hanzo Dev <[email protected]>
Its 40Gi chain volume is a local-path PV on spark, so spark is the only node
that can mount it. The amd64 pin named evo and the pod went Pending on a
PersistentVolume node-affinity mismatch. The image being amd64-only argues for
evo, but spark runs it under its registered qemu-x86_64 binfmt handler, the same
way the object store does there. Placement follows the disk; the disk does not
move.
Co-authored-by: Hanzo Dev <[email protected]>
Every image in zoo-mainnet except consul and the ingress is published for
linux/amd64 alone, and none of these files said so. Unpinned, the scheduler is
free to place them on spark, where the kubelet refuses the pull with `no match
for platform in manifest` -- which reads as a broken image and is a misplaced
pod. zoo-ui, zoo-gym, zoo-bot and zoo-build had been sitting in
ImagePullBackOff there.
The pin follows the artifact: an image that exists for one architecture may
only be scheduled where that architecture is. consul and the ingress are
multi-arch and stay free to land anywhere.
Co-authored-by: Hanzo Dev <[email protected]>
Every image in zoo-mainnet except consul and the ingress is published for
linux/amd64 alone, and none of these files said so. Unpinned, the scheduler is
free to place them on spark, where the kubelet refuses the pull with `no match
for platform in manifest` -- which reads as a broken image and is a misplaced
pod. zoo-ui, zoo-gym, zoo-bot and zoo-build had been sitting in
ImagePullBackOff there.
The pin follows the artifact: an image that exists for one architecture may
only be scheduled where that architecture is. consul and the ingress are
multi-arch and stay free to land anywhere.
Co-authored-by: Hanzo Dev <[email protected]>
Ten workloads named `ghcr-luxfi`, and zoo-mainnet has no secret by that name.
A pod naming a secret that does not exist gets no credential at all: the kubelet
falls back to anonymous, and GHCR answers a private package 403 -- which the pod
reports as ImagePullBackOff / NotFound, so it reads as a missing image and is a
missing secret. zoo-ui, zoo-gym, zoo-bot, zoo-build and zoo-wallet have been
sitting in that state.
ghcr-zooai is the name the namespace has, and it is the right one by ownership
as well: zoo's own credential for zoo's own images. Verified it reads every repo
these files reference -- zooai/gym, zoo-bot, ui, foundation, computer and the
luxfi/build-zoo, bounty-bot, work-board, wallet-web, explorer they share.
Co-authored-by: Hanzo Dev <[email protected]>
Ten workloads named `ghcr-luxfi`, and zoo-mainnet has no secret by that name.
A pod naming a secret that does not exist gets no credential at all: the kubelet
falls back to anonymous, and GHCR answers a private package 403 -- which the pod
reports as ImagePullBackOff / NotFound, so it reads as a missing image and is a
missing secret. zoo-ui, zoo-gym, zoo-bot, zoo-build and zoo-wallet have been
sitting in that state.
ghcr-zooai is the name the namespace has, and it is the right one by ownership
as well: zoo's own credential for zoo's own images. Verified it reads every repo
these files reference -- zooai/gym, zoo-bot, ui, foundation, computer and the
luxfi/build-zoo, bounty-bot, work-board, wallet-web, explorer they share.
Co-authored-by: Hanzo Dev <[email protected]>
Universe CI / Resolve Environment (push) Waiting to run
Universe CI / Update Manifest (push) Blocked by required conditions
Universe CI / E2E (push) Blocked by required conditions
CI/CD / render (push) Waiting to run
The primary-network chains answer to either case; one spelling beats two.
/v1/chain/c/rpc, /v1/chain/p, /v1/chain/x across the estate.
Co-authored-by: Hanzo AI <[email protected]>
Both declarations pinned 1.2.18; the release is 1.5.49. It is stateless
— DISABLE_INDEXER, no database, an emptyDir — so there is no migration
between them, only 30 releases of drift.
zoo-mainnet goes to zero replicas. Nothing routes to it: no Ingress or
route in any namespace names zoo-explorer, and the chain it reads,
https://api.zoo.ngo/, answers 404. The chain is down at the source —
zood-mv-0 sits Init:0/1 with its staking secret and its startup and
genesis ConfigMaps all missing — so two replicas were rendering a chain
that does not answer, for nobody.
The declaration stays. When zood has its keys the explorer comes back on
one line, at the current release rather than a year behind it.
Co-authored-by: Hanzo AI <[email protected]>
zoo.vote and zoo.industries are static bundles a pod was re-serving on every
request. Both move to the site plane: the `public/` tree of the image digest
each Deployment ran is synced to s3://hanzo-sites/zoo/vote and
s3://hanzo-sites/zoo/industries, and each file now declares the Middleware that
reads the prefix together with the route that names it. zoo-industries' server
generated /public/config.json at boot from three SPA_* variables; the plane has
no boot, so that file travels in the prefix carrying the values the pod served.
zoo-ngo is deleted rather than converted. zoo.ngo and www.zoo.ngo already route
to s3://hanzo-sites/zoo/ngo, and that prefix holds a newer build than the pod
was serving to nobody -- no route in this namespace named it. A site declared
twice is a site that goes stale in one of them.
Co-authored-by: Hanzo AI <[email protected]>
Universe CI / Resolve Environment (push) Successful in 0s
Universe CI / Update Manifest (push) Skipped
Universe CI / E2E (push) Failing after 10m20s
hanzo-build-linux-amd64, lux-build-linux-amd64, zoo-build-linux-amd64 and
every other org-prefixed spelling all meant the same machine. Which org a
job belongs to does not change which host can build it, so the org bought
one pool name per org per arch and each could go stale on its own -- the
evo host alone answered to fourteen names for one label.
One vocabulary now: linux-amd64, linux-arm64, macos-arm64, win-amd64. The
generic labels every plain workflow uses (ubuntu-latest, self-hosted) are
untouched, so naive CI keeps working.
Co-authored-by: Hanzo AI <[email protected]>
Its chain is 200203 and it declared itself network 200202 — the
devnet's — so the two halves of one document disagreed about which
network it describes, and a node reading it would announce itself on a
network whose chain it does not carry.
Co-authored-by: Hanzo AI <[email protected]>
The KMS fetch steps called `/api/v1/auth/universal-auth/login` and
`/api/v3/secrets/raw?workspaceSlug=...` — the Infisical surface. We do not run
Infisical; kms.hanzo.ai serves the Hanzo-native door and 404s those paths, so
every secret fetch failed and the build died at "KMS auth failed".
Rewritten to the canonical contract the working repos use:
`/v1/kms/auth/login` returning `.accessToken`, then
`/v1/kms/secrets/ci/<name>?env=prod` returning `.value`. The org is read from
the credential, so the `zoo-platform` workspace slug is gone.
Co-authored-by: Hanzo AI <[email protected]>
All five lived in .hanzo/workflows, a path github.com does not read, so with
github.com canonical they dispatched nothing.
cicd.yml also asked for hanzo-build-linux-amd64 — another org's pool. A
cross-org label does not fail a job, it queues it until the 24h timeout, which
reads as a slow build rather than a misrouted one. It asks for
zoo-build-linux-amd64 now, which every other workflow here already used.
The step embedded multi-line Python at column zero — a heredoc terminator in
one repo, an inline -c script in the other — and both dedented straight out of
the run: block scalar, so the workflow did not parse as YAML at all and the
job it belongs to could never start.
It is one indented line now. The lookup is unchanged: read the ApplicationSet,
find the source whose chart is app, print its repository and target revision.
A missing source raises instead of returning nothing, which is the point of
parsing the document rather than grepping it — the comment above already said
a silent miss renders against whatever the registry calls latest.
The fleet runs both architectures and this lane published linux/amd64 only,
so an image deployed on half of it.
Both platforms come out of one build on the amd64 pool under binfmt rather
than a second native job: the arm64 host advertises `zoo-linux-arm64` and NOT
`zoo-build-linux-arm64` (hanzo/universe infra/runners/spark/config.yaml), and
a job asking for a label nothing serves queues forever — this reusable is
called by every zoo service repo, so that would stall the org. A pure-Go
Dockerfile honouring $TARGETARCH cross-compiles and pays nothing for the
emulation.
The single-entry arch matrix goes with it, since a two-platform build has no
one arch to put in a tag. The three `-amd64` names are published unchanged —
deploy/ and k8s/base pin `main-amd64` — and now carry both platforms. Beside
them the semver name is published without an arch: that is the tag
notify-universe already dispatches to universe, and nothing created it, so
the dispatched image did not resolve. A step then proves the pushed tag
carries both platforms, so multi-arch is a fact rather than an intention.
notify-universe also regains a runs-on. Its runner label had been reduced to
a whitespace-only line, which leaves the job unschedulable.
deploy/zoo carries the four services that had no declaration here at all --
exchange-api, status, and both exchanges -- so Zoo's own repo is where they are
declared. zoo.exchange among them serves its published bundle with no pod.
deploy/hanzo declared Hanzo's iam, billing, cloud, cloud-api and login. Hanzo
declares those too, so two Applications from two repos claimed the same objects
and CD said so: Deployment/iam is part of applications hanzo-iam and
zoo-hanzo-iam, and the same for billing. Whichever synced last decided what ran.
cert-manager was declared here, in Lux and in Hanzo. It is one cluster-scoped
install; Hanzo declares it.
otel-agent's ClusterRole and ClusterRoleBinding are cluster-scoped and carry one
name, so two brands declaring them is a collision by construction. One
declaration remains.
Three records added alongside the sweep still named a retired address. Same
treatment as the rest of the zone: the origin the edge forwards to, proxied.
Co-authored-by: Hanzo Dev <[email protected]>
Records across these zones named an address this estate does not answer on.
Each now names the origin the edge forwards to.
They are proxied deliberately: Cloudflare answers, the origin is reached only
from Cloudflare, and the address never appears in a public answer. Unproxied, a
record both publishes the origin and makes a reassigned address a live takeover
of the name.
Co-authored-by: Hanzo Dev <[email protected]>
Same shape as the three before them: one changed object each, their Deployment,
stateless, two replicas, RollingUpdate, Healthy, managed-by Helm. The first
batch replaced six pods in about ninety seconds with no eviction and nothing
left Pending, which is the question the memory headroom raised.
hanzo-login is in this batch and is the one worth naming: it is a sign-in
surface, so it rolls with two replicas and a surge, never a gap.
Still out of this fleet: the ingress and the operator, both of which also
render a changed Service, and the explorer pair whose pod template still names
zoo-operator as its manager -- two controllers writing one template do not
settle, they take turns.
Co-authored-by: Hanzo Dev <[email protected]>
First enforcement in this fleet, so it is deliberately three. Each is
stateless, two replicas, RollingUpdate, Healthy, managed-by Helm already, and
renders exactly one changed object -- its Deployment -- so the first sync
applies the pod template the chart adds HANZO_VERSION to and nothing else.
Batch size here is set by memory, not by confidence. The four nodes carry
memory requests at 88, 97, 98 and 99 percent of allocatable: 1013Mi free in
total and only 3Mi of it on one node. A RollingUpdate surges by one pod before
it retires one, and these request 64Mi each, so a wide batch would leave
replacement pods Pending with the old ones still serving -- a stalled rollout
rather than a lost one, but not progress either. Three fits with room.
Co-authored-by: Hanzo Dev <[email protected]>
Declared v1.36.58 against v1.36.68 running. This is the one that could have
taken the chain down: zood-mv is the only luxd StatefulSet on RollingUpdate with
podManagementPolicy Parallel, so a sync would not have rolled back gradually —
it would have restarted all five Zoo validators at once, at an older version.
Two facts make that unrecoverable rather than merely bad. luxd waits for peers
before it listens, so a whole fleet restarting together deadlocks: every node is
waiting for a node that is also waiting. And N=5 with alpha=4 means one node
down is already a de-facto halt, so there is no slack to absorb it.
The value is copied from the live StatefulSet. Nothing changes; the app simply
stops carrying an instruction that would have been obeyed.
Co-authored-by: Hanzo Dev <[email protected]>
The zone lived only under lux/universe/k8s/dns, where the sync script
globs every *.yaml and so held write authority over a zone belonging to
another brand. cf-zones/sync.sh globs this directory the same way, so
moving the file is the whole change -- zoo.cloud now syncs from the
universe whose brand it is.
Co-authored-by: Hanzo Dev <[email protected]>
The social card asked for Inter first. That is not a face this estate
licenses, ships or ever drew, so a renderer with Inter installed painted
the wrong face and one without it silently fell through to system-ui.
Zen leads now; the same tail follows it.
Same schema error as hanzod-mv: additionalProperties:false makes the
wrong key a render failure, not a no-op.
Co-authored-by: Hanzo Dev <[email protected]>
Same gap as hanzo: zoo-mainnet's validators were hand-applied with no
CD tracking. Generated from the LIVE StatefulSet and render-verified.
Not auto-synced: adoption would roll a producing mainnet.
Co-authored-by: Hanzo Dev <[email protected]>
Not restart noise: an OOM landing mid-write leaves a node holding a
CONSENSUS-CERTIFIED chain that is not the network's, which it then
gossips. That poisoned every peer on hanzo-mainnet 2026-08-06 and took
a node-by-node state transplant to clear. zood-mv carried no env at
all under a 3Gi limit; 2GiB soft ceiling makes Go GC before the kernel
kills.
Co-authored-by: Hanzo Dev <[email protected]>
1.12 is the consensus-native KMS: it constructs a ConsensusAuthorizer at start
and refuses to run without one — 'consensus validator authority is empty
(refusing to boot fail-open)'. The refusal is right; there is no open mode by
design. But nothing provides this pod the authority snapshot the operator is
meant to drop at KMS_CONSENSUS_FILE, so the upgrade crash-looped and took the
Zoo KMS down for ~90 minutes until it was rolled back.
Adopting 1.12 needs the consensus authority provisioned first. Pinning the
generation that runs is not a preference here — it is the difference between a
KMS and a CrashLoopBackOff.
Co-authored-by: Hanzo Dev <[email protected]>
zoo-build runs in a 512Mi container, but V8 sizes its own old space and caps around
258 MB — so half the granted memory is unreachable and the process dies inside its
own heap:
FATAL ERROR: Ineffective mark-compacts near heap limit
Mark-Compact 251.9 (258.3) -> 251.1 (258.8) MB
exit 139
The container is never out of memory (kubelet saw ~225Mi in use), so raising
limits.memory would not have moved a V8-internal ceiling. `--max-old-space-size=384`
leaves ~128Mi for RSS outside the heap, the usual headroom for a Next.js server at
this size.
This is the SAME app and the SAME shape as the lux-k8s copy, which crashed roughly
every 13 hours until it got this setting (19 restarts; now 0 since the fix landed).
This copy has NODE_OPTIONS unset and the identical 512Mi limit — it has simply not
tripped yet, most likely because it takes less traffic. Applied proactively so both
copies of one app are configured the same way.
Image tag deliberately left at 0.1.7: this is a latent-crash fix, not a version bump.
Hanzo Dev <[email protected]>
Co-authored-by: Hanzo Dev <[email protected]>
Gitea collects workflows from the FIRST of WORKFLOW_DIRS that exists in the
pushed commit and ignores the rest (modules/actions/workflows.go,
listWorkflowsInDirs breaks on the first hit), so creating .hanzo/workflows
switches .github/workflows off wholesale — no run, no queued job, no error.
Every file therefore moves or dies in this one commit; nothing is left to look
like a fallback it could not be.
MIGRATED, unchanged apart from the label form and the notes below:
ci.yml (the Layer-2 deploy + E2E orchestrator), build-service.yml (the Layer-1
reusable), service-released.yml and update-image.yml (the two repository_dispatch
auto-bump shapes). Their runner label was already right — zoo-build-linux-amd64
is what the fleet advertises — and is now written in the list form the rest of
the fleet uses.
FIXED — service-released.yml HAS NEVER PARSED. Its `gh pr create --body`
markdown table sat at column 0 inside a block scalar indented to column 10, so
the scalar terminated at the table and the `|` was read as a second block
indicator:
yaml.scanner.ScannerError: while scanning a block scalar
expected a comment or a line break, but found 'f' (line 132, col 3)
That is why every single run of it is a failure. The table is now indented into
the scalar, which YAML strips back off, so the shell and the rendered PR body
are byte-identical to what was intended and the file loads.
DELETED — renovate.yml, and renovate.json with it. A `*/15 * * * *` cron, so 96
runs a day against the shared amd64 pool, for an inherited dependency bot we do
not use; it was measured at 83 runs in the last 24 hours, more than any real
pipeline on this forge. Its config file goes too rather than sit behind a bot
that is not there.
The reusable's PATH is part of its contract, so moving it is a caller-visible
change: services/actions.ResolveUses enforces the WORKFLOW_DIRS allowlist on
`uses:` paths, which means .github/workflows would have stopped resolving at the
narrowing anyway. Searched across the zooai org by both spellings — zoolabs/ and
zooai/ — and nothing outside this repo referenced it; the only hits were the
file itself and a comment in ci.yml. Callers now use
zooai/universe/.hanzo/workflows/build-service.yml@main.
build-push's matrix no longer carries the runner label. It still carries `arch`,
which is part of the published image tag (…-amd64), but which label the fleet
advertises is a fleet fact rather than a tag fact, and behind
${{ matrix.runner }} the one thing worth grepping for could not be grepped.
Concurrency groups added where they were missing: one in-flight bump per
service, never cancelled between `git push` and `gh pr create`, and one build
per calling service so two services do not serialise behind each other.
Co-authored-by: Hanzo Dev <[email protected]>
mpc.zoo.network served the same unguarded Base admin console as kms.zoo.network,
and that page carried <script src="https://analytics.hanzo.ai/script.js">. A Zoo
surface was reporting usage into a Hanzo analytics property, using the direct
script tag that was retired everywhere else. Blocking the console kills the
beacon at the edge, no image rebuild required to stop the leak.
Safe: mpc-api-svc has ZERO in cluster consumers and MPC node to node traffic
rides the headless service on 9999/9800, not this host.
Reuses the existing kms-block-console middleware — one deny, applied twice.
Verified: /_/ /api/health / all 403 on mpc.zoo.network, analytics references
served drop 2 -> 0, and api/explore/rpc/bridge/work.zoo.network all still 200.
Co-authored-by: Hanzo Dev <[email protected]>
kms.zoo.network published the Zoo KMS admin console to the internet with no
middleware at all. The cutover that moved this host from hanzo-k8s to zoo-k8s
(DNS -> 209.38.118.103 zoo-ingress, zoo-kms StatefulSet up 12d) left the guard
behind: the hanzo-side route carried kms-admin-guard, a forwardAuth to
admin-guard, and the zoo-side route carries nothing. The console rendered —
assets load, so the "broken assets" that were previously the only thing keeping
it non-functional no longer apply.
kms-zoo-console-block denies everything that is not /v1 via an ipAllowList on
TEST-NET-1 (RFC5737, never routable) => nothing matches => 403. This also lands
the /v1-everywhere rule at the edge: /api/* is now refused outright.
Verified live: / /admin /_/ /api/status /api/v1/auth/login and the console assets
all 403; /v1/status /v1/health /v1/ping /v1/ all 200; in-cluster kms.zoo-kms.svc
200; api.zoo.ngo, explore.zoo.ngo, rpc/bridge/api/work.zoo.network all still 200.
Also capturing zoo-dynamic itself: it is the Traefik file provider that really
serves these hosts, it was live-only, and k8s/zoo-kms/ingress.yaml (which claims
this cutover is still "staged — HOLD for owner go") was never applied.
Co-authored-by: Hanzo Dev <[email protected]>
Zoo had exactly one working OAuth client (zoo-cloud) and no declaration of it
anywhere. This ports the Lux shape into this repo: the brand-neutral mechanism
stays in hanzoai/iam (internal/provision), Zoo owns its DATA here, and
`make iam-provision` is the only supported way that data reaches the IdP.
ISSUER. id.zoo.network serves iss=https://zoolabs.id — host and issuer really do
disagree, and it is resolved in favour of zoolabs.id: IAM_ISSUER_MAP already
maps all three Zoo hosts to it, discovery agrees from either host, every sibling
brand's issuer is its <brand>.id apex (hanzo.id / lux.id / pars.id) with
id.<brand>.network as the alias, and zoo.id has no DNS at all so it cannot be
the issuer. The remaining defect is the DISCOVERY DOCUMENT (OIDC Discovery §4.3
wants issuer == fetch prefix), which is a frontend/ingress fix, not a data one.
Reasoning is in the file header, not just here.
Five clients, each declared only on evidence measured against the live surface:
zoo-cloud, zoo-wallet, zoo-dao, zoo-explore, zoo-id. Hosts that carry no OAuth
wiring, hosts served by shared Hanzo backends, and hosts with no DNS are listed
too, so each omission is a recorded decision.
Two defects fixed rather than described:
cert. The upsert copies req.Cert verbatim on CREATE, so an omitted cert means
an EMPTY cert, and a client with no cert cannot mint an id_token — the login
dies AFTER a successful code exchange. Four Zoo clients were in exactly that
state. Every app now names cert-zoo (convention cert-<org>: lux-cloud has
cert-lux, hanzo-cloud has cert-hanzo); for zoo-cloud it is a no-op.
cloud.zoo.network is no longer declared. It has an ingress but NO DNS record
at 1.1.1.1 or 8.8.8.8, so no browser can reach it; a redirect URI for a host
that cannot exist is the drift this document exists to prevent. Lux keeps
cloud.lux.network because that host does resolve.
Image pin moved forward to sha-f9813f6: sha-4bec5ef predates the `cert` field
and the strict parser rejects the document outright. f9813f6 is a strict
descendant of the public-client fixes and of 4b0f4e93, and is the newest sha-
tag in GHCR that resolves to a known commit.
PROVEN, not asserted:
- converge run twice; the five records are byte-identical apart from
updatedAt, and no client secret was issued (all five stay public/PKCE)
- real login as [email protected] -> authorization code -> PKCE token exchange with
NO client secret, on all five clients
- id_token minted for each: kid=cert-zoo, iss=https://zoolabs.id,
aud=<the client>, and the signature verifies against the published JWKS
- the dropped redirect now correctly 400s: invalid redirect_uri
Co-authored-by: Hanzo Dev <[email protected]>
kms/mpc.zoo.network cut over to the zoo cluster (Traefik file-provider CM +
Cloudflare DNS-01 + DNS flip to zoo LB; hanzo CR entry + ingresses removed).
Verified: public kms.zoo.network health 200 + zoo-kms login 200.
Zoo DAO + Z Safes deployed on 200200 (0x9011 @ live nonce 91, default salts):
DAO 0x229599f227231d8C90fcF1a78589F5DC4b7A6962 (1/1 owner 0x9011)
Z 0xb8BbCdde5E8713f9849D1520D71Bc08Af94E6F00 (1/1 owner 0x9011)
TL 0x31C89A0509F7783b742B01Be9103c05BbF4B9510 (minDelay 86400, DAO proposer)
Block-production PROVEN: 5-set produced 13348-13354, all 5 converged.
(nonce-0 predictions 0x5232/0x3Be3 were unachievable; tool adapts to live nonce.)
Co-authored-by: Hanzo Dev <[email protected]>
- chain.yaml genesisFile: ../genesis/... → ./configs/genesis/... (each
brand owns its L1 genesis per Task #109 move from luxfi/genesis)
- {brand} template token → {name} (CLI dropped the legacy alias)
- configs/genesis/localnet/genesis.json: seed from devnet so
`lux up zoo/localnet` resolves
- Set top-level networkID in each env genesis to match the sovereign-L1
ID per env (200200/200201/200202)
Migrate Zoo from L2-on-Lux to sovereign L1. Per-env primary networkID
equals per-env EVM chainID (200200 / 200201 / 200202 / 200203).
Explicit httpPort + stakingPort per env (no collisions; the old 9680
dev/local clash is gone — now 9720 / 9730).
chain.yaml:
* networks.<env>.{networkID, primaryEvmChainID, httpPort, stakingPort,
genesisFile, rpcUrl, cluster, namespace} explicit per env
* runtime: block — dataDir / snapshotName / serviceLabel templates
that the lux CLI (pkg/brand) expands per (network, env)
Makefile: collapsed from 16 hardcoded targets (4 verbs × 4 envs with
NID_MAIN := 200200 duplication) to 6 parametric lines:
make up env=devnet ≡ lux up zoo/devnet
make cycle env=mainnet ≡ lux cycle zoo/mainnet
make cycle-all
Per-env constants live in chain.yaml — single source of truth.
deploy.sh removed: replaced by `lux <verb>` calls via Makefile.
LLM.md: documents the new sovereign-L1 model, CLI grammar
(`lux up/down/snap/cycle <network>/<env>`), chain.lock invariant,
per-brand DOKS cluster topology, and the zoo-private/universe sibling.
zeekayandHanzo Dev