The five Lux mainnet validators are docker containers on the LAN host evo,
started by hand: no compose file, no unit, no script. deploy/lux-mainnet/luxd.yaml
addresses a Service on a cluster this estate no longer has, so the only record of
the flags they carry was the running processes themselves.
scripts/validators.sh carries those flags and rolls the fleet one node at a time,
verifying each is back on the version asked for with four peers before touching
the next -- sample 5 / quorum 4 means two down is the network.
Co-authored-by: Hanzo Dev <[email protected]>
github.com/luxfi/universe is canonical from here. git.hanzo.ai/lux/universe is
unreachable, and this merge carries its lineage over. The rule applied, per
area:
Workflows: one set, under .hanzo/workflows; .github/workflows is gone.
LLM.md, README.md, deploy/README.md: the forge's. GitHub's sections were the
ones the forge had already rewritten (the /mainnet/v1/bc path table, the
auto-bump flow), not divergent content. Counts brought to what this tree
holds; LLM.md records the rules below.
deploy/applicationset.yaml: the forge's object (apps.hanzo.ai/v1, destination
name lux-cluster) with repoURL https://github.com/luxfi/universe.
deploy/*.yaml modified on both sides: a three-way merge, not a whole-file pick.
Where both moved the pin, the newer semver wins; at the same semver, the side
that pinned a digest (exchange-api 2.3.34 is sha256:3eae48ac, which the
registry confirms; GitHub's zoo copy carried 007beb83, the pre-repush bytes).
Where both wrote cd.automated, GitHub's false wins: it is the later decision
(909d3bd) and the one about the cluster CD now runs on. true stays only on the
static sites GitHub verified there. The forge's /v1/bc -> /v1/chain rename and
its digests are carried into files whose other hunks came from GitHub
(explorer, exchange-api x3, rpc-tip-gate, lux-market).
lux-bank/lux-financial: GitHub's v1.5.5 is not in the registry (the newest
published tag is v1.5.3), so the pin is v1.5.3 with its digest. A pin never
names an image the registry does not have.
Static sites (lux-audio, lux-cloud/cloud, lux-credit, lux-fund, lux-lps,
lux-network, zoo/zoo-exchange): GitHub's three-object declaration — a
staticFiles Middleware, the route that names it, the endpoint-less Service,
no image — replaces the forge's pod. A union would put an Ingress and a
priority-100 IngressRoute on one host with a pod behind it serving nothing.
The forge's last pins, for the record:
lux-audio ghcr.io/luxfi/audio:latest@sha256:48265c37c91f
lux-cloud ghcr.io/luxfi/cloud-web:v1.5.34@sha256:54a586fbd4d6
lux-credit ghcr.io/lux-apps/credit:36b1c2dffd62
lux-fund ghcr.io/lux-apps/fund:20cb7cb988b3 (amd64 only)
lux-lps ghcr.io/luxfi/lps:1.0.2
lux-network ghcr.io/lux-apps/network:latest@sha256:b52dd80f59d7
zoo-exchange ghcr.io/luxfi/exchange:5.149.73@sha256:baefb1903851
Lifted objects (lux-mainnet/{explorer,lux-build,lux-market,rpc-tip-gate}):
the forge's file — the claim, routes, middlewares and rbac it lifted out of
k8s/ — with GitHub's cd.automated: false.
deploy/hanzo/{cloud,commerce-site,iam,lux-bitcoin,lux-coin,lux-finance,
lux-industries,lux-market,lux-safe}.yaml, deploy/lux-ns/lux-exchange.yaml,
deploy/lux-web/lux-network.yaml: deleted, as GitHub did in 2079731. Each Lux
site is declared once, as files, in its own namespace; hanzo's own services
are hanzo/universe's to declare. What the forge had changed in them, so it is
not lost: hanzo/cloud v1.801.476@sha256:0743f907; hanzo/iam IAM_DATABASE_URL
from Secret iam-db; lux-safe ghcr.io/luxfi/safe:v2.2.14; lux-exchange
5.149.83@sha256:7c1bd0e9 with a rateLimit Middleware and the hosts
app.lux.exchange, trade.lux.exchange, exchange.lux.network,
markets.lux.network; lux-web/lux-network at sha 1028f82e.
deploy/zoo-mainnet/: removed. Every Zoo workload is declared in zooai/universe.
Floating tags: every one carries a digest. lux-market and rpc-tip-gate take
the forge's frozen digests back; the rest already had theirs.
k8s/lux-*: the forge lifted k8s duplicates into deploy/ values and deleted
them. luxd-statefulset x3, lux-mainnet/explorer, lux-testnet/{chain-traffic,
exchange-api} stay deleted; what GitHub changed in them (node v1.36.125 pins,
exchange-api 2.3.28, the 50Gi claim, a comment) is behind the values files
(node 1.36.181, exchange-api 2.3.34, the explorer claim). The gateway
configs, luxd-startup ConfigMaps, lux-testnet/explorer and the mainnet
kustomization are the forge's: GitHub's content is a subset of them up to
/v1/bc -> /v1/chain and /v1/health/liveness -> /v1/health/ops/liveness, and
network.yaml, which GitHub's kustomization still lists, no longer exists.
scripts/roll_luxd.sh: the forge's quorum-preserving roll (eviction API, one
pod at a time, digest + version + every-chain bootstrap, devnet -> testnet ->
mainnet) plus GitHub's closing height-and-restart report. The tag is written
in the spelling the values files use (bare 1.36.181) and the digest resolved
for that spelling; a leading v is accepted and stripped. The push remote is
GIT_REMOTE (default origin).
Validated: 123 deploy YAMLs parse; 122 values files render against chart app
0.1.11; no floating tag without a digest; every kustomization that built
before builds now; pin.sh --verify reads 92 pins, 79 agree, 6 are frozen
moving tags whose pointer has since moved (their digests still name real
bytes), 6 are registries a luxfi token cannot read, and lux-financial is the
one it caught.
Co-authored-by: Hanzo Dev <[email protected]>
The bulk retag moved published tags. go.sum and sum.golang.org hold the
original bytes, proxy.golang.org answers not-found, and the direct fetch
disagrees, so builds stopped at a checksum before reaching a compiler.
Pin forward to versions the proxy still serves.
manifests/{devnet,production,testnet}.yml claimed to be "source of truth for
what is deployed on lux-k8s", last edited 2026-04-20, listing tag: latest and
sha: latest per service. Nothing has read them since; the ApplicationSet reads
deploy/, and the only writer that touched them was update-image.yml, which was
sed-ing into a tree CD does not open.
manifests/safes.yaml was the one live file in there and is not a deployment
manifest at all — it is Lux's Safe topology. It moves to SAFES.yaml, beside
NETWORKS.yaml and chain.yaml, where this repo keeps its fact tables.
scripts/deploy-do.sh creates three DigitalOcean droplets, cloud-inits docker on
them and runs compose over ssh. That is the arrangement DOKS replaced.
sync-pars-universe.sh no longer copies it into the Pars bundle.
renovate.json stops matching manifests/. It still does not match deploy/, and
that is now on purpose rather than an oversight: pin.sh resolves a tag and its
digest from one registry read, and a second writer offering a bare tag would
leave the pair disagreeing — which deploys the digest.
Co-authored-by: Hanzo Dev <[email protected]>
The route table lives in configs/gateway/*.json, where each endpoint names both
the public path and the url_pattern sent upstream. Both moved: luxd answers
under /v1/chain and nothing translates between them.
The operational scripts went with it — Makefiles, the roll and validator
helpers, the e2e specs and the market-maker config all composed /v1/bc
addresses that now reach nothing. A test that dials a dead path fails in a way
that looks like the chain is down.
Lines that record an observation keep their original spelling.
Co-authored-by: Hanzo Dev <[email protected]>
The zoo MPC nodes have served their API for as long as they have been up, with
no name pointing at them. Nothing in an Ingress creates a record and no
external-dns runs here, so the host existed only as far as the cluster.
dns-sync.sh could not have created it either: it spoke only Bearer tokens, and
what the estate has deployed for cert-manager is a global key with an email. It
now takes whichever is present, preferring the token, so one credential serves
both jobs instead of a second one being minted to do the same work.
Proxied, like every other zoo.cloud host that resolves. The origin serves the
ingress default certificate, so a direct record reaches a server whose
certificate no client accepts -- on this zone TLS is terminated at the edge or
it is not terminated at all.
Co-authored-by: Hanzo Dev <[email protected]>
Three fleets pinned the same tag under three different digests, which is how a
roll ships old bytes under a new name. The pin is what the CD plane reconciles,
so it is set once — one tag, one digest, every fleet, one commit — and the three
applications are synced together.
Then the fleets drain in parallel with each other and one pod at a time within
themselves. The StatefulSets are OnDelete on purpose: five validators restarting
together take the chain below α and halt it. The estate moves in one motion; the
validators of one chain do not.
A pod is back when it answers its own C-Chain RPC. Not when it reads Ready,
which these fleets never do, and not from a height alone, which an unstarted pod
does not answer and which therefore reads as a lagging one.
Co-authored-by: Hanzo Dev <[email protected]>
A node's C-Chain is a plugin, copied onto the volume by an init container at
boot. Those containers named their own image, and drifted: devnet staged from
v1.36.25 under a v1.36.123 node, mainnet from v1.36.88 under v1.36.116, testnet
from v1.34.0 — a tag with no image at all — under v1.36.123. Between 28 and 98
releases apart.
So every roll moved luxd and left the C-Chain built from an old evm. The plugin
files are rewritten on each boot, which makes them look current by any timestamp
you can check; they were written from a stale source. That is how four fleets ran
a two-month-old plugin under a current binary, and why a state-sync default
shipped in the node image did not reach the chain that needed it.
An artifact check cannot see this. The image is internally consistent — the
deployment swaps the plugin out afterwards — so it is a fact about the manifest
and is checked there: scripts/check_image_match.sh refuses a StatefulSet whose
init containers stage from anything other than the image its node runs.
Co-authored-by: Hanzo Dev <[email protected]>
The zoo MPC nodes have served their API for as long as they have been up, with
no name pointing at them. Nothing in an Ingress creates a record and no
external-dns runs here, so the host existed only as far as the cluster.
dns-sync.sh could not have created it either: it spoke only Bearer tokens, and
what the estate has deployed for cert-manager is a global key with an email. It
now takes whichever is present, preferring the token, so one credential serves
both jobs instead of a second one being minted to do the same work.
Proxied, like every other zoo.cloud host that resolves. The origin serves the
ingress default certificate, so a direct record reaches a server whose
certificate no client accepts -- on this zone TLS is terminated at the edge or
it is not terminated at all.
Co-authored-by: Hanzo Dev <[email protected]>
Three fleets pinned the same tag under three different digests, which is how a
roll ships old bytes under a new name. The pin is what the CD plane reconciles,
so it is set once — one tag, one digest, every fleet, one commit — and the three
applications are synced together.
Then the fleets drain in parallel with each other and one pod at a time within
themselves. The StatefulSets are OnDelete on purpose: five validators restarting
together take the chain below α and halt it. The estate moves in one motion; the
validators of one chain do not.
A pod is back when it answers its own C-Chain RPC. Not when it reads Ready,
which these fleets never do, and not from a height alone, which an unstarted pod
does not answer and which therefore reads as a lagging one.
Co-authored-by: Hanzo Dev <[email protected]>
A node's C-Chain is a plugin, copied onto the volume by an init container at
boot. Those containers named their own image, and drifted: devnet staged from
v1.36.25 under a v1.36.123 node, mainnet from v1.36.88 under v1.36.116, testnet
from v1.34.0 — a tag with no image at all — under v1.36.123. Between 28 and 98
releases apart.
So every roll moved luxd and left the C-Chain built from an old evm. The plugin
files are rewritten on each boot, which makes them look current by any timestamp
you can check; they were written from a stale source. That is how four fleets ran
a two-month-old plugin under a current binary, and why a state-sync default
shipped in the node image did not reach the chain that needed it.
An artifact check cannot see this. The image is internally consistent — the
deployment swaps the plugin out afterwards — so it is a fact about the manifest
and is checked there: scripts/check_image_match.sh refuses a StatefulSet whose
init containers stage from anything other than the image its node runs.
Co-authored-by: Hanzo Dev <[email protected]>
The .network host was retired 2026-07-27 and no longer resolves at all, so every
reference to it named an address that cannot answer. The brand host is .cloud,
matching kms.zoo.cloud and the rest of the white-label convention.
Left alone deliberately, because rewriting them would invert what they say: the
LLM.md line that RECORDS the retirement, the e2e spec that ASSERTS
kms.lux.network must not resolve, and the recorded applies + cluster backups,
which are faithful accounts of what was applied and are not config.
Co-authored-by: Hanzo Dev <[email protected]>
Drop the Avalanche-heritage /ext prefix; /v1 is the single canonical route
surface (one way, no backward compat). The node's baseURL is the source of
truth; clients, SDKs, CLI, indexer, maker, genesis, netrunner, and the
k8s/compose/gateway/explorer configs are updated to match.
Co-authored-by: Hanzo Dev <[email protected]>
Replaces .gitmodules + git-submodule-based init/update/sync targets with
a manifest-driven sibling-clone model. No nested submodule checkouts
inside universe; each ecosystem repo lives at ~/work/lux/<name> as a
free-standing clone pinned to a semver tag (no main/latest/dev floating
tags in production).
- REPOS.yaml: source-of-truth manifest. 11 live repos pinned, 1
(graph-node) still on main with TODO marker. 6 dead refs removed
(explorer-lux, explorer-zoo, explore-lux, graph-node-lux, v2-subgraph,
v3-subgraph — all 404 on github.com/luxfi; the explorer trio is
folded into luxfi/explore which ships unified Next.js SPA + Go
indexer).
- scripts/repos.sh {init|update|list}: reads REPOS.yaml via yq. init
shallow-clones missing repos at the pinned tag. update fetches +
checks out the pinned tag (optionally for a single repo via
name=<repo>).
- Makefile: drops CORE/APP/DATA_SUBMODULES + ALL_SUBMODULES static
lists. init/update/sync/list delegate to scripts/repos.sh. Section
retitled 'Sibling Repos (REPOS.yaml — no submodules)'.
- .gitmodules: deleted (was empty 0-entry leftover).
To bump a pin: edit REPOS.yaml → make update name=<repo> → verify →
commit.
Phase A status: BLOCKED on luxfi/operator v0.6.3 + v0.7.0 binaries
(see luxfi/operator d2ecca5 — duplicate Network reconciler
registration makes both crashloop at startup). Cluster rolled back
to v0.3.12 on lux.network/v1alpha1 reconcile, all 3 chains 5/5
healthy.
What this commit lays down for the next attempt (once a v0.6.4 or
v0.7.1 image ships with the operator fix):
- `scripts/migrate-luxnetwork-crds.sh <ctx>` — race-clean CR
migration script. Walks lux.network/v1alpha1 LuxNetwork,
LuxChain, LuxGateway, LuxIndexer, LuxExplorer, LuxMPC CRs,
apiVersion-swaps each to lux.cloud/v1 (dropping the Lux
prefix on every Kind except LuxNetwork which v0.6.x also
calls LuxNetwork in the canonical group). Spec/status pass
through opaque — both CRDs use
x-kubernetes-preserve-unknown-fields: true. Strips server-side
metadata. Idempotent.
- `k8s/backups/pre-phase-a-20260602-020926/` — full live state
snapshot for the 3 DOKS lux clusters, captured BEFORE any
in-cluster mutation. Used to verify migration and as the rollback
source if the next attempt needs to re-restore CRs.
- `k8s/PHASE_A_ROLLBACK.md` — step-by-step rollback procedure.
Inventory (lux-k8s only — test-k8s / dev-k8s have zero lux CRs):
- 3 LuxNetworks (luxnetworks.lux.network) — mainnet/testnet/devnet luxd
- 11 LuxChains (luxchains.lux.network) — mainnet a/b/d/g/i/k/m/o/q/r/z
- 3 LuxGateways (luxgateways.lux.network) — */api-gateway
- 1 LuxMPC (luxmpcs.lux.network) — mainnet/lux-mpc
- 0 LuxIndexers, 0 LuxExplorers
In-cluster prep that survived rollback:
- lux.cloud/v1 CRDs installed alongside lux.network/v1alpha1 (idempotent
apply of `~/work/lux/operator/spec/crd/bases/lux.cloud/`).
- Migrated lux.cloud/v1 CRs already created (apply succeeded, no
controller bound — inert until v0.6.4+ operator ships).
Next attempt blocked on:
1. CI build of v0.6.4 or v0.7.1 operator image with luxfi/operator
d2ecca5 fix (push triggers via `hanzo-build-linux-amd64` ARC pool).
2. Bump `k8s/lux-k8s/operator/deployment.yaml` image tag.
3. Re-run scale-down → image-bump → scale-up cycle.
Forward-only, no AI slop, no API breakage, current chains stay alive
during entire procedure.
* scripts/kms-crd-host-migrate.sh — bulk-patch every KMSSecret CRD's
hostAPI to point at kms.lux-kms-go.svc.cluster.local. Idempotent.
Already executed against lux-k8s + hanzo-k8s (147 CRDs).
* scripts/kms-orphan-audit.sh — audit which K8s Secrets are NOT backed
by a KMSSecret CRD or whitelisted bootstrap creds. Output:
lux-k8s 323 non-system Secrets, 268 orphan
hanzo-k8s 461 non-system Secrets, 367 orphan
Most orphans are ARC runner state, cert-manager cert content, and
operator-internal config — not appropriate for KMS migration.
Run before sweeping anything.
zeekayandHanzo Dev