The fix-suffix workaround (hanzofix/zoofix/spcfix/parsfix) is dead.
luxfi/genesis bootstrap-chain now derives the P-chain CreateChainTx
name from canonical genesis content (commit 3a072e7 there), so
re-bootstraps are idempotent and never collide on P-chain. The
user-facing brand identifier ("hanzo", "zoo", "spc", "pars") lives
exclusively at this chain-aliases ConfigMap layer.
Per-cluster changes:
- lux-testnet + lux-devnet chain-aliases.yaml: alias map now has
only brand names; legacy "fix" suffix removed. BIDs point at the
current live backing chains (live cluster CMs already updated via
kubectl apply 2026-06-02).
- testnet explorer.yaml: stale "re-bootstrapped as zoofix" comment
removed (the comment leaked a workaround name into config).
- testnet ingress-chains.yaml: "backed by zoofix chain" → "backed
by blockchain ID LXuiihy…WeQ".
- testnet network.yaml, api-gateway-gateway-config.yaml,
dead-chains-cm.yaml, lux-indexer.yaml: comments updated to refer
to live BIDs/aliases instead of stale fix-suffix names.
- lux-mainnet/lux-indexer.yaml: stale "parsv4/spcv4/zoofix" example
list removed.
- lux-k8s/api-gateway/ingresses.yaml: "LXuiihy... zoofix dormant"
note rewritten to reflect promotion to live zoo backing chain.
- lux-k8s/validators/crd-resources.yaml: indexer-zoo CR no longer
sets chainAlias: zoofix or routes /ext/bc/zoofix/rpc; routes
/ext/bc/LXuiihy…WeQ/rpc + relies on alias resolution for /zoo/rpc.
No BID changes here — only naming hygiene. Live cluster CM updates
were applied imperatively on 2026-06-02 (chain-aliases + explorer-
chains in lux-testnet and lux-devnet namespaces).
The fix-suffix workaround (hanzofix/zoofix/spcfix/parsfix) is dead.
luxfi/genesis bootstrap-chain now derives the P-chain CreateChainTx
name from canonical genesis content (commit 3a072e7 there), so
re-bootstraps are idempotent and never collide on P-chain. The
user-facing brand identifier ("hanzo", "zoo", "spc", "pars") lives
exclusively at this chain-aliases ConfigMap layer.
Per-cluster changes:
- lux-testnet + lux-devnet chain-aliases.yaml: alias map now has
only brand names; legacy "fix" suffix removed. BIDs point at the
current live backing chains (live cluster CMs already updated via
kubectl apply 2026-06-02).
- testnet explorer.yaml: stale "re-bootstrapped as zoofix" comment
removed (the comment leaked a workaround name into config).
- testnet ingress-chains.yaml: "backed by zoofix chain" → "backed
by blockchain ID LXuiihy…WeQ".
- testnet network.yaml, api-gateway-gateway-config.yaml,
dead-chains-cm.yaml, lux-indexer.yaml: comments updated to refer
to live BIDs/aliases instead of stale fix-suffix names.
- lux-mainnet/lux-indexer.yaml: stale "parsv4/spcv4/zoofix" example
list removed.
- lux-k8s/api-gateway/ingresses.yaml: "LXuiihy... zoofix dormant"
note rewritten to reflect promotion to live zoo backing chain.
- lux-k8s/validators/crd-resources.yaml: indexer-zoo CR no longer
sets chainAlias: zoofix or routes /ext/bc/zoofix/rpc; routes
/ext/bc/LXuiihy…WeQ/rpc + relies on alias resolution for /zoo/rpc.
No BID changes here — only naming hygiene. Live cluster CM updates
were applied imperatively on 2026-06-02 (chain-aliases + explorer-
chains in lux-testnet and lux-devnet namespaces).
v1.28.19 was tagged from 9df3e16f30 (LP-023 ZAP-native activation
cutover) BEFORE Blue's v3.x safety net commits landed. v1.28.20 is
rebuilt from main HEAD 2e7012af0e which includes:
- Blue v3.0 (15b25a2f16) — R6V4 CRITICAL: CreateSovereignL1Tx
zero-validator / zero-chain gate (must reject in Verify())
- Blue v3.5 (d51942d18d) — RESERVED bytes zero-gate + AddressList
CI audit gate + cross-blob aliasing contract
- Blue v3.6 (30c9608255) — R7V5 mempool admission gate for
zap_native + R7V7 Verify() for RegisterL1ValidatorTx and
ConvertNetworkToL1Tx + R7V8 MustVerify rename + CI gate
- bench fix (2e7012af0e) — disable_legacy_test underflow on
ZAPActivationUnix=0
Without R7V5 the mempool would accept zap_native L1 txs that the
current executor cannot process under always-on activation — this
bump closes that consensus risk before the per-env rollout.
v1.28.19 was tagged from 9df3e16f30 (LP-023 ZAP-native activation
cutover) BEFORE Blue's v3.x safety net commits landed. v1.28.20 is
rebuilt from main HEAD 2e7012af0e which includes:
- Blue v3.0 (15b25a2f16) — R6V4 CRITICAL: CreateSovereignL1Tx
zero-validator / zero-chain gate (must reject in Verify())
- Blue v3.5 (d51942d18d) — RESERVED bytes zero-gate + AddressList
CI audit gate + cross-blob aliasing contract
- Blue v3.6 (30c9608255) — R7V5 mempool admission gate for
zap_native + R7V7 Verify() for RegisterL1ValidatorTx and
ConvertNetworkToL1Tx + R7V8 MustVerify rename + CI gate
- bench fix (2e7012af0e) — disable_legacy_test underflow on
ZAPActivationUnix=0
Without R7V5 the mempool would accept zap_native L1 txs that the
current executor cannot process under always-on activation — this
bump closes that consensus risk before the per-env rollout.
Pins the gha-runner-scale-set values for the lux-build pool on
do-sfo3-lux-k8s. The pool was missing 2026-06-02 (helm release gone,
unclear when removed) — v1.28.19 build sat queued 28+ minutes until
reinstalled. Committing the values file so reinstall is reproducible.
Org separation: luxfi/* workflows MUST runs-on: lux-build (this pool);
hanzoai/* workflows MUST runs-on: hanzo-build-linux-amd64. Cross-org
runs-on queues silently and times out at 24h.
Install:
helm pull oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set --version 0.14.0 --destination /tmp
helm --kube-context=do-sfo3-lux-k8s -n arc-system install lux-build \\
/tmp/gha-runner-scale-set-0.14.0.tgz \\
-f infra/k8s/arc/values-build-amd64.yaml
Pins the gha-runner-scale-set values for the lux-build pool on
do-sfo3-lux-k8s. The pool was missing 2026-06-02 (helm release gone,
unclear when removed) — v1.28.19 build sat queued 28+ minutes until
reinstalled. Committing the values file so reinstall is reproducible.
Org separation: luxfi/* workflows MUST runs-on: lux-build (this pool);
hanzoai/* workflows MUST runs-on: hanzo-build-linux-amd64. Cross-org
runs-on queues silently and times out at 24h.
Install:
helm pull oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set --version 0.14.0 --destination /tmp
helm --kube-context=do-sfo3-lux-k8s -n arc-system install lux-build \\
/tmp/gha-runner-scale-set-0.14.0.tgz \\
-f infra/k8s/arc/values-build-amd64.yaml
LP-023 ZAP-native activation cutover. ZAPActivationUnix=0 (always-on)
— the forward-dated 2026-07-01 activation is dead. ZAP wire is
mandatory from genesis; legacy linearcodec is reachable only via
LUXD_ENABLE_LEGACY_CODEC=1 for read-path back-compat.
Also includes EVM v0.18.18 HashScheme default fix for the fresh L2
chain-create wedge (hanzo/zoo/spc/pars) — same fix as v1.28.18.
Operator v0.3.12 / v0.6.2 does not reconcile image tag on adoptExisting
StatefulSets — explicit pod-by-pod restart required after this sync
(kubectl delete pod luxd-N, wait for 1/1, repeat).
Authorized by 2026-06-02 destructive recovery sweep.
LP-023 ZAP-native activation cutover. ZAPActivationUnix=0 (always-on)
— the forward-dated 2026-07-01 activation is dead. ZAP wire is
mandatory from genesis; legacy linearcodec is reachable only via
LUXD_ENABLE_LEGACY_CODEC=1 for read-path back-compat.
Also includes EVM v0.18.18 HashScheme default fix for the fresh L2
chain-create wedge (hanzo/zoo/spc/pars) — same fix as v1.28.18.
Operator v0.3.12 / v0.6.2 does not reconcile image tag on adoptExisting
StatefulSets — explicit pod-by-pod restart required after this sync
(kubectl delete pod luxd-N, wait for 1/1, repeat).
Authorized by 2026-06-02 destructive recovery sweep.
Adds A records pointing to 134.199.138.27 (lux-k8s hanzoai/ingress LB)
for the 8 brand explorer SPA hosts and 8 backend API hosts that were
declared in k8s/lux-{testnet,devnet}/ingress-chains.yaml but had no
DNS records to back them.
Added (16 total):
explore-{hanzo,zoo,spc,pars}-{test,dev}.lux.network (8 SPA)
api-explore-{hanzo,zoo,spc,pars}-{test,dev}.lux.network (8 API)
IngressRoutes + explorer Service+Deployment already existed in
lux-testnet and lux-devnet namespaces (mainnet cluster, 84d old).
The unified Go explorer (ghcr.io/luxfi/explorer:1.2.11) was already
indexing all 5 chains per env into per-slug SQLite. The only blocker
was DNS.
Records pushed to Cloudflare zone 287bdfd07cf016bd102f6394892e4759
via global API key (script bash scripts/dns-sync.sh cannot create
new records under the existing API token; used the cloudflare-api-
credentials k8s Secret from lux-system ns).
Verified live (curl --resolve to bypass local resolver lag):
All 8 SPA hosts: HTTP 200
All 8 API hosts: HTTP 200 on /stats with real per-chain data
hanzo-test: 696 blocks, 679 txs
spc-test: 398 blocks, 394 txs
pars-test: 381 blocks, 353 txs
zoo-test: 1 block, 0 txs (chain bootstrapped, no traffic)
hanzo-dev: 6 blocks, 5 txs
spc-dev: 165 blocks, 164 txs
pars-dev: 89 blocks, 86 txs
zoo-dev: 0 blocks, 0 txs (chain bootstrapped, no traffic)
Sister commit: 18863e19 (mainnet api-explore-* routing).
Adds A records pointing to 134.199.138.27 (lux-k8s hanzoai/ingress LB)
for the 8 brand explorer SPA hosts and 8 backend API hosts that were
declared in k8s/lux-{testnet,devnet}/ingress-chains.yaml but had no
DNS records to back them.
Added (16 total):
explore-{hanzo,zoo,spc,pars}-{test,dev}.lux.network (8 SPA)
api-explore-{hanzo,zoo,spc,pars}-{test,dev}.lux.network (8 API)
IngressRoutes + explorer Service+Deployment already existed in
lux-testnet and lux-devnet namespaces (mainnet cluster, 84d old).
The unified Go explorer (ghcr.io/luxfi/explorer:1.2.11) was already
indexing all 5 chains per env into per-slug SQLite. The only blocker
was DNS.
Records pushed to Cloudflare zone 287bdfd07cf016bd102f6394892e4759
via global API key (script bash scripts/dns-sync.sh cannot create
new records under the existing API token; used the cloudflare-api-
credentials k8s Secret from lux-system ns).
Verified live (curl --resolve to bypass local resolver lag):
All 8 SPA hosts: HTTP 200
All 8 API hosts: HTTP 200 on /stats with real per-chain data
hanzo-test: 696 blocks, 679 txs
spc-test: 398 blocks, 394 txs
pars-test: 381 blocks, 353 txs
zoo-test: 1 block, 0 txs (chain bootstrapped, no traffic)
hanzo-dev: 6 blocks, 5 txs
spc-dev: 165 blocks, 164 txs
pars-dev: 89 blocks, 86 txs
zoo-dev: 0 blocks, 0 txs (chain bootstrapped, no traffic)
Sister commit: 18863e19 (mainnet api-explore-* routing).
Split the inline UPGRADE_JSON shell heredoc out of startup.sh into a
dedicated `cchain-upgrade.json` data key in the luxd-startup ConfigMap.
The startup script now cp's /scripts/cchain-upgrade.json to the
C-Chain alias dir only (CHAIN_CONFIG_DIR/C/upgrade.json), removing
the prior loop that overwrote EVERY EVM chain's upgrade.json (incl.
brand L2s like hanzo/zoo/pars/spc which carry their own activations
baked at block 0).
Per Red round-2 review vector 5 (HIGH): overwriting brand L2
upgrade.json with the C-Chain schedule would either be a redundant
no-op or — worse — re-schedule an already-live precompile to a
forward timestamp and brick the chain via
extras.ChainConfig.checkPrecompileCompatible. C-Chain ONLY scope
eliminates the foot-gun.
The embedded cchain-upgrade.json data block is byte-identical
(semantically, modulo whitespace) to luxfi/genesis
configs/mainnet/upgrade.json — sync rule documented in-line.
Split the inline UPGRADE_JSON shell heredoc out of startup.sh into a
dedicated `cchain-upgrade.json` data key in the luxd-startup ConfigMap.
The startup script now cp's /scripts/cchain-upgrade.json to the
C-Chain alias dir only (CHAIN_CONFIG_DIR/C/upgrade.json), removing
the prior loop that overwrote EVERY EVM chain's upgrade.json (incl.
brand L2s like hanzo/zoo/pars/spc which carry their own activations
baked at block 0).
Per Red round-2 review vector 5 (HIGH): overwriting brand L2
upgrade.json with the C-Chain schedule would either be a redundant
no-op or — worse — re-schedule an already-live precompile to a
forward timestamp and brick the chain via
extras.ChainConfig.checkPrecompileCompatible. C-Chain ONLY scope
eliminates the foot-gun.
The embedded cchain-upgrade.json data block is byte-identical
(semantically, modulo whitespace) to luxfi/genesis
configs/mainnet/upgrade.json — sync rule documented in-line.
The luxd-startup ConfigMap was previously hand-applied to the lux-devnet
namespace (do-sfo3-lux-k8s); pin it here so future reconciles converge.
Two runtime limitations now documented inline:
1. info.isBootstrapped(chain="hanzo"|"zoo"|"pars"|"spc") returns
"there is no chain with alias/ID 'hanzo'" because the canonical
chain-aliases.json file trips a json.Unmarshal failure on the
production luxd v1.28.16 binary (errMissingQuotes from
ids.ID.UnmarshalJSON). The same 262-byte JSON parses fine in
local unit tests; production fails. Workaround: omit
--chain-aliases-file*; chains are reachable via blockchain.name
so /ext/bc/<brand>/rpc keeps working — only the info.* alias
lookup is degraded (cosmetic). Tracked in luxfi/ids.
2. Writing bootstrap-l2-chain-configs blobs as ongoing per-chain
/data/configs/chains/<blockchainID>/config.json triggers
PANIC in eth.New: triedb parent [...] layer missing. Those
configs are CreateChainTx-time only. Startup now aggressively
removes any stale L2 config dirs from previous reconciler runs.
L2 mapping (consistent with chain-aliases CM in this overlay and
luxfi/genesis@06dc203 configs/<brand>-devnet/genesis.json):
blockchainID brand chainID
i3AxuEfLAqzAvabgpYvXC7QCvqXnJAHr3AYiVzfK2FDauyJSa hanzo 36964
5YP3iV6ytRjtscF8dcXWQTqd8dwjAaoZnW6oNVfDM3d2mwwvS zoo 200202
JxDPdhGde9PSrZ3nLwe7shwZaXEFBoAJ8V9rGqbcKXy9FEQEm pars 494951
LHU4rNBLr1Zi2pNJXL47djkCmmyDSgTXmaB4oNQBYyGfqPtg9 spc 36912
Verified end-to-end via luxd-0..3 LBs: eth_chainId returns the
correct chainID per brand and matches luxfi/genesis@06dc203.
Refs #168.
The luxd-startup ConfigMap was previously hand-applied to the lux-devnet
namespace (do-sfo3-lux-k8s); pin it here so future reconciles converge.
Two runtime limitations now documented inline:
1. info.isBootstrapped(chain="hanzo"|"zoo"|"pars"|"spc") returns
"there is no chain with alias/ID 'hanzo'" because the canonical
chain-aliases.json file trips a json.Unmarshal failure on the
production luxd v1.28.16 binary (errMissingQuotes from
ids.ID.UnmarshalJSON). The same 262-byte JSON parses fine in
local unit tests; production fails. Workaround: omit
--chain-aliases-file*; chains are reachable via blockchain.name
so /ext/bc/<brand>/rpc keeps working — only the info.* alias
lookup is degraded (cosmetic). Tracked in luxfi/ids.
2. Writing bootstrap-l2-chain-configs blobs as ongoing per-chain
/data/configs/chains/<blockchainID>/config.json triggers
PANIC in eth.New: triedb parent [...] layer missing. Those
configs are CreateChainTx-time only. Startup now aggressively
removes any stale L2 config dirs from previous reconciler runs.
L2 mapping (consistent with chain-aliases CM in this overlay and
luxfi/genesis@06dc203 configs/<brand>-devnet/genesis.json):
blockchainID brand chainID
i3AxuEfLAqzAvabgpYvXC7QCvqXnJAHr3AYiVzfK2FDauyJSa hanzo 36964
5YP3iV6ytRjtscF8dcXWQTqd8dwjAaoZnW6oNVfDM3d2mwwvS zoo 200202
JxDPdhGde9PSrZ3nLwe7shwZaXEFBoAJ8V9rGqbcKXy9FEQEm pars 494951
LHU4rNBLr1Zi2pNJXL47djkCmmyDSgTXmaB4oNQBYyGfqPtg9 spc 36912
Verified end-to-end via luxd-0..3 LBs: eth_chainId returns the
correct chainID per brand and matches luxfi/genesis@06dc203.
Refs #168.
Brand explorer API hosts were targeting dead per-brand services
(explorer-{brand}:4000) replaced by unified Go indexer at
explorer-mainnet-v2:8090 (lux-explorer ns). Cross-ns via existing
explorer-unified ExternalName Service in lux-mainnet ns.
Adds brand-prefix middleware chain (mainpage-rewrite +
blockscout-prefix-{brand} + cors-allow-all) so unified backend
dispatches to per-slug SQLite via /v1/indexer/{brand} prefix —
same pattern as lux-explorer/lux-explorer-mainnet-chains for X/P/D.
Verified: api-explore-hanzo.lux.network now hits unified backend
(content-security-policy header present); 404 on /v1/indexer/hanzo
expected until per-brand indexer pods come up (chain-layer work).
Brand explorer API hosts were targeting dead per-brand services
(explorer-{brand}:4000) replaced by unified Go indexer at
explorer-mainnet-v2:8090 (lux-explorer ns). Cross-ns via existing
explorer-unified ExternalName Service in lux-mainnet ns.
Adds brand-prefix middleware chain (mainpage-rewrite +
blockscout-prefix-{brand} + cors-allow-all) so unified backend
dispatches to per-slug SQLite via /v1/indexer/{brand} prefix —
same pattern as lux-explorer/lux-explorer-mainnet-chains for X/P/D.
Verified: api-explore-hanzo.lux.network now hits unified backend
(content-security-policy header present); 404 on /v1/indexer/hanzo
expected until per-brand indexer pods come up (chain-layer work).
Duplicates lux-mainnet/explorer-chains with wrong pars chainId 494949
(canonical: 7070) and stale blockchainID. Unmounted — lux-explorer ns
has no pods, only routing layer. Single source of truth now
lux-mainnet/explorer-chains (consumed by explorer Deployment in
lux-mainnet).
Duplicates lux-mainnet/explorer-chains with wrong pars chainId 494949
(canonical: 7070) and stale blockchainID. Unmounted — lux-explorer ns
has no pods, only routing layer. Single source of truth now
lux-mainnet/explorer-chains (consumed by explorer Deployment in
lux-mainnet).
CreateChainTx fired on testnet + devnet P-chains for all 4 brand L2 EVMs
(hanzo / zoo / spc / pars) — 8 chains total, all isBootstrapped=true on
all 3 pods per env (24/24 verified). Each chain uses the canonical EVM
plugin (vmID mgj786NP7uDwBCcq6YwThhaN8FLyybkCa4zBWTQbNgmK6k9A6) and its
own per-chain validator-network (CreateNetworkTx ID, L2-anchor pattern).
All 5 primary validators added as chain validators via
IssueAddChainValidatorTx.
EVM chain IDs match the LP-182 canonical map per (brand, env):
hanzo: test=36962 dev=36964
zoo : test=200201 dev=200202
spc : test=36910 dev=36912
pars : test=494950 dev=494951
Required a second wipe-and-rebootstrap of both primaries because #186's
fresh genesis funded all 50T LUX into the validator stake (zero spendable
P-chain balance for the deployer). Added a 6th allocation per genesis (a
deterministic SHA-256 deployer key — 100K LUX spendable, NOT in
initialStakedFunds). Reapplied lux-{testnet,devnet}-genesis ConfigMap,
scaled luxd StatefulSet 3→0→3, P + C re-bootstrapped in ~75s on all pods.
Snapshots from #186 retained (data-luxd-{0..2}-20260602).
Inventory committed at
k8s/lux-{test,dev}-k8s/{testnet,devnet}/chains-inventory.json (8 chains:
blockchainID + per-chain networkID + evmChainID + bootstrap timestamps).
chain-aliases ConfigMap committed for each env mapping
brand-name → blockchainID for operator + gateway convenience.
Tool used: ~/work/lux/genesis/cmd/bootstrap-chain (lux/genesis@HEAD —
adds --key-file flag) against ~/work/lux/node/wallet/network/primary
(lux/node@HEAD — fail-soft FetchState when X-Chain disabled).
Followups:
- Roll block 1 on each L2 (eth_blockNumber > 0) via heartbeat tx when
downstream contract deploys land.
- Update lux-gateway/gateway.yaml routing for /testnet/* and /devnet/*
paths to point at the new lux-{test,dev}-k8s namespaces (currently
still routes to legacy lux-testnet/lux-devnet namespaces).
- Migrate ~/work/lux/keys/deployer-test-dev/private.key to KMS once
kms-operator is deployed in test+dev clusters.
CreateChainTx fired on testnet + devnet P-chains for all 4 brand L2 EVMs
(hanzo / zoo / spc / pars) — 8 chains total, all isBootstrapped=true on
all 3 pods per env (24/24 verified). Each chain uses the canonical EVM
plugin (vmID mgj786NP7uDwBCcq6YwThhaN8FLyybkCa4zBWTQbNgmK6k9A6) and its
own per-chain validator-network (CreateNetworkTx ID, L2-anchor pattern).
All 5 primary validators added as chain validators via
IssueAddChainValidatorTx.
EVM chain IDs match the LP-182 canonical map per (brand, env):
hanzo: test=36962 dev=36964
zoo : test=200201 dev=200202
spc : test=36910 dev=36912
pars : test=494950 dev=494951
Required a second wipe-and-rebootstrap of both primaries because #186's
fresh genesis funded all 50T LUX into the validator stake (zero spendable
P-chain balance for the deployer). Added a 6th allocation per genesis (a
deterministic SHA-256 deployer key — 100K LUX spendable, NOT in
initialStakedFunds). Reapplied lux-{testnet,devnet}-genesis ConfigMap,
scaled luxd StatefulSet 3→0→3, P + C re-bootstrapped in ~75s on all pods.
Snapshots from #186 retained (data-luxd-{0..2}-20260602).
Inventory committed at
k8s/lux-{test,dev}-k8s/{testnet,devnet}/chains-inventory.json (8 chains:
blockchainID + per-chain networkID + evmChainID + bootstrap timestamps).
chain-aliases ConfigMap committed for each env mapping
brand-name → blockchainID for operator + gateway convenience.
Tool used: ~/work/lux/genesis/cmd/bootstrap-chain (lux/genesis@HEAD —
adds --key-file flag) against ~/work/lux/node/wallet/network/primary
(lux/node@HEAD — fail-soft FetchState when X-Chain disabled).
Followups:
- Roll block 1 on each L2 (eth_blockNumber > 0) via heartbeat tx when
downstream contract deploys land.
- Update lux-gateway/gateway.yaml routing for /testnet/* and /devnet/*
paths to point at the new lux-{test,dev}-k8s namespaces (currently
still routes to legacy lux-testnet/lux-devnet namespaces).
- Migrate ~/work/lux/keys/deployer-test-dev/private.key to KMS once
kms-operator is deployed in test+dev clusters.
luxfi now served by host-level arcd daemons (dbc Mac + evo WSL +
evo Windows) alongside hanzoai, zooai, parsdao, zenlm. See
hanzoai/universe/infra/runners/README.md for the fleet topology.
luxfi now served by host-level arcd daemons (dbc Mac + evo WSL +
evo Windows) alongside hanzoai, zooai, parsdao, zenlm. See
hanzoai/universe/infra/runners/README.md for the fleet topology.
v1.28.18 ships EVM plugin v0.18.18, which fixes the
"panic in eth.New: triedb parent [<EmptyRootHash>] layer missing" wedge
on freshly-provisioned L2 EVM chains (hanzo, zoo, spc, pars). Root cause
is plugin/evm/vm.go letting eth/backend.go inherit geth's path-by-default
on empty DBs while simultaneously hard-refusing path mode upfront —
detail in evm@1dea806f8 (v0.18.18). The fix locks the operator-empty
StateScheme to HashScheme before backend.go can pick "path".
v1.28.18 ships EVM plugin v0.18.18, which fixes the
"panic in eth.New: triedb parent [<EmptyRootHash>] layer missing" wedge
on freshly-provisioned L2 EVM chains (hanzo, zoo, spc, pars). Root cause
is plugin/evm/vm.go letting eth/backend.go inherit geth's path-by-default
on empty DBs while simultaneously hard-refusing path mode upfront —
detail in evm@1dea806f8 (v0.18.18). The fix locks the operator-empty
StateScheme to HashScheme before backend.go can pick "path".
luxd v1.24+ removed --track-subnets in favor of --track-chains (the
flag accepts both chain and blockchain IDs). compose.local.yml was
still passing the dead flag; switching to --track-chains=all preserves
behavior.
docs/PRODUCTION.md: rewrite stale ./wallet/network/primary/examples/
deploy-subnets/ paths to deploy-chains/ which is the actual example
directory in luxfi/node.
luxd v1.24+ removed --track-subnets in favor of --track-chains (the
flag accepts both chain and blockchain IDs). compose.local.yml was
still passing the dead flag; switching to --track-chains=all preserves
behavior.
docs/PRODUCTION.md: rewrite stale ./wallet/network/primary/examples/
deploy-subnets/ paths to deploy-chains/ which is the actual example
directory in luxfi/node.
Phase A status: BLOCKED on luxfi/operator v0.6.3 + v0.7.0 binaries
(see luxfi/operator d2ecca5 — duplicate Network reconciler
registration makes both crashloop at startup). Cluster rolled back
to v0.3.12 on lux.network/v1alpha1 reconcile, all 3 chains 5/5
healthy.
What this commit lays down for the next attempt (once a v0.6.4 or
v0.7.1 image ships with the operator fix):
- `scripts/migrate-luxnetwork-crds.sh <ctx>` — race-clean CR
migration script. Walks lux.network/v1alpha1 LuxNetwork,
LuxChain, LuxGateway, LuxIndexer, LuxExplorer, LuxMPC CRs,
apiVersion-swaps each to lux.cloud/v1 (dropping the Lux
prefix on every Kind except LuxNetwork which v0.6.x also
calls LuxNetwork in the canonical group). Spec/status pass
through opaque — both CRDs use
x-kubernetes-preserve-unknown-fields: true. Strips server-side
metadata. Idempotent.
- `k8s/backups/pre-phase-a-20260602-020926/` — full live state
snapshot for the 3 DOKS lux clusters, captured BEFORE any
in-cluster mutation. Used to verify migration and as the rollback
source if the next attempt needs to re-restore CRs.
- `k8s/PHASE_A_ROLLBACK.md` — step-by-step rollback procedure.
Inventory (lux-k8s only — test-k8s / dev-k8s have zero lux CRs):
- 3 LuxNetworks (luxnetworks.lux.network) — mainnet/testnet/devnet luxd
- 11 LuxChains (luxchains.lux.network) — mainnet a/b/d/g/i/k/m/o/q/r/z
- 3 LuxGateways (luxgateways.lux.network) — */api-gateway
- 1 LuxMPC (luxmpcs.lux.network) — mainnet/lux-mpc
- 0 LuxIndexers, 0 LuxExplorers
In-cluster prep that survived rollback:
- lux.cloud/v1 CRDs installed alongside lux.network/v1alpha1 (idempotent
apply of `~/work/lux/operator/spec/crd/bases/lux.cloud/`).
- Migrated lux.cloud/v1 CRs already created (apply succeeded, no
controller bound — inert until v0.6.4+ operator ships).
Next attempt blocked on:
1. CI build of v0.6.4 or v0.7.1 operator image with luxfi/operator
d2ecca5 fix (push triggers via `hanzo-build-linux-amd64` ARC pool).
2. Bump `k8s/lux-k8s/operator/deployment.yaml` image tag.
3. Re-run scale-down → image-bump → scale-up cycle.
Forward-only, no AI slop, no API breakage, current chains stay alive
during entire procedure.
Phase A status: BLOCKED on luxfi/operator v0.6.3 + v0.7.0 binaries
(see luxfi/operator d2ecca5 — duplicate Network reconciler
registration makes both crashloop at startup). Cluster rolled back
to v0.3.12 on lux.network/v1alpha1 reconcile, all 3 chains 5/5
healthy.
What this commit lays down for the next attempt (once a v0.6.4 or
v0.7.1 image ships with the operator fix):
- `scripts/migrate-luxnetwork-crds.sh <ctx>` — race-clean CR
migration script. Walks lux.network/v1alpha1 LuxNetwork,
LuxChain, LuxGateway, LuxIndexer, LuxExplorer, LuxMPC CRs,
apiVersion-swaps each to lux.cloud/v1 (dropping the Lux
prefix on every Kind except LuxNetwork which v0.6.x also
calls LuxNetwork in the canonical group). Spec/status pass
through opaque — both CRDs use
x-kubernetes-preserve-unknown-fields: true. Strips server-side
metadata. Idempotent.
- `k8s/backups/pre-phase-a-20260602-020926/` — full live state
snapshot for the 3 DOKS lux clusters, captured BEFORE any
in-cluster mutation. Used to verify migration and as the rollback
source if the next attempt needs to re-restore CRs.
- `k8s/PHASE_A_ROLLBACK.md` — step-by-step rollback procedure.
Inventory (lux-k8s only — test-k8s / dev-k8s have zero lux CRs):
- 3 LuxNetworks (luxnetworks.lux.network) — mainnet/testnet/devnet luxd
- 11 LuxChains (luxchains.lux.network) — mainnet a/b/d/g/i/k/m/o/q/r/z
- 3 LuxGateways (luxgateways.lux.network) — */api-gateway
- 1 LuxMPC (luxmpcs.lux.network) — mainnet/lux-mpc
- 0 LuxIndexers, 0 LuxExplorers
In-cluster prep that survived rollback:
- lux.cloud/v1 CRDs installed alongside lux.network/v1alpha1 (idempotent
apply of `~/work/lux/operator/spec/crd/bases/lux.cloud/`).
- Migrated lux.cloud/v1 CRs already created (apply succeeded, no
controller bound — inert until v0.6.4+ operator ships).
Next attempt blocked on:
1. CI build of v0.6.4 or v0.7.1 operator image with luxfi/operator
d2ecca5 fix (push triggers via `hanzo-build-linux-amd64` ARC pool).
2. Bump `k8s/lux-k8s/operator/deployment.yaml` image tag.
3. Re-run scale-down → image-bump → scale-up cycle.
Forward-only, no AI slop, no API breakage, current chains stay alive
during entire procedure.
v1.28.16 = v1.28.15 + Dockerfile EVM_VERSION bump only. luxfi/evm v0.18.16
fixes the nil-chainConfig panic at the PQ gate: vm.chainConfig.PQ was set
BEFORE vm.chainConfig was assigned from g.Config, panic on dereference.
Repro path is fresh-PVC bootstrap with chain-config.json pq:true causing
"panic in eth.New: ... triedb parent ... layer missing" — exactly the
wedge currently blocking lux-mainnet C-Chain.
Files touched (one canonical pin per net):
k8s/lux-mainnet/network.yaml v1.28.15 -> v1.28.16
k8s/lux-testnet/network.yaml v1.23.31 -> v1.28.16 (manifest was
stale; live pods on v1.28.7)
k8s/lux-devnet/network.yaml v1.28.15 -> v1.28.16
k8s/lux-k8s/validators/statefulset.yaml 3x v1.28.7 -> v1.28.16
k8s/lux-test-k8s/testnet/statefulset.yaml 2x v1.28.7 -> v1.28.16
Note: operator on lux-k8s is v0.3.12 (manifest pins v0.6.3 but cluster
has not been reconciled). v0.3.12 does NOT reconcile spec.image.tag on
LuxNetwork CRs, so a CR-only apply will not change running pods. Direct
StatefulSet patch or operator bump is needed to actually roll forward.
v1.28.16 = v1.28.15 + Dockerfile EVM_VERSION bump only. luxfi/evm v0.18.16
fixes the nil-chainConfig panic at the PQ gate: vm.chainConfig.PQ was set
BEFORE vm.chainConfig was assigned from g.Config, panic on dereference.
Repro path is fresh-PVC bootstrap with chain-config.json pq:true causing
"panic in eth.New: ... triedb parent ... layer missing" — exactly the
wedge currently blocking lux-mainnet C-Chain.
Files touched (one canonical pin per net):
k8s/lux-mainnet/network.yaml v1.28.15 -> v1.28.16
k8s/lux-testnet/network.yaml v1.23.31 -> v1.28.16 (manifest was
stale; live pods on v1.28.7)
k8s/lux-devnet/network.yaml v1.28.15 -> v1.28.16
k8s/lux-k8s/validators/statefulset.yaml 3x v1.28.7 -> v1.28.16
k8s/lux-test-k8s/testnet/statefulset.yaml 2x v1.28.7 -> v1.28.16
Note: operator on lux-k8s is v0.3.12 (manifest pins v0.6.3 but cluster
has not been reconciled). v0.3.12 does NOT reconcile spec.image.tag on
LuxNetwork CRs, so a CR-only apply will not change running pods. Direct
StatefulSet patch or operator bump is needed to actually roll forward.
Three independent fixes wedged together unblock 5 of 7 mainnet chains:
1. genesis.json alloc field names ethAddr/luxAddr → evmAddr/utxoAddr.
luxd v1.28.7 uses luxfi/genesis v1.12.19+ which requires the canonical
names. The embedded base64 blob is the source of truth for amounts
and addresses — a sed translates field names in-place post-decode.
Idempotent.
2. track-chains adds C-Chain's content-derived blockchain ID
(9SUt666By9zgBK2PUw6scnsW1xHCCEuFVNT6HkjTK6mHGeENG, computed by
migrateNewGenesisChains from the cChainGenesis blob in P-Chain
genesis). Without this the SybilProtection gate in
platformvm/config/internal.go.CreateChain silently drops C-Chain
even though it is in P-Chain state.
3. tenant-import loop sentinels only on curl success (rc=0). The prior
unconditional touch/rm-marker consumed markers on transient failures
(luxd not ready, plugin missing, chain not registered) and left the
import un-applied with no retry. Now failed imports preserve the
marker for the next reconcile.
State after deploy:
P-Chain: bootstrapped (preserved h~260, sybil-protected, never wiped)
C-Chain: live at h=0 (new genesis, RLP replay blocked by genesis
hash mismatch with archival snapshot — separate task)
Hanzo L1 EVM: alive at h=0 after per-chain DB wipe
Zoo L1 EVM: preserved at h=799 (RLP-imported, untouched)
Pars/SPC L1 EVMs: wedged on shared EVM triedb cache (orphan refs
persist after chainData wipe — separate task)
X-Chain: P-only mode by design (XChainGenesis empty in mainnet genesis)
Three independent fixes wedged together unblock 5 of 7 mainnet chains:
1. genesis.json alloc field names ethAddr/luxAddr → evmAddr/utxoAddr.
luxd v1.28.7 uses luxfi/genesis v1.12.19+ which requires the canonical
names. The embedded base64 blob is the source of truth for amounts
and addresses — a sed translates field names in-place post-decode.
Idempotent.
2. track-chains adds C-Chain's content-derived blockchain ID
(9SUt666By9zgBK2PUw6scnsW1xHCCEuFVNT6HkjTK6mHGeENG, computed by
migrateNewGenesisChains from the cChainGenesis blob in P-Chain
genesis). Without this the SybilProtection gate in
platformvm/config/internal.go.CreateChain silently drops C-Chain
even though it is in P-Chain state.
3. tenant-import loop sentinels only on curl success (rc=0). The prior
unconditional touch/rm-marker consumed markers on transient failures
(luxd not ready, plugin missing, chain not registered) and left the
import un-applied with no retry. Now failed imports preserve the
marker for the next reconcile.
State after deploy:
P-Chain: bootstrapped (preserved h~260, sybil-protected, never wiped)
C-Chain: live at h=0 (new genesis, RLP replay blocked by genesis
hash mismatch with archival snapshot — separate task)
Hanzo L1 EVM: alive at h=0 after per-chain DB wipe
Zoo L1 EVM: preserved at h=799 (RLP-imported, untouched)
Pars/SPC L1 EVMs: wedged on shared EVM triedb cache (orphan refs
persist after chainData wipe — separate task)
X-Chain: P-only mode by design (XChainGenesis empty in mainnet genesis)
luxd-startup.yaml drops the hardcoded c-chain (.cchain_rlp_imported_v4)
and zoo (.zoo_rlp_imported_v1) blocks. Replaced with a generic loop
over operator-written JSON request markers at
/data/db/.tenant-import-request-*.json. The loop parses chainAlias,
rlpPath, and sentinelPath out of each marker, calls admin_importChain,
touches the sentinel, removes the marker. Zero hardcoded tenant names.
network.yaml adds C-Chain to tenantImports[] alongside Zoo, with
chainAlias on each (required by luxfi/operator v0.7.0).
Pairs with luxfi/[email protected]: spec change → new content hash →
operator emits new request marker → loop re-fires → fresh sentinel.
Re-firing after a fix is automatic. No version-string bumping.
luxd-startup.yaml drops the hardcoded c-chain (.cchain_rlp_imported_v4)
and zoo (.zoo_rlp_imported_v1) blocks. Replaced with a generic loop
over operator-written JSON request markers at
/data/db/.tenant-import-request-*.json. The loop parses chainAlias,
rlpPath, and sentinelPath out of each marker, calls admin_importChain,
touches the sentinel, removes the marker. Zero hardcoded tenant names.
network.yaml adds C-Chain to tenantImports[] alongside Zoo, with
chainAlias on each (required by luxfi/operator v0.7.0).
Pairs with luxfi/[email protected]: spec change → new content hash →
operator emits new request marker → loop re-fires → fresh sentinel.
Re-firing after a fix is automatic. No version-string bumping.
The previous commit bumped the image tag from v0.3.12 → v0.6.3 but
left three other surfaces stale, any of which would have crashlooped
the pod on `kubectl apply`:
1. Deployment name. YAML said `metadata.name: operator` but the
cluster's actual deployment is `lux-operator`. `kubectl apply -f
deployment.yaml` would have created a parallel `operator`
Deployment alongside the running `lux-operator`, never updating
the running one. Fixed: rename to `lux-operator` to match the
cluster.
2. Flag surface. v0.6.3 (go/cmd/operator/main.go) takes
--api-group / --namespace / --operator-namespace /
--metrics-bind-address / --health-probe-bind-address /
--leader-elect / --leader-election-id. v0.3.x took
--log-level / --metrics-port / --health-port / --leader-election.
None of the v0.3.x flags exist in v0.6.3 — pflag.Parse would
error and the pod would crashloop. Args rewritten to the
v0.6.3 surface.
3. ClusterRole. v0.3.x used `apiGroups: ["lux.network"]` with
prefixed plurals (luxchains, luxindexers, luxexplorers,
luxgateways). v0.6.3 canonical group is `lux.cloud` (see
api/v1/groupversion_info.go DefaultGroup) with unprefixed
plurals (chains, networks, validators, etc.). The CRD list
matches the 20 canonical kinds + Network/LuxNetwork emitted
by cmd/generate-crd-yaml at HEAD. Also added `leases` /
`serviceaccounts` / Rbac for the controllers that need them.
Other deltas:
- Added `imagePullSecrets: [{name: ghcr-luxfi}]` (cluster already has
the secret, age 96d; the previous YAML relied on the cluster
having a node-level credential which works today but is brittle).
- `imagePullPolicy: IfNotPresent` (was implicit; explicit is better
for semver-pinned tags — no need to re-pull when the same digest
is already present).
Post-apply readiness probe:
kubectl --context=do-sfo3-lux-k8s -n lux-system rollout status \
deploy/lux-operator --timeout=180s
The previous commit bumped the image tag from v0.3.12 → v0.6.3 but
left three other surfaces stale, any of which would have crashlooped
the pod on `kubectl apply`:
1. Deployment name. YAML said `metadata.name: operator` but the
cluster's actual deployment is `lux-operator`. `kubectl apply -f
deployment.yaml` would have created a parallel `operator`
Deployment alongside the running `lux-operator`, never updating
the running one. Fixed: rename to `lux-operator` to match the
cluster.
2. Flag surface. v0.6.3 (go/cmd/operator/main.go) takes
--api-group / --namespace / --operator-namespace /
--metrics-bind-address / --health-probe-bind-address /
--leader-elect / --leader-election-id. v0.3.x took
--log-level / --metrics-port / --health-port / --leader-election.
None of the v0.3.x flags exist in v0.6.3 — pflag.Parse would
error and the pod would crashloop. Args rewritten to the
v0.6.3 surface.
3. ClusterRole. v0.3.x used `apiGroups: ["lux.network"]` with
prefixed plurals (luxchains, luxindexers, luxexplorers,
luxgateways). v0.6.3 canonical group is `lux.cloud` (see
api/v1/groupversion_info.go DefaultGroup) with unprefixed
plurals (chains, networks, validators, etc.). The CRD list
matches the 20 canonical kinds + Network/LuxNetwork emitted
by cmd/generate-crd-yaml at HEAD. Also added `leases` /
`serviceaccounts` / Rbac for the controllers that need them.
Other deltas:
- Added `imagePullSecrets: [{name: ghcr-luxfi}]` (cluster already has
the secret, age 96d; the previous YAML relied on the cluster
having a node-level credential which works today but is brittle).
- `imagePullPolicy: IfNotPresent` (was implicit; explicit is better
for semver-pinned tags — no need to re-pull when the same digest
is already present).
Post-apply readiness probe:
kubectl --context=do-sfo3-lux-k8s -n lux-system rollout status \
deploy/lux-operator --timeout=180s
Bug: wipe-check on .cchain_rlp_imported_v4 (correct since 57f0119d)
but the sentinel-write line still touched .cchain_rlp_imported_v3.
Result: import succeeds, writes v3 sentinel, next reconcile sees
v4 absent, wipes + re-imports — infinite loop.
This whole shell-script-sentinel approach is a kludge. Tracked for
removal in #122 (decomplect into operator-driven content-hash
addressed tenantImports[]).
Bug: wipe-check on .cchain_rlp_imported_v4 (correct since 57f0119d)
but the sentinel-write line still touched .cchain_rlp_imported_v3.
Result: import succeeds, writes v3 sentinel, next reconcile sees
v4 absent, wipes + re-imports — infinite loop.
This whole shell-script-sentinel approach is a kludge. Tracked for
removal in #122 (decomplect into operator-driven content-hash
addressed tenantImports[]).
ghcr.io/luxfi/operator was stuck at v0.3.12 since 2026-04-26 because
every Docker build since the canonical-reusable migration produced a
startup_failure (zero artifacts published — verified GHCR empty for
v0.4.x, v0.5.x, v0.6.0, v0.6.1, v0.6.2).
luxfi/[email protected] ships:
- Declarative VM plugin fetch + chain-id symlinks (PluginSource).
Replaces the hand-managed `cp -f /luxd/build/plugins/* + ln -sf` step
from luxd-startup.yaml.
- Per-(tenant, version) TenantImports with sentinel + request-marker
discipline for first-boot RLP imports (closes#50).
- GenesisConfigMap mount support — bake-in path for #91 (declarative
canonical genesis CMs replace the embedded base64 blobs in the
luxd-startup ConfigMap).
- CRD generator dedup (Network was listed twice, second overwrote
first → 23 files instead of 24; CI count updated to 23).
- Regenerated CRD bases for all four universes (lux.cloud, hanzo.ai,
zoo.cloud, osage.cloud); metadata.name now matches `<plural>.<group>`
(previously stale `nets.<group>` from the Subnet→Net era would have
been rejected by k8s).
- Inline Docker workflow (cross-org workflow_call to hanzoai/.github
remains broken in practice; replicated luxfi/kms self-contained
pattern → first successful Docker build in the repo's history).
Image verified pullable:
$ docker manifest inspect ghcr.io/luxfi/operator:v0.6.3
schemaVersion: 2, mediaType: application/vnd.docker.distribution.manifest.v2+json
Bring-up sequence (the swarm may fire these directly):
kubectl --context=do-sfo3-lux-k8s -n lux-system apply -f k8s/lux-k8s/operator/deployment.yaml
kubectl --context=do-sfo3-lux-k8s -n lux-system rollout restart deploy/operator
kubectl --context=do-sfo3-lux-k8s -n lux-system rollout status deploy/operator --timeout=180s
ghcr.io/luxfi/operator was stuck at v0.3.12 since 2026-04-26 because
every Docker build since the canonical-reusable migration produced a
startup_failure (zero artifacts published — verified GHCR empty for
v0.4.x, v0.5.x, v0.6.0, v0.6.1, v0.6.2).
luxfi/[email protected] ships:
- Declarative VM plugin fetch + chain-id symlinks (PluginSource).
Replaces the hand-managed `cp -f /luxd/build/plugins/* + ln -sf` step
from luxd-startup.yaml.
- Per-(tenant, version) TenantImports with sentinel + request-marker
discipline for first-boot RLP imports (closes#50).
- GenesisConfigMap mount support — bake-in path for #91 (declarative
canonical genesis CMs replace the embedded base64 blobs in the
luxd-startup ConfigMap).
- CRD generator dedup (Network was listed twice, second overwrote
first → 23 files instead of 24; CI count updated to 23).
- Regenerated CRD bases for all four universes (lux.cloud, hanzo.ai,
zoo.cloud, osage.cloud); metadata.name now matches `<plural>.<group>`
(previously stale `nets.<group>` from the Subnet→Net era would have
been rejected by k8s).
- Inline Docker workflow (cross-org workflow_call to hanzoai/.github
remains broken in practice; replicated luxfi/kms self-contained
pattern → first successful Docker build in the repo's history).
Image verified pullable:
$ docker manifest inspect ghcr.io/luxfi/operator:v0.6.3
schemaVersion: 2, mediaType: application/vnd.docker.distribution.manifest.v2+json
Bring-up sequence (the swarm may fire these directly):
kubectl --context=do-sfo3-lux-k8s -n lux-system apply -f k8s/lux-k8s/operator/deployment.yaml
kubectl --context=do-sfo3-lux-k8s -n lux-system rollout restart deploy/operator
kubectl --context=do-sfo3-lux-k8s -n lux-system rollout status deploy/operator --timeout=180s
zeekay
Hanzo AI