* feat(chain): canonical Lux primary L1 chain.yaml
Source-of-truth declaration for the Lux Network: a sovereign primary
L1 with 11 chains under one networkID per env (1/2/3 for
mainnet/testnet/devnet). Hosts L2 children (Hanzo, Zoo) that opt in
via primary-network staking.
Model:
Network (this file's `network` block) is the sovereign primary L1.
Chains live under it; CreateChainTx(networkID, vmID, genesis) each.
All chains share the Lux validator set and the same networkID.
Spelled out:
- networkID (uint32) vs evmChainID (uint64) — distinct fields at
distinct scopes; Lux deliberately keeps them different (mainnet
networkID=1, C-Chain evmChainID=96369), unlike single-brand forks
(<tenant> / Pars / Osage) that match by convention.
- Lux chain set: P + X + C + Q + Z + A + B + D + G + K + T.
- Minimum required for a Lux primary is P + Q + Z (per the PQZ
design); the rest are opt-in but ship in stock mainnet today.
- L2 children listed for inventory only; canonical spec lives in
each L2's own brand universe.
Per-env identifiers sourced from lux/genesis/configs/<env>/{network,
*chain}.json — single source of truth.
* chore(lux-devnet): rebuild on luxd 1.23.31, L2 bootstrap blocked
Orchestrator Option (c) executed: STS scaled to 0, all 5 PVCs wiped,
image bumped 1.27.21 -> 1.23.31, scaled to 5 (Ready in 84s after fresh
PVC bind). P-X-C-D-chains bootstrapped on all 5 nodes, full mesh peers
(4/each), sybil-protection-enabled=true throughout.
bootstrap-l2 blocked: control key (BIP44 idx=5) has 0 nLUX unlocked
vs 1.5 LUX required for CreateNetworkTx. Genesis funds only idx 0-4
(consumed by initialStakedFunds for 8 stakers). All other addrs have
unlockSchedule=[] which v1.23.31 treats as locked-forever. v1.27.21
treated [] as fully-vested at startTime — that's why 5/27 worked.
Mainnet & testnet untouched, both already on 1.23.31 5/5 Ready.
Hand-off options in chain-aliases.yaml annotation.
* fix(lux-devnet): sync luxd-genesis CM with canonical source genesis
The cluster's luxd-genesis ConfigMap had drifted from the canonical source
(~/work/lux/genesis/configs/devnet/genesis.json): 1000 non-staker
allocations had unlockSchedule:[] which v1.23.31 treats as locked-forever
(no UTXOs minted). This included the BIP44 idx=5 deployer key
(P-dev1jx9p2hxfemcgdlylth05zj20klsn4t06s0t5kv) at 0 nLUX unlocked, blocking
all L2 EVM bootstrap (CreateNetworkTx needs 1.5 LUX fee).
The fix is to re-emit the CM from the canonical source genesis which
already has the correct unlockSchedule={amount,locktime:0} on all 1000
non-staker allocs (and preserves the 5 vesting allocs unchanged).
After kubectl replace + STS scale 0→5 + PVC recreate, the deployer key has
50000000000000 nLUX = 50,000 LUX unlocked at genesis time, sufficient for
4× L2 CreateNetworkTx + CreateChainTx + AddChainValidatorTx flows.
Note: kubectl apply hits the 256KB metadata.annotations limit on this file
(it stuffs the entire genesis into last-applied-configuration). Use
kubectl replace -f instead.
Refs: lux/genesis/configs/devnet/genesis.json (canonical, unchanged)
Refs: issue #88 control-key funding (devnet-status annotation in
chain-aliases CM)
* feat(lux-devnet): record 3/4 L2 EVM chains live after #88 fix
After the luxd-genesis re-sync (commit 84417785), the deployer key has
50,000 LUX unlocked at genesis and the bootstrap-l2 tool can run.
Chains bootstrapped:
- hanzo ganKVYMn5f3DmeEdHLvSwTMSRW9BP2zSSy4VaFX1Qye4JKGVb
- zoo 2EkjfV4YimDz9uY3VNo5BUeCWKfdc2Bt4rvN9vtap1sbokoSBT
- pars HRgWS1AYU5rMxw5ADqXx4HLfREkLnCy7SRaJveScSiRSc1bt3
spc CreateChainTx hits intermittent consensus stall: in the proposer
logs I see numValidators=4 (not 5) on the proposed block, so the alpha
threshold (3 votes of 5 sample with all-or-nothing peers) isn't met.
Same proposer block (height 19, blockDataLen 3641) replays from mempool
but doesn't finalize. luxd-0 restarted once during the run while the
others stayed up — likely the genesis-vs-status reconciliation
heisenbug. spc retry is a no-op for the genesis fix; needs a window
where all 5 luxd pods are simultaneously responsive.
Refs: lux/genesis/cmd/bootstrap-l2 (idempotent, will skip the 3 live
chains on next run and only attempt spc).
* feat(chain): canonical Lux primary L1 chain.yaml
Source-of-truth declaration for the Lux Network: a sovereign primary
L1 with 11 chains under one networkID per env (1/2/3 for
mainnet/testnet/devnet). Hosts L2 children (Hanzo, Zoo) that opt in
via primary-network staking.
Model:
Network (this file's `network` block) is the sovereign primary L1.
Chains live under it; CreateChainTx(networkID, vmID, genesis) each.
All chains share the Lux validator set and the same networkID.
Spelled out:
- networkID (uint32) vs evmChainID (uint64) — distinct fields at
distinct scopes; Lux deliberately keeps them different (mainnet
networkID=1, C-Chain evmChainID=96369), unlike single-brand forks
(<tenant> / Pars / Osage) that match by convention.
- Lux chain set: P + X + C + Q + Z + A + B + D + G + K + T.
- Minimum required for a Lux primary is P + Q + Z (per the PQZ
design); the rest are opt-in but ship in stock mainnet today.
- L2 children listed for inventory only; canonical spec lives in
each L2's own brand universe.
Per-env identifiers sourced from lux/genesis/configs/<env>/{network,
*chain}.json — single source of truth.
* chore(lux-devnet): rebuild on luxd 1.23.31, L2 bootstrap blocked
Orchestrator Option (c) executed: STS scaled to 0, all 5 PVCs wiped,
image bumped 1.27.21 -> 1.23.31, scaled to 5 (Ready in 84s after fresh
PVC bind). P-X-C-D-chains bootstrapped on all 5 nodes, full mesh peers
(4/each), sybil-protection-enabled=true throughout.
bootstrap-l2 blocked: control key (BIP44 idx=5) has 0 nLUX unlocked
vs 1.5 LUX required for CreateNetworkTx. Genesis funds only idx 0-4
(consumed by initialStakedFunds for 8 stakers). All other addrs have
unlockSchedule=[] which v1.23.31 treats as locked-forever. v1.27.21
treated [] as fully-vested at startTime — that's why 5/27 worked.
Mainnet & testnet untouched, both already on 1.23.31 5/5 Ready.
Hand-off options in chain-aliases.yaml annotation.
* fix(lux-devnet): sync luxd-genesis CM with canonical source genesis
The cluster's luxd-genesis ConfigMap had drifted from the canonical source
(~/work/lux/genesis/configs/devnet/genesis.json): 1000 non-staker
allocations had unlockSchedule:[] which v1.23.31 treats as locked-forever
(no UTXOs minted). This included the BIP44 idx=5 deployer key
(P-dev1jx9p2hxfemcgdlylth05zj20klsn4t06s0t5kv) at 0 nLUX unlocked, blocking
all L2 EVM bootstrap (CreateNetworkTx needs 1.5 LUX fee).
The fix is to re-emit the CM from the canonical source genesis which
already has the correct unlockSchedule={amount,locktime:0} on all 1000
non-staker allocs (and preserves the 5 vesting allocs unchanged).
After kubectl replace + STS scale 0→5 + PVC recreate, the deployer key has
50000000000000 nLUX = 50,000 LUX unlocked at genesis time, sufficient for
4× L2 CreateNetworkTx + CreateChainTx + AddChainValidatorTx flows.
Note: kubectl apply hits the 256KB metadata.annotations limit on this file
(it stuffs the entire genesis into last-applied-configuration). Use
kubectl replace -f instead.
Refs: lux/genesis/configs/devnet/genesis.json (canonical, unchanged)
Refs: issue #88 control-key funding (devnet-status annotation in
chain-aliases CM)
* feat(lux-devnet): record 3/4 L2 EVM chains live after #88 fix
After the luxd-genesis re-sync (commit 84417785), the deployer key has
50,000 LUX unlocked at genesis and the bootstrap-l2 tool can run.
Chains bootstrapped:
- hanzo ganKVYMn5f3DmeEdHLvSwTMSRW9BP2zSSy4VaFX1Qye4JKGVb
- zoo 2EkjfV4YimDz9uY3VNo5BUeCWKfdc2Bt4rvN9vtap1sbokoSBT
- pars HRgWS1AYU5rMxw5ADqXx4HLfREkLnCy7SRaJveScSiRSc1bt3
spc CreateChainTx hits intermittent consensus stall: in the proposer
logs I see numValidators=4 (not 5) on the proposed block, so the alpha
threshold (3 votes of 5 sample with all-or-nothing peers) isn't met.
Same proposer block (height 19, blockDataLen 3641) replays from mempool
but doesn't finalize. luxd-0 restarted once during the run while the
others stayed up — likely the genesis-vs-status reconciliation
heisenbug. spc retry is a no-op for the genesis fix; needs a window
where all 5 luxd pods are simultaneously responsive.
Refs: lux/genesis/cmd/bootstrap-l2 (idempotent, will skip the 3 live
chains on next run and only attempt spc).
Source-of-truth declaration for the Lux Network: a sovereign primary
L1 with 11 chains under one networkID per env (1/2/3 for
mainnet/testnet/devnet). Hosts L2 children (Hanzo, Zoo) that opt in
via primary-network staking.
Model:
Network (this file's `network` block) is the sovereign primary L1.
Chains live under it; CreateChainTx(networkID, vmID, genesis) each.
All chains share the Lux validator set and the same networkID.
Spelled out:
- networkID (uint32) vs evmChainID (uint64) — distinct fields at
distinct scopes; Lux deliberately keeps them different (mainnet
networkID=1, C-Chain evmChainID=96369), unlike single-brand forks
(<tenant> / Pars / Osage) that match by convention.
- Lux chain set: P + X + C + Q + Z + A + B + D + G + K + T.
- Minimum required for a Lux primary is P + Q + Z (per the PQZ
design); the rest are opt-in but ship in stock mainnet today.
- L2 children listed for inventory only; canonical spec lives in
each L2's own brand universe.
Per-env identifiers sourced from lux/genesis/configs/<env>/{network,
*chain}.json — single source of truth.
Source-of-truth declaration for the Lux Network: a sovereign primary
L1 with 11 chains under one networkID per env (1/2/3 for
mainnet/testnet/devnet). Hosts L2 children (Hanzo, Zoo) that opt in
via primary-network staking.
Model:
Network (this file's `network` block) is the sovereign primary L1.
Chains live under it; CreateChainTx(networkID, vmID, genesis) each.
All chains share the Lux validator set and the same networkID.
Spelled out:
- networkID (uint32) vs evmChainID (uint64) — distinct fields at
distinct scopes; Lux deliberately keeps them different (mainnet
networkID=1, C-Chain evmChainID=96369), unlike single-brand forks
(<tenant> / Pars / Osage) that match by convention.
- Lux chain set: P + X + C + Q + Z + A + B + D + G + K + T.
- Minimum required for a Lux primary is P + Q + Z (per the PQZ
design); the rest are opt-in but ship in stock mainnet today.
- L2 children listed for inventory only; canonical spec lives in
each L2's own brand universe.
Per-env identifiers sourced from lux/genesis/configs/<env>/{network,
*chain}.json — single source of truth.
The bootno.de TLS+rate-limit cutover (hanzo/universe#87) intentionally
skipped exchange hostnames since they live on lux-k8s + bootnode-k8s, not
hanzo-k8s. Those hosts are now also off Cloudflare's grey-cloud WAF, so
they need the same in-cluster floor. This wires it in:
api.lux.exchange (+ api.lux.network, api.zoo.network) — bootnode-k8s,
ingress-nginx (class `nginx`). Codifies the live `web3-platform-api-apex`
Ingress (previously kubectl-applied only) into k8s/bootnode/ingress.yaml
with the canonical nginx.ingress.kubernetes.io/limit-rps=50, limit-rpm=1000,
limit-burst-multiplier=2 (effective burst=100), limit-connections=20.
lux.exchange / www.lux.exchange / trade.lux.exchange / exchange.lux.network
— lux-k8s lux-ns, hanzoai/ingress (Traefik). Adds Middleware
`lux-exchange-rate-limit` (100 rps / 200 burst — SPA is loose because no
body POSTs hit it) and binds it to a new Ingress manifest at
k8s/lux-ns/lux-exchange.yaml.
zoo.exchange + 3 white-label hosts — lux-k8s zoo, same SPA pattern.
Middleware `zoo-exchange-rate-limit` bound via traefik.ingress
.kubernetes.io/router.middlewares.
api.zoo.exchange — lux-k8s lux-mainnet. KrakenD-backed (same engine that
drives api.lux.network). 50 rps / 100 burst to match the upstream apex.
Drive-by fix: `api-gateway-gateway` service port was paste-typo'd at 8080
in this file; live state was 80 and Traefik was logging "service port not
found" for it (which is why the original audit reported the 404). Pinned
to 80 to match the cluster.
Verification (in-cluster, bootnode-k8s ingress-nginx):
1020 reqs at ~113 rps sustained → 554x404 (passed through to upstream)
466x429 (rate-limited at limit-rps=50)
Cluster-wide ingress-nginx-controller cm patched to return 429 instead
of the 503 default (limit-req-status-code, limit-conn-status-code).
The bootno.de TLS+rate-limit cutover (hanzo/universe#87) intentionally
skipped exchange hostnames since they live on lux-k8s + bootnode-k8s, not
hanzo-k8s. Those hosts are now also off Cloudflare's grey-cloud WAF, so
they need the same in-cluster floor. This wires it in:
api.lux.exchange (+ api.lux.network, api.zoo.network) — bootnode-k8s,
ingress-nginx (class `nginx`). Codifies the live `web3-platform-api-apex`
Ingress (previously kubectl-applied only) into k8s/bootnode/ingress.yaml
with the canonical nginx.ingress.kubernetes.io/limit-rps=50, limit-rpm=1000,
limit-burst-multiplier=2 (effective burst=100), limit-connections=20.
lux.exchange / www.lux.exchange / trade.lux.exchange / exchange.lux.network
— lux-k8s lux-ns, hanzoai/ingress (Traefik). Adds Middleware
`lux-exchange-rate-limit` (100 rps / 200 burst — SPA is loose because no
body POSTs hit it) and binds it to a new Ingress manifest at
k8s/lux-ns/lux-exchange.yaml.
zoo.exchange + 3 white-label hosts — lux-k8s zoo, same SPA pattern.
Middleware `zoo-exchange-rate-limit` bound via traefik.ingress
.kubernetes.io/router.middlewares.
api.zoo.exchange — lux-k8s lux-mainnet. KrakenD-backed (same engine that
drives api.lux.network). 50 rps / 100 burst to match the upstream apex.
Drive-by fix: `api-gateway-gateway` service port was paste-typo'd at 8080
in this file; live state was 80 and Traefik was logging "service port not
found" for it (which is why the original audit reported the 404). Pinned
to 80 to match the cluster.
Verification (in-cluster, bootnode-k8s ingress-nginx):
1020 reqs at ~113 rps sustained → 554x404 (passed through to upstream)
466x429 (rate-limited at limit-rps=50)
Cluster-wide ingress-nginx-controller cm patched to return 429 instead
of the 503 default (limit-req-status-code, limit-conn-status-code).
Live do-sfo3-lux-k8s/lux-system carries two `status*` deployments:
- `status` — orphan, 92d old, image ghcr.io/hanzoai/status:latest
(no source-of-truth in this repo, references CMs
status-config/status-frontend/status-brand/status-logo
that also have no source; second pod has been
ContainerCreating for 5+ days). Policy violation —
floating :latest tag.
- `status-lux` — canonical, sha-pinned, declared in k8s/status/.
Add k8s/status/kustomization.yaml so the canonical stack is Kustomize-
applyable (was previously raw YAML).
Add k8s/status/orphan-cleanup.sh — production-impacting cleanup
script with all the kubectl delete commands needed to remove the
orphan deployment + its dangling ConfigMaps + PVC. NOT executed in
this commit; flagged in LLM.md for confirmation before run.
LLM.md additions:
- Decomplection audit section recording the lux-system orphan state
- Decomplection audit section recording the luxd-startup.yaml bash bag
+ the operator-side decomplection plan (see luxfi/operator
decomplect/luxnetwork-declarative-startup@d5b3d26)
Live do-sfo3-lux-k8s/lux-system carries two `status*` deployments:
- `status` — orphan, 92d old, image ghcr.io/hanzoai/status:latest
(no source-of-truth in this repo, references CMs
status-config/status-frontend/status-brand/status-logo
that also have no source; second pod has been
ContainerCreating for 5+ days). Policy violation —
floating :latest tag.
- `status-lux` — canonical, sha-pinned, declared in k8s/status/.
Add k8s/status/kustomization.yaml so the canonical stack is Kustomize-
applyable (was previously raw YAML).
Add k8s/status/orphan-cleanup.sh — production-impacting cleanup
script with all the kubectl delete commands needed to remove the
orphan deployment + its dangling ConfigMaps + PVC. NOT executed in
this commit; flagged in LLM.md for confirmation before run.
LLM.md additions:
- Decomplection audit section recording the lux-system orphan state
- Decomplection audit section recording the luxd-startup.yaml bash bag
+ the operator-side decomplection plan (see luxfi/operator
decomplect/luxnetwork-declarative-startup@d5b3d26)
The testnet LuxGateway and LuxIndexer manifests pointed at four blockchain
IDs that no longer exist on the live testnet P-Chain (a prior testnet
generation). All four routes were 404-ing; the indexers were tailing
nothing. Update to the IDs currently registered on the testnet P-Chain:
zoo -> 2BSjBuuRHUMGdhw9HzcmiRMtWEBsVcS98tjeF3x5F99T1BfbQo (alias zoofix)
hanzo -> 2ZUpGYPyaDsmuwnMUrhkcP3C24RYHvGWTZZYPaNr4oPPdUmnh3
pars -> 26a3qkQ5VzTwFDBerz2a9kte8GPqS1wJpguc9x4BHz26YXrtL9
spc -> Sd9bKzGu5Mzh6nafNecxicWJLdDmc9DB2aQPN6EUb1KfknReh
(testnet C-Chain stays B76NXURYwskvstwnyQfEpaaTpSgLrWBNxadVt3Q11JTc3vbGD,
verified live via /ext/info — no change needed.)
The legacy testnet "zoo" chain
(2Uw1b5tf63vamHsWVeSW9uHcGBVbF5CYHS9JiLWttusY4Vvjsn) was committed via
CreateChainTx with malformed EVM genesis (alloc address with 41 hex
chars instead of 40). subnet-evm's parseGenesis fails with "hex string
of odd length" on every restart; CreateChainTx is immutable on P-Chain.
That ghost has been pinning the LuxNetwork CR in "Bootstrapping 0/5"
for days. Replacement is the new zoofix subnet listed above. The
ghost's blockchain ID
(ZAKwMk5cPp5B6RkHbxxoZWh2xBGMqd7TBR8UPBdAkaV7MGKWU) is tombstoned in
the new dead-chains ConfigMap so the operator (see devops:fix/tombstone-
dead-chains) skips it in `check_bootstrap_progress`.
Resource name `indexer-zoo` retained so gateway.indexerRefs / explorer
indexerRefs don't need touch-ups across the rest of the stack; the
indexer's chainAlias and label both move to `zoofix`.
Scope: testnet only. Mainnet and devnet manifests unchanged.
The testnet LuxGateway and LuxIndexer manifests pointed at four blockchain
IDs that no longer exist on the live testnet P-Chain (a prior testnet
generation). All four routes were 404-ing; the indexers were tailing
nothing. Update to the IDs currently registered on the testnet P-Chain:
zoo -> 2BSjBuuRHUMGdhw9HzcmiRMtWEBsVcS98tjeF3x5F99T1BfbQo (alias zoofix)
hanzo -> 2ZUpGYPyaDsmuwnMUrhkcP3C24RYHvGWTZZYPaNr4oPPdUmnh3
pars -> 26a3qkQ5VzTwFDBerz2a9kte8GPqS1wJpguc9x4BHz26YXrtL9
spc -> Sd9bKzGu5Mzh6nafNecxicWJLdDmc9DB2aQPN6EUb1KfknReh
(testnet C-Chain stays B76NXURYwskvstwnyQfEpaaTpSgLrWBNxadVt3Q11JTc3vbGD,
verified live via /ext/info — no change needed.)
The legacy testnet "zoo" chain
(2Uw1b5tf63vamHsWVeSW9uHcGBVbF5CYHS9JiLWttusY4Vvjsn) was committed via
CreateChainTx with malformed EVM genesis (alloc address with 41 hex
chars instead of 40). subnet-evm's parseGenesis fails with "hex string
of odd length" on every restart; CreateChainTx is immutable on P-Chain.
That ghost has been pinning the LuxNetwork CR in "Bootstrapping 0/5"
for days. Replacement is the new zoofix subnet listed above. The
ghost's blockchain ID
(ZAKwMk5cPp5B6RkHbxxoZWh2xBGMqd7TBR8UPBdAkaV7MGKWU) is tombstoned in
the new dead-chains ConfigMap so the operator (see devops:fix/tombstone-
dead-chains) skips it in `check_bootstrap_progress`.
Resource name `indexer-zoo` retained so gateway.indexerRefs / explorer
indexerRefs don't need touch-ups across the rest of the stack; the
indexer's chainAlias and label both move to `zoofix`.
Scope: testnet only. Mainnet and devnet manifests unchanged.
Devnet L2 EVM bootstrap completed 2026-05-27 against luxd v1.27.21:
hanzo: tA86uxgQkopjwnajTfHXWzqNuv5BAAhkkdCHYFaeC33t57c2P (chainId 36964)
zoo: qVJmAaCF77XCxM6i7chgEUs3hEip8v2MY5FtwR5LJjyWj8Gno (chainId 200202)
pars: sh7CvVVAYezEiDHD6Soj1XtPVdTjhUWr4wQ9rnNPvmGgLbBzt (chainId 494951)
spc: 2azCUDCq6jWn5C9VDm1jbaTTtE9uGmaUqGvyoxBcV5i8Yz4QW5 (chainId 36912)
All four created via /tmp/bootstrap-l2-v127 (rebuilt from luxd/main HEAD)
using BIP44 idx=5 of LUX_MNEMONIC. VM ID for all four is the canonical
subnet-evm `nyGCobireNhxFB7iM5bxV74hAY6j9nQX6wizxfWomnMMtztkr` — devnet
operators must symlink that to the actual binary at
`mgj786NP7uDwBCcq6YwThhaN8FLyybkCa4zBWTQbNgmK6k9A6` (the embedded EVM
plugin in the luxd image) before pods restart, otherwise luxd logs
"chain VM plugin not loaded — opting out of this chain" and the L2
EVMs stay un-instantiated. Workaround for now: exec into each pod and
`ln -sf` in /data/plugins; long-term fix is task #47 (bake symlinks
into LuxNetwork.spec.pluginSource).
Verified end-to-end via api.lux-dev.network gateway:
hanzo: eth_chainId → 0x9064 ✓
pars: eth_chainId → 0x78d67 ✓
spc: eth_chainId → 0x9030 ✓
zoo: gateway 404 (krakend route needs /ext/bc/zoo/rpc rewrite)
Devnet L2 EVM bootstrap completed 2026-05-27 against luxd v1.27.21:
hanzo: tA86uxgQkopjwnajTfHXWzqNuv5BAAhkkdCHYFaeC33t57c2P (chainId 36964)
zoo: qVJmAaCF77XCxM6i7chgEUs3hEip8v2MY5FtwR5LJjyWj8Gno (chainId 200202)
pars: sh7CvVVAYezEiDHD6Soj1XtPVdTjhUWr4wQ9rnNPvmGgLbBzt (chainId 494951)
spc: 2azCUDCq6jWn5C9VDm1jbaTTtE9uGmaUqGvyoxBcV5i8Yz4QW5 (chainId 36912)
All four created via /tmp/bootstrap-l2-v127 (rebuilt from luxd/main HEAD)
using BIP44 idx=5 of LUX_MNEMONIC. VM ID for all four is the canonical
subnet-evm `nyGCobireNhxFB7iM5bxV74hAY6j9nQX6wizxfWomnMMtztkr` — devnet
operators must symlink that to the actual binary at
`mgj786NP7uDwBCcq6YwThhaN8FLyybkCa4zBWTQbNgmK6k9A6` (the embedded EVM
plugin in the luxd image) before pods restart, otherwise luxd logs
"chain VM plugin not loaded — opting out of this chain" and the L2
EVMs stay un-instantiated. Workaround for now: exec into each pod and
`ln -sf` in /data/plugins; long-term fix is task #47 (bake symlinks
into LuxNetwork.spec.pluginSource).
Verified end-to-end via api.lux-dev.network gateway:
hanzo: eth_chainId → 0x9064 ✓
pars: eth_chainId → 0x78d67 ✓
spc: eth_chainId → 0x9030 ✓
zoo: gateway 404 (krakend route needs /ext/bc/zoo/rpc rewrite)
The operator's CRD struct only has `upgrade_config` (mapped from
`upgradeConfig`). `chainUpgradeConfig` was silently dropped by the
strict decoder on every reconcile — the field never reached the
generated StatefulSet. Live cluster CRs already cleaned via kubectl
patch; this aligns the source manifest so a future kustomize-apply
doesn't reintroduce the dead block.
Precompile activations live in the genesis JSON files at
luxfi/genesis@06dc203 (cChain: 43 precompiles incl. corona, pulsar,
magnetar; L2 EVMs: ai-coin, all PQ precompiles). The hand-managed
luxd-startup.yaml writes those into per-chain upgrade.json at boot
via the EVM_CHAIN_IDS loop (canonical IDs fixed in commit 5dc650e0).
The operator's CRD struct only has `upgrade_config` (mapped from
`upgradeConfig`). `chainUpgradeConfig` was silently dropped by the
strict decoder on every reconcile — the field never reached the
generated StatefulSet. Live cluster CRs already cleaned via kubectl
patch; this aligns the source manifest so a future kustomize-apply
doesn't reintroduce the dead block.
Precompile activations live in the genesis JSON files at
luxfi/genesis@06dc203 (cChain: 43 precompiles incl. corona, pulsar,
magnetar; L2 EVMs: ai-coin, all PQ precompiles). The hand-managed
luxd-startup.yaml writes those into per-chain upgrade.json at boot
via the EVM_CHAIN_IDS loop (canonical IDs fixed in commit 5dc650e0).
The hardcoded list had stale IDs from pre-rebootstrap chains:
hanzo: cjhijar... → y5HBrn6E... (current)
zoo: 2Y625Kvdd... → 2wbPWcyu... (current)
pars: 2GiQb73Ce... → 2wYWjBHziTjf... (parsv4, 7B PARS)
spc: rtjwvtE1... and 2pUskxqaL5... → 2YTuJW9b1k... (spcv4, 11B SPC)
The stale IDs caused the per-pod upgrade.json to be written to chain-config
directories for blockchain IDs that no longer exist, while the actually-
running L2 EVMs at canonical IDs got no upgrade.json injection. EVM_CHAIN_IDS
now matches the same canonical IDs already in the chain-aliases stanza
below (and Agent D's lux-indexer ConfigMap commit 5f2a22ee).
NOTE: this changes which chain config dir gets the upgrade.json on next
pod restart. The 20-precompile + coronaThreshold list itself is
unchanged; reconciling it against the canonical 43-precompile +
coronaThreshold genesis is a separate (state-sensitive) operation that
needs planned downtime to avoid block-validity divergence.
The hardcoded list had stale IDs from pre-rebootstrap chains:
hanzo: cjhijar... → y5HBrn6E... (current)
zoo: 2Y625Kvdd... → 2wbPWcyu... (current)
pars: 2GiQb73Ce... → 2wYWjBHziTjf... (parsv4, 7B PARS)
spc: rtjwvtE1... and 2pUskxqaL5... → 2YTuJW9b1k... (spcv4, 11B SPC)
The stale IDs caused the per-pod upgrade.json to be written to chain-config
directories for blockchain IDs that no longer exist, while the actually-
running L2 EVMs at canonical IDs got no upgrade.json injection. EVM_CHAIN_IDS
now matches the same canonical IDs already in the chain-aliases stanza
below (and Agent D's lux-indexer ConfigMap commit 5f2a22ee).
NOTE: this changes which chain config dir gets the upgrade.json on next
pod restart. The 20-precompile + coronaThreshold list itself is
unchanged; reconciling it against the canonical 43-precompile +
coronaThreshold genesis is a separate (state-sensitive) operation that
needs planned downtime to avoid block-validity divergence.
Audit found the unified explorer + lux-indexer ConfigMaps drifting from
the canonical post-rebootstrap blockchain IDs, plus testnet/devnet
ingress-chains.yaml still pointing at dead Blockscout services.
Changes (manifest-only, no luxd touching):
- lux-mainnet/lux-indexer.yaml: zoo + hanzo rpc URLs now point at the
current blockchain IDs (2wbPWcyu... for zoo, y5HBrn6E... for hanzo).
Old IDs (2Y625Kvdd... / 2GiQb73Ce...) were 404ing.
- lux-mainnet/explorer.yaml: switched per-chain rpc URLs from slug
aliases (/ext/bc/{slug}/rpc) to literal blockchain IDs, since the
chain-aliases ConfigMap is declarative state luxd has not yet
loaded (verified: /ext/bc/pars/rpc → 404, /ext/bc/2wYWjBHziTjf.../rpc → 200).
- lux-testnet/lux-indexer.yaml + explorer.yaml: added hanzo, zoo
(zoofix LXuiihyi...), pars (26a3qkQ5...), and spc (Sd9bKzGu...) chains.
Testnet zoo now backed by the 2026-05-27 zoofix re-bootstrap.
- lux-devnet/lux-indexer.yaml + explorer.yaml: added hanzo (2XSToxn...),
zoo (2jhVoHB5...), pars (SbgsF19z...), spc (2YmQ1abS...) chains.
Chain IDs use the on-chain values from chain-aliases annotations.
- lux-testnet/ingress-chains.yaml + lux-devnet/ingress-chains.yaml:
rewrote per-chain IngressRoute backends from dead
explorer-{chain}-{net}:4000 Blockscout services to the unified
explorer:8090 service with blockscout-prefix-{slug} middlewares
mirroring the mainnet pattern. Added explore-{chain}-{net}.lux.network
SPA routes (will work once DNS records are created; existing
explorer.{net}.lux.network already covers all chains via path
/v1/indexer/{slug}).
Verified end-to-end (heights from /v1/indexer/{slug}/blocks):
mainnet/cchain=1083589, hanzo=0, zoo=799, pars=0, spc=0
testnet/cchain=923, hanzo=682, zoo=0, pars=377, spc=389
devnet/cchain=163, hanzo=5, zoo=0, pars=88, spc=164
Audit found the unified explorer + lux-indexer ConfigMaps drifting from
the canonical post-rebootstrap blockchain IDs, plus testnet/devnet
ingress-chains.yaml still pointing at dead Blockscout services.
Changes (manifest-only, no luxd touching):
- lux-mainnet/lux-indexer.yaml: zoo + hanzo rpc URLs now point at the
current blockchain IDs (2wbPWcyu... for zoo, y5HBrn6E... for hanzo).
Old IDs (2Y625Kvdd... / 2GiQb73Ce...) were 404ing.
- lux-mainnet/explorer.yaml: switched per-chain rpc URLs from slug
aliases (/ext/bc/{slug}/rpc) to literal blockchain IDs, since the
chain-aliases ConfigMap is declarative state luxd has not yet
loaded (verified: /ext/bc/pars/rpc → 404, /ext/bc/2wYWjBHziTjf.../rpc → 200).
- lux-testnet/lux-indexer.yaml + explorer.yaml: added hanzo, zoo
(zoofix LXuiihyi...), pars (26a3qkQ5...), and spc (Sd9bKzGu...) chains.
Testnet zoo now backed by the 2026-05-27 zoofix re-bootstrap.
- lux-devnet/lux-indexer.yaml + explorer.yaml: added hanzo (2XSToxn...),
zoo (2jhVoHB5...), pars (SbgsF19z...), spc (2YmQ1abS...) chains.
Chain IDs use the on-chain values from chain-aliases annotations.
- lux-testnet/ingress-chains.yaml + lux-devnet/ingress-chains.yaml:
rewrote per-chain IngressRoute backends from dead
explorer-{chain}-{net}:4000 Blockscout services to the unified
explorer:8090 service with blockscout-prefix-{slug} middlewares
mirroring the mainnet pattern. Added explore-{chain}-{net}.lux.network
SPA routes (will work once DNS records are created; existing
explorer.{net}.lux.network already covers all chains via path
/v1/indexer/{slug}).
Verified end-to-end (heights from /v1/indexer/{slug}/blocks):
mainnet/cchain=1083589, hanzo=0, zoo=799, pars=0, spc=0
testnet/cchain=923, hanzo=682, zoo=0, pars=377, spc=389
devnet/cchain=163, hanzo=5, zoo=0, pars=88, spc=164
The networkUpgradeOverrides + 16-entry precompileUpgrades in the CR were
duplicating state that now lives canonically in the per-chain genesis files
(luxfi/genesis@06dc203 bakes Durango/Etna/Fortuna/Granite + 43 precompiles
into all 15 chain genesis JSONs).
Keeping that block in the CR was the source of CR-vs-genesis drift: the
genesis JSON now lists 43 precompiles, the CR only 16, and once luxd reads
both layers the union/override semantics become a foot-gun. Single source
of truth wins.
- DROPPED: spec.chainUpgradeConfig.networkUpgradeOverrides
- DROPPED: spec.chainUpgradeConfig.precompileUpgrades (16 entries)
- KEPT: spec.upgradeConfig (platform-level Durango/Etna/Fortuna/Granite)
- KEPT: spec.chainTracking.trackAllChains (transitional — flips to a
baked L2-ID list once the canonical L2 IDs land at genesis)
All other CR fields (consensus, staking KMS, plugins, ports, seedRestore,
snapshotSchedule, upgradeStrategy, startupGate) are preserved unchanged.
The networkUpgradeOverrides + 16-entry precompileUpgrades in the CR were
duplicating state that now lives canonically in the per-chain genesis files
(luxfi/genesis@06dc203 bakes Durango/Etna/Fortuna/Granite + 43 precompiles
into all 15 chain genesis JSONs).
Keeping that block in the CR was the source of CR-vs-genesis drift: the
genesis JSON now lists 43 precompiles, the CR only 16, and once luxd reads
both layers the union/override semantics become a foot-gun. Single source
of truth wins.
- DROPPED: spec.chainUpgradeConfig.networkUpgradeOverrides
- DROPPED: spec.chainUpgradeConfig.precompileUpgrades (16 entries)
- KEPT: spec.upgradeConfig (platform-level Durango/Etna/Fortuna/Granite)
- KEPT: spec.chainTracking.trackAllChains (transitional — flips to a
baked L2-ID list once the canonical L2 IDs land at genesis)
All other CR fields (consensus, staking KMS, plugins, ports, seedRestore,
snapshotSchedule, upgradeStrategy, startupGate) are preserved unchanged.
Cluster has 1500-line krakend.json with stale routes from prior
bootstrap iterations; not committing verbatim. This file documents
the canonical edits applied 2026-05-26 to hide the parsv4/spcv4
chain-name suffix from external clients.
Cluster has 1500-line krakend.json with stale routes from prior
bootstrap iterations; not committing verbatim. This file documents
the canonical edits applied 2026-05-26 to hide the parsv4/spcv4
chain-name suffix from external clients.
Replace operator-default --track-all-chains with explicit
--track-chains=<4 canonical blockchain IDs>, so luxd validators stop
serving the orphan pars/spc/parsm chains that were left behind by
prior bootstrap iterations:
hanzo y5HBrn6EWRMysSmH4nHA2sgpwgcjChJjQ7upSjWSwsrPeg4Cc (chainId 36963)
zoo 2wbPWcyuNUimWW7YMq3gymTTpHjP9Lsm7GpmcyWimEs9kyTAAH (chainId 200200)
pars 2wYWjBHziTjfMJmUPZwsDdWhm96cWHo4vov1HPBmCYLXdbAZfM (chainId 7070, parsv4)
spc 2YTuJW9b1k7gDGgqZC58Wj6cT1qTzobx6SV8hiN22zT1L5qfHF (chainId 36911, spcv4)
Add --chain-aliases-file=/data/aliases.json so info.* APIs accept
human-readable names. luxd's HTTP router still keys by chain-name —
the user-facing rename pars←parsv4 is done at the api-gateway krakend
layer via backend url_pattern rewrites (committed separately to its CM
in cluster; source manifest TODO).
Net result: api.lux.network/ext/bc/{pars,spc,hanzo,zoo}/rpc return
the correct canonical chains with the correct chainIds and treasury
balances; no v3/v4 suffix exposed externally.
Replace operator-default --track-all-chains with explicit
--track-chains=<4 canonical blockchain IDs>, so luxd validators stop
serving the orphan pars/spc/parsm chains that were left behind by
prior bootstrap iterations:
hanzo y5HBrn6EWRMysSmH4nHA2sgpwgcjChJjQ7upSjWSwsrPeg4Cc (chainId 36963)
zoo 2wbPWcyuNUimWW7YMq3gymTTpHjP9Lsm7GpmcyWimEs9kyTAAH (chainId 200200)
pars 2wYWjBHziTjfMJmUPZwsDdWhm96cWHo4vov1HPBmCYLXdbAZfM (chainId 7070, parsv4)
spc 2YTuJW9b1k7gDGgqZC58Wj6cT1qTzobx6SV8hiN22zT1L5qfHF (chainId 36911, spcv4)
Add --chain-aliases-file=/data/aliases.json so info.* APIs accept
human-readable names. luxd's HTTP router still keys by chain-name —
the user-facing rename pars←parsv4 is done at the api-gateway krakend
layer via backend url_pattern rewrites (committed separately to its CM
in cluster; source manifest TODO).
Net result: api.lux.network/ext/bc/{pars,spc,hanzo,zoo}/rpc return
the correct canonical chains with the correct chainIds and treasury
balances; no v3/v4 suffix exposed externally.
Pars and SPC L2 EVMs were re-bootstrapped 2026-05-25 with corrected
genesis (LUX_MNEMONIC-funded treasury 0x9011E... at ~2T LUX, matching
hanzo/zoo). New blockchain IDs:
pars: fcWycpdtBR8o1qAGcSRwsfcap9cckzHr5zGDfa7dFijy2HRWG (chain name parsv3)
spc: YLsRfVmN1Jfy8A1gXk7uWF8GcRbwSdjeRkAvmUAnVVUDLCrQT (chain name spcv3)
Both controlled by BIP44 m/44'/9000'/0'/0/5 (P-lux1jx9p2hxfemcgdlylth05zj20klsn4t06zddpv7).
L2 validator-set IDs + chain names recorded as annotations on the
lux-deployer secret in lux-mainnet ns (lux.network/pars-* and
lux.network/spc-*).
The old pars (2B17v...) and spc (24SAg...) chains remain registered on
P-chain forever but no longer canonical; alias-only routing.
Pars and SPC L2 EVMs were re-bootstrapped 2026-05-25 with corrected
genesis (LUX_MNEMONIC-funded treasury 0x9011E... at ~2T LUX, matching
hanzo/zoo). New blockchain IDs:
pars: fcWycpdtBR8o1qAGcSRwsfcap9cckzHr5zGDfa7dFijy2HRWG (chain name parsv3)
spc: YLsRfVmN1Jfy8A1gXk7uWF8GcRbwSdjeRkAvmUAnVVUDLCrQT (chain name spcv3)
Both controlled by BIP44 m/44'/9000'/0'/0/5 (P-lux1jx9p2hxfemcgdlylth05zj20klsn4t06zddpv7).
L2 validator-set IDs + chain names recorded as annotations on the
lux-deployer secret in lux-mainnet ns (lux.network/pars-* and
lux.network/spc-*).
The old pars (2B17v...) and spc (24SAg...) chains remain registered on
P-chain forever but no longer canonical; alias-only routing.
Marks the Zoo block as tenant-contributed; source of truth at
zooai/universe k8s/zoo-mainnet/luxd-rlp-import.yaml. Notes the
long-term fix (operator-level tenantImports[] composition).
Marks the Zoo block as tenant-contributed; source of truth at
zooai/universe k8s/zoo-mainnet/luxd-rlp-import.yaml. Notes the
long-term fix (operator-level tenantImports[] composition).
Previously the lux-mainnet luxd-startup ConfigMap existed only in the
live cluster (no source manifest); a rebuild would have lost the C-Chain
RLP import logic and the new Zoo (200200) EVM L2 RLP import block I
added 2026-05-25.
Sentinels under /data/:
.cchain_rlp_imported_v3 → /data/lux-mainnet-96369.rlp (1.08M blocks)
.zoo_rlp_imported_v1 → /data/zoo-mainnet-200200.rlp (799 blocks)
Both are idempotent: import fires only when the sentinel is absent and
the RLP is present. After import, sentinel is written so subsequent pod
restarts skip. The same RLP file is staged on all 5 luxd-N pods (each
imports independently into its own SQLite-backed chainData).
Validated 2026-05-25: all 5 mainnet validators report Zoo block 0x31f
(799), hash 0xd6f92941bb...; public api.lux.network/ext/bc/zoo/rpc
returns the same.
Previously the lux-mainnet luxd-startup ConfigMap existed only in the
live cluster (no source manifest); a rebuild would have lost the C-Chain
RLP import logic and the new Zoo (200200) EVM L2 RLP import block I
added 2026-05-25.
Sentinels under /data/:
.cchain_rlp_imported_v3 → /data/lux-mainnet-96369.rlp (1.08M blocks)
.zoo_rlp_imported_v1 → /data/zoo-mainnet-200200.rlp (799 blocks)
Both are idempotent: import fires only when the sentinel is absent and
the RLP is present. After import, sentinel is written so subsequent pod
restarts skip. The same RLP file is staged on all 5 luxd-N pods (each
imports independently into its own SQLite-backed chainData).
Validated 2026-05-25: all 5 mainnet validators report Zoo block 0x31f
(799), hash 0xd6f92941bb...; public api.lux.network/ext/bc/zoo/rpc
returns the same.
Replaces the per-chain + per-network indexer deployments (each backed by
its own postgres) with one indexerd pod per network running every chain
in that network's chains.yaml. Storage is embedded SQLite under /data,
one .db per chain at /data/{slug}/query/indexer.db (see indexer
storage/unified.go DefaultUnifiedConfig).
Per-network footprint:
mainnet: 13 chains (C, P, X, A, B, Q, T, Z, K, Zoo, Hanzo, SPC, Pars)
testnet: 3 chains (C, P, X)
devnet: 7 chains (C, P, X, A, Q, T, Z)
Deleted:
k8s/lux-mainnet/indexer-all-chains.yaml 13 deployments + indexer-postgres StatefulSet
k8s/lux-mainnet/indexer-mainnet.yaml single-chain pointing at dead lux-sql
k8s/lux-mainnet/indexer.yaml legacy
k8s/lux-testnet/indexer-testnet.yaml single-chain pointing at dead lux-sql
k8s/lux-devnet/indexer-devnet.yaml single-chain pointing at dead lux-sql
Added:
k8s/lux-{mainnet,testnet,devnet}/lux-indexer.yaml
- ConfigMap with chains.yaml
- PVC: 100Gi (do-block-storage-retain)
- Deployment: 1 replica, securityContext fsGroup=65532 (indexer uid)
- Service: ClusterIP :8091
Image pinned to ghcr.io/luxfi/indexer:1.4.4 (immutable semver tag per
universe/CLAUDE.md image-pin policy; :latest forbidden).
Cluster cleanup applied:
- deleted 5 LuxIndexer CRs on hanzo-k8s/lux-mainnet + 5 on
/lux-devnet (controller was removed from lux-operator)
- deleted 10 indexer-{c,hanzo,pars,spc,zoo}-pg StatefulSets +
associated PVCs on hanzo-k8s (were running but DNS-broken since
the luxd they pointed at is on lux-k8s, not hanzo-k8s)
- deleted lux-sql StatefulSet in lux ns on lux-k8s (0/1 for 84d)
- deleted parked lux-indexer-{achain,bchain,cchain,kchain,hanzo,
pars,pchain,qchain,spc,tchain,xchain,zchain,zoo} deployments on
lux-k8s/lux-mainnet (zero replicas anyway, now superseded)
Smoke: all three lux-indexer pods 1/1 Running, /health returns
{"status":"ok"}. Chains begin indexing immediately on first sync;
SQLite tables created lazily per chain when first block arrives.
Replaces the per-chain + per-network indexer deployments (each backed by
its own postgres) with one indexerd pod per network running every chain
in that network's chains.yaml. Storage is embedded SQLite under /data,
one .db per chain at /data/{slug}/query/indexer.db (see indexer
storage/unified.go DefaultUnifiedConfig).
Per-network footprint:
mainnet: 13 chains (C, P, X, A, B, Q, T, Z, K, Zoo, Hanzo, SPC, Pars)
testnet: 3 chains (C, P, X)
devnet: 7 chains (C, P, X, A, Q, T, Z)
Deleted:
k8s/lux-mainnet/indexer-all-chains.yaml 13 deployments + indexer-postgres StatefulSet
k8s/lux-mainnet/indexer-mainnet.yaml single-chain pointing at dead lux-sql
k8s/lux-mainnet/indexer.yaml legacy
k8s/lux-testnet/indexer-testnet.yaml single-chain pointing at dead lux-sql
k8s/lux-devnet/indexer-devnet.yaml single-chain pointing at dead lux-sql
Added:
k8s/lux-{mainnet,testnet,devnet}/lux-indexer.yaml
- ConfigMap with chains.yaml
- PVC: 100Gi (do-block-storage-retain)
- Deployment: 1 replica, securityContext fsGroup=65532 (indexer uid)
- Service: ClusterIP :8091
Image pinned to ghcr.io/luxfi/indexer:1.4.4 (immutable semver tag per
universe/CLAUDE.md image-pin policy; :latest forbidden).
Cluster cleanup applied:
- deleted 5 LuxIndexer CRs on hanzo-k8s/lux-mainnet + 5 on
/lux-devnet (controller was removed from lux-operator)
- deleted 10 indexer-{c,hanzo,pars,spc,zoo}-pg StatefulSets +
associated PVCs on hanzo-k8s (were running but DNS-broken since
the luxd they pointed at is on lux-k8s, not hanzo-k8s)
- deleted lux-sql StatefulSet in lux ns on lux-k8s (0/1 for 84d)
- deleted parked lux-indexer-{achain,bchain,cchain,kchain,hanzo,
pars,pchain,qchain,spc,tchain,xchain,zchain,zoo} deployments on
lux-k8s/lux-mainnet (zero replicas anyway, now superseded)
Smoke: all three lux-indexer pods 1/1 Running, /health returns
{"status":"ok"}. Chains begin indexing immediately on first sync;
SQLite tables created lazily per chain when first block arrives.
Per separation-of-concerns: hanzo universe was holding 11 manifests
that target lux-k8s. Moved them here where they belong.
Imported into k8s/lux-k8s/:
- agnost/, api-gateway/, bootnode-lux/, bridge/, exchange/,
explorer/, operator/ (unique to hanzo universe, fresh copies)
- gateway/ (hanzo had newer pins than lux's local copy)
Skipped: validators/ — lux universe already has the newer
(luxd v1.27.4) version. Hanzo's older v1.23.31 copy is being
dropped, not merged.
Imported into k8s/:
- status/{deployment,configmap,ingressroute}.yaml
(Hanzo status SPA in lux-system ns on lux-k8s)
- nchain/crs/mainnet.yaml
(Network CR for lux-mainnet ns on lux-k8s)
Per separation-of-concerns: hanzo universe was holding 11 manifests
that target lux-k8s. Moved them here where they belong.
Imported into k8s/lux-k8s/:
- agnost/, api-gateway/, bootnode-lux/, bridge/, exchange/,
explorer/, operator/ (unique to hanzo universe, fresh copies)
- gateway/ (hanzo had newer pins than lux's local copy)
Skipped: validators/ — lux universe already has the newer
(luxd v1.27.4) version. Hanzo's older v1.23.31 copy is being
dropped, not merged.
Imported into k8s/:
- status/{deployment,configmap,ingressroute}.yaml
(Hanzo status SPA in lux-system ns on lux-k8s)
- nchain/crs/mainnet.yaml
(Network CR for lux-mainnet ns on lux-k8s)
zeekayandGitHub