universe/LLM.md

LLM.md

This file provides guidance to AI tools when working with code in this repository.

Project Overview

This is a multi-blockchain infrastructure project managing 4 Lux app chain EVM networks (LUX, ZOO, SPC, and Hanzo). The infrastructure includes blockchain nodes, block explorers, DeFi exchange, Graph Protocol indexers, cross-chain bridges, and IPFS storage.

Deterministic validator deployment

The live luxd StatefulSets MUST equal the three CD values files under deploy/lux-{mainnet,testnet,devnet}/luxd.yaml. Those files are the only node deployment source. Do not add luxd workloads under k8s/, hand-edit a live StatefulSet, or restore a LuxNetwork operator as an alternate reconciliation path.

The exact release and quorum-preserving rollout procedure is RUNBOOK.md and its sole entry point is scripts/roll_luxd.sh. Every fleet has five validators with a four-vote quorum. Roll devnet, then testnet, then mainnet; within a fleet, replace only one validator at a time and require Ready + bootstrapped + exact version + exact digest before continuing. Mainnet storage is never wiped.

Infrastructure: ZAP-native forks only — NEVER vanilla OSS

All data/storage/messaging backends MUST be the Hanzo ZAP-native forks, never upstream OSS (they speak the native ZAP protocol — note the extra zap port on each):

Need	USE (Hanzo fork)	NOT
analytics DB	Datastore ghcr.io/hanzoai/datastore (svc insights-datastore; secret datastore/DATASTORE_PASSWORD; data /var/lib/hanzo-datastore)	ClickHouse
object store	hanzoai/s3 (lux-system s3 deploy, S3_ROOT_USER=hanzo/hanzo-s3-secret; aliases lux-assets/lux-cdn)	MinIO
KV / cache	hanzoai/kv (kv/kv-0)	Redis
SQL	hanzoai/sql (sql/sql-0)	vanilla Postgres
KMS	hanzoai/kms (zapdb)	Infisical/Vault
IAM	hanzoai/iam (Casdoor fork)	—
pub/sub	hanzoai/pubsub (nats)	Kafka

Deploy by replicating the canonical fork workload from a working cluster (e.g. copy the datastore StatefulSet + datastore-config ConfigMap + datastore Secret from hanzo-k8s). Do NOT kubectl run clickhouse/minio or pull clickhouse/clickhouse-server etc. Refer to them by the Hanzo name (Datastore, not ClickHouse).

Architecture

Network Configuration

CRITICAL: Network ID != C-Chain ID. Network IDs are well-known (1/2/3), C-Chain IDs are EVM chain IDs.

Network	Network ID	C-Chain ID	Start Time
Mainnet	1	96369	Dec 25, 2025 4:20 PM PDT (1766704800)
Testnet	2	96368	Dec 25, 2025 4:20 PM PDT (1766704800)
Devnet	3	96367	Dec 25, 2025 4:20 PM PDT (1766704800)

app chain EVM Chain IDs -- source of truth: NETWORKS.yaml

app chain EVM	Coin	Mainnet	Testnet	Devnet
Zoo	ZOO	200200	200201	200202
Hanzo	AI	36963	36962	36964
SPC	SPC	36911	36910	36912
Pars	PARS	494949	494950	494951

Note: Hanzo coin is AI, not HANZO. Hanzo testnet EVM chain ID is 36962; devnet is 36964.

Service Ports

Service	Mainnet	Testnet	Description
Node 1-5 HTTP	9630-9638	9740-9748	RPC endpoints
Node 1-5 Staking	9631-9639	9741-9749	P2P staking
gRPC	8369	8368	gRPC API
LUX Explorer	4010	4020	Lux Explorer (Go binary)
Zoo Explorer	4011	4021	Lux Explorer (Go binary)
Hanzo Explorer	4012	4022	Lux Explorer (Go binary)
SPC Explorer	4013	4023	Lux Explorer (Go binary)
LXD Gateway	8080	8081	DEX API
PostgreSQL	15432	15433	Database
Redis	16379	16380	Cache

Directory Structure

/home/z/work/lux
├── node/           # Lux Network node
├── geth/           # C-Chain implementation (formerly coreth)
├── evm/            # App chain EVM implementation
├── cli/            # Lux CLI for managing networks
├── vms/            # Virtual machine implementations
│   └── cchainvm/   # C-Chain VM utilities
├── exchange/       # Uniswap V2/V3 DEX (lux.exchange)
├── explorer-*/     # Blockchain explorers per network
├── monitoring/     # Health check and monitoring system
├── genesis-*/      # Genesis configurations per network
└── scripts/        # Management scripts

Recent Updates (2025)

Port Standardization

• All services now use port 9630 (previously 9650)
• Staking port: 9631
• Consistent across all configurations

Fixed Issues

1. cchainvm Package: Created missing implementation at /vms/cchainvm/slurp.go
2. Error Handling: Exported error constants in ids package (ErrNoAliasForID, ErrAliasAlreadyMapped)
3. Hanzo Network: Deployment script ready at /deploy-hanzo-network.sh
4. Database Connections: Fixed all explorer PostgreSQL connections
5. Health Monitoring: Comprehensive system at /monitoring/health_check_system.sh
6. Q-Chain Integration: Added quantum-resistant chain configuration with Crystal-Dilithium signatures
7. admin_importChain Deadlock Fix (2026-01-05): Fixed P2P degradation after block import

Critical Fix: admin_importChain Deadlock (2026-01-05)

Problem: After admin_importChain, P-chain and Info APIs timeout while C-chain continues working.

Root Cause: PostImportCallback was called synchronously while holding chainmu.Lock(), causing cross-chain mutex contention with P-chain/Info API handlers.

Solution (coreth/eth/api_admin.go:180-196):

// CRITICAL: Run asynchronously to avoid deadlock
go func() {
    if err := api.eth.CallPostImportCallback(lastInsertedBlock.Hash(), lastInsertedBlock.NumberU64()); err != nil {
        log.Error("PostImportCallback failed", "error", err)
        return
    }
    log.Info("ImportChain: post-import callback completed asynchronously")
}()

Build after fix:

cd ~/work/lux/node && CGO_ENABLED=0 ./scripts/build.sh
cd ~/work/lux/coreth && CGO_ENABLED=0 ./scripts/build.sh

Key Technologies

• Blockchain: Go (node, geth, evm), Lux-based
• Explorers: Go (Lux Explorer single binary: indexer+GraphQL+SPA)
• DeFi: TypeScript/React (Uniswap fork)
• Indexing: Rust (Graph Protocol)
• Infrastructure: Docker, Docker Compose, PostgreSQL

Gateway Security Hardening (2026-03-24)

Five security fixes applied across all gateway configs:

1. info.peers/getNodeIP/getNodeID blocked -- CEL validation on all /v1/info, /mainnet/v1/info, /testnet/v1/info, /devnet/v1/info endpoints across all config layers (gateway.json, gateway-mainnet.json, testnet/devnet JSONs, K8s ConfigMap). K8s gateway.yaml was missing CEL on /mainnet/v1/info, /testnet/v1/info, /devnet/v1/info -- now fixed.
2. Response headers stripped -- modifier/response-headers-blacklist blocks X-Gateway, X-Gateway-Completed, node-id, Node-Id, Server in all configs. Was already correct.
3. Per-endpoint rate limiting -- Every endpoint now has qos/ratelimit/router: 100 req/s for RPC, 50 for WS, 30 for health, 10 for info. K8s gateway.yaml previously had zero per-endpoint limits.
4. CORS wildcard removed -- IngressRoute cors-allow-all middleware replaced with gateway-cors (regex-based origin allowlist: *.lux.network, *.lux.exchange, lux.market). LuxGateway CRDs also updated from ["*"] to explicit origin list.
5. XVM (exchangevm) routes added -- /v1/chain/exchange alias routes map to /v1/chain/X on all networks (mainnet/testnet/devnet) in both K8s gateway and standalone configs. Follows same pattern as /v1/chain/dex/rpc -> /v1/chain/D/rpc.

Files changed: configs/gateway/gateway.json, configs/gateway/gateway-mainnet.json, configs/gateway/gateway-testnet.json, configs/gateway/gateway-devnet.json, k8s/lux-gateway/gateway.yaml, k8s/lux-k8s/validators/crd-resources.yaml.

lux.exchange TLS: app.lux.exchange added to Ingress (2026-07-14)

Symptom: https://app.lux.exchange → ERR_SSL_PROTOCOL_ERROR (SNI returned no cert).

Root cause: k8s/lux-ns/lux-exchange.yaml Ingress omitted app.lux.exchange from both spec.tls[0].hosts and spec.rules, so hanzoai/ingress (Traefik) had no router/cert for that SNI. The lux-exchange-tls cert-manager secret already covers app.lux.exchange in its SAN (Ready), and DNS app.lux.exchange (Cloudflare grey-cloud, DNS-only) points at the LB 134.199.138.27.

Fix: Added app.lux.exchange to the Ingress tls.hosts + a rules entry → lux-exchange:3000. Traefik now serves the valid LE cert (verified in-cluster: CN=lux.exchange, SAN incl. app.lux.exchange, valid → 2026-09-29). DEX confirmed rendering via the unfiltered exchange.lux.network alias (same backend/cert). Cloudflare lux.exchange zone is healthy: active, SSL full, Universal cert covers lux.exchange + *.lux.exchange; apex/www proxied, app/trade grey-cloud.

Residual (NOT origin/ingress/cert): the lux.exchange domain is intercepted by safebrowse-style SNI/DNS egress filters on some client networks (crypto-exchange reputation lists) — proven: to the same origin IP, SNI zoo.exchange/bridge.lux.network handshake OK while SNI *.lux.exchange returns non-TLS bytes (wrong version number). Needs a domain-reputation/delisting request; not a k8s/Cloudflare fault. See header note in k8s/lux-ns/lux-exchange.yaml.

api.lux.cloud accepted no API key at all

Symptom: every sk-/pk- presented to api.lux.cloud answered 403 X-Org-Id required, which reads as "bad key". A JWT worked, so only machine credentials were dead — CI could never deploy.

Three independent causes, all in k8s/lux-k8s/cloud/deployment.yaml:

1. No IAM client. IAM_MINT_CLIENT_ID/IAM_MINT_CLIENT_SECRET were unset, so cloud's iamCred() returned "" and the key resolver short-circuited to nil before any network call. Now set from Secret cloud-iam-client (client hanzo-cloud — the binary's own service identity, and one of the two IAM admits at get-user?accessKey via IAM_KEY_RESOLVE_APPS). lux-cloud cannot be used: it is a SPA and holds no secret.
2. Image too old. v1.801.210 predates "a machine credential resolves its org from the subject" (v1.801.245). Before that an sk- resolved to a principal with no orgs claim, and the identity boundary — right to refuse a human token that lost its claim — refused the key too. Pinned v1.801.476: the newest tag that is both above that floor and below v1.801.487, where the console became a published site and boot began requiring a released lux/hanzo-console this front door does not have (v1.801.496 crash-looped on exactly that). Tags above v1.801.496 have no image at all — a pin must be checked against ghcr, never git tag.
3. Probes pointed at a port the image does not open. Liveness/readiness hit :9090/healthz + /readyz; this binary serves /healthz on the product listener only and reads CLOUD_HEALTH_LISTEN nowhere, so liveness killed a process that was serving traffic. Probes now use http (8000) /healthz, and the phantom ops port is gone.

Also: the cluster ghcr-secret held a dead credential (403 Forbidden from ghcr's token endpoint). Nothing noticed because imagePullPolicy: IfNotPresent served every pod from node cache — until a pod landed on a node without the layer. Replaced with the working credential.

Store moved with the version: v1.801.476 keeps projects at /var/lib/cloud/orgs/_platform/projects.db; the pre-bump rows in /var/lib/cloud/projects.db are invisible to it, so the tel project was recreated (new id proj_4vW2NGyLKpHImFgVpg9Msg, org lux). Nothing had ever deployed to the old row.

Still owed on this host (not changed here — both are decisions, not defects): CLOUD_HOSTING_FEE_CENTS=0 (hanzo prod sets it; without it every deploy hits a billing gate this deployment has no commerce backend to answer, so 503 balance_unavailable), and S3_ADMIN_ACCESS_KEY/S3_ADMIN_SECRET_KEY + a reachable endpoint for CLOUD_PROJECTS_BUCKET (hanzo's S3 is in-cluster on hanzo-k8s and unreachable from lux-k8s).

Development Guidelines

Important Rules

• ALWAYS use luxfi packages, never use go-ethereum or upstream legacy packages
• Use luxfi/geth for Ethereum compatibility, NOT go-ethereum
• Use luxfi/ids NOT luxfi/node/ids (regression)
• NEVER commit local replace directives in go.mod
• Always use semantic versioned tags, never make up git tags
• Port 9630 for luxd, NOT 9650

Running Networks

Option 1: Docker Compose (Recommended for Production)

cd ~/work/lux/universe

# Build and start mainnet (5 validators)
docker compose -f compose.mainnet.yml up -d --build

# Start testnet (5 validators)
docker compose -f compose.testnet.yml up -d --build

# Add block explorers
docker compose -f compose.mainnet.yml --profile explorers up -d

# View logs
docker compose -f compose.mainnet.yml logs -f mainnet-node-1

# Stop
docker compose -f compose.mainnet.yml down

Option 2: Lux CLI (Development)

# Build binaries with deadlock fix
cd ~/work/lux/node && CGO_ENABLED=0 ./scripts/build.sh
cd ~/work/lux/coreth && CGO_ENABLED=0 ./scripts/build.sh
cp ~/work/lux/node/build/luxd ~/.luxd/

# Start networks
lux network start --mainnet
lux network start --testnet

# Check status
lux network status

# Stop
lux network stop --mainnet --force
lux network stop --testnet --force

Block Import (After Network is Running)

# Write genesis state spec first
curl -s -X POST -H "Content-Type: application/json" \
  --data '{"jsonrpc":"2.0","method":"admin_writeGenesisStateSpec","params":["/Users/z/work/lux/genesis/configs/mainnet/cchain.json"],"id":1}' \
  http://127.0.0.1:9630/v1/chain/C/rpc

# Import blocks (mainnet C-Chain: 1,082,780 blocks)
curl -s -X POST -H "Content-Type: application/json" \
  --data '{"jsonrpc":"2.0","method":"admin_importChain","params":["/Users/z/work/lux/state/rlp/lux-mainnet/lux-mainnet-96369.rlp"],"id":1}' \
  http://127.0.0.1:9630/v1/chain/C/rpc

# Verify import
curl -s -X POST -H "Content-Type: application/json" \
  --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
  http://127.0.0.1:9630/v1/chain/C/rpc

RLP Block Data Available

Chain	File	Blocks
C-Chain Mainnet	state/rlp/lux-mainnet/lux-mainnet-96369.rlp	1,082,780
C-Chain Testnet	state/rlp/lux-testnet/lux-testnet-96368.rlp	218
Zoo Mainnet	state/rlp/zoo-mainnet/zoo-mainnet-200200.rlp	799
Zoo Testnet	state/rlp/zoo-testnet/zoo-testnet-200201.rlp	84
SPC Mainnet	state/rlp/spc-mainnet/spc-mainnet-36911.rlp	11

POA Implementation

The node implements Proof of Authority consensus with:

• Modified Wave parameters (K=1, Alpha=1, Beta=1)
• Single-node blockchain operation
• No validator staking requirements
• Immediate block finalization
• Automining enabled for development

Common Tasks

Service Management

# Start monitoring daemon
./monitoring/health_check_system.sh --daemon

# Fix all RPC endpoints
./fix_all_rpc_endpoints.sh

# Deploy new app chain EVM
./deploy-hanzo-network.sh

# Start Q-Chain network
./launch-qchain.sh start

Database Operations

# Access database
psql -h localhost -p 5432 -U postgres -d explorer_lux  # legacy DB; new Lux Explorer is DB-free

# Fix explorer connections
./fix_explorer_databases.sh

C-Chain Migration

# Import ancient blockchain data
./cmd/import-ancient/import-ancient \
  -source /old/db \
  -target /new/db \
  -start 0 \
  -end 1000000

API Endpoints

RPC Endpoints (Port 9630)

• Info API: http://localhost:9630/v1/info
• C-Chain RPC: http://localhost:9630/v1/chain/C/rpc
• C-Chain WebSocket: ws://localhost:9630/v1/chain/C/ws
• Health API: http://localhost:9630/v1/health
• Metrics: http://localhost:9630/v1/metrics

Public RPC Endpoints (via Gateway)

A network is a HOST, not a path prefix. There is no /mainnet/ or /testnet/ segment — measured 2026-09-01, every such form returns 404. The path is /v1/<segment>/<chain>/rpc on that network's own host.

Network	Host	C-Chain RPC
Lux mainnet	api.lux.network	https://api.lux.network/v1/chain/C/rpc
Lux testnet	api.lux-test.network	https://api.lux-test.network/v1/chain/C/rpc
Zoo	api.zoo.network	https://api.zoo.network/v1/chain/C/rpc
Hanzo	api.hanzo.network	https://api.hanzo.network/v1/chain/C/rpc

A brand chain on the Lux host is addressed by its alias or blockchain ID in the same position: https://api.lux.network/v1/chain/zoo/rpc.

The segment is chain, and that is now what answers. This paragraph used to say the opposite — that the fleet still answered the older bc spelling and that /v1/chain/C/rpc returned 404 — and it has been the wrong way round since deploy/lux-api/route.yaml landed. Measured 2026-09-09:

POST https://api.lux.network/v1/chain/C/rpc   eth_chainId -> 0x17871   200
POST https://api.lux.cloud/v1/chain/C/rpc     eth_chainId -> 0x17871   200
POST https://api.lux.network/v1/bc/C/rpc                               404
POST https://api.lux.network/                 eth_chainId -> 0x17871   200

The bare host IS the C-chain, on both api.lux.network and api.lux.cloud: a wallet is handed a host, so that host answers RPC at /. .../rpc is not a third spelling of it and returns 404.

The trading surface, on the same two hosts

ghcr.io/luxfi/dex's gateway binary runs as lux-mainnet/trade and answers everything under /v1/trade on api.lux.cloud and api.lux.network — eight paths: venues, quote, quotes, swap, and the four allowance ones. Its declaration is deploy/lux-mainnet/trade.yaml; api/openapi.yaml in luxfi/dex is the contract, and a test in that repo holds the document to the route table so neither can drift from the other.

The other half of the surface is the indexer, and that is not an accident. Enumerating a chain — its tokens, pools, positions, TVL, price history — is an index read, and api-explore.lux.cloud answers it: measured, 16 pools and $39.9M of TVL for 96369 with real symbols and decimals. Those paths were on the gateway too and answered 500 on every request, because a quoting surface reads ONE pool by address and cannot walk every pool on Ethereum. A screen wanting a market list asks the indexer; a screen wanting a price for a size asks the gateway.

A market on 96369 is quoted from our own contracts and a market anywhere else from the pools that chain already holds — read over its own JSON-RPC, no vendor API and no upstream account. The shape is the same either way, so a screen never learns which arm answered:

POST /v1/trade/quote  {"chainId":96369,"tokenIn":WLUX,"tokenOut":LUSD,…}
  -> 997001483075056712  providerName uniswap_v3
POST /v1/trade/quote  {"chainId":1,"tokenIn":WETH,"tokenOut":USDC,…}
  -> 2512401632          providerName uniswap_v3

GET /v1/trade/venues is what a screen asks first — which chains this deployment can price, and from what. 96369 comes back native with the precompile arm beside the V3 quoter; the other six carry V2 and V3.

POST /v1/trade/swap is built by the venue that gave the price, against the SwapRouter02 beside the quoter that answered — LUX_V3_ROUTER in the values file for 96369, and Uniswap's own published router on every other chain. It used to encode every quote for the PoolManager precompile at 0x9010, which answers 0x on this chain. Proved by replaying the gateway's own calldata through eth_call on the live router from the account holding the allowance:

POST /v1/trade/swap  1 WLUX -> LUSD  ->  to 0xbbD8d9…, fee 3000
eth_call that calldata                ->  997001483075056712
the quote said                            997001483075056712

Permit2 is not on 96369. 0x000000000022D473030F116dDEE9F6B43aC78BA3 holds 9152 bytes on Ethereum and nothing here, so both permit2 paths answer 404 on this chain naming the reason rather than calldata addressed to an empty account. Approve the spender directly; /v1/trade/approval/* does that and reads the allowance from the chain.

Two routers on one host, and the chain's is the more specific. /v1/chain keeps priority 300 and the root 310; trade takes 200, which is above the bare-host router that answers these hosts out of hanzoai/cloud — api-hanzo in the ingress ingress-dynamic ConfigMap, no priority stated, so Traefik gives it its rule length, about 118. A cloud 404 on one of these hosts is recognisable by the x-api-version: header it carries.

⚠ Today the route names BOTH shapes, and that is deliberate. The pin is still the image that answers the older spellings, so a rule naming /v1/trade alone made every quote a hanzoai/cloud 404 — a surface reachable, running, and addressed by a name it does not answer to. A union is safe in both directions. strip-role stays for the same reason: the pinned image still authorises /v1/admin/* on a header the caller writes, and no rule here should be the only thing between that and the internet. Both come out in the commit that moves the pin past that version.

One prefix, where there used to be a nine-clause allowlist. The allowlist existed for a real reason: the binary published /v1/admin/pool/freeze, so PathPrefix(/v1) would have put an irreversible switch on the internet. Those handlers are deleted and the whole surface moved under /v1/trade, so the allowlist and the thing it guarded against went away together. /v1/leads and /v1/events keep their old paths and stay reachable from inside the cluster and nowhere else — not because the route omits them, but because they are not under this prefix.

⚠ The file provider IS enabled on this ingress and its routers are live — --providers.file.directory=/etc/traefik/dynamic, mounted from the ingress-dynamic ConfigMap in the hanzo namespace, alongside kubernetesingress and kubernetescrd. A router that seems to come from nowhere is in that ConfigMap.

Production Infrastructure (March 2026)

Deployment Model

Each component deploys independently via semantic version tags and container images (GHCR). No submodules. Update image tags in k8s manifests to deploy.

DOKS Cluster: lux-k8s (LB: 134.199.138.27)

All Lux production services run on the lux-k8s DOKS cluster (do-sfo3-lux-k8s):

• lux-mainnet: 5 luxd validators (StatefulSet, ports 9630-9631)
• lux-testnet: 5 luxd validators (StatefulSet, ports 9640-9641)
• lux-devnet: 5 luxd validators (StatefulSet, ports 9650-9651)
• lux-gateway: Gateway API gateway
• lux-explorer: Lux Explorer (single Go binary, indexer+GraphQL+SPA in one process) — 5 chains: C, Zoo, Hanzo, SPC, Pars
• lux-bridge: Bridge UI + API server
• lux-mpc: MPC dashboard + API (5 MPC nodes, threshold 3)
• lux-system: hanzo-ingress LoadBalancer, lux-operator

DNS (Source of truth: k8s/dns/lux-network.yaml)

• LB: 134.199.138.27 — ALL *.lux.network services (DNS-only, no CF proxy)
• Cloud: 134.199.136.205 — cloud/api.cloud/ws.cloud/web3.lux.network
• Cloudflare Zone: 287bdfd07cf016bd102f6394892e4759

White-Label Explorers

URL	Chain	Logo Source
explore.lux.network	C-Chain	~/work/lux/logo
explore-zoo.lux.network / explore.zoo.network / explore.zoo.ngo	Zoo	~/work/zoo/logo
explore-hanzo.lux.network / explore.hanzo.network / explore.hanzo.ai	Hanzo	~/work/hanzo/logo
explore-spc.lux.network	SPC	unicorn
explore-pars.lux.network / explore.pars.network	Pars	~/work/pars/logo

Key Images (GHCR)

Image	Purpose	Current
ghcr.io/luxfi/node	Blockchain node	exact tag + digest in deploy/lux-*/luxd.yaml
ghcr.io/luxfi/explorer	Explorer (single Go binary: indexer+GraphQL+SPA)	1.2.1
ghcr.io/luxfi/operator	K8s operator	v0.3.13
ghcr.io/luxfi/indexer	Chain indexer	v1.0.2
ghcr.io/luxfi/gateway	Gateway	latest

In-Cluster Databases

• postgres.hanzo.svc (user=hanzo) — DBs: cloud, commerce, console, gateway, hanzo, kms
• lux-sql-0.lux.svc — MPC API database (mpc_api)
• indexer-postgres.lux-explorer.svc — Indexer database

Hanzo Services: NOT on lux-k8s

All Hanzo services (IAM, KMS, Platform, etc.) run on the separate hanzo-k8s cluster.

Local Development

Docker Compose files in this repo are for local development only.

White-Label Architecture

See ARCHITECTURE.md for the canonical service topology, hostname-to-service mapping, and namespace layout. Key points:

• All services run on lux-k8s (do-sfo3-lux-k8s), LB: 134.199.138.27
• White-label brands (Lux, Zoo, Hanzo, SPC, Pars) share backends, differ by frontend ConfigMap
• Exchange, Explorer, Cloud, Bridge, Faucet all support per-brand env var overrides
• Brand config pattern: config/brand.ts reads NEXT_PUBLIC_BRAND_* env vars, defaults to Lux

E2E Tests

21 spec files in e2e/ (Playwright), ~488 test cases total. Run via:

pnpm exec playwright test

Spec	Tests	Area
00-health	27	Node health, chain sync
01-explorer	13	Lux Explorer UI
02-exchange	13	DEX swap flow
03-bridge	9	Bridge UI
04-staking	11	Staking operations
05-cloud	25	Cloud platform
06-wallet	13	Wallet operations
07-validator	19	Validator management
08-validator-lifecycle	17	Validator lifecycle
09-defi-flow	31	Full DeFi flow
10-stats-dashboard	15	Stats dashboard
11-wallet-ops	20	Wallet operations (deep)
12-staking-rpc	17	Staking via RPC
13-bridge-mpc	15	Bridge + MPC
14-dex-contracts	19	DEX contract interactions
15-nft-marketplace	11	NFT marketplace
16-app-chains	18	App chain operations
17-governance	11	Governance flow
18-broker	35	Broker operations
19-broker-deep	97	Broker deep tests
20-exchange-deep	30	Exchange deep tests

Plus e2e/tests/lux-cloud.spec.ts (22 tests).

Troubleshooting

Service Issues

1. Ensure networks are running: ./monitoring/health_check_system.sh
2. Check Docker logs: docker compose logs {service}
3. Verify port 9630 is open: netstat -tulpn | grep 9630
4. Run database fixes: ./fix_explorer_databases.sh

Build Failures

1. Check Go version (1.26.1+ required for node)
2. Clear build cache: go clean -cache
3. Ensure no local replace directives in go.mod
4. For JS projects: Remove node_modules and reinstall

Important Files

• Genesis Generator: unified_genesis.py - Creates genesis files
• Network Status: docs/NETWORK_STATUS.md - Current deployment status
• POA Details: docs/POA_IMPLEMENTATION_SUMMARY.md - POA guide
• Health Monitoring: monitoring/health_check_system.sh - System health
• Token Data: genesis-*/allocations_combined.json - Account balances

Ecosystem Architecture

Lux provides the base consensus layer (L0) for the AI compute ecosystem:

Lux (L0 Consensus) → Hanzo (L1 AI Compute) → Zoo (App Layer) → Zen (Models)

Related Organizations

• Lux Network: Base L0 multi-consensus blockchain
• Hanzo AI Inc (Techstars '17): L1 AI compute infrastructure (HMM, ASO)
• Zoo Labs Foundation (501c3): Decentralized training (DSO, PoAI)
• Zen LM: Open frontier models (600M-480B params)

Full Stack Development (Hanzo Ecosystem)

For complete integrated development with AI services:

cd ~/work/hanzo/experiments/stack
make dev        # Starts Router, Chat, Cloud, IAM, Search, Runtime

See ~/work/hanzo/experiments/stack/LLM.md for full documentation.

Rules for AI Assistants

1. ALWAYS update LLM.md with significant discoveries
2. NEVER commit symlinked files (.AGENTS.md, CLAUDE.md, etc.)
3. NEVER create random summary files - update THIS file
4. ALWAYS use luxfi packages, NOT go-ethereum or upstream legacy
5. NEVER bump Go packages above v1.x.x

Troubleshooting

P-Chain/Info API Timeout After Block Import

Symptom: After admin_importChain, P-chain and Info APIs timeout. Solution: Rebuild with deadlock fix (see "Critical Fix" section above).

GPU Library Not Found

Symptom: Linker errors for lux_gpu_* symbols. Solution: Build without CGO: CGO_ENABLED=0 ./scripts/build.sh

Network Controller Crash

Symptom: lux network status shows "not running". Solution:

lux network stop --mainnet --force
lux network stop --testnet --force
lux network start --mainnet

Block Import Fails

Symptom: admin_importChain returns error. Solution: Write genesis state spec first, then import.

---

K8s manifests + auto-bump

Registry convention

All Lux images live at ghcr.io/luxfi/<service>:<tag>. Never push to GAR. Mixing in shared upstream images (ghcr.io/hanzoai/{kms,gateway,ingress,sql,kv,static}) is allowed for shared platform components — Lux universe pins them where used.

Manifest layout

Path	Owner	Notes
k8s/lux-k8s/<service>/deployment.yaml	per-service	Style reference: vote/, safe/
k8s/lux-k8s/validators/{statefulset,crd-resources}.yaml	LuxNetwork CRD	Operator-managed
k8s/lux-mpc/{luxmpc,dashboard,ingress}.yaml	MPC	LuxMPC CRD + dashboard Deployment
k8s/lux-ns/dex-{api,docs}.yaml	DEX gateway/docs
k8s/lux-mainnet/, lux-explorer/	Network slices	Indexer/explorer fanout per chain

Tag style — pin reality, not aesthetics

GHCR is the source of truth. Two known styles in use:

• node (ghcr.io/luxfi/node): bare semver, e.g. 1.23.31. No v prefix exists in GHCR.
• mpc, broker, indexer: v-prefix semver, e.g. v1.10.1, v1.5.0, v1.0.1.

Never invent the other form. Verify with:

TOKEN=$(curl -s "https://ghcr.io/token?service=ghcr.io&scope=repository:luxfi/<svc>:pull" | jq -r .token)
curl -sL "https://ghcr.io/v2/luxfi/<svc>/tags/list?n=2000" -H "Authorization: Bearer $TOKEN" | jq .

Never pin :latest in main-branch manifests. If a service has no semver yet, leave a # TODO: pin once a semver tag is published comment above its image: line.

Where a values file carries both tag: and digest:, the digest is what kubelet pulls and the tag is only a label. Bumping one without the other deploys the old bits under the new name — the rollout succeeds, the pods are healthy, and the footer reports a version that is not what is running. Do not advance them by hand: deploy/pin.sh resolves both from one registry read and writes them in one edit, so there is no second lookup for them to drift against.

The release train

Service repo: build → push ghcr.io/luxfi/<svc>:<tag> → dispatch <svc>-released
universe:     .hanzo/workflows/image-published.yml → deploy/pin.sh → commit to main
cd.hanzo.ai:  git generator over deploy/*/*.yaml → reconcile

.hanzo/workflows/image-published.yml is the only listener and moving the pin is the only thing it does. Which files move is DERIVED — pin.sh --which <repo> reads every values file whose image.repository is the one that was published. The per-service table this replaced named four k8s/ paths that did not exist, and k8s/ is not read by CD anyway.

Dispatch types wired: image-published, plus <svc>-released for mpc, mpc-api, mpc-dashboard, dex-gateway, dex-docs, dex, broker, cex, bank-api, bank-admin, bank-dash, kms.

payload: { service, tag, image? } — image is a REPOSITORY, never repo:tag, and defaults to ghcr.io/luxfi/<service>. pin.sh reads the tag convention (v-prefixed or bare) out of the values file, so a sender does not have to know it. It refuses a non-semver version, an unpublished tag, and a version older than the current pin (PIN_ROLLBACK=1 to go back deliberately).

ghcr.io/luxfi/node is refused outright: luxd is promoted only by scripts/roll_luxd.sh, which enforces quorum, all-chain bootstrap and the staged-network order that a tag bump cannot express.

IAM is upstream (ghcr.io/hanzoai/iam); not in the dispatch list.

Known drift and TODOs

Service	Drift	Action
dex-gateway, dex-docs, market, wallet, pricing, explorer	No semver in GHCR	TODO comments left inline; cluster runs :latest. Publish tags from source repos.
broker, cex, dex (matching engine)	No live deployment in any Lux cluster	Dispatch types reserved; manifests to be authored once a service goes live.
bank-api, bank-admin, bank-dash	Live but commit-sha pinned	Manifests authored at sha; switch to semver once Bank repo cuts releases.
node	Universe pins 1.23.31, GHCR latest stable 1.24.28	Validate then run scripts/roll_luxd.sh once mainnet is ready to advance.
~100 values files	Concrete version, no digest	A tag names a release but a registry tag is mutable: zoo/exchange-api was pinned at bytes 2.3.34 no longer meant. pin.sh --verify lists them.
explore-fe-pars (x3), hanzo/lux-admin	pin.sh --verify cannot read them	403 from ghcr.io/parsdao with a luxfi token; no DO registry credential. Not defects — grants.
chart app	ApplicationSet pins 0.1.11; 0.1.13 is published	3 values files use kms.hostAPI/credentialsSecret, which 0.1.13 removed for a ZAP transport, and its KMSSecret group changes to kms.hanzo.ai/v1. Cluster-side prerequisite — see deploy/README.md.

---

Decomplection audit (2026-05-29) — lux-system orphan

Live cluster do-sfo3-lux-k8s ns lux-system carries an orphan status Deployment that violates several rules and must be removed:

• Image policy violation: ghcr.io/hanzoai/status:latest — no :latest per CLAUDE.md "no floating tags".
• Second pod stuck: 5+ day ContainerCreating because it mounts a ConfigMap status-config that has no source-of-truth.
• Duplicate concern: status-lux (sourced from k8s/status/deployment.yaml) serves the same role and is correctly pinned by sha.

The orphan + its ConfigMaps (status-config, status-frontend, status-brand, status-logo) and PVC (status-data) have no source in this repo — deployed by hand 92d ago before the lux/hanzo brand split. They are safely removable.

Cleanup script: k8s/status/orphan-cleanup.sh. Production-impacting; confirm before running:

KUBECTL_CONTEXT=do-sfo3-lux-k8s ./k8s/status/orphan-cleanup.sh

Recorded outside this entry — k8s/status/kustomization.yaml added so the canonical status-lux stack is Kustomize-applyable.

Decomplection audit (2026-05-29) — luxd-startup bash bag

k8s/lux-mainnet/luxd-startup.yaml inlines a single bash script that braids 7 distinct concerns:

• staking-key copy
• plugin install
• per-chain upgrade.json write
• chain-aliases aliases.json write
• bootstrap-peer DNS computation
• one-time RLP imports (mainnet only)
• luxd flag assembly

The operator OWNS the StatefulSet (ownerReferences: LuxNetwork) but does NOT manage the command/args/volumeMounts for /scripts — those survive on the live spec via SSA from a hand-applied StatefulSet patch. Dual-source pattern; the hand patches and the operator reconciliation co-exist.

Decomplection landed in luxfi/operator@decomplect/luxnetwork- declarative-startup (commit d5b3d26):

• Two of the seven concerns moved to CR fields:
  • LuxChainSpec.Aliases []string
  • LuxChainSpec.UpgradeConfig string
• One new init container slot:
  • chain-setup between seed-restore and startup-gate
• New manifest builder:
  • manifests.BuildChainSetupInitContainer (10 tests)
• Reconciler wired to emit chain-setup from CR-declared state

Migration plan (production-impacting, NOT yet run):

1. Land operator decomplection PR → bump operator image
2. Add aliases: and upgradeConfig: to chain entries in k8s/lux-mainnet/network.yaml
3. Wait for operator reconcile to write chain-setup init container
4. Verify aliases.json + upgrade.json on disk match what luxd-startup.yaml would have written
5. Remove inline cat > aliases.json and per-chain upgrade.json blocks from luxd-startup.yaml
6. After multiple successful pod restarts, scrub remaining bash concerns (bootstrap-peer DNS → operator-emitted env var; plugin install → init container with declarative VMID→objectKey map; etc.)

The five remaining concerns (staking-key, plugin install, RLP import, bootstrap-peer DNS, luxd flag assembly) are NOT YET decomplected. Tracked for follow-up; not landed this session.

---

Fresh-bootstrap primary (testnet + devnet) — 2026-06-02

testnet and devnet had been wedged with classic BLS-key drift:

node has BLS key 0xa9503da5... but is registered to the validator set
with 0x95bc37b9... (testnet)

i.e. the luxd-staking Secret signer.key didn't match the BLS pubkey encoded in the lux-{net}-genesis ConfigMap → 0 peers, 0 progress.

The fix is a clean rebuild of (Secret + genesis ConfigMap + PVC wipe), codified as a re-runnable procedure. Source of truth for the 5-validator key set: ~/work/lux/keys/node{1..5}/{staker.{crt,key}, bls/signer.key, ec/private.key} — the canonical local validator identities used by the genesis CLI's KEYS_DIR discovery path. Same keys for both testnet and devnet; per-env isolation comes from the different networkID (2 vs 3) baked into the genesis JSON.

Source-of-truth genesis ConfigMaps now committed at:

• k8s/lux-test-k8s/testnet/genesis.yaml — lux-testnet-genesis (networkID=2, C-Chain ID 96368)
• k8s/lux-dev-k8s/devnet/genesis.yaml — lux-devnet-genesis (networkID=3, C-Chain ID 96367)

Both genesis blobs:

• 5 initialStakers from node{1..5} certs (NodeIDs deterministic via NodeIDFromCert over the constant-time PEM cert bytes)
• 5 allocations, each with unlockSchedule: [{amount:50T, locktime:0}] (CRITICAL — empty unlockSchedule yields zero validator weight at genesis parse and the node refuses to boot)
• cChainGenesis sourced verbatim from ~/work/lux/genesis/configs/{testnet,devnet}/cchain.json (canonical)
• No L2 chains (chains: []) — brand L2s (Hanzo/Zoo/SPC/Pars) re-created via CreateChainTx as a separate follow-up (that follow-up was dropped 2026-06-27; brands run their own network — see "Zoo and Hanzo are not chains on the Lux primary network")

Genesis CLI gotchas:

1. LoadKeysFromDir scans EVERY subdirectory of KEYS_DIR including .git/, cmd/, proofs/. Use a clean staging dir (e.g. /tmp/lux-keys-clean/) containing ONLY the five nodeN/ folders.
2. buildConfigFromKeyInfos emits unlockSchedule: null — post-process with jq to add the matching [{amount: X, locktime: 0}].
3. -cchain ~/work/lux/genesis/configs/{net}/cchain.json is required to get the canonical EVM chainID (96368 testnet / 96367 devnet) — without it the C-Chain defaults to chainID == networkID.
4. devnet startup script previously hardcoded --track-chains=nQCV... (old L2 ID). After fresh-bootstrap the L2 doesn't exist, so C-Chain wouldn't bootstrap. Patched live CM to --track-all-chains=true (testnet pattern).

NodeID ↔ key index (deterministic):

pod	NodeID	BLS pubkey prefix
0	NodeID-Jc1ACAWeKDZR4N5PgxNsPrrcKrKMTh4ct	0xb57029264c…
1	NodeID-CTyFvG1HYdcMLqdmKwqA8EFPRQdmrNat3	0xa8c919018c…
2	NodeID-Hvsc6vanABE3ufcMhpMVEMVGpZ68DhoMb	0x8311f8eb15…
3	NodeID-44e37hMkUbjpoWsJrWcvBB4kcMK3ab8qg	0xa1efbd8916…
4	NodeID-HcFvgYUxXJx4gzQ7Sgsj8grYK6FDEgBrr	0xa07f54a097…

Verification probe (P + C bootstrap on all three networks):

for ctx_ns_port in \
    "do-sfo3-lux-k8s lux-mainnet 9630" \
    "do-sfo3-lux-test-k8s testnet 9650" \
    "do-sfo3-lux-dev-k8s devnet 9650"; do
  set -- $ctx_ns_port
  for chain in P C; do
    kubectl --context=$1 -n $2 exec luxd-0 -- \
      curl -sS -X POST -H 'Content-Type: application/json' \
      -d "{\"jsonrpc\":\"2.0\",\"method\":\"info.isBootstrapped\",\"params\":{\"chain\":\"$chain\"},\"id\":1}" \
      http://localhost:$3/v1/info
  done
done

All six results expect isBootstrapped: true. Confirmed 2026-06-02.

Brand L2 chain registration on test+dev primaries — 2026-06-02 (closes #168)

Superseded 2026-06-27. The record below is what was done and is kept for provenance; it is not the model. Brands run their own network — see "Zoo and Hanzo are not chains on the Lux primary network".

The 4 brand L2 EVMs (hanzo/zoo/spc/pars) are now registered on the fresh testnet + devnet primary networks via IssueCreateChainTx (P-chain canonical tx, codec name unchanged from upstream). One pair = one chain across both envs (the parent networkID is per-env: testnet=2, devnet=3; the per-chain validator-network (CreateNetworkTx ID) is per-(env, brand)).

Tool: ~/work/lux/genesis/cmd/bootstrap-chain (now accepts --key-file <path-to-32-byte-hex-secp256k1> in addition to MNEMONIC+BIP44). Tool source cmd/bootstrap-chain/main.go — modeled on lux/node/wallet/network/primary/examples/bootstrap-hanzo/main.go and extended to be data-driven over --track-chain-ids.

Wallet patch: lux/node/wallet/network/primary/api.go now fails-soft on "no X alias" — required because testnet+devnet primaries run in P-only mode (no XVM in platform genesis). FetchState degrades the X-chain context to ids.Empty instead of erroring; the chain-pair UTXO scan skips X entirely when X is unavailable. Drop the local replace in genesis/cmd/go.mod once luxfi/node ships a tag carrying this change.

Per-env, per-brand registration

Deployer (shared test+dev):

• private key: ~/work/lux/keys/deployer-test-dev/private.key (deterministic SHA-256 of "lux-test-dev-l2-bootstrap-deployer-2026-06-02")
• test P-chain: P-test19zc4fcupelfkue4z9wujxmu2tv9505q27ph88j
• dev P-chain: P-dev19zc4fcupelfkue4z9wujxmu2tv9505q20pelm6
• EVM addr (shared): 0x28b154e381cfd36e66a22bb9236f8a5b0b47d00a

Funded via a 6th allocations entry in lux-{testnet,devnet}-genesis (100B nLUX = 100K LUX per network, unlockSchedule:[{amount, locktime:0}], NOT in initialStakedFunds so the UTXO is spendable at genesis).

Adding this allocation required a second wipe-and-rebootstrap of the primaries (fresh #186 rebuild was funded entirely into stake). Genesis ConfigMap reapplied → StatefulSet scaled to 0 → PVCs deleted → scaled to 3 → re-bootstrapped. P + C bootstrapped on all 3 pods per env in ~75s. Snapshots from #186 retained for forensic comparison (data-luxd-{0..2}-20260602).

Common tx parameters per chain:

• vmID: mgj786NP7uDwBCcq6YwThhaN8FLyybkCa4zBWTQbNgmK6k9A6 (canonical EVM plugin)
• chain authoriser: own per-chain CreateNetworkTx (L2 anchor pattern; the bootstrap-chain tool issues CreateNetworkTx first, then CreateChainTx against that network ID)
• validators added per chain: all 5 primary validators via IssueAddChainValidatorTx (chain start = +60s, end = primary's earliest stake-end minus 1h)

Per-env inventory (also committed at k8s/lux-{test,dev}-k8s/{testnet,devnet}/chains-inventory.json):

brand	env	blockchainID	per-chain networkID	evmChainID
hanzo	test	2Cgj3Re98MfRB44SneJcfF4K6uctP2shAZKayrLpAZenzyBH7d	2NLMq51Kz7HzTDvu42r9CwfE5wxhmPoTpF8NAkYJwnVHGjqwFQ	36962
zoo	test	2UHBMA2XfiB1azgNANbby9oexbMV4aQ5JMDuLPEGvMNSLB2BrN	Wdfqb7qunsabQCxfsLnS6RCApdZxij2SXcF9oTLZR2SoYYgHH	200201
spc	test	2uKb33jCcZWCBurRG6UYtvgKd5reycscN6yXToHjhLUWzkdXYc	m7TyVpMgyi99dRF74ywP9epXY1Ta3XpNMcV6U8yA8zhqKfjZG	36910
pars	test	WhWbGHTi8SFRjHrTYDWSxhPFgUvQW2oiwzNvHgLxEdeWVS9ca	FuRQspw3hcqtWQhD8nUNTiC9rb9KTeKFaZvrKHEph8iUfejfz	494950
hanzo	dev	2gSYzcV4Jo3ATAPrsJnjhmUjPVtmMNFqEMCpDjZ2hroAtzz3Ko	sz2oFLPvGz7KQ7BHrs2tcY5WTFwP9ad2UgorsCyEM4jc563jn	36964
zoo	dev	GP4ZLRFjoTcZSmuhPvJWuvRdHC4ZhiYp5M2RSPQxc33sNjkuD	UXckap5QxjViKuKePQp6mwaWrD19is7foFsqrRDfJKDc85HGo	200202
spc	dev	2bgKkULQx7nuNPD5tVKfHjMiaqKbQWykwNVECKAL4EDuTmHGyE	EMQBtaiRSoerFsnyiHpU2F1dRfkzGw6KU6NtTA1C4E9JG4ArH	36912
pars	dev	YNcHsEAEy4tRjGi4TmEvgKXk6bykh7BSRpeCRjWJKNkYkRdB6	2kEVjgH5AquPxVFvogoi4LbtQyRg5WfGXH2XqphonzNfCpcBzx	494951

Verification (24/24 pods × chains: isBootstrapped: true):

for cid in <chainID>; do
  for pod in 0 1 2; do
    kubectl --context=do-sfo3-lux-{test,dev}-k8s -n {testnet,devnet} exec luxd-$pod -- \
      curl -sS -X POST -H 'Content-Type: application/json' \
      -d "{\"jsonrpc\":\"2.0\",\"method\":\"info.isBootstrapped\",\"params\":{\"chain\":\"$cid\"},\"id\":1}" \
      http://localhost:9650/v1/info
  done
done

eth_chainId confirms canonical map on every L2 (hex matches dec from the LP-182 EVMChainID map in luxfi/genesis).

Followups (separate tasks)

• Each new EVM chain currently sits at block 0 — --evm-heartbeat-key flag of bootstrap-chain can roll block 1 if funded. Hold for downstream contract deploys to roll their own blocks.
• Gateway path-based routing (/testnet/v1/chain/<brand>/rpc etc) in k8s/lux-gateway/gateway.yaml still points at OLD lux-testnet/ lux-devnet namespaces with stale chainIDs. Update to route to the lux-testnet / lux-devnet namespaces on lux-k8s with new blockchainIDs. (Mainnet entries are correct.) NOTE: the lux-test-k8s / lux-dev-k8s CLUSTERS no longer exist — see the consolidation section at the end of this file.
• kms-operator deployment in test+dev clusters → migrate file-on-disk ~/work/lux/keys/deployer-test-dev/private.key to KMS-stored per-env deployer secret. Same key, encrypted-at-rest path.

---

Historical network upgrade record — node version + precompiles (2026-06-23)

This section records the June 2026 migration and is not an operating procedure. Use RUNBOOK.md for the current, sole deployment path. At the time of this record, all three networks ran ghcr.io/luxfi/node:v1.30.40 (latest PQ crypto: consensus v1.25.19 ConsensusCert, pulsar v1.1.5, magnetar v1.2.3, corona v0.7.9, threshold v1.9.9) with the migrated DEX precompiles.

Historical declared state

File	Controls	Live-of-record
k8s/lux-{net}/network.yaml	LuxNetwork CR image.tag (node version)	v1.30.40
k8s/lux-{net}/luxd-startup.yaml	luxd-startup ConfigMap → C-Chain upgrade.json (precompile activations)	migrated

These are captured from the live cluster so a rebuild / operator reconcile reproduces the exact deployed state. Never hand-edit the live configmap without mirroring it here.

DEX precompile migration (dexConfig → granular, for full C↔D compat)

The monolithic dexConfig (old DEX at 0x0400) was split in node ≥ the Uniswap-v4 refactor into four granular precompiles at 0x9996-0x9999: dexPositionManagerConfig (0x9996), dexStateViewConfig (0x9997), dexQuoterConfig (0x9998), dexSettleConfig (0x9999). Each config is just {"blockTimestamp": <ts>} (stateless adapters; the old params — protocolFeeController/maxPools/flashLoans/hooks — are now on-chain governance).

• Activation ts = 1766708400 (4:20pm PST Dec 25 2025) — matches the 29 other precompiles already at that ts in the chain config. (The universe network-start table says 1766704800/PDT; that's a separate 1hr-earlier metadata value — precompile activations use 1766708400.)
• precompileUpgrades MUST be ascending by blockTimestamp — insert the new entries in sorted position, never append at the end (coreth rejects out-of-order: timestamp < previous timestamp).

Retired procedure (do not use)

1. Edit declared state here: bump network.yaml tag: and/or edit the upgrade.json inside luxd-startup.yaml.
2. Deploy via platform.hanzo.ai (PaaS-driven operator reconcile) — or the deploy-k8s.yml workflow. NEVER hand-kubectl set image/apply ad-hoc.
3. Coordinated cutover when binary↔config are incompatible: a node version that renames precompiles (e.g. v1.30.3 dexConfig vs v1.30.40 dex* ) CANNOT roll one-pod-at-a-time through one shared configmap — the configs are mutually exclusive. Switch image and config together and restart all validators (brief C-chain downtime); verify C-chain re-bootstraps to full height (= replay-safe) before declaring done; auto-rollback to the prior image+config if it doesn't.
4. DNS / public hosts are Cloudflare (zone 287bdfd07cf016bd102f6394892e4759 for *.lux.network). *.lux-dev.network / *.lux-test.network are separate CF zones (scoped token needed). Explorer frontends (explore.lux-test.network) are CF-served Next.js apps, NOT in-cluster ingresses — devnet's explore.lux-dev.network (404) needs the same CF Pages/frontend deploy, not a k8s ingress (the in-cluster devnet explorer pod serves nothing on :80/:3000).

Verify all 3 (P/X/C bootstrap + version)

for n in "lux-mainnet api.lux.network" "lux-testnet api.lux-test.network" "lux-devnet api.lux-dev.network"; do
  set -- $n; echo "$1:"; for ch in P X C; do
    curl -s -X POST -H 'content-type:application/json' \
      -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"info.isBootstrapped\",\"params\":{\"chain\":\"$ch\"}}" \
      "https://$2/v1/info"; done; echo; done

Confirmed 2026-06-23: all P/X/C isBootstrapped:true; mainnet C-chain at 1,082,800 blocks preserved through the v1.30.3→v1.30.40 coordinated cutover.

---

Last Updated: 2026-06-23 Note: This file should be committed. Symlinks (CLAUDE.md, GEMINI.md, etc.) are for local use only.

Lux as a Hanzo Cloud customer org — DOKS + billing (2026-07-02)

Lux runs as a first-class customer org on the shared Hanzo Cloud control plane (platform.hanzo.ai / console.hanzo.ai), NOT a bespoke path. This is how lux-k8s and future Lux compute are managed + billed centrally.

• Identity: [email protected] is the lux-org owner in IAM (lux.id). Logging in to platform.hanzo.ai as a lux user auto-creates the lux org + owner membership + wallet in the platform DB (tested upsertUserFromIam flow).
• DOKS: per-org DOKS provisioning/attach is enabled (Hanzo side wired PAAS_DO_API_TOKEN 2026-07-02). lux-k8s (DOKS id 04c46df5-5986-429a-9c7f-3f11a81c8539, since retired — the fleet is addressed to the hanzo-cd cluster NAME lux-cluster, not to an apiserver URL, so a replacement is registered once there and nothing in this repo changes) attached to the lux org via POST /v1/org/{luxOrgId}/cluster/attach; new clusters via POST /v1/org/{luxOrgId}/cluster. Full runbook: hanzo/universe LLM.md "Per-org DOKS provisioning".
• Billing: lux org gets an organization_wallet (seed $10,000 credits; admin top-up via admin.hanzo.ai). DO-resell usage meters DO cost → commerce → wallet, tracked + invoiced like any paying customer.
• Console: visible to the lux-org user at console.hanzo.ai (org-switch to lux) today; the branded home is console.lux.cloud (white-label, pending the SSR-brand-host fix + DNS cutover — see hanzo/universe white-label notes).

Branded web3-surface regression — e2e/branded/ (2026-07-20)

The durable "is every public surface live · functional · branded" gate. 26 surfaces across Lux/Zoo/Hanzo/Pars (explorers, wallets, id, votes, work boards, bridges, safe, clouds, exchanges) × 4 breakpoints (390/768/1280/1920) = 104 cells, real Chrome/Blink (channel:'chrome'), read-only (no txns, no wallet-connect, no auth — the login form is asserted, never submitted).

• Run: npm run test:branded (config playwright.branded.config.ts, breakpoints = projects, custom matrix reporter). Screenshots → ~/Desktop/lux-shots-e2e/<surface>-<bp>.png; matrix → branded-report/matrix.{md,txt,json}. Wired into CI as the branded job in .hanzo/workflows/e2e.yml (installs the chrome channel, uploads the matrix).
• Per cell (owner's 6-point rubric → 8 checks): HTTP 200 · branded <title>+favicon+zero foreign brand in <head> (the white-label invariant — body chain references are legit, only the head gates) · no horizontal overflow · no app-origin console errors (3rd-party telemetry + benign resource-4xx allow-listed) · key control visible · brand font (broken Times/serif fallback fails; accent is an observational note).
• Gate vs flag (the model): e2e/branded/surfaces.ts is the ONE source of truth; the spec+reporter are a pure function of it. A live surface gates every check unless a documented flag (with a reason + flip condition, optionally bps-scoped) demotes it to ⚑. down / foreign-domain postures never gate. A flagged check that starts passing is reported STALE so the table self-cleans. Only a NEW un-flagged failure turns the run red ✗.
• Baseline 2026-07-20: 67 pass · 37 flagged · 0 regressions. Documented flags (each carries a flip): bridge zoo/pars/lux Times-font fallback + zoo/pars head <meta description> still "…Lux Network." (white-label incomplete) + bridge-zoo CORS→bridge-api.lux.network; zoo/hanzo vote safe-client console errors; zoo.cloud @390 ~110px overflow; exchange.lux Vite-preload/$0-quotes + @390/@768 marketing-band overflow. Owner-gated (posture down/foreign): hanzo.cloud = JP company, hanzo.exchange = GoDaddy parking, lux.exchange = SSL/SNI-blocked. Confirmed-fixed since the 2026-07-18/19 sweeps: wallet.zoo (was 100% Lux) now Zoo-branded; zoo.vote head-leak gone; all 4 work boards + safe.lux white-labeled; explorers no longer overflow @1280.

---

Cluster consolidation — verified inventory + teardown plan (measured 2026-07-27)

One cluster per org, one NAMESPACE per blockchain network. The per-network part is already done and needs no work: doctl kubernetes cluster list returns 8 clusters and none of them is a per-network split. mainnet/testnet/devnet is a namespace everywhere it exists — lux-{mainnet,testnet,devnet}, zoo-{mainnet,testnet,devnet}, pars-{mainnet,testnet,devnet}. The old per-network CLUSTERS are already gone from DigitalOcean; only the stale k8s/lux-dev-k8s/ and k8s/lux-test-k8s/ dirs (and the Followups note above, which still names them as if live) survive them.

Fleet as scanned by clients/admin/infra (the code behind GET /v1/admin/infra): 8 clusters · 44 nodes · 183 volumes · 10 load balancers · $4,274.10/mo, complete=true.

Cluster	Nodes	Pods	What it actually runs	Verdict
hanzo-k8s	17	370	the platform; LB ingress 129.212.164.5	keep — org cluster
lux-k8s	10	343	lux-{mainnet,testnet,devnet} + apps	keep — org cluster
zoo-k8s	4	91	zoo-{mainnet,testnet,devnet}	keep — org cluster
pars-k8s	3	35	parsd-mv 5/5 ghcr.io/luxfi/node:v1.36.2 + 4 app deploys; 0 ingresses	collapse → lux-k8s
osage-k8s	3	27	osaged-mv 5/5 ghcr.io/luxfi/node:v1.36.2; 0 ingresses	collapse → lux-k8s
bootnode-k8s	2	40	8 deploys, ghcr.io/hanzoai/bootnode; 29 live hosts; ingress-nginx	collapse → hanzo-k8s bootnode
adnexus-k8s	3	46	14 deploys; 10 *.ad.nexus hosts; ingress-nginx	collapse → hanzo-k8s adnexus
hanzo-dev-k8s	2	56	dev tier; 70 ingress hosts, mostly dormant	collapse → hanzo-k8s hanzo-devnet

Removing the five candidates retires ~13 nodes (3+3+2+3+2) — at DO list price (s-4vcpu-8gb $48, s-2vcpu-4gb $24) about $504/mo of droplets plus their load balancers.

Done (safe half)

pars-mainnet, pars, osage-mainnet created on lux-k8s and declared here. pars-testnet / pars-devnet deliberately NOT created — both are empty on pars-k8s (0 deploy / 0 sts / 0 pods). bootnode, adnexus and hanzo-devnet already exist on hanzo-k8s.

Not done — needs a human decision before anything is deleted

1. pars / osage first (lowest risk: zero ingresses, so no DNS step). Move the StatefulSet + its PVCs; validator identity keys live in KMS, so re-key nothing. Capacity is the real constraint — lux-k8s already runs 345 pods on 10 nodes at 27–87% memory, so add a node pool BEFORE the move, not after.
2. bootnode-k8s is not idle. It answers 29 production hosts (lux.cloud, api.lux.cloud, bootno.de, zoo.cloud, api.lux.network, …) and several of them are ALSO declared on hanzo-k8s ns bootnode — dual-homed. Public DNS is Cloudflare-proxied, so the origin cannot be read from dig; the Cloudflare records must be read per host before its LB (134.199.136.205) is retired.
3. hanzo-dev-k8s declares production identity hosts (lux.id, pars.id, bootno.de, kms.*, s3.*) but does NOT serve them: lux.id/pars.id resolve to 129.212.164.5 = the hanzo-k8s ingress LB. The declarations are dormant duplicates; still verify per host before deleting.
4. adnexus-k8s is a clean lift-and-shift of 14 deploys + 10 hosts onto the already-existing (and currently empty) hanzo-k8s adnexus namespace.
5. Both bootnode-k8s and adnexus-k8s run ingress-nginx, which is banned — collapsing them onto hanzoai/ingress removes it as a side effect.
6. Correlated-failure note: putting pars/osage validators on lux-k8s makes one cluster a shared failure domain for three sovereign L1s. Sovereignty is a network-layer property and is unaffected, but availability is not — this is the trade to accept explicitly, not by accident.

No cluster is deleted by any commit in this repo. DOKS deletion is effectively irreversible; each teardown is its own reviewed change after its migration is verified green.

One repository: github.com/luxfi/universe (2026-09-04)

github.com/luxfi/universe is the canonical remote. git.hanzo.ai/lux/universe is unreachable; its lineage was merged into GitHub's in one commit, and The lux-fleet generator and its fence live in hanzo/universe infra/k8s/hanzo-cd/project-lux.yaml; they read this repo from GitHub and deploy to the cluster named lux-cluster. from here on:

• A static site is three objects and no image: a staticFiles Middleware, the route that names it, and the endpoint-less Service the route schema requires (deploy/lux-credit/lux-credit.yaml is the reference). The DigitalOcean-era pod declarations of those sites are gone; their last image pins are recorded in the merge commit's message.
• cd.automated: true only on a file verified to run on the current cluster. Every pod-based file is false until that is shown; a newer pin does not opt a file back in.
• A chain answers under /v1/chain/<alias>/rpc and the gateway asks luxd for health at /v1/health/ops/liveness; every declaration and script says so.
• luxd moves only through scripts/roll_luxd.sh, which writes the tag in the spelling the values files already use (bare 1.36.181) and one digest to all three fleets.
• k8s/ never redeclares what deploy/ declares. A k8s manifest that duplicated a values file is deleted, not merged back.

Zoo and Hanzo are not chains on the Lux primary network — measured 2026-09-09

Asked to create zoo and hanzo with CreateChainTx on Lux mainnet and bootstrap them. Neither was created. Three independent reasons, each measured against the live fleet on evo (10.0.0.21, five validators on 9730–9770, ghcr.io/luxfi/node:v1.36.181, --network-id=1).

They already exist, elsewhere, with their own identities. hanzod-mv-0..4 (ns hanzo-mainnet) serves eth_chainId 0x9063 = 36963 with info.isBootstrapped(C)=true, on its own primary network 36963 — its genesis ConfigMap hanzod-mv-genesis is a primary genesis (networkID: 36963, initialStakers, cChainGenesis.config.chainId: 36963), not a CreateChainTx blob. Zoo mainnet's C-chain is 23hNkQEfXaGfVVJjDPEPGgvHNfGW5BXWx47kxe67fn1qgL2fsP on sovereign network 200200, carrying 13,348 blocks (genesis 0x7c548af4, tip 13,347 0x891a539c, 729 settlements) — see zoo/universe LLM.md, cut over 2026-07-14. Creating either on the Lux P-chain mints a second, competing chain identity for the same brand and abandons that history.

The primary network's chain set is closed to brands by decision. brand_sovereign_node_one_way (2026-06-27): luxd runs the Lux primary network only. cmd/bootstrap-chain enforces it — the names hanzo/zoo/pars/spc/osage are refused before any P-chain spend. k8s/lux-mainnet/chain-aliases.yaml records the same finding: the brand aliases pointed at chains never created.

The P-chain has no JSON-RPC surface left, so the tooling cannot reach it. platformvm.CreateHandlers returns exactly one handler, at /ops (node/vms/platformvm/vm.go) — the base is deliberately unserved. Measured through luxfi/[email protected], the client path both tools use:

platform.getHeight  → /v1/chain/P  → 404      (also via api.lux.cloud)
info.getNetworkID   → /v1/info     → 404
primary.FetchState  → received status code: 404

createchaintx and bootstrap-chain both open with primary.MakeWallet, which begins at FetchState, so neither gets past its first network call. The live surface is typed operations under /v1/chain/P/ops (OpenAPI 3.1 at /ops/.well-known/openapi.json): GET /blockchains, GET /utxos, POST /tx taking already-signed bytes. Reaching the P-chain from Go again means a client that speaks that, not a newer path string.

What the primary network carries, read from /v1/chain/P/ops/blockchains. All eleven sit on net 11111111111111111111111111111111LpoYY; the P-chain itself is 11111111111111111111111111111111P. No T — it became F and M.

chain	blockchain id
C	TZEa3jDMCY5iQWqm5brNhqdszHza3e2qvzSGCC63uQNbVA3z3
X	2fdnRbzs9sWj5cVqny2sdShXzZXgvm96DyzvzgB2Z7j3VKZukd
A	2NXHomv4gbu8i6JTqALHvcbdeb8gb8r7jzbxqXvqZRd1UzorhF
B	JNF8xWg2eMam5nAq9MQaE9LuXYo9Ccv2WcY8hEts7bDdmNGtJ
D	1AFS4GPsBYrzG3KLvjEfJfWAcs5AgTs2XfFoG4xBQed95p3tT
F	Ec9QkdmFnmhPSpuvXenwQ798mFH9gYCgJjy6V4Wfg336wFaX1
G	24C9zm36x43T7LqcaKF1ikHxSeuQeTXnstzi5Gwh2apo18rXNE
K	2sK7efGAyZJcfLMKPzHg39NPJnL4SwYmFUsPDHMYVLMVYqP4Fw
M	2erizn2pu4gbafsFzbyfqwzhkR2RT1qFdwr7kKKwn5T3BH3wW8
Q	2wiiMEaeLPnxQH3gs5GU1QWCBakE4axd9rJgAqwzkWfr5BwhyM
Z	ddz6d4PFUhuxzYdLDCiKyMfcxsuboT67KqQ7mwPt9baudAoKs

Only C is an EVM: eth_chainId 0x17871 = 96369, head 0x3b = 59 on all five validators. Aliases resolve, but only under /ops — /v1/chain/P alone is a 404 and /v1/chain/P/ops/blockchains is a 200. The P-chain's /height answers 204 with no body, which is why height reads look empty.

Nothing on this P-chain can pay a fee anyway. All 100 genesis allocations carry initialAmount: 0; allocations 0–4 are initialStakedFunds and hold nothing until their stake ends (~2026-12-12). Slots 5–98 each hold 3,500 LUX unlocked against 46,500 stakeable-locked, and spending one needs the master mnemonic at BIP-44 m/44'/9000'/0'/0/<slot>, which lives in KMS and was not on this machine.

What Zoo mainnet needs here instead

zood-mv (ns zoo-mainnet) is declared and scheduled but has never started: zood-mv-0 has been Init:0/1 since it was created, on three missing objects — ConfigMaps zood-mv-genesis and zood-mv-startup, and Secret zood-mv-staking. The startup script is already committed at k8s/zoo-mainnet/zood-mv-startup.yaml (--network-id=200200, /genesis/genesis.json + /genesis/upgrade.json); the other two are not.

The genesis is the one to be careful about. zoo/universe/configs/genesis/mainnet/genesis.json is an EVM genesis (chainId 200200) with a networkID field added, which is not the shape --genesis-file wants — compare hanzod-mv-genesis, which is a primary genesis carrying its EVM under cChainGenesis. Zoo needs the same: a primary genesis for network 200200 with five stakers, that EVM blob verbatim as cChainGenesis, and the existing upgrade.json. Anything else changes the genesis hash and orphans the history.

Two facts to carry into that work. Zoo's canonical genesis hash is 0x7c548af47de27560779ccc67dda32a540944accc71dac3343da3b9cd18f14933; a genesis that does not reproduce it is a different chain. And zoo/universe LLM.md records that admin_importChain of the RLP diverges at block 800 on both v1.34.9 and v1.36.2 — the state moves by direct zapdb copy of the coreth C-db, not by RLP import. ~/work/lux/state/rlp/zoo-mainnet/zoo-mainnet-200200-0-13347.rlp (14.7 MB, blocks 0–13347, sentinel hashes matching the canonical tip) is the only copy of that history on this machine; the data-zood-0 PVC and its snapshot were on the DigitalOcean cluster.