universe/k8s/PHASE_A_ROLLBACK.md

Phase A — Operator Migration Rollback Procedure

The Phase A migration (lux.network/v1alpha1 LuxNetwork etc. → lux.cloud/v1 managed by operator v0.6.3) hit an upstream operator bug at first attempt on 2026-06-02 (see luxfi/operator d2ecca5).

This file documents how to roll back if a future attempt fails.

What Phase A does

1. Backs up live CRs to k8s/backups/pre-phase-a-<ts>/
2. Installs lux.cloud CRDs alongside lux.network CRDs
3. Runs scripts/migrate-luxnetwork-crds.sh <ctx> which apiVersion-swaps each CR (LuxNetwork stays LuxNetwork; LuxChain→Chain; LuxGateway→Gateway; LuxIndexer→Indexer; LuxExplorer→Explorer; LuxMPC→MPC)
4. Scales legacy operator (v0.3.x) to 0 replicas
5. Applies k8s/lux-k8s/operator/deployment.yaml (v0.6.x image, --api-group=lux.cloud)
6. Verifies new operator reconciles new CRs and StatefulSets stay healthy

Rollback (in order)

A) Scale v0.6.x operator to 0

kubectl --context=<ctx> -n lux-system scale deploy lux-operator --replicas=0

B) Patch deployment back to v0.3.12 + legacy args

kubectl --context=<ctx> -n lux-system patch deploy lux-operator --type=json -p='[
  {"op":"replace","path":"/spec/template/spec/containers/0/image","value":"ghcr.io/luxfi/operator:v0.3.12"},
  {"op":"replace","path":"/spec/template/spec/containers/0/args","value":["--namespace","","--log-level","info","--metrics-port","8080","--health-port","8081","--leader-election"]}
]'

C) Restore ClusterRole rules for the legacy lux.network apiGroup

(v0.6.x manifest replaces the ClusterRole to only cover lux.cloud.)

kubectl --context=<ctx> apply -f - <<'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: lux-operator
rules:
- apiGroups: ["lux.network"]
  resources:
  - luxnetworks
  - luxnetworks/status
  - luxchains
  - luxchains/status
  - luxgateways
  - luxgateways/status
  - luxindexers
  - luxindexers/status
  - luxexplorers
  - luxexplorers/status
  - luxmpcs
  - luxmpcs/status
  verbs: ["*"]
- apiGroups: ["lux.cloud"]
  resources:
  - networks
  - networks/status
  - luxnetworks
  - luxnetworks/status
  - chains
  - chains/status
  - gateways
  - gateways/status
  - mpcs
  - mpcs/status
  - indexers
  - indexers/status
  - explorers
  - explorers/status
  verbs: ["*"]
- apiGroups: ["apps"]
  resources: ["statefulsets", "deployments", "daemonsets", "replicasets"]
  verbs: ["*"]
- apiGroups: [""]
  resources: ["pods", "services", "configmaps", "secrets", "persistentvolumeclaims", "events", "serviceaccounts", "pods/exec"]
  verbs: ["*"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["batch"]
  resources: ["jobs", "cronjobs"]
  verbs: ["*"]
- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["*"]
- apiGroups: ["rbac.authorization.k8s.io"]
  resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses"]
  verbs: ["*"]
- apiGroups: ["policy"]
  resources: ["poddisruptionbudgets"]
  verbs: ["*"]
- apiGroups: ["secrets.lux.network"]
  resources: ["kmssecrets"]
  verbs: ["*"]
EOF

D) Scale legacy operator back to 1

kubectl --context=<ctx> -n lux-system scale deploy lux-operator --replicas=1

E) (Optional) delete migrated lux.cloud CRs

The legacy v1alpha1 CRs were left untouched, so legacy reconcile resumes immediately. The new lux.cloud CRs become inert (no reconciler) and are safe to leave in place — but for a clean rollback:

for k in luxnetworks chains gateways mpcs indexers explorers; do
  kubectl --context=<ctx> get ${k}.lux.cloud -A --no-headers \
    | awk '{print $1,$2}' \
    | while read ns name; do
      kubectl --context=<ctx> -n "$ns" delete ${k}.lux.cloud "$name"
    done
done

F) (Optional) uninstall lux.cloud CRDs

If a clean rollback is needed:

kubectl --context=<ctx> delete -f ~/work/lux/operator/spec/crd/bases/lux.cloud/

CRDs that still have CRs in them block deletion until step E completes for that Kind.

Verification after rollback

# Operator pod healthy
kubectl --context=<ctx> -n lux-system get pods -l app=lux-operator
kubectl --context=<ctx> -n lux-system logs -l app=lux-operator --tail=20

# StatefulSets still 5/5 ready
for ns in lux-mainnet lux-testnet lux-devnet; do
  kubectl --context=<ctx> -n $ns get sts luxd
done