mirror of
https://github.com/luxfi/migration.git
synced 2026-10-30 15:25:07 +00:00
zeekayandHanzo Dev
e5bf04908e
roll_pathA.sh rolled the fleet with `kubectl delete pod` and hand-patched the
luxd-startup ConfigMap. Both are wrong, and both fail quietly:
- DO's kubelet TTL-caches ConfigMaps, so a recreated pod can mount the STALE
copy and run the old config — the exact thing the roll exists to prevent.
- The ConfigMap is owned by hanzo-cd. A hand patch survives until the
reconciler notices. Two StateUpgrades applied that way on 2026-08-05 were
reverted in config while still live on-chain, leaving a chain whose own
config no longer described its state — invisible until a node replays.
Replaced with four pieces, one concern each:
fleet.sh per-pod RPC, in-place restart, stranded-node heal
fleet_verify.sh converged? direction + stateRoot at a common height
fleet_heal.sh rejoin stranded validators, one at a time
state_upgrade.sh stage an upgrade into the GitOps source, with guards
The guards are the point. state_upgrade.sh refuses unless precompileUpgrades and
every prior stateUpgrade are byte-identical, appends exactly one entry, stamps
activation off the CHAIN clock rather than wall clock, and flags a bare
left-padded address as a possible packed-slot write — the defect that put six
tokens at decimals=189 while slot-readback verification stayed green.
Parameterised by env (NS/STS/CTX/N), so the same scripts drive lux mainnet,
testnet, devnet, zoo and hanzo rather than one fleet each.
Verified against live lux-mainnet: fleet_verify reports CONVERGED 5/5 with one
distinct state root; state_upgrade --dry-run correctly refuses a manifest with
nothing new to add.
Co-authored-by: Hanzo Dev <[email protected]>
38 lines
1.5 KiB
Bash
Executable File
38 lines
1.5 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Rejoin stranded validators, one at a time.
|
|
#
|
|
# ./fleet_heal.sh luxd-4 # heal named pods
|
|
# NS=lux-testnet ./fleet_heal.sh luxd-2 luxd-3
|
|
#
|
|
# A validator that missed a gossiped block under --skip-bootstrap=true (the
|
|
# default) never re-fetches it and freezes at that height, while still logging
|
|
# "quorum: CERT assembled ... sentToPeers=4". It looks alive from every angle
|
|
# except the one that matters. startup.sh honours /data/db/.allow-bootstrap as a
|
|
# per-pod opt-in that flips --skip-bootstrap=false and self-consumes once the
|
|
# frontier is reached.
|
|
#
|
|
# ONE AT A TIME is not politeness. Quorum is 4-of-5; a fleet with one node
|
|
# already stranded has zero margin, and taking a second one down halts the chain.
|
|
set -uo pipefail
|
|
cd "$(dirname "$0")" && . ./fleet.sh
|
|
|
|
[ $# -gt 0 ] || die "usage: fleet_heal.sh <pod> [pod...]"
|
|
PORT=${BASE_PORT:-9760}
|
|
|
|
for pod in "$@"; do
|
|
hdr "healing $pod"
|
|
before=$(pod_read "$pod" "$PORT" tip)
|
|
say "tip before: ${before:-unreachable}"
|
|
heal_stranded "$pod"
|
|
wait_healthy "$pod" "$PORT" 24 || die "$pod never came back — stop here, do NOT touch another node"
|
|
if is_advancing "$pod" "$PORT" 45; then
|
|
say "tip after: $(pod_read "$pod" "$PORT" tip) advancing"
|
|
else
|
|
die "$pod answers but is still frozen. Do not heal another node; investigate this one
|
|
(check its image matches the fleet, and read its logs for the bootstrap path)."
|
|
fi
|
|
done
|
|
|
|
hdr "done"
|
|
say "re-run fleet_verify.sh to confirm state roots agree across all $N"
|