Three things went wrong when a public network took a while to come up.
The node config set track-chains to the string "all"; that key takes
chain IDs, and the boolean that means every chain is track-all-chains,
so a chain deployed after boot was not served until the nodes were
restarted. The start budget was two minutes, sized for three nodes
rather than for creating and bootstrapping every chain in P-chain
state. And network state was written only after the network reported
healthy, so a run that timed out left processes running that status,
stop and clean could no longer see.
State is now written as soon as the backend answers, on the timeout
path as well, and the budget is ten minutes.
Co-authored-by: Hanzo Dev <[email protected]>