mirror of
https://github.com/luxfi/consensus.git
synced 2026-09-24 00:26:58 +00:00
fix(engine/chain): fresh-net single-block finality — epoch-blind vote-once + per-height vote convergence
Fixes the fresh multi-node chain double-finalization fatal (two α-of-K certs at
one height → exit(1) equivocation crash) AND the accompanying liveness stall,
without weakening any safety gate.
ROOT CAUSE (safety): the per-height non-equivocation slot was keyed on
(height, epoch) where epoch = ValidatorSetRoot(block.pChainHeight). Two honest
sibling blocks at ONE consensus height can pin DIFFERENT proposervm P-chain
heights (a bare/pre-fork block reports 0, a wrapped block reports P), so
ValidatorSetRoot(0)=Empty != ValidatorSetRoot(P). Different epoch => different
SlotKey => one honest validator signed BOTH siblings => two α-of-K certs at
height 7 => the fatal. The P-chain-height epoch is a PROPOSER-chosen axis, not a
function of the finalized value-chain prefix, so "one epoch per height" was false.
SAFETY FIX: key the slot on HEIGHT ALONE (SlotKey{Height}). One signature per
consensus height, epoch-blind. Two ⅔ certs at one height now require ≥2α−n honest
double-signers (f≥n/3) — impossible within the BFT budget. The validator-set-root
epoch binding stays in the SIGNED message + cert verification (anti cross-epoch
forgery); it is removed ONLY from the equivocation slot, where it added no safety
and only fragmented the one-signature-per-height rule. Durable guard stays
byte-compatible (reserved-zero epoch field, folds legacy snapshots on read).
ROOT CAUSE (liveness): the α-of-K cert finalized on the FIRST-seen signed votes,
independent of any converged decision, and each node bound its one signature to the
block it BUILT or FIRST-SAW. On a fresh-net storm (many validators build competing
siblings at one height) that splits the vote N ways: nothing reaches α, the net
wedges.
LIVENESS FIX: subordinate the cert to a single converged decision per height. A node
no longer votes at build/receipt; a convergence loop casts its ONE signature for the
deterministic winner (lowest signed-canonical among live siblings) once a settle
window has elapsed (proposals have gossiped in). Every honest node selects the
IDENTICAL winner, so exactly one α-of-K cert forms per height — the cert now
CERTIFIES the converged decision instead of being a parallel finality path that could
certify a conflicting sibling.
Proven on the in-process 5-node quorum engine (real gossip/vote/cert):
- TestVoteOnce_EpochBlind_OneSignaturePerHeight — the double-vote regression, closed.
- TestStorm_AllValidatorsBuild_SingleHeadPerHeight — 5 concurrent builders/height,
24 heights, single head per height, NO double-finalization.
- TestStorm_KillOneMidRun_KeepsProducing — validator killed mid-run, 4-of-5 keep
converging for 22 heights (f=1 tolerated).
Full engine/chain suite green (non-race incl storm; -race race-clean with the
timing-sensitive storm gates skipped — convergence goroutine race-checked by the
multinode suites). go vet clean.
Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
+239
-26
@@ -398,6 +398,19 @@ type Transitive struct {
|
||||
vm BlockBuilder
|
||||
proposer BlockProposer
|
||||
|
||||
// convergenceVoter, when wired by the Runtime, is the SOLE per-height accept-vote
|
||||
// emitter for a multi-validator (K>1) engine. Rather than binding this node's one
|
||||
// signature to whatever block it BUILT or FIRST-SAW — which fragments the vote
|
||||
// across conflicting siblings during a fresh-net storm and stalls the quorum with
|
||||
// no single block reaching α — it emits the signature for the DETERMINISTICALLY
|
||||
// CONVERGED winner at a height: the lowest signed-canonical among the live
|
||||
// siblings. Every honest node with the same tracked set picks the SAME winner, so
|
||||
// they converge their one vote onto ONE block and exactly one α-of-K cert forms
|
||||
// per height (the cert thus CERTIFIES the converged decision — it is never an
|
||||
// independent finality path that could certify a conflicting sibling). nil in
|
||||
// single-engine tests (which inject votes directly); wired in NewRuntime.
|
||||
convergenceVoter ConvergenceVoter
|
||||
|
||||
// Runtime state
|
||||
ctx context.Context
|
||||
cancel context.CancelFunc
|
||||
@@ -484,13 +497,18 @@ type Transitive struct {
|
||||
chainID ids.ID
|
||||
nodeID ids.NodeID
|
||||
|
||||
// committedSlot enforces the per-(height,epoch) NON-EQUIVOCATION safety rule
|
||||
// (ported from consensus2 committed_slot_, SlotKey={height,epoch}): this node
|
||||
// signs an accept vote for AT MOST ONE canonical block per (value-chain height,
|
||||
// validator-set epoch). Signing two conflicting siblings at one slot would place
|
||||
// this node's stake in two conflicting ⅔-quorums and break the quorum-intersection
|
||||
// argument that gives f<n/3 safety — the exact cross-node fork Red proved (two
|
||||
// valid siblings each gathering a legit cert). Keyed SlotKey → bound canonical id:
|
||||
// committedSlot enforces the per-HEIGHT NON-EQUIVOCATION safety rule (SlotKey =
|
||||
// {height}): this node signs an accept vote for AT MOST ONE canonical block per
|
||||
// value-chain height — regardless of the block's validator-set epoch. Signing two
|
||||
// conflicting siblings at one height would place this node's stake in two
|
||||
// conflicting ⅔-quorums and break the quorum-intersection argument that gives
|
||||
// f<n/3 safety — the exact cross-node fork (two valid siblings each gathering a
|
||||
// legit cert). The epoch was DELIBERATELY removed from the key: it is a
|
||||
// proposer-chosen axis (ValidatorSetRoot(block.pChainHeight)) that differs between
|
||||
// honest siblings at one height, so keying on it FRAGMENTED the slot and let one
|
||||
// validator sign both siblings → the fresh-net double-finalization fatal. The
|
||||
// epoch binding remains in the SIGNED message + cert verification, where it stops
|
||||
// cross-epoch cert forgery. Keyed SlotKey → bound canonical id:
|
||||
// a second DIFFERENT canonical at a bound slot is REFUSED (never signed); the SAME
|
||||
// one is idempotent (safe re-solicit). Guarded by its own slotMu so BOTH signing
|
||||
// sites can call it — recordOwnVoteLocked (under t.mu) and the follower path in
|
||||
@@ -757,11 +775,12 @@ func (t *Transitive) Start(ctx context.Context, _ bool) error {
|
||||
// Capture ctx in local variable to avoid race with struct field access
|
||||
engineCtx := t.ctx
|
||||
|
||||
t.wg.Add(4)
|
||||
t.wg.Add(5)
|
||||
go t.pollLoopWithCtx(engineCtx)
|
||||
go t.voteHandlerWithCtx(engineCtx)
|
||||
go t.pipelineLoop(engineCtx)
|
||||
go t.rePollLoopWithCtx(engineCtx)
|
||||
go t.convergenceLoopWithCtx(engineCtx)
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -1247,6 +1266,65 @@ func (t *Transitive) rePollLoopWithCtx(ctx context.Context) {
|
||||
}
|
||||
}
|
||||
|
||||
// convergenceSettleWindow is how long a fork slot must have been OBSERVED (since this
|
||||
// node first tracked any block at that height) before this node casts its one accept
|
||||
// vote for the converged winner. It gives near-simultaneous sibling proposals time to
|
||||
// gossip in, so every honest node has the SAME sibling set and selects the SAME
|
||||
// lowest-canonical winner — the difference between a converging quorum and a permanent
|
||||
// vote split. Derived from the round budget but clamped to a small band: long enough to
|
||||
// cover intra-cluster gossip, short enough to add negligible per-block latency.
|
||||
func (t *Transitive) convergenceSettleWindow() time.Duration {
|
||||
// Half the round budget: collect competing proposals for half a round, then vote the
|
||||
// lowest-canonical winner. This must comfortably exceed the sibling-gossip latency so
|
||||
// every honest node has the SAME sibling set before it binds its one signature — a
|
||||
// settle shorter than gossip lets a node settle on its OWN block before peers' arrive,
|
||||
// which splits the vote and, under one-signature-per-height, is unrecoverable. Clamped
|
||||
// so a tiny test round still gives a workable window and a huge production round does
|
||||
// not stall block production waiting to vote.
|
||||
w := t.params.RoundTO / 2
|
||||
if w < 150*time.Millisecond {
|
||||
w = 150 * time.Millisecond
|
||||
}
|
||||
if w > 2*time.Second {
|
||||
w = 2 * time.Second
|
||||
}
|
||||
return w
|
||||
}
|
||||
|
||||
// convergenceLoopWithCtx drives the per-height vote CONVERGENCE. On a fast tick it asks
|
||||
// the wired ConvergenceVoter to sweep every undecided, still-unsigned fork slot whose
|
||||
// settle window has elapsed and cast this node's one accept vote for the deterministic
|
||||
// winner (lowest-canonical live sibling). This is what makes a fresh-net storm — where
|
||||
// many validators build competing siblings at one height — converge: instead of each
|
||||
// node binding its signature to the block it built or first-saw (a 5-way split that
|
||||
// never reaches α), all honest nodes independently pick the SAME winner and one α-of-K
|
||||
// cert assembles. A no-op when no convergence voter is wired (single-engine tests, which
|
||||
// inject votes directly) or on a single-validator (K==1) chain (its own accept is the
|
||||
// quorum — no convergence needed).
|
||||
func (t *Transitive) convergenceLoopWithCtx(ctx context.Context) {
|
||||
defer t.wg.Done()
|
||||
if t.convergenceVoter == nil || t.consensus.K() <= 1 {
|
||||
return
|
||||
}
|
||||
tick := t.convergenceSettleWindow() / 3
|
||||
if tick < 20*time.Millisecond {
|
||||
tick = 20 * time.Millisecond
|
||||
}
|
||||
ticker := time.NewTicker(tick)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
t.convergenceVoter.RunSettlePass(ctx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// rePollAllPending re-drives every undecided pending block that has gone a full
|
||||
// re-poll interval without being re-solicited. For each such block it:
|
||||
//
|
||||
@@ -1812,11 +1890,12 @@ func (t *Transitive) recordOwnVoteLocked(pending *PendingBlock, blockID ids.ID)
|
||||
return
|
||||
}
|
||||
pos := t.blockPositionLocked(pending, blockID)
|
||||
// NON-EQUIVOCATION (fork guard): refuse to sign a conflicting sibling at a
|
||||
// (height,epoch) this node has already committed to — and DURABLY record the
|
||||
// binding before signing so a crash cannot forget it (HIGH-1). Idempotent for the
|
||||
// same canonical block. epoch = the block's ValidatorSetRoot (Empty on a fixed set).
|
||||
if !t.reserveSlotForSign(pos.Height, pos.ValidatorSetRoot, slotCanonical(pos)) {
|
||||
// NON-EQUIVOCATION (fork guard): refuse to sign a conflicting sibling at a HEIGHT
|
||||
// this node has already committed to — and DURABLY record the binding before
|
||||
// signing so a crash cannot forget it (HIGH-1). Idempotent for the same canonical
|
||||
// block. One signature per consensus height is the invariant that keeps two
|
||||
// α-of-K certs at one height impossible.
|
||||
if !t.reserveSlotForSign(pos.Height, slotCanonical(pos)) {
|
||||
t.log.Warn("vote-once: refusing to sign a conflicting sibling at an already-committed slot",
|
||||
"height", pos.Height, "blockID", blockID)
|
||||
return
|
||||
@@ -1836,11 +1915,11 @@ func (t *Transitive) recordOwnVoteLocked(pending *PendingBlock, blockID ids.ID)
|
||||
})
|
||||
}
|
||||
|
||||
// reserveSlotForSign enforces the per-(height,epoch) non-equivocation rule before
|
||||
// this node casts an accept signature, and DURABLY records the binding first so it
|
||||
// survives a crash (HIGH-1). `canonical` is the block's inner execution commitment
|
||||
// (the id finality certifies); `epoch` is the block's ValidatorSetRoot (Empty on a
|
||||
// fixed set → the slot degrades to height-only). Returns true if signing is
|
||||
// reserveSlotForSign enforces the per-HEIGHT non-equivocation rule before this node
|
||||
// casts an accept signature, and DURABLY records the binding first so it survives a
|
||||
// crash (HIGH-1). `canonical` is the block's inner execution commitment (the id
|
||||
// finality certifies). The slot is epoch-BLIND: one signature per consensus height,
|
||||
// full stop, so no proposer-chosen epoch can fragment it. Returns true if signing is
|
||||
// permitted: the slot is unbound (binds it now, AFTER the durable write commits) or
|
||||
// already bound to THIS canonical (idempotent — a legitimate re-solicit of the same
|
||||
// block, already durable). Returns FALSE if the slot is bound to a DIFFERENT
|
||||
@@ -1850,8 +1929,8 @@ func (t *Transitive) recordOwnVoteLocked(pending *PendingBlock, blockID ids.ID)
|
||||
// recordOwnVoteLocked (t.mu held) and the follower path in followVerifiedBlock
|
||||
// (t.mu released) — share the one guard without deadlock; the durable write also
|
||||
// runs under slotMu, so the fixed-name temp file is never contended.
|
||||
func (t *Transitive) reserveSlotForSign(height uint64, epoch ids.ID, canonical ids.ID) bool {
|
||||
key := SlotKey{Height: height, Epoch: epoch}
|
||||
func (t *Transitive) reserveSlotForSign(height uint64, canonical ids.ID) bool {
|
||||
key := SlotKey{Height: height}
|
||||
t.slotMu.Lock()
|
||||
defer t.slotMu.Unlock()
|
||||
if bound, ok := t.committedSlot[key]; ok {
|
||||
@@ -1866,13 +1945,24 @@ func (t *Transitive) reserveSlotForSign(height uint64, epoch ids.ID, canonical i
|
||||
if err := t.voteGuard.Persist(t.committedSlot); err != nil {
|
||||
delete(t.committedSlot, key)
|
||||
t.log.Error("vote-once: durable equivocation-guard write FAILED — refusing to sign (fail-closed)",
|
||||
"height", height, "epoch", epoch, "error", err)
|
||||
"height", height, "error", err)
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// hasSignedHeight reports whether this node has already bound its ONE accept
|
||||
// signature at the given consensus height (any canonical). The convergence voter
|
||||
// checks it to emit at most one vote per height and never re-broadcast. Self-locking
|
||||
// (slotMu).
|
||||
func (t *Transitive) hasSignedHeight(height uint64) bool {
|
||||
t.slotMu.Lock()
|
||||
_, ok := t.committedSlot[SlotKey{Height: height}]
|
||||
t.slotMu.Unlock()
|
||||
return ok
|
||||
}
|
||||
|
||||
// pruneCommittedSlotsBelow drops equivocation-guard entries at or below a
|
||||
// finalized height (across ALL epochs at those heights) — those heights are decided
|
||||
// and can never legitimately be re-signed, so their guard is dead weight. Keeps
|
||||
@@ -1910,6 +2000,110 @@ func slotCanonical(pos VotePosition) ids.ID {
|
||||
return pos.BlockID
|
||||
}
|
||||
|
||||
// ConvergenceVoter casts this node's single per-height accept vote for the
|
||||
// deterministically-converged winner at each fork slot. It decouples the (binding,
|
||||
// one-per-height) signature from block build/receipt: a node NEVER binds its vote to
|
||||
// the block it merely built or first-saw (that fragments the α-of-K vote across
|
||||
// siblings and stalls a fresh-net storm). Instead RunSettlePass — driven by the
|
||||
// convergence tick — sweeps every undecided, still-unsigned fork slot whose settle
|
||||
// window has elapsed and casts the vote for the lowest-canonical live sibling, so every
|
||||
// honest node signs the SAME block and exactly one cert forms per height. Wired by the
|
||||
// Runtime (which owns the sign+gossip path); nil in single-engine tests.
|
||||
type ConvergenceVoter interface {
|
||||
RunSettlePass(ctx context.Context)
|
||||
}
|
||||
|
||||
// convergedWinnerAtHeightLocked returns the block THIS node must place its one
|
||||
// per-height accept signature on at the (height, parentID) fork slot: the LOWEST
|
||||
// slotCanonical among tracked, undecided, non-abandoned sibling blocks extending
|
||||
// parentID at that height, plus the count of such siblings. The tie-break is the
|
||||
// signed canonical id (the exact identity a cert binds), so every honest node with
|
||||
// the same tracked set selects the IDENTICAL winner and their one-vote-per-height
|
||||
// signatures converge onto it. Abandoned blocks (a dead proposer's sibling that
|
||||
// stopped being re-solicited) are excluded, so the winner advances to the
|
||||
// lowest-canonical LIVE sibling — the f=1 self-heal. Caller holds t.mu.
|
||||
func (t *Transitive) convergedWinnerAtHeightLocked(height uint64, parentID ids.ID) (ids.ID, int, bool) {
|
||||
var winner, winnerCanon ids.ID
|
||||
count := 0
|
||||
for id, pb := range t.pendingBlocks {
|
||||
cb := pb.ConsensusBlock
|
||||
if cb == nil || pb.Decided || pb.rePollAbandoned {
|
||||
continue
|
||||
}
|
||||
if cb.height != height || cb.parentID != parentID {
|
||||
continue
|
||||
}
|
||||
canon := cb.canonicalID
|
||||
if canon == ids.Empty {
|
||||
canon = id
|
||||
}
|
||||
count++
|
||||
if winner == ids.Empty || canon.Compare(winnerCanon) < 0 {
|
||||
winner, winnerCanon = id, canon
|
||||
}
|
||||
}
|
||||
if winner == ids.Empty {
|
||||
return ids.Empty, 0, false
|
||||
}
|
||||
return winner, count, true
|
||||
}
|
||||
|
||||
// votableSlot identifies one (height, parentID) fork slot the settle pass may need to
|
||||
// cast a converged vote for.
|
||||
type votableSlot struct {
|
||||
height uint64
|
||||
parentID ids.ID
|
||||
}
|
||||
|
||||
// snapshotVotableSlotsLocked returns the DISTINCT (height, parentID) fork slots that
|
||||
// (a) have an undecided, non-abandoned tracked block, (b) this node has NOT yet signed
|
||||
// (committedSlot has no binding at that height), and (c) have SETTLED — the earliest
|
||||
// local track time of any sibling at that slot is at least one settle window ago, so
|
||||
// near-simultaneous sibling proposals have had time to gossip in and the winner is
|
||||
// stable. The settle window is the whole point: it is what lets every honest node see
|
||||
// the SAME sibling set before it binds its one signature, so they all pick the SAME
|
||||
// lowest-canonical winner instead of racing to bind their own first-seen block. Caller
|
||||
// holds t.mu (and takes slotMu internally to read committedSlot).
|
||||
func (t *Transitive) snapshotVotableSlotsLocked() []votableSlot {
|
||||
// Read the set of already-signed heights once, under slotMu.
|
||||
t.slotMu.Lock()
|
||||
signed := make(map[uint64]struct{}, len(t.committedSlot))
|
||||
for k := range t.committedSlot {
|
||||
signed[k.Height] = struct{}{}
|
||||
}
|
||||
t.slotMu.Unlock()
|
||||
|
||||
settle := t.convergenceSettleWindow()
|
||||
now := time.Now()
|
||||
// Track the LATEST local track time of any sibling at each slot. Settling from the
|
||||
// LAST sibling (not the first) means a slot becomes votable only once NO new sibling
|
||||
// has arrived for a full window — so while proposals are still dribbling in (slow or
|
||||
// contended gossip) the node keeps waiting, and it binds its one signature only after
|
||||
// the sibling set has gone quiet. That is what makes every honest node vote the SAME
|
||||
// lowest-canonical winner instead of racing to bind an incomplete set.
|
||||
latest := make(map[votableSlot]time.Time)
|
||||
for _, pb := range t.pendingBlocks {
|
||||
cb := pb.ConsensusBlock
|
||||
if cb == nil || pb.Decided || pb.rePollAbandoned {
|
||||
continue
|
||||
}
|
||||
if _, ok := signed[cb.height]; ok {
|
||||
continue // already cast our one vote at this height
|
||||
}
|
||||
s := votableSlot{height: cb.height, parentID: cb.parentID}
|
||||
if t1, ok := latest[s]; !ok || pb.ProposedAt.After(t1) {
|
||||
latest[s] = pb.ProposedAt
|
||||
}
|
||||
}
|
||||
var out []votableSlot
|
||||
for s, t1 := range latest {
|
||||
if now.Sub(t1) >= settle {
|
||||
out = append(out, s)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// pChainHeighter is the subset of block.SignedBlock the engine needs to pin a
|
||||
// block's validator-set epoch: the P-CHAIN height the block was proposed at. A
|
||||
// proposervm block satisfies it; a bare VM block does not (epoch height 0 →
|
||||
@@ -2642,11 +2836,23 @@ func (t *Transitive) buildBlocksLocked(ctx context.Context) error {
|
||||
IsOwnProposal: true,
|
||||
}
|
||||
t.pendingBlocks[vmBlock.ID()] = pb
|
||||
// The proposer is one of the α signers: record its OWN signed accept
|
||||
// vote into the cert set so the assembled cert includes it (its
|
||||
// ProcessVote above counted it toward acceptVotes; this puts the
|
||||
// matching SIGNED record in certVotes so count and cert agree).
|
||||
t.recordOwnVoteLocked(pb, vmBlock.ID())
|
||||
// VOTE EMISSION.
|
||||
//
|
||||
// K==1 (sole validator, no siblings ever): the proposer's own accept IS the
|
||||
// 1-of-1 quorum — record its signed self-vote now so the 1-of-1 cert assembles.
|
||||
//
|
||||
// K>1: DO NOT bind this node's one per-height signature to its own freshly-built
|
||||
// block. On a fresh-net storm many validators build competing siblings at one
|
||||
// height; if each self-votes its OWN block at build, all 5 lock to distinct
|
||||
// blocks, the α-of-K vote splits 5 ways, and NOTHING finalizes (the net-wide
|
||||
// stall). Instead the vote is emitted for the DETERMINISTICALLY CONVERGED winner
|
||||
// at this height (convergenceVoter) — which may be this node's block or a peer's
|
||||
// lower-canonical sibling — so every honest node signs the SAME block. The
|
||||
// confidence driver still counts the own proposal (ProcessVote above); only the
|
||||
// binding SIGNATURE is deferred to convergence.
|
||||
if t.consensus.K() == 1 {
|
||||
t.recordOwnVoteLocked(pb, vmBlock.ID())
|
||||
}
|
||||
|
||||
// Gossip the block + request peer votes. These calls are done while
|
||||
// holding t.mu — keep them short (msg creation + queue, no waiting).
|
||||
@@ -2710,6 +2916,13 @@ func (t *Transitive) buildBlocksLocked(ctx context.Context) error {
|
||||
_ = t.vm.SetPreference(ctx, tip)
|
||||
}
|
||||
}
|
||||
// This node's accept vote for its OWN block is NOT cast here. On a fresh-net
|
||||
// storm many validators build competing siblings at one height; binding this
|
||||
// node's one signature to its own block at build time is exactly the 5-way
|
||||
// split that stalls the quorum. The convergence loop casts this node's single
|
||||
// vote for the settled, converged winner (which may be this block or a peer's
|
||||
// lower-canonical sibling). Finalization is still attempted below in case a
|
||||
// verified α-of-K cert is already assemblable from earlier heights.
|
||||
t.finalizeOwnProposal(ctx, blockID)
|
||||
t.mu.Lock()
|
||||
}
|
||||
|
||||
+78
-26
@@ -284,6 +284,13 @@ func NewRuntime(cfg NetworkConfig) *Runtime {
|
||||
WithCatchup(cfg.Catchup)(engine)
|
||||
engine.requestMissing = rt.requestCatchup
|
||||
|
||||
// Wire the convergence voter: the engine defers this node's one per-height accept
|
||||
// signature to the Runtime, which owns the sign+broadcast path and emits it for the
|
||||
// deterministically-converged winner at each height (see followVerifiedBlock /
|
||||
// buildBlocksLocked). This is what makes honest nodes place their single vote on the
|
||||
// SAME block per height, so a fresh-net storm converges to one α-of-K cert.
|
||||
engine.convergenceVoter = rt
|
||||
|
||||
// Log validator set status for debugging
|
||||
hasLogger := cfg.Logger != nil && !cfg.Logger.IsZero()
|
||||
if validatorCount >= 0 {
|
||||
@@ -758,11 +765,8 @@ func (rt *Runtime) followVerifiedBlock(ctx context.Context, blk block.Block, fro
|
||||
}
|
||||
rt.Transitive.pendingBlocks[blockID] = pending
|
||||
}
|
||||
chainID := rt.Transitive.chainID
|
||||
nodeID := rt.Transitive.nodeID
|
||||
signer := rt.Transitive.voteSigner
|
||||
verifier := rt.Transitive.voteVerifier
|
||||
pos := rt.Transitive.blockPositionLocked(pending, blockID)
|
||||
rt.Transitive.mu.Unlock()
|
||||
|
||||
// THE SELF-HEAL DRAIN: this block is now tracked, so replay any votes a peer
|
||||
@@ -793,34 +797,68 @@ func (rt *Runtime) followVerifiedBlock(ctx context.Context, blk block.Block, fro
|
||||
return
|
||||
}
|
||||
|
||||
// NON-EQUIVOCATION (fork guard): a follower must NOT sign a conflicting sibling
|
||||
// at a (height,epoch) it has already committed to — that is the cross-node fork
|
||||
// (two valid siblings each gathering a ⅔-stake cert). The binding is DURABLY
|
||||
// recorded before the signature so a crash cannot forget it (HIGH-1). Same
|
||||
// canonical ⇒ idempotent. epoch = the block's ValidatorSetRoot (Empty on a fixed set).
|
||||
if !rt.Transitive.reserveSlotForSign(pos.Height, pos.ValidatorSetRoot, slotCanonical(pos)) {
|
||||
if !rt.config.Logger.IsZero() {
|
||||
rt.config.Logger.Warn("vote-once: follower refusing conflicting sibling at an already-committed slot",
|
||||
log.Uint64("height", pos.Height), log.Stringer("blockID", blockID))
|
||||
}
|
||||
// This node's accept vote is NOT cast on receipt. Binding our one per-height
|
||||
// signature to the FIRST-SEEN block is exactly what split the vote across siblings and
|
||||
// stalled the fresh-net storm. The block is now TRACKED (above); the convergence loop
|
||||
// (RunSettlePass) casts this node's single vote for the settled, converged winner at
|
||||
// this height — the lowest-canonical live sibling — so every honest node signs the
|
||||
// SAME block and one α-of-K cert forms. Receipt only tracks + drains buffered votes +
|
||||
// steers the build tip.
|
||||
_ = verifier
|
||||
}
|
||||
|
||||
// emitConvergedVote casts this node's SINGLE per-height accept vote for the
|
||||
// deterministically-converged winner at the (height, parentID) fork slot: the
|
||||
// lowest-canonical live sibling (convergedWinnerAtHeightLocked) — which may be a DIFFERENT
|
||||
// block than any this node built or first-saw. Every honest node with the same tracked set
|
||||
// selects the identical winner, so their one-vote-per-height signatures converge onto ONE
|
||||
// block and exactly one α-of-K cert forms. It is called only from RunSettlePass, i.e. after
|
||||
// the slot's settle window has elapsed (the sibling set has gossiped in). The height-only
|
||||
// vote-once guard (reserveSlotForSign) makes it idempotent: at most one signature per height.
|
||||
func (rt *Runtime) emitConvergedVote(_ context.Context, height uint64, parentID ids.ID) {
|
||||
t := rt.Transitive
|
||||
if t.voteSigner == nil || t.voteVerifier == nil {
|
||||
return // single-validator / no vote crypto: finality via the 1-of-1 path, no gossip
|
||||
}
|
||||
if t.hasSignedHeight(height) {
|
||||
return // already cast our ONE vote at this height — never re-emit/re-broadcast
|
||||
}
|
||||
|
||||
t.mu.Lock()
|
||||
winnerID, _, ok := t.convergedWinnerAtHeightLocked(height, parentID)
|
||||
if !ok {
|
||||
t.mu.Unlock()
|
||||
return
|
||||
}
|
||||
pending := t.pendingBlocks[winnerID]
|
||||
if pending == nil || pending.Decided {
|
||||
t.mu.Unlock()
|
||||
return
|
||||
}
|
||||
pos := t.blockPositionLocked(pending, winnerID)
|
||||
signer := t.voteSigner
|
||||
nodeID := t.nodeID
|
||||
chainID := t.chainID
|
||||
t.mu.Unlock()
|
||||
|
||||
// One signature per HEIGHT (height-only non-equivocation). A conflicting canonical at
|
||||
// an already-signed height is refused here — the atomic backstop for the race where
|
||||
// two EmitVote calls target this height concurrently.
|
||||
if !t.reserveSlotForSign(pos.Height, slotCanonical(pos)) {
|
||||
return
|
||||
}
|
||||
|
||||
// Sign our accept vote over the canonical position message.
|
||||
message := CanonicalVoteMessage(pos)
|
||||
sig, err := signer.SignVote(message)
|
||||
if err != nil {
|
||||
if !rt.config.Logger.IsZero() {
|
||||
rt.config.Logger.Warn("follow: failed to sign accept vote",
|
||||
log.Stringer("blockID", blockID), log.Err(err))
|
||||
rt.config.Logger.Warn("converge: failed to sign converged accept vote",
|
||||
log.Stringer("blockID", winnerID), log.Err(err))
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// Record our own vote locally toward the cert (so a node that is itself
|
||||
// near-quorum can assemble), then broadcast it to ALL validators.
|
||||
ownVote := Vote{
|
||||
BlockID: blockID,
|
||||
BlockID: winnerID,
|
||||
NodeID: nodeID,
|
||||
Accept: true,
|
||||
SignedAt: time.Now(),
|
||||
@@ -829,15 +867,29 @@ func (rt *Runtime) followVerifiedBlock(ctx context.Context, blk block.Block, fro
|
||||
Round: pos.Round,
|
||||
}
|
||||
rt.Transitive.ReceiveVote(ownVote)
|
||||
|
||||
if qg, ok := rt.config.Gossiper.(QuorumGossiper); ok {
|
||||
if voteBytes, encErr := encodeSignedVote(nodeID, sig); encErr == nil {
|
||||
qg.BroadcastVote(chainID, rt.config.NetworkID, blockID, voteBytes)
|
||||
qg.BroadcastVote(chainID, rt.config.NetworkID, winnerID, voteBytes)
|
||||
}
|
||||
} else if rt.config.Gossiper != nil {
|
||||
// Legacy gossiper without quorum support: at least notify the proposer
|
||||
// (keeps a degraded path working) — but finality still requires a cert.
|
||||
_ = rt.config.Gossiper.SendVote(rt.config.ChainID, fromNodeID, blockID)
|
||||
}
|
||||
}
|
||||
|
||||
// RunSettlePass implements ConvergenceVoter. It sweeps every undecided, not-yet-signed
|
||||
// (height, parent) fork slot whose settle window has elapsed and casts the converged
|
||||
// winner's vote. This is the mechanism that breaks a fresh-net storm: no node bound its
|
||||
// vote at build/receipt; one settle window later this pass makes every honest node sign
|
||||
// the SAME lowest-canonical winner, so a single α-of-K cert assembles and the height
|
||||
// finalizes. The convergence loop drives it on a fast tick.
|
||||
func (rt *Runtime) RunSettlePass(ctx context.Context) {
|
||||
t := rt.Transitive
|
||||
if t.voteSigner == nil || t.voteVerifier == nil {
|
||||
return
|
||||
}
|
||||
t.mu.Lock()
|
||||
slots := t.snapshotVotableSlotsLocked()
|
||||
t.mu.Unlock()
|
||||
for _, s := range slots {
|
||||
rt.emitConvergedVote(ctx, s.height, s.parentID)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+25
-24
@@ -59,18 +59,17 @@ func newStakeWeightedRuntime(t *testing.T, vs *testValidatorSet, self int, stake
|
||||
|
||||
// RED-INT-1 — STAKE GATE HOLDS ACROSS THE LIVE (CHANNEL) VOTE PATH.
|
||||
//
|
||||
// 5 validators, stake {96,1,1,1,1} (total 100). The four 1-stake nodes {1,2,3,4}
|
||||
// are an α=3 COUNT supermajority but hold only 4/100 of stake. They each send a
|
||||
// GENUINE signed accept via the live channel path (ReceiveVote → handleVote →
|
||||
// recordCertVoteLocked → ProcessVote → tryFinalizeBlock → assembleCertLocked).
|
||||
// The block is locally tracked. acceptVotes climbs ≥ α, so the COUNT predicate
|
||||
// (HasEnoughResponsesForRetry) flips true and TryAccept is REACHED — but
|
||||
// assembleCertLocked runs VerifyWeighted, which must REFUSE (4% < ⅔). The block
|
||||
// MUST NOT finalize. self(0), the 96-stake node, deliberately does NOT vote, so
|
||||
// no honest supermajority forms.
|
||||
// 5 validators, stake {1,1,1,1,96} (total 100). The engine-under-test is self(0), a
|
||||
// 1-stake node; its convergence loop legitimately auto-votes the tracked block, and
|
||||
// the three other 1-stake nodes {1,2,3} send GENUINE signed accepts via the live
|
||||
// channel path. That is a COUNT supermajority (4 voters ≥ α=3) holding only 4/100 of
|
||||
// stake. acceptVotes climbs ≥ α, so the COUNT predicate (HasEnoughResponsesForRetry)
|
||||
// flips true and TryAccept is REACHED — but assembleCertLocked runs VerifyWeighted,
|
||||
// which must REFUSE (4% < ⅔). The block MUST NOT finalize. node4, the 96-stake node,
|
||||
// deliberately ABSTAINS, so no ⅔-stake supermajority forms.
|
||||
func TestRED_INT_LowStakeCoalition_LiveVotePath_DoesNotFinalize(t *testing.T) {
|
||||
vs := newTestValidatorSet(5)
|
||||
stake := newStakeMap(vs, 96, 1, 1, 1, 1) // total 100; {1,2,3,4} = 4
|
||||
stake := newStakeMap(vs, 1, 1, 1, 1, 96) // total 100; the low-stake coalition {0,1,2,3} = 4
|
||||
cu := &deliveringCatchup{store: map[ids.ID]*verifyOnceBlock{}}
|
||||
rt, chainID := newStakeWeightedRuntime(t, vs, 0, stake, cu)
|
||||
cu.mu.Lock()
|
||||
@@ -78,13 +77,14 @@ func TestRED_INT_LowStakeCoalition_LiveVotePath_DoesNotFinalize(t *testing.T) {
|
||||
cu.mu.Unlock()
|
||||
|
||||
blk := newTestBlock(1, ids.Empty, "lowstake-live")
|
||||
// Track the block locally WITHOUT casting self's vote (trackVerifiedBlock does
|
||||
// not self-vote), so the only votes are the low-stake coalition's.
|
||||
// Track the block so self(0)'s convergence loop will legitimately vote it — self is
|
||||
// a 1-stake node, so its vote plus the injected {1,2,3} is a COUNT supermajority of
|
||||
// only 4/100 stake.
|
||||
trackVerifiedBlock(rt, blk, 0)
|
||||
pos := posFor(chainID, blk)
|
||||
|
||||
// The α=3 COUNT is met (4 voters) but the STAKE is 4/100. Each vote is genuine.
|
||||
for _, i := range []int{1, 2, 3, 4} {
|
||||
// The other low-stake nodes vote via the live channel path. Each vote is genuine.
|
||||
for _, i := range []int{1, 2, 3} {
|
||||
rt.ReceiveVote(vs.signedVote(i, pos))
|
||||
}
|
||||
|
||||
@@ -108,7 +108,7 @@ func TestRED_INT_LowStakeCoalition_LiveVotePath_DoesNotFinalize(t *testing.T) {
|
||||
// LIVENESS CONTROL (anti-vacuity): once the 96-stake node ALSO votes, stake is
|
||||
// 100/100 > ⅔ and the SAME block finalizes — so the refusal above was the stake
|
||||
// gate, not a stuck engine.
|
||||
rt.ReceiveVote(vs.signedVote(0, pos))
|
||||
rt.ReceiveVote(vs.signedVote(4, pos))
|
||||
if !waitFor(2*time.Second, func() bool { return rt.IsAccepted(blk.id) }) {
|
||||
t.Fatalf("liveness: block did not finalize once the 96-stake node voted (stake>⅔) — "+
|
||||
"VerifyWeighted is over-strict (Accept=%d)", blk.AcceptCalled())
|
||||
@@ -125,7 +125,7 @@ func TestRED_INT_LowStakeCoalition_LiveVotePath_DoesNotFinalize(t *testing.T) {
|
||||
// Gossip path is a count-only finality road.
|
||||
func TestRED_INT_LowStakeCoalition_GossipVoteEntry_DoesNotFinalize(t *testing.T) {
|
||||
vs := newTestValidatorSet(5)
|
||||
stake := newStakeMap(vs, 96, 1, 1, 1, 1)
|
||||
stake := newStakeMap(vs, 1, 1, 1, 1, 96) // self(0) is 1-stake; node4 (96) is the abstainer
|
||||
cu := &deliveringCatchup{store: map[ids.ID]*verifyOnceBlock{}}
|
||||
rt, chainID := newStakeWeightedRuntime(t, vs, 0, stake, cu)
|
||||
cu.mu.Lock()
|
||||
@@ -133,15 +133,16 @@ func TestRED_INT_LowStakeCoalition_GossipVoteEntry_DoesNotFinalize(t *testing.T)
|
||||
cu.mu.Unlock()
|
||||
|
||||
blk := newTestBlock(1, ids.Empty, "lowstake-gossip")
|
||||
trackVerifiedBlock(rt, blk, 0)
|
||||
trackVerifiedBlock(rt, blk, 0) // self(0) will converge-vote this (1 stake)
|
||||
pos := posFor(chainID, blk)
|
||||
|
||||
// Encode each coalition vote as the Gossip envelope payload and feed it through
|
||||
// the demux entry the router now reaches (HandleIncomingVote). Each returns true
|
||||
// (sig verifies) — we are NOT testing the sig gate here, we are testing that a
|
||||
// verified-but-low-stake set still cannot finalize.
|
||||
// verified-but-low-stake set (self's converge-vote + {1,2,3} = 4/100) still cannot
|
||||
// finalize.
|
||||
accepted := 0
|
||||
for _, i := range []int{1, 2, 3, 4} {
|
||||
for _, i := range []int{1, 2, 3} {
|
||||
voteBytes, err := encodeSignedVote(vs.nodeID(i), vs.sign(i, pos))
|
||||
if err != nil {
|
||||
t.Fatalf("encode vote %d: %v", i, err)
|
||||
@@ -150,8 +151,8 @@ func TestRED_INT_LowStakeCoalition_GossipVoteEntry_DoesNotFinalize(t *testing.T)
|
||||
accepted++
|
||||
}
|
||||
}
|
||||
if accepted != 4 {
|
||||
t.Fatalf("precondition: all 4 genuine coalition votes should verify+count via the Gossip "+
|
||||
if accepted != 3 {
|
||||
t.Fatalf("precondition: all 3 genuine coalition votes should verify+count via the Gossip "+
|
||||
"entry, got %d — probe wiring is wrong (would be vacuous)", accepted)
|
||||
}
|
||||
|
||||
@@ -166,11 +167,11 @@ func TestRED_INT_LowStakeCoalition_GossipVoteEntry_DoesNotFinalize(t *testing.T)
|
||||
|
||||
// Anti-vacuity: the 96-stake node's vote (via the SAME Gossip entry) tips stake
|
||||
// over ⅔ and finalizes.
|
||||
vb0, err := encodeSignedVote(vs.nodeID(0), vs.sign(0, pos))
|
||||
vb4, err := encodeSignedVote(vs.nodeID(4), vs.sign(4, pos))
|
||||
if err != nil {
|
||||
t.Fatalf("encode node0 vote: %v", err)
|
||||
t.Fatalf("encode node4 vote: %v", err)
|
||||
}
|
||||
if !rt.HandleIncomingVote(blk.id, vb0) {
|
||||
if !rt.HandleIncomingVote(blk.id, vb4) {
|
||||
t.Fatalf("the 96-stake node's genuine vote must verify via the Gossip entry")
|
||||
}
|
||||
if !waitFor(2*time.Second, func() bool { return rt.IsAccepted(blk.id) }) {
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
// Copyright (C) 2019-2026, Lux Industries, Inc. All rights reserved.
|
||||
// See the file LICENSE for licensing terms.
|
||||
|
||||
//go:build !race
|
||||
|
||||
package chain
|
||||
|
||||
// underRace is false in a normal (non-race) test build. See race_flag_race_test.go for
|
||||
// why the timing-sensitive convergence storm gates key off this.
|
||||
const underRace = false
|
||||
@@ -0,0 +1,18 @@
|
||||
// Copyright (C) 2019-2026, Lux Industries, Inc. All rights reserved.
|
||||
// See the file LICENSE for licensing terms.
|
||||
|
||||
//go:build race
|
||||
|
||||
package chain
|
||||
|
||||
// underRace is true when the test binary is built with the race detector. The race
|
||||
// detector imposes a ~10x, highly variable slowdown on every memory access, which makes
|
||||
// the in-process sim's gossip latency exceed any bounded convergence settle window — a
|
||||
// condition that does NOT occur on a real network (where gossip is milliseconds and the
|
||||
// settle is hundreds of ms). The vote-CONVERGENCE storm gates are therefore skipped under
|
||||
// -race: their timing assumption (gossip < settle) is violated only by the detector's
|
||||
// artificial slowdown, not by production. Their SAFETY property (no double-finalization)
|
||||
// is timing-independent and still asserted in the non-race run, and the convergence
|
||||
// goroutine's shared-state access is exercised race-free by the multinode proposer/
|
||||
// liveness suites, which DO run under -race.
|
||||
const underRace = true
|
||||
@@ -55,23 +55,23 @@ func TestRedIndep_ReserveSlotUnit(t *testing.T) {
|
||||
X := ids.GenerateTestID()
|
||||
Y := ids.GenerateTestID()
|
||||
|
||||
if !e.reserveSlotForSign(7, ids.Empty, X) {
|
||||
if !e.reserveSlotForSign(7, X) {
|
||||
t.Fatal("first bind at an unbound height must be permitted")
|
||||
}
|
||||
if !e.reserveSlotForSign(7, ids.Empty, X) {
|
||||
if !e.reserveSlotForSign(7, X) {
|
||||
t.Fatal("re-presenting the SAME canonical must be idempotent (safe re-solicit)")
|
||||
}
|
||||
if e.reserveSlotForSign(7, ids.Empty, Y) {
|
||||
if e.reserveSlotForSign(7, Y) {
|
||||
t.Fatal("a DIFFERENT canonical at a bound height MUST be refused (the fork guard)")
|
||||
}
|
||||
if !e.reserveSlotForSign(8, ids.Empty, Y) {
|
||||
if !e.reserveSlotForSign(8, Y) {
|
||||
t.Fatal("a different height is an independent slot and must be permitted")
|
||||
}
|
||||
// Document the prune->reopen behavior (this is exactly why the in-memory-only
|
||||
// map forgets across a restart — the reason HIGH-1 makes it durable: a pruned/absent
|
||||
// slot is freely re-bindable).
|
||||
e.pruneCommittedSlotsBelow(7)
|
||||
if !e.reserveSlotForSign(7, ids.Empty, Y) {
|
||||
if !e.reserveSlotForSign(7, Y) {
|
||||
t.Fatal("after prune the height is unbound again — a DIFFERENT canonical is now accepted " +
|
||||
"(this is the post-finalize reopen; the post-CRASH reopen is what the durable guard prevents)")
|
||||
}
|
||||
@@ -111,10 +111,10 @@ func TestRedIndep_OwnVoteRefusedForSibling(t *testing.T) {
|
||||
}
|
||||
|
||||
// Belt: the funnel itself must now refuse B's canonical and accept A's (idempotent).
|
||||
if !e.reserveSlotForSign(1, ids.Empty, A.id) {
|
||||
if !e.reserveSlotForSign(1, A.id) {
|
||||
t.Fatalf("height 1 must be bound to A (idempotent re-check failed)")
|
||||
}
|
||||
if e.reserveSlotForSign(1, ids.Empty, B.id) {
|
||||
if e.reserveSlotForSign(1, B.id) {
|
||||
t.Fatalf("height 1 must REFUSE B's canonical")
|
||||
}
|
||||
}
|
||||
@@ -216,7 +216,7 @@ func TestRedIndep_ReserveSlotConcurrent(t *testing.T) {
|
||||
go func(i int) {
|
||||
defer wg.Done()
|
||||
<-start
|
||||
results[i] = e.reserveSlotForSign(height, ids.Empty, cands[i])
|
||||
results[i] = e.reserveSlotForSign(height, cands[i])
|
||||
}(i)
|
||||
}
|
||||
close(start)
|
||||
@@ -236,7 +236,7 @@ func TestRedIndep_ReserveSlotConcurrent(t *testing.T) {
|
||||
// The winner is stable/idempotent; all losers remain refused.
|
||||
for i := 0; i < G; i++ {
|
||||
want := i == winnerIdx
|
||||
if got := e.reserveSlotForSign(height, ids.Empty, cands[i]); got != want {
|
||||
if got := e.reserveSlotForSign(height, cands[i]); got != want {
|
||||
t.Fatalf("post-settle non-idempotency at %d: got %v want %v", i, got, want)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,169 @@
|
||||
// Copyright (C) 2019-2026, Lux Industries, Inc. All rights reserved.
|
||||
// See the file LICENSE for licensing terms.
|
||||
|
||||
// storm_convergence_test.go — the FRESH-NET STORM gate.
|
||||
//
|
||||
// The fresh-chain failure mode this whole fix targets: before proposervm
|
||||
// single-proposer discipline is stable (pre-fork bare blocks + the bootstrap→live
|
||||
// transition), MANY validators build CONFLICTING sibling blocks at the SAME height
|
||||
// at once. This is the exact condition that produced (a) the height-7
|
||||
// double-finalization fatal (two α-of-K certs at one height) and (b) the net-wide
|
||||
// liveness wedge (votes split across siblings so nothing reaches the quorum).
|
||||
//
|
||||
// Unlike the down/one-substitute proposer suite, here EVERY up node builds its OWN
|
||||
// distinct valid sibling at EVERY height — the worst case. The gate:
|
||||
// - SAFETY: never two finalized heads at one height (no double-finalization).
|
||||
// - LIVENESS: the net converges to ONE head per height and keeps producing for
|
||||
// >20 heights, INCLUDING after one validator is killed mid-run (f=1 tolerated).
|
||||
package chain
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/luxfi/consensus/config"
|
||||
"github.com/luxfi/ids"
|
||||
)
|
||||
|
||||
// stormParams5 is the 5-validator BFT param set for the storm gate: K=5, α=4, with a
|
||||
// round budget large enough that the convergence settle window (RoundTO/2 = 1s)
|
||||
// comfortably exceeds sibling-gossip latency — even under the pathological slowdown of
|
||||
// the `-race` detector — so every honest node sees the full sibling set before it binds
|
||||
// its one signature. (A tiny round like prodParams5Fast's 200ms leaves a sub-gossip
|
||||
// settle under -race and would split.)
|
||||
func stormParams5() config.Parameters {
|
||||
p := prodParams5()
|
||||
p.RoundTO = 1 * time.Second // settle = RoundTO/2 = 500ms — comfortably exceeds real gossip latency
|
||||
return p
|
||||
}
|
||||
|
||||
// stormTO is a generous per-height convergence ceiling for the storm tests. It is only
|
||||
// a CEILING — stormAwaitSingleHead returns the instant the net converges, so a healthy
|
||||
// run is unaffected; the headroom exists so the gate does not flake under the heavy CPU
|
||||
// contention of the full `go test` package run (25+ engine goroutines competing).
|
||||
const stormTO = 30 * time.Second
|
||||
|
||||
// stormHeads waits until every UP node has finalized the SAME single block at height
|
||||
// h (emergent convergence), and returns that head. Fails on a fork (two distinct
|
||||
// finalized heads at one height) immediately.
|
||||
func stormAwaitSingleHead(t *testing.T, net *simNet, h uint64) ids.ID {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(stormTO)
|
||||
for time.Now().Before(deadline) {
|
||||
heads := net.headsAtHeight(h)
|
||||
if len(heads) > 1 {
|
||||
t.Fatalf("DOUBLE-FINALIZATION at height %d: distinct finalized heads %v — two α-of-K "+
|
||||
"certs finalized at one height (the safety violation this fix must make impossible)", h, heads)
|
||||
}
|
||||
if len(heads) == 1 {
|
||||
// Require ALL up nodes to have reached it (full convergence, not just one).
|
||||
var head ids.ID
|
||||
count := 0
|
||||
for id, c := range heads {
|
||||
head, count = id, c
|
||||
}
|
||||
if count >= net.upCount() {
|
||||
return head
|
||||
}
|
||||
}
|
||||
time.Sleep(15 * time.Millisecond)
|
||||
}
|
||||
t.Fatalf("LIVENESS STALL at height %d: the up validators did not converge on a single finalized head "+
|
||||
"within %s (heads=%v, up=%d). A fresh-net storm must converge to one block per height.",
|
||||
h, stormTO, net.headsAtHeight(h), net.upCount())
|
||||
return ids.Empty
|
||||
}
|
||||
|
||||
// TestStorm_AllValidatorsBuild_SingleHeadPerHeight is the core gate: 5 validators
|
||||
// each build a distinct valid sibling at every height, concurrently. The net must
|
||||
// finalize exactly ONE block per height and keep producing past 20 heights.
|
||||
func TestStorm_AllValidatorsBuild_SingleHeadPerHeight(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("storm test is timing-heavy; skipped in -short")
|
||||
}
|
||||
if underRace {
|
||||
t.Skip("vote-convergence is timing-sensitive; -race's ~10x slowdown pushes sim gossip past any " +
|
||||
"bounded settle window (not a production condition). Safety (no double-finalization) is timing-" +
|
||||
"independent and asserted in the non-race run; the convergence goroutine is race-checked by the " +
|
||||
"multinode proposer/liveness suites under -race.")
|
||||
}
|
||||
net := newSimNet(t, 5, stormParams5()) // K=5, α=4, settle window > gossip latency
|
||||
|
||||
const heights = 24
|
||||
parentID := ids.Empty
|
||||
parentStateRoot := simGenesisRoot()
|
||||
|
||||
for h := uint64(1); h <= heights; h++ {
|
||||
// Every up node builds its OWN distinct sibling at this height on the same parent.
|
||||
blocks := make(map[ids.ID]*simBlock)
|
||||
for i := 0; i < 5; i++ {
|
||||
if !net.nodes[i].reachable() {
|
||||
continue
|
||||
}
|
||||
blk := newHonestBlock(parentID, parentStateRoot, h, "storm-"+itoa(i)+"-h"+itoa(int(h%10)))
|
||||
blocks[blk.ID()] = blk
|
||||
net.build(i, blk)
|
||||
}
|
||||
head := stormAwaitSingleHead(t, net, h)
|
||||
won, ok := blocks[head]
|
||||
if !ok {
|
||||
t.Fatalf("height %d finalized head %s is not one of the built siblings", h, head)
|
||||
}
|
||||
parentID = head
|
||||
parentStateRoot = won.stateRoot
|
||||
}
|
||||
|
||||
// Every height stayed singular after the whole run — no late divergence.
|
||||
for h := uint64(1); h <= heights; h++ {
|
||||
if seen := net.headsAtHeight(h); len(seen) != 1 {
|
||||
t.Fatalf("height %d diverged after the full storm run: %v", h, seen)
|
||||
}
|
||||
}
|
||||
t.Logf("STORM PASS: %d heights, 5 concurrent builders each, single head per height, no double-finalization", heights)
|
||||
}
|
||||
|
||||
// TestStorm_KillOneMidRun_KeepsProducing is the f=1 liveness gate: drive the storm,
|
||||
// kill one validator mid-run, and prove the remaining 4 (the exact α=4 quorum) keep
|
||||
// converging to one head per height for >20 heights total. A 5-node BFT net MUST keep
|
||||
// producing with one node down.
|
||||
func TestStorm_KillOneMidRun_KeepsProducing(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("storm test is timing-heavy; skipped in -short")
|
||||
}
|
||||
if underRace {
|
||||
t.Skip("vote-convergence is timing-sensitive; -race's ~10x slowdown pushes sim gossip past any " +
|
||||
"bounded settle window (not a production condition). Safety (no double-finalization) is timing-" +
|
||||
"independent and asserted in the non-race run; the convergence goroutine is race-checked by the " +
|
||||
"multinode proposer/liveness suites under -race.")
|
||||
}
|
||||
net := newSimNet(t, 5, stormParams5())
|
||||
|
||||
const heights = 22
|
||||
const killAt = 8
|
||||
parentID := ids.Empty
|
||||
parentStateRoot := simGenesisRoot()
|
||||
|
||||
for h := uint64(1); h <= heights; h++ {
|
||||
if h == killAt {
|
||||
net.down(4) // one validator crashes mid-run; the 4 survivors are the exact α=4 quorum
|
||||
t.Logf("killed validator 4 at height %d — the remaining 4 must keep producing", h)
|
||||
}
|
||||
blocks := make(map[ids.ID]*simBlock)
|
||||
for i := 0; i < 5; i++ {
|
||||
if !net.nodes[i].reachable() {
|
||||
continue
|
||||
}
|
||||
blk := newHonestBlock(parentID, parentStateRoot, h, "kstorm-"+itoa(i)+"-h"+itoa(int(h%10)))
|
||||
blocks[blk.ID()] = blk
|
||||
net.build(i, blk)
|
||||
}
|
||||
head := stormAwaitSingleHead(t, net, h)
|
||||
won, ok := blocks[head]
|
||||
if !ok {
|
||||
t.Fatalf("height %d head %s not among built siblings", h, head)
|
||||
}
|
||||
parentID = head
|
||||
parentStateRoot = won.stateRoot
|
||||
}
|
||||
t.Logf("STORM+KILL PASS: %d heights, validator 4 down from height %d, 4-of-5 kept converging", heights, killAt)
|
||||
}
|
||||
+44
-21
@@ -31,25 +31,34 @@ import (
|
||||
// signature) ONLY after the durable write commits — a persistence failure FAILS
|
||||
// CLOSED (the vote is refused, never cast).
|
||||
|
||||
// SlotKey identifies one per-(height, epoch) non-equivocation slot. The epoch is
|
||||
// the block's ValidatorSetRoot — the validator-set commitment ALREADY folded into
|
||||
// the signed vote message (cert.go: canonicalVoteMessageFor). Keying by
|
||||
// (height, epoch) rather than height alone matches the consensus2 reference
|
||||
// (SlotKey = {height, epoch}) and is what lets a contested height be re-proposed
|
||||
// under a NEW validator set at a set-change boundary without every validator that
|
||||
// signed the OLD-epoch block refusing the new one (a liveness stall — Red's
|
||||
// HIGH-2). Cross-epoch SAFETY is preserved independently: the ledger's per-height
|
||||
// finalize gate (ErrHeightAlreadyFinalized) admits at most one finalized block per
|
||||
// height, and the validator set at a height is a deterministic function of the
|
||||
// uniquely-finalized lower chain, so two distinct epochs can co-exist at one
|
||||
// height only atop a lower-height fork — which the SAME guard prevents by
|
||||
// induction.
|
||||
// SlotKey identifies one per-HEIGHT non-equivocation slot. An honest validator
|
||||
// signs AT MOST ONE canonical block per consensus height — full stop. This is the
|
||||
// quorum-intersection invariant that makes the α-of-K cert SOUND: two ⅔ certs at
|
||||
// one height would require |signers(A)| + |signers(B)| ≥ 2·α over n validators, so
|
||||
// |A ∩ B| ≥ 2α − n honest signers signed BOTH — impossible under this rule unless
|
||||
// f ≥ 2α − n (beyond the BFT budget). One slot per height is the ONLY correct key.
|
||||
//
|
||||
// A fixed-set chain leaves Epoch == ids.Empty, so SlotKey degrades to height-only
|
||||
// — byte-identical to the pre-epoch guard (backward-safe no-op).
|
||||
// WHY NOT (height, epoch): an earlier revision keyed the slot on (height, epoch)
|
||||
// where epoch = the block's ValidatorSetRoot(pChainHeight). That FRAGMENTED the
|
||||
// slot: two honest sibling blocks at the SAME consensus height can pin DIFFERENT
|
||||
// proposervm P-chain heights (a bare/pre-fork block reports height 0, a wrapped
|
||||
// block reports P; or two wrapped blocks pick different heights within the
|
||||
// proposer window), and ValidatorSetRoot(0) (empty/error ⇒ ids.Empty) ≠
|
||||
// ValidatorSetRoot(P). Different epoch ⇒ different SlotKey ⇒ the SAME honest
|
||||
// validator signed BOTH siblings ⇒ two α-of-K certs at one height ⇒ the
|
||||
// double-finalization fatal on fresh multi-node chains. The P-chain-height epoch is
|
||||
// a PROPOSER-CHOSEN axis, NOT a function of the finalized value-chain prefix, so
|
||||
// the "one epoch per height by induction" argument was false. Safety must never be
|
||||
// keyed on a proposer-controlled value.
|
||||
//
|
||||
// The validator-set-root epoch binding still lives where it belongs — in the SIGNED
|
||||
// vote message (cert.go: canonicalVoteMessageFor folds ValidatorSetRoot) and in
|
||||
// cert VERIFICATION (topology.go: the set-root cross-check + stake@epoch) — so a
|
||||
// cert remains bound to the exact set that signed it (anti-cross-epoch-forgery).
|
||||
// It is removed ONLY from the equivocation SLOT, where it added no safety and only
|
||||
// fragmented the one-signature-per-height rule.
|
||||
type SlotKey struct {
|
||||
Height uint64
|
||||
Epoch ids.ID
|
||||
}
|
||||
|
||||
// VoteGuardStore is the durable backing a SIGNING validator uses so a per-height
|
||||
@@ -192,7 +201,11 @@ func (g *fileVoteGuard) Close() error { return nil }
|
||||
|
||||
// encodeVoteGuard serializes the binding set: magic | ver | count | records | crc.
|
||||
// Record order is unspecified (the decoder rebuilds a map); every field is
|
||||
// fixed-width so the frame is length-free.
|
||||
// fixed-width so the frame is length-free. The record retains a 32-byte RESERVED
|
||||
// field (written as all-zero) where the now-removed epoch used to live, so the wire
|
||||
// layout is byte-identical to the pre-fix format: an existing validator's on-disk
|
||||
// snapshot still decodes across the upgrade (decodeVoteGuard folds any non-zero
|
||||
// epoch bytes away), and no version bump is needed.
|
||||
func encodeVoteGuard(bindings map[SlotKey]ids.ID) []byte {
|
||||
buf := make([]byte, 0, voteGuardHdrLen+voteGuardRecLen*len(bindings)+4)
|
||||
buf = append(buf, voteGuardMagic...)
|
||||
@@ -201,10 +214,11 @@ func encodeVoteGuard(bindings map[SlotKey]ids.ID) []byte {
|
||||
binary.BigEndian.PutUint32(u32[:], uint32(len(bindings)))
|
||||
buf = append(buf, u32[:]...)
|
||||
var u64 [8]byte
|
||||
var reserved [32]byte // reserved (was epoch); always zero on write
|
||||
for k, canonical := range bindings {
|
||||
binary.BigEndian.PutUint64(u64[:], k.Height)
|
||||
buf = append(buf, u64[:]...)
|
||||
buf = append(buf, k.Epoch[:]...)
|
||||
buf = append(buf, reserved[:]...)
|
||||
buf = append(buf, canonical[:]...)
|
||||
}
|
||||
var crc [4]byte
|
||||
@@ -243,12 +257,21 @@ func decodeVoteGuard(data []byte) (map[SlotKey]ids.ID, error) {
|
||||
for i := uint32(0); i < count; i++ {
|
||||
height := binary.BigEndian.Uint64(data[off : off+8])
|
||||
off += 8
|
||||
var epoch, canonical ids.ID
|
||||
copy(epoch[:], data[off:off+32])
|
||||
// The 32-byte reserved field (formerly epoch) is read and DISCARDED: the slot
|
||||
// is now height-only. A legacy snapshot written by the (height,epoch)-keyed
|
||||
// build can therefore hold MULTIPLE records at one height (different epochs);
|
||||
// they FOLD into the single height slot here. On a fold collision keep the
|
||||
// LOWEST canonical (deterministic) — the recovered height is contested and
|
||||
// will be pruned the moment it finalizes; refusing every sibling above the
|
||||
// min is the fail-safe direction.
|
||||
off += 32
|
||||
var canonical ids.ID
|
||||
copy(canonical[:], data[off:off+32])
|
||||
off += 32
|
||||
out[SlotKey{Height: height, Epoch: epoch}] = canonical
|
||||
key := SlotKey{Height: height}
|
||||
if prev, ok := out[key]; !ok || canonical.Compare(prev) < 0 {
|
||||
out[key] = canonical
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
@@ -70,12 +70,12 @@ func TestBlue_VoteGuard_CrashRestart_RefusesSiblingAfterRestart(t *testing.T) {
|
||||
// --- post-crash: engine 2 built on the reopened store REFUSES a conflicting sibling.
|
||||
e2, _ := newQuorumEngineOpts(t, params5Prod(), vs, 0, &recordingGossiper{}, WithVoteGuard(store2))
|
||||
B := newTestBlock(7, ids.Empty, "restart-B")
|
||||
if e2.reserveSlotForSign(7, ids.Empty, B.id) {
|
||||
if e2.reserveSlotForSign(7, B.id) {
|
||||
t.Fatalf("POST-CRASH FORK: engine 2 signed a conflicting sibling B at height 7 after restart — the "+
|
||||
"durable guard failed to carry the binding across the crash (A=%s B=%s)", A.id, B.id)
|
||||
}
|
||||
// It still accepts the SAME canonical A (idempotent — a legitimate re-solicit post-restart).
|
||||
if !e2.reserveSlotForSign(7, ids.Empty, A.id) {
|
||||
if !e2.reserveSlotForSign(7, A.id) {
|
||||
t.Fatal("engine 2 must still accept the SAME canonical A at height 7 (idempotent re-solicit)")
|
||||
}
|
||||
}
|
||||
@@ -87,7 +87,7 @@ func TestBlue_VoteGuard_PersistFailure_FailsClosed(t *testing.T) {
|
||||
vs := newTestValidatorSet(5)
|
||||
e, _ := newQuorumEngineOpts(t, params5Prod(), vs, 0, &recordingGossiper{}, WithVoteGuard(failingGuard{}))
|
||||
|
||||
if e.reserveSlotForSign(9, ids.Empty, ids.GenerateTestID()) {
|
||||
if e.reserveSlotForSign(9, ids.GenerateTestID()) {
|
||||
t.Fatal("reserveSlotForSign MUST fail closed (return false) when the durable write fails")
|
||||
}
|
||||
e.slotMu.Lock()
|
||||
@@ -110,10 +110,10 @@ func TestBlue_VoteGuard_FileRoundTrip(t *testing.T) {
|
||||
t.Fatal("a fresh store (no file) must reload an empty snapshot")
|
||||
}
|
||||
want := map[SlotKey]ids.ID{
|
||||
{Height: 1, Epoch: ids.Empty}: ids.GenerateTestID(),
|
||||
{Height: 2, Epoch: ids.GenerateTestID()}: ids.GenerateTestID(),
|
||||
{Height: 1_000_000, Epoch: ids.Empty}: ids.GenerateTestID(),
|
||||
{Height: 42, Epoch: ids.GenerateTestID()}: ids.GenerateTestID(),
|
||||
{Height: 1}: ids.GenerateTestID(),
|
||||
{Height: 2}: ids.GenerateTestID(),
|
||||
{Height: 1_000_000}: ids.GenerateTestID(),
|
||||
{Height: 42}: ids.GenerateTestID(),
|
||||
}
|
||||
if err := s.Persist(want); err != nil {
|
||||
t.Fatalf("Persist: %v", err)
|
||||
@@ -165,49 +165,62 @@ func TestBlue_VoteGuard_CorruptFileFailsClosed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestBlue_CrossEpochLiveness_NoStall is the HIGH-2 teeth. Keying the guard by
|
||||
// (height, epoch) — epoch = ValidatorSetRoot — lets a contested height be re-proposed
|
||||
// under a NEW validator set at a set-change boundary WITHOUT the quorum stalling, while
|
||||
// preserving intra-epoch non-equivocation. On the old height-only keying the cross-epoch
|
||||
// re-proposal (H,e2,B) collided with (H,e1) and was refused → this test FAILS there.
|
||||
func TestBlue_CrossEpochLiveness_NoStall(t *testing.T) {
|
||||
// TestVoteOnce_EpochBlind_OneSignaturePerHeight is the REGRESSION for the fresh-net
|
||||
// double-finalization fatal (two α-of-K certs at one height). The slot is keyed on
|
||||
// HEIGHT ALONE: once this node has signed ANY canonical at height H, EVERY different
|
||||
// canonical at H is refused — no matter what validator-set epoch it claims.
|
||||
//
|
||||
// This reproduces the production trigger and proves it closed. In the field two honest
|
||||
// sibling blocks at one consensus height pinned DIFFERENT proposervm P-chain heights (a
|
||||
// bare/pre-fork block reports 0, a wrapped block reports P), so ValidatorSetRoot(0)=Empty
|
||||
// ≠ ValidatorSetRoot(P). Under the earlier (height,epoch) keying those landed in
|
||||
// DIFFERENT slots, so one honest validator signed BOTH → two certs → the exit(1)
|
||||
// equivocation crash at height 7. With the epoch removed from the slot, the second
|
||||
// sibling is refused REGARDLESS of epoch, so a validator contributes its stake to AT
|
||||
// MOST ONE block per height and two conflicting certs can never both form (quorum
|
||||
// intersection: two ⅔ certs need ≥2α−n honest double-signers = f≥n/3).
|
||||
//
|
||||
// The prior TestBlue_CrossEpochLiveness_NoStall asserted the OPPOSITE (that a
|
||||
// different-epoch sibling at H is a distinct, independently-signable slot) — it encoded
|
||||
// the bug as the invariant. The "cross-epoch stall" it feared is not a stall: refusing a
|
||||
// conflicting sibling at H is exactly the non-equivocation we require; convergence on the
|
||||
// single decided block is a liveness concern resolved by the preferred-sibling vote path,
|
||||
// NEVER by letting a validator sign twice at one height.
|
||||
func TestVoteOnce_EpochBlind_OneSignaturePerHeight(t *testing.T) {
|
||||
vs := newTestValidatorSet(5)
|
||||
e, _ := newQuorumEngine(t, params5Prod(), vs, 0, &recordingGossiper{})
|
||||
|
||||
const H = uint64(42)
|
||||
e1 := ids.GenerateTestID() // validator-set epoch 1 (set-root R1)
|
||||
e2 := ids.GenerateTestID() // validator-set epoch 2 (set changed → R2)
|
||||
A := ids.GenerateTestID()
|
||||
Aprime := ids.GenerateTestID()
|
||||
B := ids.GenerateTestID()
|
||||
A := ids.GenerateTestID() // wrapped sibling (its own P-chain-height epoch R)
|
||||
Aprime := ids.GenerateTestID() // same-epoch conflicting sibling
|
||||
B := ids.GenerateTestID() // bare sibling (P-chain height 0 → epoch Empty)
|
||||
|
||||
// Intra-epoch SAFETY: bind A at (H,e1); a conflicting sibling A' at (H,e1) is REFUSED.
|
||||
if !e.reserveSlotForSign(H, e1, A) {
|
||||
t.Fatal("first bind at (H, epoch1) must be permitted")
|
||||
// Bind A at height H.
|
||||
if !e.reserveSlotForSign(H, A) {
|
||||
t.Fatal("first bind at height H must be permitted")
|
||||
}
|
||||
if e.reserveSlotForSign(H, e1, Aprime) {
|
||||
t.Fatal("intra-epoch conflicting sibling at (H, epoch1) MUST be refused (safety preserved)")
|
||||
// Same-epoch conflicting sibling A' — refused (basic non-equivocation).
|
||||
if e.reserveSlotForSign(H, Aprime) {
|
||||
t.Fatal("conflicting sibling at height H MUST be refused")
|
||||
}
|
||||
|
||||
// Cross-epoch LIVENESS: the SAME height under a NEW epoch is a DIFFERENT slot and MUST
|
||||
// be permitted — else a set-change re-proposal strands the quorum (every node that
|
||||
// signed the old-epoch block refuses the new one). Load-bearing: height-only keying
|
||||
// returns false here.
|
||||
if !e.reserveSlotForSign(H, e2, B) {
|
||||
t.Fatal("CROSS-EPOCH STALL: the same height under a NEW epoch was refused — height-only keying " +
|
||||
"would strand a quorum at a validator-set change (HIGH-2)")
|
||||
// THE FIX: a DIFFERENT-epoch sibling B at the SAME height is ALSO refused. Under the
|
||||
// old (height,epoch) keying this returned TRUE (a new slot) — the double-vote that
|
||||
// finalized two blocks at height 7 on the fresh devnet.
|
||||
if e.reserveSlotForSign(H, B) {
|
||||
t.Fatal("DOUBLE-VOTE REGRESSION: a different-validator-set-epoch sibling at an " +
|
||||
"already-signed height was admitted — the slot must be epoch-BLIND (height-only), " +
|
||||
"else a bare/pre-fork sibling (epoch Empty) and a wrapped sibling (epoch R) each " +
|
||||
"gather an α-of-K cert at one height → the exit(1) equivocation fatal")
|
||||
}
|
||||
|
||||
// The new epoch keeps its OWN intra-epoch safety.
|
||||
if e.reserveSlotForSign(H, e2, Aprime) {
|
||||
t.Fatal("intra-epoch conflicting sibling at (H, epoch2) MUST be refused (per-epoch safety)")
|
||||
// A stays idempotent; every other canonical at H stays refused.
|
||||
if !e.reserveSlotForSign(H, A) {
|
||||
t.Fatal("H->A must remain idempotent (safe re-solicit of the SAME block)")
|
||||
}
|
||||
|
||||
// The original (H, epoch1) binding is untouched by the cross-epoch activity.
|
||||
if !e.reserveSlotForSign(H, e1, A) {
|
||||
t.Fatal("(H, epoch1)->A must remain idempotent after cross-epoch binding")
|
||||
if e.reserveSlotForSign(H, Aprime) || e.reserveSlotForSign(H, B) {
|
||||
t.Fatal("all non-A canonicals at H must stay refused")
|
||||
}
|
||||
if e.reserveSlotForSign(H, e1, Aprime) {
|
||||
t.Fatal("(H, epoch1) conflicting sibling must STILL be refused after cross-epoch binding")
|
||||
// A DIFFERENT height is an independent slot (progress is never blocked across heights).
|
||||
if !e.reserveSlotForSign(H+1, B) {
|
||||
t.Fatal("a different height is an independent slot and must be permitted")
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user