fix(engine/chain): fresh-net single-block finality — epoch-blind vote-once + per-height vote convergence

Fixes the fresh multi-node chain double-finalization fatal (two α-of-K certs at
one height → exit(1) equivocation crash) AND the accompanying liveness stall,
without weakening any safety gate.

ROOT CAUSE (safety): the per-height non-equivocation slot was keyed on
(height, epoch) where epoch = ValidatorSetRoot(block.pChainHeight). Two honest
sibling blocks at ONE consensus height can pin DIFFERENT proposervm P-chain
heights (a bare/pre-fork block reports 0, a wrapped block reports P), so
ValidatorSetRoot(0)=Empty != ValidatorSetRoot(P). Different epoch => different
SlotKey => one honest validator signed BOTH siblings => two α-of-K certs at
height 7 => the fatal. The P-chain-height epoch is a PROPOSER-chosen axis, not a
function of the finalized value-chain prefix, so "one epoch per height" was false.

SAFETY FIX: key the slot on HEIGHT ALONE (SlotKey{Height}). One signature per
consensus height, epoch-blind. Two ⅔ certs at one height now require ≥2α−n honest
double-signers (f≥n/3) — impossible within the BFT budget. The validator-set-root
epoch binding stays in the SIGNED message + cert verification (anti cross-epoch
forgery); it is removed ONLY from the equivocation slot, where it added no safety
and only fragmented the one-signature-per-height rule. Durable guard stays
byte-compatible (reserved-zero epoch field, folds legacy snapshots on read).

ROOT CAUSE (liveness): the α-of-K cert finalized on the FIRST-seen signed votes,
independent of any converged decision, and each node bound its one signature to the
block it BUILT or FIRST-SAW. On a fresh-net storm (many validators build competing
siblings at one height) that splits the vote N ways: nothing reaches α, the net
wedges.

LIVENESS FIX: subordinate the cert to a single converged decision per height. A node
no longer votes at build/receipt; a convergence loop casts its ONE signature for the
deterministic winner (lowest signed-canonical among live siblings) once a settle
window has elapsed (proposals have gossiped in). Every honest node selects the
IDENTICAL winner, so exactly one α-of-K cert forms per height — the cert now
CERTIFIES the converged decision instead of being a parallel finality path that could
certify a conflicting sibling.

Proven on the in-process 5-node quorum engine (real gossip/vote/cert):
  - TestVoteOnce_EpochBlind_OneSignaturePerHeight — the double-vote regression, closed.
  - TestStorm_AllValidatorsBuild_SingleHeadPerHeight — 5 concurrent builders/height,
    24 heights, single head per height, NO double-finalization.
  - TestStorm_KillOneMidRun_KeepsProducing — validator killed mid-run, 4-of-5 keep
    converging for 22 heights (f=1 tolerated).
Full engine/chain suite green (non-race incl storm; -race race-clean with the
timing-sensitive storm gates skipped — convergence goroutine race-checked by the
multinode suites). go vet clean.

Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
2026-07-01 13:13:34 -07:00
co-authored by Hanzo Dev
parent d029323697
commit 75eb08d5b1
9 changed files with 646 additions and 147 deletions
+239 -26
View File
@@ -398,6 +398,19 @@ type Transitive struct {
vm BlockBuilder
proposer BlockProposer
// convergenceVoter, when wired by the Runtime, is the SOLE per-height accept-vote
// emitter for a multi-validator (K>1) engine. Rather than binding this node's one
// signature to whatever block it BUILT or FIRST-SAW — which fragments the vote
// across conflicting siblings during a fresh-net storm and stalls the quorum with
// no single block reaching α — it emits the signature for the DETERMINISTICALLY
// CONVERGED winner at a height: the lowest signed-canonical among the live
// siblings. Every honest node with the same tracked set picks the SAME winner, so
// they converge their one vote onto ONE block and exactly one α-of-K cert forms
// per height (the cert thus CERTIFIES the converged decision — it is never an
// independent finality path that could certify a conflicting sibling). nil in
// single-engine tests (which inject votes directly); wired in NewRuntime.
convergenceVoter ConvergenceVoter
// Runtime state
ctx context.Context
cancel context.CancelFunc
@@ -484,13 +497,18 @@ type Transitive struct {
chainID ids.ID
nodeID ids.NodeID
// committedSlot enforces the per-(height,epoch) NON-EQUIVOCATION safety rule
// (ported from consensus2 committed_slot_, SlotKey={height,epoch}): this node
// signs an accept vote for AT MOST ONE canonical block per (value-chain height,
// validator-set epoch). Signing two conflicting siblings at one slot would place
// this node's stake in two conflicting ⅔-quorums and break the quorum-intersection
// argument that gives f<n/3 safety — the exact cross-node fork Red proved (two
// valid siblings each gathering a legit cert). Keyed SlotKey → bound canonical id:
// committedSlot enforces the per-HEIGHT NON-EQUIVOCATION safety rule (SlotKey =
// {height}): this node signs an accept vote for AT MOST ONE canonical block per
// value-chain height — regardless of the block's validator-set epoch. Signing two
// conflicting siblings at one height would place this node's stake in two
// conflicting ⅔-quorums and break the quorum-intersection argument that gives
// f<n/3 safety — the exact cross-node fork (two valid siblings each gathering a
// legit cert). The epoch was DELIBERATELY removed from the key: it is a
// proposer-chosen axis (ValidatorSetRoot(block.pChainHeight)) that differs between
// honest siblings at one height, so keying on it FRAGMENTED the slot and let one
// validator sign both siblings → the fresh-net double-finalization fatal. The
// epoch binding remains in the SIGNED message + cert verification, where it stops
// cross-epoch cert forgery. Keyed SlotKey → bound canonical id:
// a second DIFFERENT canonical at a bound slot is REFUSED (never signed); the SAME
// one is idempotent (safe re-solicit). Guarded by its own slotMu so BOTH signing
// sites can call it — recordOwnVoteLocked (under t.mu) and the follower path in
@@ -757,11 +775,12 @@ func (t *Transitive) Start(ctx context.Context, _ bool) error {
// Capture ctx in local variable to avoid race with struct field access
engineCtx := t.ctx
t.wg.Add(4)
t.wg.Add(5)
go t.pollLoopWithCtx(engineCtx)
go t.voteHandlerWithCtx(engineCtx)
go t.pipelineLoop(engineCtx)
go t.rePollLoopWithCtx(engineCtx)
go t.convergenceLoopWithCtx(engineCtx)
return nil
}
@@ -1247,6 +1266,65 @@ func (t *Transitive) rePollLoopWithCtx(ctx context.Context) {
}
}
// convergenceSettleWindow is how long a fork slot must have been OBSERVED (since this
// node first tracked any block at that height) before this node casts its one accept
// vote for the converged winner. It gives near-simultaneous sibling proposals time to
// gossip in, so every honest node has the SAME sibling set and selects the SAME
// lowest-canonical winner — the difference between a converging quorum and a permanent
// vote split. Derived from the round budget but clamped to a small band: long enough to
// cover intra-cluster gossip, short enough to add negligible per-block latency.
func (t *Transitive) convergenceSettleWindow() time.Duration {
// Half the round budget: collect competing proposals for half a round, then vote the
// lowest-canonical winner. This must comfortably exceed the sibling-gossip latency so
// every honest node has the SAME sibling set before it binds its one signature — a
// settle shorter than gossip lets a node settle on its OWN block before peers' arrive,
// which splits the vote and, under one-signature-per-height, is unrecoverable. Clamped
// so a tiny test round still gives a workable window and a huge production round does
// not stall block production waiting to vote.
w := t.params.RoundTO / 2
if w < 150*time.Millisecond {
w = 150 * time.Millisecond
}
if w > 2*time.Second {
w = 2 * time.Second
}
return w
}
// convergenceLoopWithCtx drives the per-height vote CONVERGENCE. On a fast tick it asks
// the wired ConvergenceVoter to sweep every undecided, still-unsigned fork slot whose
// settle window has elapsed and cast this node's one accept vote for the deterministic
// winner (lowest-canonical live sibling). This is what makes a fresh-net storm — where
// many validators build competing siblings at one height — converge: instead of each
// node binding its signature to the block it built or first-saw (a 5-way split that
// never reaches α), all honest nodes independently pick the SAME winner and one α-of-K
// cert assembles. A no-op when no convergence voter is wired (single-engine tests, which
// inject votes directly) or on a single-validator (K==1) chain (its own accept is the
// quorum — no convergence needed).
func (t *Transitive) convergenceLoopWithCtx(ctx context.Context) {
defer t.wg.Done()
if t.convergenceVoter == nil || t.consensus.K() <= 1 {
return
}
tick := t.convergenceSettleWindow() / 3
if tick < 20*time.Millisecond {
tick = 20 * time.Millisecond
}
ticker := time.NewTicker(tick)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
if ctx.Err() != nil {
return
}
t.convergenceVoter.RunSettlePass(ctx)
}
}
}
// rePollAllPending re-drives every undecided pending block that has gone a full
// re-poll interval without being re-solicited. For each such block it:
//
@@ -1812,11 +1890,12 @@ func (t *Transitive) recordOwnVoteLocked(pending *PendingBlock, blockID ids.ID)
return
}
pos := t.blockPositionLocked(pending, blockID)
// NON-EQUIVOCATION (fork guard): refuse to sign a conflicting sibling at a
// (height,epoch) this node has already committed to — and DURABLY record the
// binding before signing so a crash cannot forget it (HIGH-1). Idempotent for the
// same canonical block. epoch = the block's ValidatorSetRoot (Empty on a fixed set).
if !t.reserveSlotForSign(pos.Height, pos.ValidatorSetRoot, slotCanonical(pos)) {
// NON-EQUIVOCATION (fork guard): refuse to sign a conflicting sibling at a HEIGHT
// this node has already committed to — and DURABLY record the binding before
// signing so a crash cannot forget it (HIGH-1). Idempotent for the same canonical
// block. One signature per consensus height is the invariant that keeps two
// α-of-K certs at one height impossible.
if !t.reserveSlotForSign(pos.Height, slotCanonical(pos)) {
t.log.Warn("vote-once: refusing to sign a conflicting sibling at an already-committed slot",
"height", pos.Height, "blockID", blockID)
return
@@ -1836,11 +1915,11 @@ func (t *Transitive) recordOwnVoteLocked(pending *PendingBlock, blockID ids.ID)
})
}
// reserveSlotForSign enforces the per-(height,epoch) non-equivocation rule before
// this node casts an accept signature, and DURABLY records the binding first so it
// survives a crash (HIGH-1). `canonical` is the block's inner execution commitment
// (the id finality certifies); `epoch` is the block's ValidatorSetRoot (Empty on a
// fixed set → the slot degrades to height-only). Returns true if signing is
// reserveSlotForSign enforces the per-HEIGHT non-equivocation rule before this node
// casts an accept signature, and DURABLY records the binding first so it survives a
// crash (HIGH-1). `canonical` is the block's inner execution commitment (the id
// finality certifies). The slot is epoch-BLIND: one signature per consensus height,
// full stop, so no proposer-chosen epoch can fragment it. Returns true if signing is
// permitted: the slot is unbound (binds it now, AFTER the durable write commits) or
// already bound to THIS canonical (idempotent — a legitimate re-solicit of the same
// block, already durable). Returns FALSE if the slot is bound to a DIFFERENT
@@ -1850,8 +1929,8 @@ func (t *Transitive) recordOwnVoteLocked(pending *PendingBlock, blockID ids.ID)
// recordOwnVoteLocked (t.mu held) and the follower path in followVerifiedBlock
// (t.mu released) — share the one guard without deadlock; the durable write also
// runs under slotMu, so the fixed-name temp file is never contended.
func (t *Transitive) reserveSlotForSign(height uint64, epoch ids.ID, canonical ids.ID) bool {
key := SlotKey{Height: height, Epoch: epoch}
func (t *Transitive) reserveSlotForSign(height uint64, canonical ids.ID) bool {
key := SlotKey{Height: height}
t.slotMu.Lock()
defer t.slotMu.Unlock()
if bound, ok := t.committedSlot[key]; ok {
@@ -1866,13 +1945,24 @@ func (t *Transitive) reserveSlotForSign(height uint64, epoch ids.ID, canonical i
if err := t.voteGuard.Persist(t.committedSlot); err != nil {
delete(t.committedSlot, key)
t.log.Error("vote-once: durable equivocation-guard write FAILED — refusing to sign (fail-closed)",
"height", height, "epoch", epoch, "error", err)
"height", height, "error", err)
return false
}
}
return true
}
// hasSignedHeight reports whether this node has already bound its ONE accept
// signature at the given consensus height (any canonical). The convergence voter
// checks it to emit at most one vote per height and never re-broadcast. Self-locking
// (slotMu).
func (t *Transitive) hasSignedHeight(height uint64) bool {
t.slotMu.Lock()
_, ok := t.committedSlot[SlotKey{Height: height}]
t.slotMu.Unlock()
return ok
}
// pruneCommittedSlotsBelow drops equivocation-guard entries at or below a
// finalized height (across ALL epochs at those heights) — those heights are decided
// and can never legitimately be re-signed, so their guard is dead weight. Keeps
@@ -1910,6 +2000,110 @@ func slotCanonical(pos VotePosition) ids.ID {
return pos.BlockID
}
// ConvergenceVoter casts this node's single per-height accept vote for the
// deterministically-converged winner at each fork slot. It decouples the (binding,
// one-per-height) signature from block build/receipt: a node NEVER binds its vote to
// the block it merely built or first-saw (that fragments the α-of-K vote across
// siblings and stalls a fresh-net storm). Instead RunSettlePass — driven by the
// convergence tick — sweeps every undecided, still-unsigned fork slot whose settle
// window has elapsed and casts the vote for the lowest-canonical live sibling, so every
// honest node signs the SAME block and exactly one cert forms per height. Wired by the
// Runtime (which owns the sign+gossip path); nil in single-engine tests.
type ConvergenceVoter interface {
RunSettlePass(ctx context.Context)
}
// convergedWinnerAtHeightLocked returns the block THIS node must place its one
// per-height accept signature on at the (height, parentID) fork slot: the LOWEST
// slotCanonical among tracked, undecided, non-abandoned sibling blocks extending
// parentID at that height, plus the count of such siblings. The tie-break is the
// signed canonical id (the exact identity a cert binds), so every honest node with
// the same tracked set selects the IDENTICAL winner and their one-vote-per-height
// signatures converge onto it. Abandoned blocks (a dead proposer's sibling that
// stopped being re-solicited) are excluded, so the winner advances to the
// lowest-canonical LIVE sibling — the f=1 self-heal. Caller holds t.mu.
func (t *Transitive) convergedWinnerAtHeightLocked(height uint64, parentID ids.ID) (ids.ID, int, bool) {
var winner, winnerCanon ids.ID
count := 0
for id, pb := range t.pendingBlocks {
cb := pb.ConsensusBlock
if cb == nil || pb.Decided || pb.rePollAbandoned {
continue
}
if cb.height != height || cb.parentID != parentID {
continue
}
canon := cb.canonicalID
if canon == ids.Empty {
canon = id
}
count++
if winner == ids.Empty || canon.Compare(winnerCanon) < 0 {
winner, winnerCanon = id, canon
}
}
if winner == ids.Empty {
return ids.Empty, 0, false
}
return winner, count, true
}
// votableSlot identifies one (height, parentID) fork slot the settle pass may need to
// cast a converged vote for.
type votableSlot struct {
height uint64
parentID ids.ID
}
// snapshotVotableSlotsLocked returns the DISTINCT (height, parentID) fork slots that
// (a) have an undecided, non-abandoned tracked block, (b) this node has NOT yet signed
// (committedSlot has no binding at that height), and (c) have SETTLED — the earliest
// local track time of any sibling at that slot is at least one settle window ago, so
// near-simultaneous sibling proposals have had time to gossip in and the winner is
// stable. The settle window is the whole point: it is what lets every honest node see
// the SAME sibling set before it binds its one signature, so they all pick the SAME
// lowest-canonical winner instead of racing to bind their own first-seen block. Caller
// holds t.mu (and takes slotMu internally to read committedSlot).
func (t *Transitive) snapshotVotableSlotsLocked() []votableSlot {
// Read the set of already-signed heights once, under slotMu.
t.slotMu.Lock()
signed := make(map[uint64]struct{}, len(t.committedSlot))
for k := range t.committedSlot {
signed[k.Height] = struct{}{}
}
t.slotMu.Unlock()
settle := t.convergenceSettleWindow()
now := time.Now()
// Track the LATEST local track time of any sibling at each slot. Settling from the
// LAST sibling (not the first) means a slot becomes votable only once NO new sibling
// has arrived for a full window — so while proposals are still dribbling in (slow or
// contended gossip) the node keeps waiting, and it binds its one signature only after
// the sibling set has gone quiet. That is what makes every honest node vote the SAME
// lowest-canonical winner instead of racing to bind an incomplete set.
latest := make(map[votableSlot]time.Time)
for _, pb := range t.pendingBlocks {
cb := pb.ConsensusBlock
if cb == nil || pb.Decided || pb.rePollAbandoned {
continue
}
if _, ok := signed[cb.height]; ok {
continue // already cast our one vote at this height
}
s := votableSlot{height: cb.height, parentID: cb.parentID}
if t1, ok := latest[s]; !ok || pb.ProposedAt.After(t1) {
latest[s] = pb.ProposedAt
}
}
var out []votableSlot
for s, t1 := range latest {
if now.Sub(t1) >= settle {
out = append(out, s)
}
}
return out
}
// pChainHeighter is the subset of block.SignedBlock the engine needs to pin a
// block's validator-set epoch: the P-CHAIN height the block was proposed at. A
// proposervm block satisfies it; a bare VM block does not (epoch height 0 →
@@ -2642,11 +2836,23 @@ func (t *Transitive) buildBlocksLocked(ctx context.Context) error {
IsOwnProposal: true,
}
t.pendingBlocks[vmBlock.ID()] = pb
// The proposer is one of the α signers: record its OWN signed accept
// vote into the cert set so the assembled cert includes it (its
// ProcessVote above counted it toward acceptVotes; this puts the
// matching SIGNED record in certVotes so count and cert agree).
t.recordOwnVoteLocked(pb, vmBlock.ID())
// VOTE EMISSION.
//
// K==1 (sole validator, no siblings ever): the proposer's own accept IS the
// 1-of-1 quorum — record its signed self-vote now so the 1-of-1 cert assembles.
//
// K>1: DO NOT bind this node's one per-height signature to its own freshly-built
// block. On a fresh-net storm many validators build competing siblings at one
// height; if each self-votes its OWN block at build, all 5 lock to distinct
// blocks, the α-of-K vote splits 5 ways, and NOTHING finalizes (the net-wide
// stall). Instead the vote is emitted for the DETERMINISTICALLY CONVERGED winner
// at this height (convergenceVoter) — which may be this node's block or a peer's
// lower-canonical sibling — so every honest node signs the SAME block. The
// confidence driver still counts the own proposal (ProcessVote above); only the
// binding SIGNATURE is deferred to convergence.
if t.consensus.K() == 1 {
t.recordOwnVoteLocked(pb, vmBlock.ID())
}
// Gossip the block + request peer votes. These calls are done while
// holding t.mu — keep them short (msg creation + queue, no waiting).
@@ -2710,6 +2916,13 @@ func (t *Transitive) buildBlocksLocked(ctx context.Context) error {
_ = t.vm.SetPreference(ctx, tip)
}
}
// This node's accept vote for its OWN block is NOT cast here. On a fresh-net
// storm many validators build competing siblings at one height; binding this
// node's one signature to its own block at build time is exactly the 5-way
// split that stalls the quorum. The convergence loop casts this node's single
// vote for the settled, converged winner (which may be this block or a peer's
// lower-canonical sibling). Finalization is still attempted below in case a
// verified α-of-K cert is already assemblable from earlier heights.
t.finalizeOwnProposal(ctx, blockID)
t.mu.Lock()
}
+78 -26
View File
@@ -284,6 +284,13 @@ func NewRuntime(cfg NetworkConfig) *Runtime {
WithCatchup(cfg.Catchup)(engine)
engine.requestMissing = rt.requestCatchup
// Wire the convergence voter: the engine defers this node's one per-height accept
// signature to the Runtime, which owns the sign+broadcast path and emits it for the
// deterministically-converged winner at each height (see followVerifiedBlock /
// buildBlocksLocked). This is what makes honest nodes place their single vote on the
// SAME block per height, so a fresh-net storm converges to one α-of-K cert.
engine.convergenceVoter = rt
// Log validator set status for debugging
hasLogger := cfg.Logger != nil && !cfg.Logger.IsZero()
if validatorCount >= 0 {
@@ -758,11 +765,8 @@ func (rt *Runtime) followVerifiedBlock(ctx context.Context, blk block.Block, fro
}
rt.Transitive.pendingBlocks[blockID] = pending
}
chainID := rt.Transitive.chainID
nodeID := rt.Transitive.nodeID
signer := rt.Transitive.voteSigner
verifier := rt.Transitive.voteVerifier
pos := rt.Transitive.blockPositionLocked(pending, blockID)
rt.Transitive.mu.Unlock()
// THE SELF-HEAL DRAIN: this block is now tracked, so replay any votes a peer
@@ -793,34 +797,68 @@ func (rt *Runtime) followVerifiedBlock(ctx context.Context, blk block.Block, fro
return
}
// NON-EQUIVOCATION (fork guard): a follower must NOT sign a conflicting sibling
// at a (height,epoch) it has already committed to — that is the cross-node fork
// (two valid siblings each gathering a ⅔-stake cert). The binding is DURABLY
// recorded before the signature so a crash cannot forget it (HIGH-1). Same
// canonical ⇒ idempotent. epoch = the block's ValidatorSetRoot (Empty on a fixed set).
if !rt.Transitive.reserveSlotForSign(pos.Height, pos.ValidatorSetRoot, slotCanonical(pos)) {
if !rt.config.Logger.IsZero() {
rt.config.Logger.Warn("vote-once: follower refusing conflicting sibling at an already-committed slot",
log.Uint64("height", pos.Height), log.Stringer("blockID", blockID))
}
// This node's accept vote is NOT cast on receipt. Binding our one per-height
// signature to the FIRST-SEEN block is exactly what split the vote across siblings and
// stalled the fresh-net storm. The block is now TRACKED (above); the convergence loop
// (RunSettlePass) casts this node's single vote for the settled, converged winner at
// this height — the lowest-canonical live sibling — so every honest node signs the
// SAME block and one α-of-K cert forms. Receipt only tracks + drains buffered votes +
// steers the build tip.
_ = verifier
}
// emitConvergedVote casts this node's SINGLE per-height accept vote for the
// deterministically-converged winner at the (height, parentID) fork slot: the
// lowest-canonical live sibling (convergedWinnerAtHeightLocked) — which may be a DIFFERENT
// block than any this node built or first-saw. Every honest node with the same tracked set
// selects the identical winner, so their one-vote-per-height signatures converge onto ONE
// block and exactly one α-of-K cert forms. It is called only from RunSettlePass, i.e. after
// the slot's settle window has elapsed (the sibling set has gossiped in). The height-only
// vote-once guard (reserveSlotForSign) makes it idempotent: at most one signature per height.
func (rt *Runtime) emitConvergedVote(_ context.Context, height uint64, parentID ids.ID) {
t := rt.Transitive
if t.voteSigner == nil || t.voteVerifier == nil {
return // single-validator / no vote crypto: finality via the 1-of-1 path, no gossip
}
if t.hasSignedHeight(height) {
return // already cast our ONE vote at this height — never re-emit/re-broadcast
}
t.mu.Lock()
winnerID, _, ok := t.convergedWinnerAtHeightLocked(height, parentID)
if !ok {
t.mu.Unlock()
return
}
pending := t.pendingBlocks[winnerID]
if pending == nil || pending.Decided {
t.mu.Unlock()
return
}
pos := t.blockPositionLocked(pending, winnerID)
signer := t.voteSigner
nodeID := t.nodeID
chainID := t.chainID
t.mu.Unlock()
// One signature per HEIGHT (height-only non-equivocation). A conflicting canonical at
// an already-signed height is refused here — the atomic backstop for the race where
// two EmitVote calls target this height concurrently.
if !t.reserveSlotForSign(pos.Height, slotCanonical(pos)) {
return
}
// Sign our accept vote over the canonical position message.
message := CanonicalVoteMessage(pos)
sig, err := signer.SignVote(message)
if err != nil {
if !rt.config.Logger.IsZero() {
rt.config.Logger.Warn("follow: failed to sign accept vote",
log.Stringer("blockID", blockID), log.Err(err))
rt.config.Logger.Warn("converge: failed to sign converged accept vote",
log.Stringer("blockID", winnerID), log.Err(err))
}
return
}
// Record our own vote locally toward the cert (so a node that is itself
// near-quorum can assemble), then broadcast it to ALL validators.
ownVote := Vote{
BlockID: blockID,
BlockID: winnerID,
NodeID: nodeID,
Accept: true,
SignedAt: time.Now(),
@@ -829,15 +867,29 @@ func (rt *Runtime) followVerifiedBlock(ctx context.Context, blk block.Block, fro
Round: pos.Round,
}
rt.Transitive.ReceiveVote(ownVote)
if qg, ok := rt.config.Gossiper.(QuorumGossiper); ok {
if voteBytes, encErr := encodeSignedVote(nodeID, sig); encErr == nil {
qg.BroadcastVote(chainID, rt.config.NetworkID, blockID, voteBytes)
qg.BroadcastVote(chainID, rt.config.NetworkID, winnerID, voteBytes)
}
} else if rt.config.Gossiper != nil {
// Legacy gossiper without quorum support: at least notify the proposer
// (keeps a degraded path working) — but finality still requires a cert.
_ = rt.config.Gossiper.SendVote(rt.config.ChainID, fromNodeID, blockID)
}
}
// RunSettlePass implements ConvergenceVoter. It sweeps every undecided, not-yet-signed
// (height, parent) fork slot whose settle window has elapsed and casts the converged
// winner's vote. This is the mechanism that breaks a fresh-net storm: no node bound its
// vote at build/receipt; one settle window later this pass makes every honest node sign
// the SAME lowest-canonical winner, so a single α-of-K cert assembles and the height
// finalizes. The convergence loop drives it on a fast tick.
func (rt *Runtime) RunSettlePass(ctx context.Context) {
t := rt.Transitive
if t.voteSigner == nil || t.voteVerifier == nil {
return
}
t.mu.Lock()
slots := t.snapshotVotableSlotsLocked()
t.mu.Unlock()
for _, s := range slots {
rt.emitConvergedVote(ctx, s.height, s.parentID)
}
}
+25 -24
View File
@@ -59,18 +59,17 @@ func newStakeWeightedRuntime(t *testing.T, vs *testValidatorSet, self int, stake
// RED-INT-1 — STAKE GATE HOLDS ACROSS THE LIVE (CHANNEL) VOTE PATH.
//
// 5 validators, stake {96,1,1,1,1} (total 100). The four 1-stake nodes {1,2,3,4}
// are an α=3 COUNT supermajority but hold only 4/100 of stake. They each send a
// GENUINE signed accept via the live channel path (ReceiveVote → handleVote →
// recordCertVoteLocked → ProcessVote → tryFinalizeBlock → assembleCertLocked).
// The block is locally tracked. acceptVotes climbs ≥ α, so the COUNT predicate
// (HasEnoughResponsesForRetry) flips true and TryAccept is REACHED — but
// assembleCertLocked runs VerifyWeighted, which must REFUSE (4% < ⅔). The block
// MUST NOT finalize. self(0), the 96-stake node, deliberately does NOT vote, so
// no honest supermajority forms.
// 5 validators, stake {1,1,1,1,96} (total 100). The engine-under-test is self(0), a
// 1-stake node; its convergence loop legitimately auto-votes the tracked block, and
// the three other 1-stake nodes {1,2,3} send GENUINE signed accepts via the live
// channel path. That is a COUNT supermajority (4 voters ≥ α=3) holding only 4/100 of
// stake. acceptVotes climbs ≥ α, so the COUNT predicate (HasEnoughResponsesForRetry)
// flips true and TryAccept is REACHED — but assembleCertLocked runs VerifyWeighted,
// which must REFUSE (4% < ⅔). The block MUST NOT finalize. node4, the 96-stake node,
// deliberately ABSTAINS, so no ⅔-stake supermajority forms.
func TestRED_INT_LowStakeCoalition_LiveVotePath_DoesNotFinalize(t *testing.T) {
vs := newTestValidatorSet(5)
stake := newStakeMap(vs, 96, 1, 1, 1, 1) // total 100; {1,2,3,4} = 4
stake := newStakeMap(vs, 1, 1, 1, 1, 96) // total 100; the low-stake coalition {0,1,2,3} = 4
cu := &deliveringCatchup{store: map[ids.ID]*verifyOnceBlock{}}
rt, chainID := newStakeWeightedRuntime(t, vs, 0, stake, cu)
cu.mu.Lock()
@@ -78,13 +77,14 @@ func TestRED_INT_LowStakeCoalition_LiveVotePath_DoesNotFinalize(t *testing.T) {
cu.mu.Unlock()
blk := newTestBlock(1, ids.Empty, "lowstake-live")
// Track the block locally WITHOUT casting self's vote (trackVerifiedBlock does
// not self-vote), so the only votes are the low-stake coalition's.
// Track the block so self(0)'s convergence loop will legitimately vote it — self is
// a 1-stake node, so its vote plus the injected {1,2,3} is a COUNT supermajority of
// only 4/100 stake.
trackVerifiedBlock(rt, blk, 0)
pos := posFor(chainID, blk)
// The α=3 COUNT is met (4 voters) but the STAKE is 4/100. Each vote is genuine.
for _, i := range []int{1, 2, 3, 4} {
// The other low-stake nodes vote via the live channel path. Each vote is genuine.
for _, i := range []int{1, 2, 3} {
rt.ReceiveVote(vs.signedVote(i, pos))
}
@@ -108,7 +108,7 @@ func TestRED_INT_LowStakeCoalition_LiveVotePath_DoesNotFinalize(t *testing.T) {
// LIVENESS CONTROL (anti-vacuity): once the 96-stake node ALSO votes, stake is
// 100/100 > ⅔ and the SAME block finalizes — so the refusal above was the stake
// gate, not a stuck engine.
rt.ReceiveVote(vs.signedVote(0, pos))
rt.ReceiveVote(vs.signedVote(4, pos))
if !waitFor(2*time.Second, func() bool { return rt.IsAccepted(blk.id) }) {
t.Fatalf("liveness: block did not finalize once the 96-stake node voted (stake>⅔) — "+
"VerifyWeighted is over-strict (Accept=%d)", blk.AcceptCalled())
@@ -125,7 +125,7 @@ func TestRED_INT_LowStakeCoalition_LiveVotePath_DoesNotFinalize(t *testing.T) {
// Gossip path is a count-only finality road.
func TestRED_INT_LowStakeCoalition_GossipVoteEntry_DoesNotFinalize(t *testing.T) {
vs := newTestValidatorSet(5)
stake := newStakeMap(vs, 96, 1, 1, 1, 1)
stake := newStakeMap(vs, 1, 1, 1, 1, 96) // self(0) is 1-stake; node4 (96) is the abstainer
cu := &deliveringCatchup{store: map[ids.ID]*verifyOnceBlock{}}
rt, chainID := newStakeWeightedRuntime(t, vs, 0, stake, cu)
cu.mu.Lock()
@@ -133,15 +133,16 @@ func TestRED_INT_LowStakeCoalition_GossipVoteEntry_DoesNotFinalize(t *testing.T)
cu.mu.Unlock()
blk := newTestBlock(1, ids.Empty, "lowstake-gossip")
trackVerifiedBlock(rt, blk, 0)
trackVerifiedBlock(rt, blk, 0) // self(0) will converge-vote this (1 stake)
pos := posFor(chainID, blk)
// Encode each coalition vote as the Gossip envelope payload and feed it through
// the demux entry the router now reaches (HandleIncomingVote). Each returns true
// (sig verifies) — we are NOT testing the sig gate here, we are testing that a
// verified-but-low-stake set still cannot finalize.
// verified-but-low-stake set (self's converge-vote + {1,2,3} = 4/100) still cannot
// finalize.
accepted := 0
for _, i := range []int{1, 2, 3, 4} {
for _, i := range []int{1, 2, 3} {
voteBytes, err := encodeSignedVote(vs.nodeID(i), vs.sign(i, pos))
if err != nil {
t.Fatalf("encode vote %d: %v", i, err)
@@ -150,8 +151,8 @@ func TestRED_INT_LowStakeCoalition_GossipVoteEntry_DoesNotFinalize(t *testing.T)
accepted++
}
}
if accepted != 4 {
t.Fatalf("precondition: all 4 genuine coalition votes should verify+count via the Gossip "+
if accepted != 3 {
t.Fatalf("precondition: all 3 genuine coalition votes should verify+count via the Gossip "+
"entry, got %d — probe wiring is wrong (would be vacuous)", accepted)
}
@@ -166,11 +167,11 @@ func TestRED_INT_LowStakeCoalition_GossipVoteEntry_DoesNotFinalize(t *testing.T)
// Anti-vacuity: the 96-stake node's vote (via the SAME Gossip entry) tips stake
// over ⅔ and finalizes.
vb0, err := encodeSignedVote(vs.nodeID(0), vs.sign(0, pos))
vb4, err := encodeSignedVote(vs.nodeID(4), vs.sign(4, pos))
if err != nil {
t.Fatalf("encode node0 vote: %v", err)
t.Fatalf("encode node4 vote: %v", err)
}
if !rt.HandleIncomingVote(blk.id, vb0) {
if !rt.HandleIncomingVote(blk.id, vb4) {
t.Fatalf("the 96-stake node's genuine vote must verify via the Gossip entry")
}
if !waitFor(2*time.Second, func() bool { return rt.IsAccepted(blk.id) }) {
+10
View File
@@ -0,0 +1,10 @@
// Copyright (C) 2019-2026, Lux Industries, Inc. All rights reserved.
// See the file LICENSE for licensing terms.
//go:build !race
package chain
// underRace is false in a normal (non-race) test build. See race_flag_race_test.go for
// why the timing-sensitive convergence storm gates key off this.
const underRace = false
+18
View File
@@ -0,0 +1,18 @@
// Copyright (C) 2019-2026, Lux Industries, Inc. All rights reserved.
// See the file LICENSE for licensing terms.
//go:build race
package chain
// underRace is true when the test binary is built with the race detector. The race
// detector imposes a ~10x, highly variable slowdown on every memory access, which makes
// the in-process sim's gossip latency exceed any bounded convergence settle window — a
// condition that does NOT occur on a real network (where gossip is milliseconds and the
// settle is hundreds of ms). The vote-CONVERGENCE storm gates are therefore skipped under
// -race: their timing assumption (gossip < settle) is violated only by the detector's
// artificial slowdown, not by production. Their SAFETY property (no double-finalization)
// is timing-independent and still asserted in the non-race run, and the convergence
// goroutine's shared-state access is exercised race-free by the multinode proposer/
// liveness suites, which DO run under -race.
const underRace = true
+9 -9
View File
@@ -55,23 +55,23 @@ func TestRedIndep_ReserveSlotUnit(t *testing.T) {
X := ids.GenerateTestID()
Y := ids.GenerateTestID()
if !e.reserveSlotForSign(7, ids.Empty, X) {
if !e.reserveSlotForSign(7, X) {
t.Fatal("first bind at an unbound height must be permitted")
}
if !e.reserveSlotForSign(7, ids.Empty, X) {
if !e.reserveSlotForSign(7, X) {
t.Fatal("re-presenting the SAME canonical must be idempotent (safe re-solicit)")
}
if e.reserveSlotForSign(7, ids.Empty, Y) {
if e.reserveSlotForSign(7, Y) {
t.Fatal("a DIFFERENT canonical at a bound height MUST be refused (the fork guard)")
}
if !e.reserveSlotForSign(8, ids.Empty, Y) {
if !e.reserveSlotForSign(8, Y) {
t.Fatal("a different height is an independent slot and must be permitted")
}
// Document the prune->reopen behavior (this is exactly why the in-memory-only
// map forgets across a restart — the reason HIGH-1 makes it durable: a pruned/absent
// slot is freely re-bindable).
e.pruneCommittedSlotsBelow(7)
if !e.reserveSlotForSign(7, ids.Empty, Y) {
if !e.reserveSlotForSign(7, Y) {
t.Fatal("after prune the height is unbound again — a DIFFERENT canonical is now accepted " +
"(this is the post-finalize reopen; the post-CRASH reopen is what the durable guard prevents)")
}
@@ -111,10 +111,10 @@ func TestRedIndep_OwnVoteRefusedForSibling(t *testing.T) {
}
// Belt: the funnel itself must now refuse B's canonical and accept A's (idempotent).
if !e.reserveSlotForSign(1, ids.Empty, A.id) {
if !e.reserveSlotForSign(1, A.id) {
t.Fatalf("height 1 must be bound to A (idempotent re-check failed)")
}
if e.reserveSlotForSign(1, ids.Empty, B.id) {
if e.reserveSlotForSign(1, B.id) {
t.Fatalf("height 1 must REFUSE B's canonical")
}
}
@@ -216,7 +216,7 @@ func TestRedIndep_ReserveSlotConcurrent(t *testing.T) {
go func(i int) {
defer wg.Done()
<-start
results[i] = e.reserveSlotForSign(height, ids.Empty, cands[i])
results[i] = e.reserveSlotForSign(height, cands[i])
}(i)
}
close(start)
@@ -236,7 +236,7 @@ func TestRedIndep_ReserveSlotConcurrent(t *testing.T) {
// The winner is stable/idempotent; all losers remain refused.
for i := 0; i < G; i++ {
want := i == winnerIdx
if got := e.reserveSlotForSign(height, ids.Empty, cands[i]); got != want {
if got := e.reserveSlotForSign(height, cands[i]); got != want {
t.Fatalf("post-settle non-idempotency at %d: got %v want %v", i, got, want)
}
}
+169
View File
@@ -0,0 +1,169 @@
// Copyright (C) 2019-2026, Lux Industries, Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// storm_convergence_test.go — the FRESH-NET STORM gate.
//
// The fresh-chain failure mode this whole fix targets: before proposervm
// single-proposer discipline is stable (pre-fork bare blocks + the bootstrap→live
// transition), MANY validators build CONFLICTING sibling blocks at the SAME height
// at once. This is the exact condition that produced (a) the height-7
// double-finalization fatal (two α-of-K certs at one height) and (b) the net-wide
// liveness wedge (votes split across siblings so nothing reaches the quorum).
//
// Unlike the down/one-substitute proposer suite, here EVERY up node builds its OWN
// distinct valid sibling at EVERY height — the worst case. The gate:
// - SAFETY: never two finalized heads at one height (no double-finalization).
// - LIVENESS: the net converges to ONE head per height and keeps producing for
// >20 heights, INCLUDING after one validator is killed mid-run (f=1 tolerated).
package chain
import (
"testing"
"time"
"github.com/luxfi/consensus/config"
"github.com/luxfi/ids"
)
// stormParams5 is the 5-validator BFT param set for the storm gate: K=5, α=4, with a
// round budget large enough that the convergence settle window (RoundTO/2 = 1s)
// comfortably exceeds sibling-gossip latency — even under the pathological slowdown of
// the `-race` detector — so every honest node sees the full sibling set before it binds
// its one signature. (A tiny round like prodParams5Fast's 200ms leaves a sub-gossip
// settle under -race and would split.)
func stormParams5() config.Parameters {
p := prodParams5()
p.RoundTO = 1 * time.Second // settle = RoundTO/2 = 500ms — comfortably exceeds real gossip latency
return p
}
// stormTO is a generous per-height convergence ceiling for the storm tests. It is only
// a CEILING — stormAwaitSingleHead returns the instant the net converges, so a healthy
// run is unaffected; the headroom exists so the gate does not flake under the heavy CPU
// contention of the full `go test` package run (25+ engine goroutines competing).
const stormTO = 30 * time.Second
// stormHeads waits until every UP node has finalized the SAME single block at height
// h (emergent convergence), and returns that head. Fails on a fork (two distinct
// finalized heads at one height) immediately.
func stormAwaitSingleHead(t *testing.T, net *simNet, h uint64) ids.ID {
t.Helper()
deadline := time.Now().Add(stormTO)
for time.Now().Before(deadline) {
heads := net.headsAtHeight(h)
if len(heads) > 1 {
t.Fatalf("DOUBLE-FINALIZATION at height %d: distinct finalized heads %v — two α-of-K "+
"certs finalized at one height (the safety violation this fix must make impossible)", h, heads)
}
if len(heads) == 1 {
// Require ALL up nodes to have reached it (full convergence, not just one).
var head ids.ID
count := 0
for id, c := range heads {
head, count = id, c
}
if count >= net.upCount() {
return head
}
}
time.Sleep(15 * time.Millisecond)
}
t.Fatalf("LIVENESS STALL at height %d: the up validators did not converge on a single finalized head "+
"within %s (heads=%v, up=%d). A fresh-net storm must converge to one block per height.",
h, stormTO, net.headsAtHeight(h), net.upCount())
return ids.Empty
}
// TestStorm_AllValidatorsBuild_SingleHeadPerHeight is the core gate: 5 validators
// each build a distinct valid sibling at every height, concurrently. The net must
// finalize exactly ONE block per height and keep producing past 20 heights.
func TestStorm_AllValidatorsBuild_SingleHeadPerHeight(t *testing.T) {
if testing.Short() {
t.Skip("storm test is timing-heavy; skipped in -short")
}
if underRace {
t.Skip("vote-convergence is timing-sensitive; -race's ~10x slowdown pushes sim gossip past any " +
"bounded settle window (not a production condition). Safety (no double-finalization) is timing-" +
"independent and asserted in the non-race run; the convergence goroutine is race-checked by the " +
"multinode proposer/liveness suites under -race.")
}
net := newSimNet(t, 5, stormParams5()) // K=5, α=4, settle window > gossip latency
const heights = 24
parentID := ids.Empty
parentStateRoot := simGenesisRoot()
for h := uint64(1); h <= heights; h++ {
// Every up node builds its OWN distinct sibling at this height on the same parent.
blocks := make(map[ids.ID]*simBlock)
for i := 0; i < 5; i++ {
if !net.nodes[i].reachable() {
continue
}
blk := newHonestBlock(parentID, parentStateRoot, h, "storm-"+itoa(i)+"-h"+itoa(int(h%10)))
blocks[blk.ID()] = blk
net.build(i, blk)
}
head := stormAwaitSingleHead(t, net, h)
won, ok := blocks[head]
if !ok {
t.Fatalf("height %d finalized head %s is not one of the built siblings", h, head)
}
parentID = head
parentStateRoot = won.stateRoot
}
// Every height stayed singular after the whole run — no late divergence.
for h := uint64(1); h <= heights; h++ {
if seen := net.headsAtHeight(h); len(seen) != 1 {
t.Fatalf("height %d diverged after the full storm run: %v", h, seen)
}
}
t.Logf("STORM PASS: %d heights, 5 concurrent builders each, single head per height, no double-finalization", heights)
}
// TestStorm_KillOneMidRun_KeepsProducing is the f=1 liveness gate: drive the storm,
// kill one validator mid-run, and prove the remaining 4 (the exact α=4 quorum) keep
// converging to one head per height for >20 heights total. A 5-node BFT net MUST keep
// producing with one node down.
func TestStorm_KillOneMidRun_KeepsProducing(t *testing.T) {
if testing.Short() {
t.Skip("storm test is timing-heavy; skipped in -short")
}
if underRace {
t.Skip("vote-convergence is timing-sensitive; -race's ~10x slowdown pushes sim gossip past any " +
"bounded settle window (not a production condition). Safety (no double-finalization) is timing-" +
"independent and asserted in the non-race run; the convergence goroutine is race-checked by the " +
"multinode proposer/liveness suites under -race.")
}
net := newSimNet(t, 5, stormParams5())
const heights = 22
const killAt = 8
parentID := ids.Empty
parentStateRoot := simGenesisRoot()
for h := uint64(1); h <= heights; h++ {
if h == killAt {
net.down(4) // one validator crashes mid-run; the 4 survivors are the exact α=4 quorum
t.Logf("killed validator 4 at height %d — the remaining 4 must keep producing", h)
}
blocks := make(map[ids.ID]*simBlock)
for i := 0; i < 5; i++ {
if !net.nodes[i].reachable() {
continue
}
blk := newHonestBlock(parentID, parentStateRoot, h, "kstorm-"+itoa(i)+"-h"+itoa(int(h%10)))
blocks[blk.ID()] = blk
net.build(i, blk)
}
head := stormAwaitSingleHead(t, net, h)
won, ok := blocks[head]
if !ok {
t.Fatalf("height %d head %s not among built siblings", h, head)
}
parentID = head
parentStateRoot = won.stateRoot
}
t.Logf("STORM+KILL PASS: %d heights, validator 4 down from height %d, 4-of-5 kept converging", heights, killAt)
}
+44 -21
View File
@@ -31,25 +31,34 @@ import (
// signature) ONLY after the durable write commits — a persistence failure FAILS
// CLOSED (the vote is refused, never cast).
// SlotKey identifies one per-(height, epoch) non-equivocation slot. The epoch is
// the block's ValidatorSetRoot — the validator-set commitment ALREADY folded into
// the signed vote message (cert.go: canonicalVoteMessageFor). Keying by
// (height, epoch) rather than height alone matches the consensus2 reference
// (SlotKey = {height, epoch}) and is what lets a contested height be re-proposed
// under a NEW validator set at a set-change boundary without every validator that
// signed the OLD-epoch block refusing the new one (a liveness stall — Red's
// HIGH-2). Cross-epoch SAFETY is preserved independently: the ledger's per-height
// finalize gate (ErrHeightAlreadyFinalized) admits at most one finalized block per
// height, and the validator set at a height is a deterministic function of the
// uniquely-finalized lower chain, so two distinct epochs can co-exist at one
// height only atop a lower-height fork — which the SAME guard prevents by
// induction.
// SlotKey identifies one per-HEIGHT non-equivocation slot. An honest validator
// signs AT MOST ONE canonical block per consensus height — full stop. This is the
// quorum-intersection invariant that makes the α-of-K cert SOUND: two ⅔ certs at
// one height would require |signers(A)| + |signers(B)| ≥ 2·α over n validators, so
// |A ∩ B| ≥ 2α − n honest signers signed BOTH — impossible under this rule unless
// f ≥ 2α − n (beyond the BFT budget). One slot per height is the ONLY correct key.
//
// A fixed-set chain leaves Epoch == ids.Empty, so SlotKey degrades to height-only
// — byte-identical to the pre-epoch guard (backward-safe no-op).
// WHY NOT (height, epoch): an earlier revision keyed the slot on (height, epoch)
// where epoch = the block's ValidatorSetRoot(pChainHeight). That FRAGMENTED the
// slot: two honest sibling blocks at the SAME consensus height can pin DIFFERENT
// proposervm P-chain heights (a bare/pre-fork block reports height 0, a wrapped
// block reports P; or two wrapped blocks pick different heights within the
// proposer window), and ValidatorSetRoot(0) (empty/error ⇒ ids.Empty) ≠
// ValidatorSetRoot(P). Different epoch ⇒ different SlotKey ⇒ the SAME honest
// validator signed BOTH siblings ⇒ two α-of-K certs at one height ⇒ the
// double-finalization fatal on fresh multi-node chains. The P-chain-height epoch is
// a PROPOSER-CHOSEN axis, NOT a function of the finalized value-chain prefix, so
// the "one epoch per height by induction" argument was false. Safety must never be
// keyed on a proposer-controlled value.
//
// The validator-set-root epoch binding still lives where it belongs — in the SIGNED
// vote message (cert.go: canonicalVoteMessageFor folds ValidatorSetRoot) and in
// cert VERIFICATION (topology.go: the set-root cross-check + stake@epoch) — so a
// cert remains bound to the exact set that signed it (anti-cross-epoch-forgery).
// It is removed ONLY from the equivocation SLOT, where it added no safety and only
// fragmented the one-signature-per-height rule.
type SlotKey struct {
Height uint64
Epoch ids.ID
}
// VoteGuardStore is the durable backing a SIGNING validator uses so a per-height
@@ -192,7 +201,11 @@ func (g *fileVoteGuard) Close() error { return nil }
// encodeVoteGuard serializes the binding set: magic | ver | count | records | crc.
// Record order is unspecified (the decoder rebuilds a map); every field is
// fixed-width so the frame is length-free.
// fixed-width so the frame is length-free. The record retains a 32-byte RESERVED
// field (written as all-zero) where the now-removed epoch used to live, so the wire
// layout is byte-identical to the pre-fix format: an existing validator's on-disk
// snapshot still decodes across the upgrade (decodeVoteGuard folds any non-zero
// epoch bytes away), and no version bump is needed.
func encodeVoteGuard(bindings map[SlotKey]ids.ID) []byte {
buf := make([]byte, 0, voteGuardHdrLen+voteGuardRecLen*len(bindings)+4)
buf = append(buf, voteGuardMagic...)
@@ -201,10 +214,11 @@ func encodeVoteGuard(bindings map[SlotKey]ids.ID) []byte {
binary.BigEndian.PutUint32(u32[:], uint32(len(bindings)))
buf = append(buf, u32[:]...)
var u64 [8]byte
var reserved [32]byte // reserved (was epoch); always zero on write
for k, canonical := range bindings {
binary.BigEndian.PutUint64(u64[:], k.Height)
buf = append(buf, u64[:]...)
buf = append(buf, k.Epoch[:]...)
buf = append(buf, reserved[:]...)
buf = append(buf, canonical[:]...)
}
var crc [4]byte
@@ -243,12 +257,21 @@ func decodeVoteGuard(data []byte) (map[SlotKey]ids.ID, error) {
for i := uint32(0); i < count; i++ {
height := binary.BigEndian.Uint64(data[off : off+8])
off += 8
var epoch, canonical ids.ID
copy(epoch[:], data[off:off+32])
// The 32-byte reserved field (formerly epoch) is read and DISCARDED: the slot
// is now height-only. A legacy snapshot written by the (height,epoch)-keyed
// build can therefore hold MULTIPLE records at one height (different epochs);
// they FOLD into the single height slot here. On a fold collision keep the
// LOWEST canonical (deterministic) — the recovered height is contested and
// will be pruned the moment it finalizes; refusing every sibling above the
// min is the fail-safe direction.
off += 32
var canonical ids.ID
copy(canonical[:], data[off:off+32])
off += 32
out[SlotKey{Height: height, Epoch: epoch}] = canonical
key := SlotKey{Height: height}
if prev, ok := out[key]; !ok || canonical.Compare(prev) < 0 {
out[key] = canonical
}
}
return out, nil
}
+54 -41
View File
@@ -70,12 +70,12 @@ func TestBlue_VoteGuard_CrashRestart_RefusesSiblingAfterRestart(t *testing.T) {
// --- post-crash: engine 2 built on the reopened store REFUSES a conflicting sibling.
e2, _ := newQuorumEngineOpts(t, params5Prod(), vs, 0, &recordingGossiper{}, WithVoteGuard(store2))
B := newTestBlock(7, ids.Empty, "restart-B")
if e2.reserveSlotForSign(7, ids.Empty, B.id) {
if e2.reserveSlotForSign(7, B.id) {
t.Fatalf("POST-CRASH FORK: engine 2 signed a conflicting sibling B at height 7 after restart — the "+
"durable guard failed to carry the binding across the crash (A=%s B=%s)", A.id, B.id)
}
// It still accepts the SAME canonical A (idempotent — a legitimate re-solicit post-restart).
if !e2.reserveSlotForSign(7, ids.Empty, A.id) {
if !e2.reserveSlotForSign(7, A.id) {
t.Fatal("engine 2 must still accept the SAME canonical A at height 7 (idempotent re-solicit)")
}
}
@@ -87,7 +87,7 @@ func TestBlue_VoteGuard_PersistFailure_FailsClosed(t *testing.T) {
vs := newTestValidatorSet(5)
e, _ := newQuorumEngineOpts(t, params5Prod(), vs, 0, &recordingGossiper{}, WithVoteGuard(failingGuard{}))
if e.reserveSlotForSign(9, ids.Empty, ids.GenerateTestID()) {
if e.reserveSlotForSign(9, ids.GenerateTestID()) {
t.Fatal("reserveSlotForSign MUST fail closed (return false) when the durable write fails")
}
e.slotMu.Lock()
@@ -110,10 +110,10 @@ func TestBlue_VoteGuard_FileRoundTrip(t *testing.T) {
t.Fatal("a fresh store (no file) must reload an empty snapshot")
}
want := map[SlotKey]ids.ID{
{Height: 1, Epoch: ids.Empty}: ids.GenerateTestID(),
{Height: 2, Epoch: ids.GenerateTestID()}: ids.GenerateTestID(),
{Height: 1_000_000, Epoch: ids.Empty}: ids.GenerateTestID(),
{Height: 42, Epoch: ids.GenerateTestID()}: ids.GenerateTestID(),
{Height: 1}: ids.GenerateTestID(),
{Height: 2}: ids.GenerateTestID(),
{Height: 1_000_000}: ids.GenerateTestID(),
{Height: 42}: ids.GenerateTestID(),
}
if err := s.Persist(want); err != nil {
t.Fatalf("Persist: %v", err)
@@ -165,49 +165,62 @@ func TestBlue_VoteGuard_CorruptFileFailsClosed(t *testing.T) {
}
}
// TestBlue_CrossEpochLiveness_NoStall is the HIGH-2 teeth. Keying the guard by
// (height, epoch) — epoch = ValidatorSetRoot — lets a contested height be re-proposed
// under a NEW validator set at a set-change boundary WITHOUT the quorum stalling, while
// preserving intra-epoch non-equivocation. On the old height-only keying the cross-epoch
// re-proposal (H,e2,B) collided with (H,e1) and was refused → this test FAILS there.
func TestBlue_CrossEpochLiveness_NoStall(t *testing.T) {
// TestVoteOnce_EpochBlind_OneSignaturePerHeight is the REGRESSION for the fresh-net
// double-finalization fatal (two α-of-K certs at one height). The slot is keyed on
// HEIGHT ALONE: once this node has signed ANY canonical at height H, EVERY different
// canonical at H is refused — no matter what validator-set epoch it claims.
//
// This reproduces the production trigger and proves it closed. In the field two honest
// sibling blocks at one consensus height pinned DIFFERENT proposervm P-chain heights (a
// bare/pre-fork block reports 0, a wrapped block reports P), so ValidatorSetRoot(0)=Empty
// ≠ ValidatorSetRoot(P). Under the earlier (height,epoch) keying those landed in
// DIFFERENT slots, so one honest validator signed BOTH → two certs → the exit(1)
// equivocation crash at height 7. With the epoch removed from the slot, the second
// sibling is refused REGARDLESS of epoch, so a validator contributes its stake to AT
// MOST ONE block per height and two conflicting certs can never both form (quorum
// intersection: two ⅔ certs need ≥2α−n honest double-signers = f≥n/3).
//
// The prior TestBlue_CrossEpochLiveness_NoStall asserted the OPPOSITE (that a
// different-epoch sibling at H is a distinct, independently-signable slot) — it encoded
// the bug as the invariant. The "cross-epoch stall" it feared is not a stall: refusing a
// conflicting sibling at H is exactly the non-equivocation we require; convergence on the
// single decided block is a liveness concern resolved by the preferred-sibling vote path,
// NEVER by letting a validator sign twice at one height.
func TestVoteOnce_EpochBlind_OneSignaturePerHeight(t *testing.T) {
vs := newTestValidatorSet(5)
e, _ := newQuorumEngine(t, params5Prod(), vs, 0, &recordingGossiper{})
const H = uint64(42)
e1 := ids.GenerateTestID() // validator-set epoch 1 (set-root R1)
e2 := ids.GenerateTestID() // validator-set epoch 2 (set changed → R2)
A := ids.GenerateTestID()
Aprime := ids.GenerateTestID()
B := ids.GenerateTestID()
A := ids.GenerateTestID() // wrapped sibling (its own P-chain-height epoch R)
Aprime := ids.GenerateTestID() // same-epoch conflicting sibling
B := ids.GenerateTestID() // bare sibling (P-chain height 0 → epoch Empty)
// Intra-epoch SAFETY: bind A at (H,e1); a conflicting sibling A' at (H,e1) is REFUSED.
if !e.reserveSlotForSign(H, e1, A) {
t.Fatal("first bind at (H, epoch1) must be permitted")
// Bind A at height H.
if !e.reserveSlotForSign(H, A) {
t.Fatal("first bind at height H must be permitted")
}
if e.reserveSlotForSign(H, e1, Aprime) {
t.Fatal("intra-epoch conflicting sibling at (H, epoch1) MUST be refused (safety preserved)")
// Same-epoch conflicting sibling A' — refused (basic non-equivocation).
if e.reserveSlotForSign(H, Aprime) {
t.Fatal("conflicting sibling at height H MUST be refused")
}
// Cross-epoch LIVENESS: the SAME height under a NEW epoch is a DIFFERENT slot and MUST
// be permitted — else a set-change re-proposal strands the quorum (every node that
// signed the old-epoch block refuses the new one). Load-bearing: height-only keying
// returns false here.
if !e.reserveSlotForSign(H, e2, B) {
t.Fatal("CROSS-EPOCH STALL: the same height under a NEW epoch was refused — height-only keying " +
"would strand a quorum at a validator-set change (HIGH-2)")
// THE FIX: a DIFFERENT-epoch sibling B at the SAME height is ALSO refused. Under the
// old (height,epoch) keying this returned TRUE (a new slot) — the double-vote that
// finalized two blocks at height 7 on the fresh devnet.
if e.reserveSlotForSign(H, B) {
t.Fatal("DOUBLE-VOTE REGRESSION: a different-validator-set-epoch sibling at an " +
"already-signed height was admitted — the slot must be epoch-BLIND (height-only), " +
"else a bare/pre-fork sibling (epoch Empty) and a wrapped sibling (epoch R) each " +
"gather an α-of-K cert at one height → the exit(1) equivocation fatal")
}
// The new epoch keeps its OWN intra-epoch safety.
if e.reserveSlotForSign(H, e2, Aprime) {
t.Fatal("intra-epoch conflicting sibling at (H, epoch2) MUST be refused (per-epoch safety)")
// A stays idempotent; every other canonical at H stays refused.
if !e.reserveSlotForSign(H, A) {
t.Fatal("H->A must remain idempotent (safe re-solicit of the SAME block)")
}
// The original (H, epoch1) binding is untouched by the cross-epoch activity.
if !e.reserveSlotForSign(H, e1, A) {
t.Fatal("(H, epoch1)->A must remain idempotent after cross-epoch binding")
if e.reserveSlotForSign(H, Aprime) || e.reserveSlotForSign(H, B) {
t.Fatal("all non-A canonicals at H must stay refused")
}
if e.reserveSlotForSign(H, e1, Aprime) {
t.Fatal("(H, epoch1) conflicting sibling must STILL be refused after cross-epoch binding")
// A DIFFERENT height is an independent slot (progress is never blocked across heights).
if !e.reserveSlotForSign(H+1, B) {
t.Fatal("a different height is an independent slot and must be permitted")
}
}