mirror of
https://github.com/luxfi/consensus.git
synced 2026-09-24 00:26:58 +00:00
287d5a5fa63c0063d8ba767a5e9e36f541e0ee92
262
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  zeekayandHanzo Dev
|
287d5a5fa6 |
test(engine/chain): async-gossip (WAN jitter/reorder) idle-storm variant
Adds an opt-in per-message delivery latency to the sim bus (nil default = unchanged instant delivery) so a test can model async WAN gossip: heavy-tail jitter where ~10% of messages arrive after the RoundTO/2 settle window, so prevotes reach different nodes in different orders. Plus the async+zero-margin+ idle storm scenario. FINDING: PASSES at HEAD (v1.35.16), like the naive and zero-margin idle storms. THREE in-process variants (sync idle, zero-margin idle, async-gossip idle) all finalize cleanly — the in-process consensus harness robustly cannot reproduce the live VC stall. The trigger is at the node-integration/live layer (real proposervm + real transport), not the consensus engine in isolation. The epoch-churn hypothesis is separately refuted by static analysis: epochHeightLocked returns pChainHeight (0 on these nets, stable), not the churning epoch number, so the moving epoch does not affect vote pubkey/stake resolution. Existing storm tests remain green (nil-default opt-in). The bisect + forward-fix must run on a live net. Co-authored-by: Hanzo Dev <[email protected]> |
||
|
zeekayandHanzo Dev
|
54f84838c8 |
test(engine/chain): idle-inclusive re-storm gate (naive + zero-margin)
The existing storm drives blocks back-to-back and never idles — the exact coverage gap that let the VC fleet-stall ship (live testnet v1.35.15 finalized a few blocks then all 5 nodes went silent). This adds idle windows between build bursts across multiple cycles, requiring sustained finalization. FINDING: both scenarios PASS at HEAD (v1.35.16) — naive idle and zero-margin (exact alpha=4) idle. HEAD already carries the round-skip fix that handles the in-process async case, and this harness models neither the proposervm (stale-parent blocks / epochs) nor the real async ZAP transport. So the live VC stall is NOT reproducible in-process; the version bisect + forward-fix must run on a live net. This gate remains as an in-process liveness regression guard. Co-authored-by: Hanzo Dev <[email protected]> |
||
|
zeekayandHanzo Dev
|
f290db7e18 |
consensus: housekeeping — one ⅔ formula, leaf-lock doctrine (v1.35.16)
RED decomplect-review follow-ups (non-blocking), on top of v1.35.15: - ONE ⅔ supermajority formula: bftAlpha(count) = TwoThirdsStakeFloor(count)+1 is the single definition of the integer accept quorum; bftCommittee and the 1→N reclamp both call it. The count quorum and the stake quorum are now literally the same rational threshold, never two hand-inlined 2*x/3+1 copies that could drift. - Lock doctrine: the unlock-before-call-out comment now draws the exact CALL-OUT vs LEAF-LOCK boundary — external surfaces (VM, gossiper, ReceiveVote, poll delivery, callbacks) are banned under t.mu; acquiring an internal leaf lock (c.mu via t.consensus.*, the vote-guard store) is SAFE because the leaf never re-enters the engine and the order t.mu→c.mu is fixed and acyclic. - Removed the untracked round_view_red2_test.go scratch file. Formula consolidation is behavior-identical (2*x/3+1 == floor(2x/3)+1); full engine/chain suite green. Co-authored-by: Hanzo Dev <[email protected]> |
||
|
zeekayandHanzo Dev
|
17fedb9cbb |
consensus: global unlock-before-call-out invariant + n=1..10 quorum matrix
Harden the single-node decomplect into a GLOBAL invariant and prove the finality quorum scales at every denominator. LOCK DOCTRINE (the deadlock class, not one instance): - Transitive.mu documents the exact rule: never call VM.Accept, ReceiveVote, a network send, poll delivery, certificate callbacks, or any external callback while holding it — queue or release first. - Moved EVERY remaining under-lock call-out out of t.mu: Propose/RequestVotes (network sends) in buildBlocksLocked (main + rebuild paths), VM.BuildBlock (loop top), and the boot VM.LastAccepted/GetBlock seed. All call-outs now run in one unlocked region after capturing what they need under the lock. - engineMutex: a sync.RWMutex that records the write-holder goroutine when a test enables it (one atomic load per Lock/Unlock in production; no goID cost). - TestBlue_NoCallOutUnderEngineLock drives the real NewRuntime build→propose→ finalize flow (K==1 and K>1) with every VM/gossiper/block call-out instrumented and asserts NONE runs under the engine write lock. QUORUM ACCEPTANCE MATRIX (runtime level, real gossiperProposer): - n=1..10 unweighted: exactly q=floor(2n/3)+1 online DECIDES; q-1 does NOT finalize and does NOT false-accept. - Weighted stake: same-voter-count no/yes pairs where >2/3-by-STAKE is the sole discriminator ([40,30,30] 60/70; [34,33,33] 66/67; [50,20,20,10] pivotal-50). - alpha-of-K and stake are SEPARATE composed gates: a 2-of-4 cert is rejected at the threshold floor even at 70% stake (disjoint-quorum fork defence). - duplicate-counted-once, equivocation-rejected, all-online/one-down per quorum, late-vote buffered-then-drained, VM.Accept fail-closed (retained, no half-apply; recovery via the phantom-floor boot reconcile). Live finality path (VerifyWeighted, bftCommittee) UNCHANGED — frozen as verified. Full engine/chain suite green; race-clean on the concurrency-sensitive tests. Co-authored-by: Hanzo Dev <[email protected]> |
||
|
zeekayandHanzo Dev
|
0e115304bb |
consensus: DECOMPLECT single-validator finalize — remove selfVoter shortcut (one path, ava-aligned)
Owner mandate: decomplect the n=1 finalize, do it right, one path. Supersedes the async band-aid (v1.35.13) with the structural fix. The single-node selfVoter shortcut (integration.go) was the COMPLECTING element: for K==1 it called engine.ReceiveVote from inside RequestVotes while buildBlocksLocked held t.mu, self-deadlocking on the non-reentrant t.mu.RLock (the live n=1 freeze — node hangs at 'single-node mode: self-voting'), and delivered a redundant vote that could sit uncounted in handleVote's not-yet-tracked buffer. REMOVED entirely (field, wiring, RequestVotes branch). A K==1 chain now finalizes SOLELY through the inline build-path finalizer (buildBlocksLocked → buildSingleValidatorCertLocked → acceptWithCertCore) — one finalize path, matching avalanchego's 'a lone validator's own decision finalizes locally.' buildBlocksLocked skips RequestVotes for K==1 (no peers); RequestVotes returns early when the sample resolves to no peers (true no-op, not a query to nobody). Also (RED decentralization follow-up): gate reclampCommitteeLocked AND the buildSingleValidatorCertLocked count>1 synthesize-refusal on a RESOLVED set (GetFinalizedHeight set = ≥1 finalized block), so a fresh single-validator chain that transiently over-reports its validator count cannot wedge its own first block. PROOF through the REAL runtime path: TestBlue_SingleValidator_DecidesThroughFullRuntimePath now drives NewRuntime (real gossiperProposer) 0→1→2 and asserts VM.Accept + height advances — HANGS pre-decomplect (deadlock, 25s timeout), PASSES post. params_test / single_node_test converted to the decomplected invariant (K==1 polls nobody; K>1 polls network). Full engine/chain suite green incl -race. v1.35.14. Co-authored-by: Hanzo Dev <[email protected]> |
||
|
zeekayandHanzo Dev
|
24b2a32ba9 |
consensus: FIX single-validator DECIDE deadlock — selfVoter must dispatch ReceiveVote async
THE LIVE n=1 FREEZE (neo's repro; my v1.35.11 unit fix passed but the live chain still froze because the earlier tests used raw New with NO proposer, so they never exercised the selfVoter path). ROOT CAUSE — a reentrant-RWMutex self-deadlock, NOT a cert issue. NewRuntime wires the single-node selfVoter (integration.go) to call engine.ReceiveVote. RequestVotes invokes that selfVoter while buildBlocksLocked HOLDS t.mu (write); ReceiveVote then does t.mu.RLock() on the SAME goroutine — sync.RWMutex is not reentrant, so it blocks forever on RLock while holding the write lock. The node logs 'single-node mode: self-voting for proposed block' and hangs there; the inline K==1 finalize (which runs AFTER RequestVotes in buildBlocksLocked) never executes → no block ever decides → EVM head frozen at 0. Stack: buildBlocksLocked → RequestVotes → selfVoter → ReceiveVote → t.mu.RLock (deadlock). FIX: the selfVoter dispatches ReceiveVote on a fresh goroutine (go engine.ReceiveVote) so it never reenters t.mu on the lock-holding goroutine. buildBlocksLocked finishes its inline finalize and releases t.mu; the async vote then delivers (redundant for the own-proposal path, and the finality trigger for the rebuild / re-poll paths that have no inline finalize). ReceiveVote is non-blocking, so the goroutine returns immediately. PROOF through the REAL runtime path (not the cert function in isolation): TestBlue_SingleValidator_DecidesThroughFullRuntimePath builds a NewRuntime with the gossiperProposer + selfVoter wired (a single-validator sampler + recordingGossiper), drives Notify, and asserts the block DECIDES + finalized height advances to 1. It HANGS (25s timeout, deadlock stack) pre-fix and PASSES post-fix. Full engine/chain suite green incl -race. v1.35.13. Co-authored-by: Hanzo Dev <[email protected]> |
||
|
zeekayandHanzo Dev
|
69f88b10df |
consensus: re-clamp committee UP to the live validator set — 1→N decentralization fork guard
RED HIGH (the decentralization path). K is clamped to the live validator count at CONSTRUCTION (bftCommittee) and ChainConsensus.k is written ONLY at NewChainConsensus — never re-clamped. A chain that LAUNCHES single-validator (count=1 → K=1) and later ADDS validators via its PoS staking contract kept validator-1's K stuck at 1, so it would keep taking the K==1 path and finalize UNILATERALLY (a synthesized 1-of-1 cert, or a single-signer stake cert at alpha=1) while validators 2..N require a real k=N quorum and reject it → validator-1 serves a DIVERGENT finalized chain = cross-node double-cert fork. FIX (RED's structural (a) + the re-clamp): - ChainConsensus.Reclamp(k, alpha): setter to move the committee to a changed live set (beta unchanged; degenerate k/alpha ignored). - Transitive.reclampCommitteeLocked(): re-reads the LIVE validator count (wired from the sampler) and re-clamps k/alpha UP toward presetK (= the pre-construction-clamp target). UP-ONLY — never auto-shrinks, so a transient low read (syncing node / unresolved set) can never drop the quorum and enable a unilateral finalize. Called before the single-validator vs multi-validator decision in buildBlocksLocked + TryAccept. - buildSingleValidatorCertLocked: belt-and-suspenders guard — REFUSE to synthesize a 1-of-1 cert when the live count > 1 (returns ZERO → acceptWithCertCore requires a real quorum). count<1 (unwired / --dev / unresolved set) preserves the genuine single- validator + n=1 path. - Runtime wires presetK (pre-clamp K) + liveValidatorCount (sampler.Count) — entirely in the consensus layer; no node change. Unwired for tests/--dev (committee fixed). Regression tests: single→multi transition — after validator-2 joins, K re-clamps to 2 and block 2 does NOT finalize unilaterally (fail-secure stall, not fork); unwired sampler still synthesizes (n=1 fix preserved). Full engine/chain suite green incl -race. v1.35.12. Co-authored-by: Hanzo Dev <[email protected]> |
||
|
zeekayandHanzo Dev
|
b848054201 |
consensus: single-validator (K=1) block DECIDES even when its self-vote is unverifiable — n=1 stall fix
ROOT CAUSE (neo's live zood/Zoo-200200 repro): buildSingleValidatorCertLocked returned a ZERO cert when a vote verifier is wired but the sole validator's signed self-vote did not assemble into a verified cert. On a fresh single-validator sovereign L1 the preset K>1 wires a verifier, then the dynamic committee clamp drops K to the live count (1); the validator set is not yet resolvable at the block's P-chain epoch, so the self-vote's signature cannot be verified against it, assembleVerifiedCertLocked returns false, and the ZERO cert made acceptWithCertCore refuse (ErrNoVerifiedQC, discarded by the inline build-loop finalize) — so the block re-built every poll and NEVER decided (EVM head frozen at N while it rebuilds N+1). Shared single-node path → identical stall on the view-change (v1.32.x) and finality-admission (v1.34.x) lines. n>1 chains never reach the K()==1 branch. FIX: for K()==1, ALWAYS produce a valid 1-of-1 cert — prefer the verified cert when it assembles, else SYNTHESIZE it. Both callers gate on K()==1, so this is a genuinely single-validator engine (clamp sets K to the live count); the sole validator's accept IS the 1-of-1 quorum (not a fabricated multi-party agreement), and FinalizeBranch's per-height gate stays the real single-node safety. A single validator finalizing its OWN block on its OWN chain needs no external signature witness. Regression tests (fail pre-fix, pass post-fix): unverifiable self-vote DECIDES; pure no-crypto --dev still DECIDES. Full engine/chain suite green incl -race; K>1 safety tests (no-force/quorum/fork/double-finalization) unaffected. v1.35.11. Co-authored-by: Hanzo Dev <[email protected]> |
||
|
zeekayandHanzo Dev
|
5277183b7b |
consensus: reconcile fails CLOSED when the VM head is present-but-unreadable (RED fail-open fix)
RED found a fail-OPEN in ReconcilePhantomFloor: it derived the intrinsic-guard
localVM via localLastAccepted, which returns (id,0,nil) when VM.GetBlock fails —
so a transiently-unreadable VM (exactly the recovery regime: a corrupt/half-
written store) was treated as height 0, degrading the guard to max(target,0)=
target. A node whose VM is actually AHEAD would then abandon its own committed
range (a self-inflicted fork).
Fix: read the VM directly and distinguish three cases —
- VM.LastAccepted errors -> refuse (fail-closed, floor unchanged)
- LastAccepted is a real id, unreadable -> refuse (present-but-corrupt; never 0)
- LastAccepted is ids.Empty -> genuinely empty (fresh/wiped): 0 is
correct, reconcile proceeds/no-ops
Tests: unreadable head -> refuse + floor unchanged; empty VM -> proceeds (target
governs). v1.35.10.
Co-authored-by: Hanzo Dev <[email protected]>
|
||
|
zeekayandHanzo Dev
|
d1842910e2 |
consensus: bounded phantom-floor reconcile — un-freeze a decided-floor ahead of every VM's applied head
The pre-fix swallowed-VM.Accept bug let a node's durable decided-floor (vote-guard finalizedThrough) advance past every VM's last-accepted head. On reboot WithVoteGuard re-seeds that phantom floor, the sign gate treats the never-applied heights as decided, and the round-scoped view-change can never re-finalize them — the chain is wedged even after the liveness fix. Add a DELIBERATE, bounded recovery: - VoteGuardStore.Reconcile: the ONE writer permitted to LOWER finalizedThrough (Persist stays monotonic). DRY atomic write shared via writeSnapshot. - Transitive.ReconcilePhantomFloor(target): lowers decidedFloor to max(target, localVM) — an intrinsic guard that can never abandon a block THIS node applied — drops committedSlot/lockRounds/recoveredLocks above it, and durably rewrites the lowered floor. Fail-closed (nothing changes on a store error); idempotent; a zero target is a no-op. Safety (no observable double-cert): reconciling only heights strictly above the fleet-max VM-applied height abandons ONLY internal-only finalizations — no VM applied them (never externalized over RPC/bridge/light-client) and their cert material was in-memory and is gone post-restart, so no party can present a cert; re-finalizing fresh yields exactly one presentable cert. Committed heights at/below the floor are recovered by frontier-trust bootstrap, never re-finalized. Tests: floor lowered + phantom dropped + committed retained; durable monotonic bypass; intrinsic guard vs wrong-low target; behind-node target-governs; phantom-becomes-signable/committed-stays-decided; no-ops; idempotent; fail-closed. Full engine/chain suite green under -race. Co-authored-by: Hanzo Dev <[email protected]> |
||
|
zeekayandHanzo Dev
|
83fad2dd6d |
consensus: fail-closed finalize→VM-accept (fix 2b) — decided-floor never runs ahead of the EVM
Root cause of the testnet VM-layer lag: applyBranchFinalization swallowed VM.Accept's error (_ = vmb.Accept(ctx)) and marked the block finalized + advanced decidedFloor/finalizedThrough regardless. When the EVM refused a consensus-finalized block (empty-block rejection), consensus finalized it anyway and the durable floor ran AHEAD of the EVM accepted head (floor 427 while EVM stuck at 415) — a permanent divergence. FIX (fail-closed): VM.Accept is now called BEFORE committing a block's finality and its error is CHECKED. applyBranchFinalization accepts ascending, commits each block's finality only after the VM applied it, STOPS at the first VM.Accept error, and returns highestAccepted = the highest height the VM actually applied. acceptWithCertCore advances the durable floor ONLY to highestAccepted (never past it) and propagates the error — the chain HALTS at an un-appliable block rather than diverging. Losers are pruned + the serving cert stored only on a full clean accept. This kills the divergence at its source (defense-in-depth even after the VM-side fix that makes finalized-empty blocks appliable). Existing finalize/cert/accept/view-change/RED suites green. Co-authored-by: Hanzo Dev <[email protected]> |
||
|
zeekayandHanzo Dev
|
f74f073bff |
consensus: remove dead safeConfig() dup — alpha-bound enforced in ONE place (RED follow-up b)
The 2a-n>f fork-safety bound is enforced solely in runViewChangePass (fail-secure halt). safeConfig() was an unused second predicate; removed to keep one source of truth. roundView retains n/f only for noteSender's f+1 round-skip. Co-authored-by: Hanzo Dev <[email protected]> |
||
|
zeekayandHanzo Dev
|
906cc2b860 |
consensus: v3 vote-guard lock-round persistence (RED deployment item 2)
Extends the durable vote-guard to v3: persists the view-change LOCK ROUND per height (round+1 sentinel in the record's reserved field, 0=none, byte-length identical to v1/v2). A restarted view-change node seeds its roundView lock (round+value) from disk and re-converges LOCALLY at a recovered height under the unlock rule, instead of the conservative hard-lock a round-less recovery forces. Legacy v1/v2 bindings (no lock round) still hard-lock (fail-safe). decode accepts v1/v2/v3 (upgrade window; fail-closed on corrupt). Persist/Snapshot interface gains lockRounds + LockRoundsSnapshot. Tests: FileRoundTrip (lock rounds round-trip), V3LockRound_SeedsRoundViewLock (v3 seed vs legacy hard-lock), RecoveredLock (direct binding hard-locks), LegacyMultiEpochFold (v2->v3 window). Full suite green. Co-authored-by: Hanzo Dev <[email protected]> |
||
|
zeekayandHanzo Dev
|
f68978c97c |
consensus: rolling-restart lock safety for view-change (RED item 2)
A binding RECOVERED from the durable vote-guard has no persisted lock ROUND, so the unlock rule cannot be evaluated safely across a restart. Fail-safe: recoveredLocks marks recovered heights; reserveSlotForSign keeps the STRICT conflicting-canonical refusal there even under ViewChange, so a rolling upgrade / correlated crash cannot make a node precommit a second, conflicting value at a pre-crash height (double-precommit). Liveness is unharmed — the node finalizes the network's value via the alpha-of-K cert gossip, not a local re-precommit. In-session bindings still re-precommit under the round+lock rule. Released on finalize. Test: TestViewChange_RecoveredLock_NoCrossRestartDoublePrecommit (recovered height refuses a conflicting value, stays idempotent for the recovered value; in-session height re-precommits). Full lock-round persistence (v3 guard format, restoring local re-convergence at recovered heights) is the tracked enhancement. Co-authored-by: Hanzo Dev <[email protected]> |
||
|
zeekayandHanzo Dev
|
52ef297812 |
consensus: storm tests use view-change, -race skip REMOVED (converges under -race)
The -race skip in storm_convergence_test.go was hiding the freeze: the legacy single-phase convergence deadlocks when -race's slowdown pushes gossip past the settle window. Switch the storm tests to stormParams5VC() (round-scoped view-change) and DELETE both -race skips. Now: all-5-build-competing-siblings converges to one head per height over 24 heights AND kill-one-mid-run keeps producing, under -race, no double-finalization. Co-authored-by: Hanzo Dev <[email protected]> |
||
|
zeekayandHanzo Dev
|
bdbbc0185b |
consensus: integrate round-scoped view-change into engine + round-skip + alpha-bound (RED fixes)
Engine integration (params.ViewChange, default off — opt-in/owner-gated per net): - Prevote wire (domain LUX/chain/prevote/v1) + QuorumGossiper.BroadcastPrevote + HandleIncomingPrevote. - runViewChangePass/stepViewChange driver: fluid prevote -> POL -> precommit+lock; precommit reuses the existing per-block cert machinery (all round logic in the prevote/POL/lock layer). - reserveSlotForSign RELAXED under ViewChange (round+lock governs) but the DURABLE decided-floor (PART-A/HIGH-1 finalized gate) is untouched; proposer PREVOTES its own block (recordOwnVoteLocked skipped under VC); pruneViewsBelow on finalize. RED review fixes (c7b2f0081): 1. ROUND-SKIP (noteSender): jump to a round with >=f+1 senders — eliminates the residual ~3% asymmetric-gossip freeze. Non-uniform-visibility honest liveness now 0/200 stalls. 2. ALPHA-BOUND: roundView carries n/f; safeConfig()=2a-n>f; runViewChangePass REFUSES (fail-secure halt) if the committee is unsafe. RED's alpha=3@n=5 fork proof retained as the tightness gate. 3. PRECOMMIT bound to the POL round (validRound), not the drifted current round. 4. POL-cert gossip (polCert + prevoteSigs relay) + locked-precommit re-broadcast so a laggard that missed prevotes/precommits during a split still converges. Proven (clean testcache, -race): RED equivocator suite (no fork, always converges), 0/200 non-uniform stalls, alpha=3 fork tightness, split->heal converges through the FULL engine (TestViewChange_CompetingFork_AsymmetricSplit_Converges); legacy path (VC off) unchanged. Remaining: storm -race skip removal, durable lock persist, reconciliation, node v1.32.13, testnet. Co-authored-by: Hanzo Dev <[email protected]> |
||
|
zeekayandHanzo Dev
|
c6e986bfcc |
consensus: round-scoped view-change core (prevote/POL/lock) + competing-fork deadlock repro
Diagnoses + reproduces the competing-fork finalization deadlock (415->416 testnet freeze): the irrevocable one-signature-per-height guard turns any settle<gossip split into a PERMANENT vote-split (competing_fork_deadlock_test.go, gated LUX_REPRO_DEADLOCK=1; harness gains a per-link partition). Adds the safety-critical view-change CORE (round_view.go): fluid prevotes + POL + precommit-on-POL + lock/unlock + round advance — proven in isolation (round_view_test.go): split->heal CONVERGES (liveness), late lower sibling CANNOT double-finalize (safety), 200-seed randomized partition/reorder at zero margin never double-finalizes. Refines the guard one-sig-per-height -> one-precommit-per-(height,round)+lock (owner sign-off flagged). Engine integration + durable lock + reconciliation + node wiring + tags REMAIN. Co-authored-by: Hanzo Dev <[email protected]> |
||
|
zeekayandHanzo Dev
|
8958589895 |
fix(engine/chain): seed decidedFloor from vm.LastAccepted at boot — clean in-place v1->v2 upgrade (v1.35.5)
v1.35.4's decided-height floor was durable via the vote-guard file, but a node upgrading IN PLACE from the old consensus carries a LEGACY v1 guard file (no finalizedThrough -> floor 0) whose below-tip decided-height slots were already pruned, and its certified ledger is a (0,false) hint until the first post-upgrade finalize (PART-A). In that window the sign gate's floor was 0, so a re-gossiped sibling at a decided-below-tip height could be signed — the fork, now reachable on a mainnet-first in-place roll (RED fix-priority #1). Fix: seed engine.decidedFloor DIRECTLY from vm.LastAccepted at Start, BEFORE the signing goroutines launch (seedDecidedFloorFromVMLocked), and again in SyncState from the reconciled height. vm.LastAccepted is durable and a sound lower bound on the decided height (every accepted block was finalized), so the floor is real from the first instant of boot — no reliance on the transient SyncState hint, the vote-guard file, or a post-upgrade finalize. SIGN-GATE ONLY: advances only t.decidedFloor (under slotMu); never touches byHeight / ledger.Height() (PART-A intact); can only ever REFUSE MORE signing. Test: TestRed_V1ToV2Upgrade_BootSeedFromVM — legacy v1 guard (floor 0) + vm.LastAccepted at the tip + NO SyncState -> reserveSlotForSign refuses a decided-below-tip sibling immediately on boot. Mutation-verified: disabling the boot seed reproduces the fork (RED). Full engine/chain suite green non-race + -race; build clean. Declined the optional fsync-skip for a floor-only prune: it would let the authoritative never-lags vote-guard floor go stale and fall back to a possibly-lagging vm.LastAccepted — a durability regression for negligible I/O savings (a finalize normally prunes the prior tip, so it fsyncs anyway). Keeps height-only slot, strictly-below prune, durable floor, Crit->exit fatal. Co-authored-by: Hanzo Dev <[email protected]> |
||
|
zeekayandHanzo Dev
|
a14d50d4cf |
fix(engine/chain): durable decided-height floor — close CROSS-RESTART prune-then-resign (v1.35.4)
v1.35.3's decided-height gate read GetFinalizedHeight() = the certified ledger height,
which on boot is a NON-authoritative (0,false) hint until the first post-restart cert
(incident-1082814 PART-A). Meanwhile the strictly-below prune PERSISTED the removal of
below-tip slots. So after a correlated rolling restart of a storming fresh net, a
finalized-BELOW-tip height H had: no slot{H} (pruned from the durable file) AND a dead
gate (fh=(0,false)) -> a re-gossiped, differently-enveloped sibling B at decided height H
collected this node's SECOND signature -> 2nd alpha-of-K cert -> os.Exit(1) fleet-wide.
(RED PoC: confirmed failing on v1.35.3; the exact rolling-deploy-on-fresh-net trigger.)
Fix — make the decided-through floor DURABLE and reconstructed on boot, WITHOUT
regressing PART-A:
- Vote-guard file v2: fsync the decided-through floor (finalizedThrough) in the SAME
atomic write as the pruned binding set, so the floor and the slot removal are atomic
on disk. decodeVoteGuard reads legacy v1 files (floor=0) too — no brick on upgrade.
- engine.decidedFloor: seeded on boot from the vote-guard file's floor (WithVoteGuard),
advanced at each finalize (pruneCommittedSlotsBelow). This is authoritative: the
vote-guard is written by the consensus engine at finalize time, so it NEVER lags the
decision (unlike vm.LastAccepted, which a fire-and-forget Accept can leave frozen).
- consensus.GetDecidedFloor() = max(certified height, vm.LastAccepted hint height): a
complementary lower bound, and the sole floor for a signer with a fresh vote-guard.
HARD CONSTRAINT (PART-A): SIGN-GATE-ONLY — reads the hint HEIGHT only, never its
identity; never writes to / enters byHeight / the equivocation index / the
authoritative ledger.Height() (which stays (0,false)-until-cert). It can only ever
REFUSE MORE signing (fail-safe direction).
- reserveSlotForSign AND snapshotVotableSlots now refuse/skip height <=
max(decidedFloor, GetDecidedFloor()).
Test: TestRed_CrossRestart_DecidedHeightUnsignable reproduces the PoC — finalize H then
H+1 (prune deletes slot{H} from memory AND the fsync'd file), reopen the store into a
fresh engine, and prove a sibling at decided height H is refused via the durable floor
alone (no SyncState hint), and via the vm.LastAccepted hint alone (fresh vote-guard).
Mutation-verified: disabling the floor seeding reproduces the fork (RED). Full
engine/chain suite green non-race + -race; vote-guard round-trip asserts the floor.
Corrected the false 'refusal spans a crash' comment (v1.35.3) to describe the durable
sources. Keeps height-only slot, strictly-below prune, atomic reserve, Crit->exit fatal.
Co-authored-by: Hanzo Dev <[email protected]>
|
||
|
zeekayandHanzo Dev
|
edb076bc1e |
fix(engine/chain): decided-height gate closes prune-then-resign double-finalization
The fresh-net double-finalization fatal that survived the epoch-blind height-only
slot fix was a guard-LIFECYCLE defect, not a keying one:
pruneCommittedSlotsBelow used k.Height <= height (INCLUSIVE), deleting the
just-finalized height's equivocation slot. A losing sibling at that height with
a DIFFERENT outer parent than the winner (bare vs proposervm-wrapped envelope) is
NOT in losingSubtrees(winner) — losingSubtrees only rejects the other children of
the winner's parent — so it stays tracked and undecided after the winner finalizes.
With the slot deleted, the convergence pass re-offered the height and
reserveSlotForSign found an EMPTY slot, admitting this node's SECOND signature at
the height. Enough nodes doing this assembled a second alpha-of-K cert at one
height -> fatal EQUIVOCATION -> os.Exit(1) fleet-wide (live devnet artifact: the
conflicting cert's canonical == its envelope, i.e. exactly the bare sibling).
Fix (avalanchego-grounded, two layers):
1. DECIDED-HEIGHT GATE (load-bearing, durable): reserveSlotForSign refuses any
height <= GetFinalizedHeight(). A decided height's blocks are permanently
unsignable, exactly as topological.go's acceptPreferredChild+rejectTransitively
makes a decided height's siblings unreachable and drops their votes. The
certified frontier is monotonic, regression-guarded, and reconstructed from the
VM's last-accepted head on restart, so the refusal spans a crash.
2. PRUNE STRICTLY BELOW (k.Height < height): retain the finalized tip's slot as an
in-memory belt, mirroring ava keeping the last accepted block in its tree. The
tip's slot is reclaimed only when the next height finalizes, by when the gate
covers it durably.
3. Convergence pass skips slots at/below the finalized frontier (never busy-replay
a decided, permanently-refused height).
Tests: new TestFinalizeThenResign_DecidedHeightIsUnsignable drives the real finalizer
functions (FinalizeBranch + pruneCommittedSlotsBelow) and proves a sibling at a
decided height is refused both while the tip slot is retained AND after it is pruned
by a higher finalize (isolating the durable gate). Fails on pre-fix code, passes on
fixed. Two RED probe tests updated from the old inclusive-prune 'reopen' invariant to
the corrected strictly-below + decided-height-gate invariant. Full engine/chain suite
green, incl. storm (24 heights, 5 concurrent builders, no double-finalization) and
f=1 kill-one liveness; -race clean.
Co-authored-by: Hanzo Dev <[email protected]>
|
||
|
zeekayandHanzo Dev
|
57e3e43a4f |
fix(engine/chain): RED-round convergence hardening — tunable settle (N4), K-recheck (N2), proven-loser parent filter (N1)
RED review of v1.35.1 returned 0 critical / 0 high (SAFETY SOUND — height-only slot = 2α>n impossibility; cross-epoch forgery refuted; wire byte-identical). This closes the actionable liveness/robustness findings ahead of testnet/mainnet promotion. The core slot/cert path is UNCHANGED — no re-review needed there. N4 (M-liveness gate): the vote-convergence settle window is now an INDEPENDENTLY tunable Params.ConvergenceSettleWindow (0 = auto RoundTO/2). Prod RoundTO 250-400ms yields a 150ms auto window — too tight for a WAN validator set whose p99 sibling gossip is 100-300ms, where a node could mis-settle and bind its one height-only vote to a non-global-min block (an unrecoverable split under vote-once). Operators can now lengthen the settle for WAN latency WITHOUT slowing the round cadence. A 150ms floor still guards a misconfigured near-zero value. N2: convergenceLoopWithCtx re-checks K() every tick instead of capturing it once at goroutine start — a net that grows K==1→K>1 without an engine restart now begins converging the moment the set expands. N1: snapshotVotableSlotsLocked skips a fork slot whose PARENT is a proven loser (a strictly-lower-canonical H-1 sibling is tracked), so a node never wastes its one height-H signature on a child of an eventually-pruned H-1 parent (the transient H-1-fork stall). Conservative: an UNTRACKED parent (finalized tip / behind) is never filtered, so the normal H=finalizedHeight+1 path cannot be starved. M-grind (DOCUMENTED, testnet/mainnet gate — not safety/halt): the winner tie-break is the proposer-controlled content hash (CanonicalID), grindable under multi-proposer contention to censor/extract; bounded to the fresh-net/down-proposer transient (no effect at a single-proposer height); grind-resistant VRF tie-break (proposervm eligibility VRF) is the tracked follow-up before adversarial-validator mainnet. Tests: TestConvergence_SettleWindow_OperatorOverride, ..._ParentProvenLoser_ ExcludesDeadBranch, TestVoteGuard_LegacyMultiEpochFold (N3). Storm gates still converge (5-way, 24 heights, single head; f=1); full engine/chain suite green; vet clean. Co-authored-by: Hanzo Dev <[email protected]> |
||
|
zeekayandHanzo Dev
|
75eb08d5b1 |
fix(engine/chain): fresh-net single-block finality — epoch-blind vote-once + per-height vote convergence
Fixes the fresh multi-node chain double-finalization fatal (two α-of-K certs at
one height → exit(1) equivocation crash) AND the accompanying liveness stall,
without weakening any safety gate.
ROOT CAUSE (safety): the per-height non-equivocation slot was keyed on
(height, epoch) where epoch = ValidatorSetRoot(block.pChainHeight). Two honest
sibling blocks at ONE consensus height can pin DIFFERENT proposervm P-chain
heights (a bare/pre-fork block reports 0, a wrapped block reports P), so
ValidatorSetRoot(0)=Empty != ValidatorSetRoot(P). Different epoch => different
SlotKey => one honest validator signed BOTH siblings => two α-of-K certs at
height 7 => the fatal. The P-chain-height epoch is a PROPOSER-chosen axis, not a
function of the finalized value-chain prefix, so "one epoch per height" was false.
SAFETY FIX: key the slot on HEIGHT ALONE (SlotKey{Height}). One signature per
consensus height, epoch-blind. Two ⅔ certs at one height now require ≥2α−n honest
double-signers (f≥n/3) — impossible within the BFT budget. The validator-set-root
epoch binding stays in the SIGNED message + cert verification (anti cross-epoch
forgery); it is removed ONLY from the equivocation slot, where it added no safety
and only fragmented the one-signature-per-height rule. Durable guard stays
byte-compatible (reserved-zero epoch field, folds legacy snapshots on read).
ROOT CAUSE (liveness): the α-of-K cert finalized on the FIRST-seen signed votes,
independent of any converged decision, and each node bound its one signature to the
block it BUILT or FIRST-SAW. On a fresh-net storm (many validators build competing
siblings at one height) that splits the vote N ways: nothing reaches α, the net
wedges.
LIVENESS FIX: subordinate the cert to a single converged decision per height. A node
no longer votes at build/receipt; a convergence loop casts its ONE signature for the
deterministic winner (lowest signed-canonical among live siblings) once a settle
window has elapsed (proposals have gossiped in). Every honest node selects the
IDENTICAL winner, so exactly one α-of-K cert forms per height — the cert now
CERTIFIES the converged decision instead of being a parallel finality path that could
certify a conflicting sibling.
Proven on the in-process 5-node quorum engine (real gossip/vote/cert):
- TestVoteOnce_EpochBlind_OneSignaturePerHeight — the double-vote regression, closed.
- TestStorm_AllValidatorsBuild_SingleHeadPerHeight — 5 concurrent builders/height,
24 heights, single head per height, NO double-finalization.
- TestStorm_KillOneMidRun_KeepsProducing — validator killed mid-run, 4-of-5 keep
converging for 22 heights (f=1 tolerated).
Full engine/chain suite green (non-race incl storm; -race race-clean with the
timing-sensitive storm gates skipped — convergence goroutine race-checked by the
multinode suites). go vet clean.
Co-authored-by: Hanzo Dev <[email protected]>
|
||
|
zeekayandHanzo Dev
|
d029323697 |
fix(engine/chain): durable + epoch-keyed + funnel-pruned vote-once guard (Red HIGH-1/HIGH-2/MEDIUM-1)
Fast-follow hardening of the per-height vote-once fork fix (v1.33.2). Red's
re-review found three residuals that leave post-crash equivocation reachable;
this closes all three and adopts Red's harness-free suite as the standing gate.
HIGH-1 — PERSIST committedSlot. The guard was an in-memory map: a crash/restart
between casting a vote and finalizing the height forgot every unfinalized binding,
so the restarted node could sign a conflicting sibling → a cross-node fork with
ZERO Byzantine intent under a rolling upgrade / k8s eviction / OOM. New
vote_guard.go adds a durable VoteGuardStore (Tendermint priv_validator_state.json
pattern): the (height,epoch)→canonical binding is fsync'd BEFORE the signature is
cast (reserveSlotForSign returns true only after the durable write commits — a
write failure FAILS CLOSED) and reloaded on startup (WithVoteGuard seeds
committedSlot). The one concrete store is an atomically-replaced, crc32c-checked
snapshot file (write-temp → fsync → rename → fsync-dir); a corrupt file is a hard
error at open (a signer must not start with equivocation memory it cannot trust).
HIGH-2 — key (height,epoch), matching the consensus2 reference (SlotKey={height,
epoch}). epoch = the block's ValidatorSetRoot, already folded into the signed vote
message. Height-only keying is SAFE but can STALL a quorum at a validator-set
change (a contested height re-proposed under a new epoch is refused by everyone who
signed the old-epoch block). Cross-epoch safety is preserved independently by the
per-height finalize gate (ErrHeightAlreadyFinalized) plus the induction that the
set at a height is a function of the uniquely-finalized lower chain. A fixed-set
chain leaves epoch=Empty, so the slot degrades to height-only (backward-safe).
MEDIUM-1 — fix the prune leak. pruneCommittedSlotsBelow ran ONLY from
HandleIncomingCert, which the dominant LOCAL finalize path never reaches (it
short-circuits at pending.Decided). Moved the prune into the sole finalizer
acceptWithCertCore (beside pruneSlashingBelowWindow), the one funnel EVERY finality
path shares, so committedSlot stays bounded on every validator; removed the now-
redundant HandleIncomingCert prune.
Tests (all -race): Red's red_independent_fork_test.go adopted verbatim as the gate
(mechanical API adaptations only — epoch arg, SlotKey; leak test promoted from Skip
to a hard MEDIUM-1 assert); TestBlue_VoteGuard_CrashRestart proves a reopened store
refuses a sibling after a simulated crash; TestBlue_CrossEpochLiveness_NoStall
proves no stall at a set-change boundary while intra-epoch safety holds;
fail-closed / corrupt-file / round-trip store gates; a multi-round down-proposer
liveness test with an ACTIVE short RoundTO (fork-fix × liveness-fix interaction).
Verified: neutering the guard flips OwnVoteRefusedForSibling + equivocators=2 RED
(teeth); full engine/chain suite green; 50× emergent fork probe = 0 forks.
Co-authored-by: Hanzo Dev <[email protected]>
|
||
|
zeekay
|
5be093c149 |
fix(engine/chain): per-height vote-once discipline — closes the cross-node fork (Red CRITICAL)
Red proved a deterministic cross-node fork in v1.33.1: the finality layer had NO signing-side per-height equivocation lock, so an honest node would sign accept for two conflicting valid siblings at one height; each gathered a ≥⅔-stake cert → different honest nodes finalized different blocks → fork (reachable f=0 via the down-proposer transient the liveness fix opens; ~1-2/8 emergent). Port of consensus2 committed_slot_ to Go: a per-height committedSlot map (height → bound canonical id), reserved through ONE self-locking funnel reserveSlotForSign that BOTH signing sites route through — recordOwnVoteLocked (own proposal, t.mu held) and followVerifiedBlock (follower, t.mu released). A second DIFFERENT canonical at a bound height is REFUSED (never signed); the same one is idempotent (safe re-solicit). Pruned below the finalized height (bounded). This makes honest double-vote IMPOSSIBLE BY CONSTRUCTION — the quorum-intersection overlap that a fork needs can no longer be honest, so a real fork requires f≥n/3 (beyond the guarantee). Tests: all 3 Red repros (red_sibling_fork_test.go) flip RED→GREEN; the shared test validator set now models honest per-validator vote-once (equivocation needs actual Byzantine validators); emergent probe 15× no fork; full engine/chain suite green under -race. BFT ≥ avalanche, f<n/3 safety restored; extends, does not break. |
||
|
zeekayandHanzo Dev
|
3c4a202bca |
fix(engine/chain): re-solicit undecided own proposal on rebuild + real multi-node down/wedged/forked proposer suite
buildBlocksLocked previously dropped a rebuild of an already-tracked block with a bare continue, so a peer that missed the first PushQuery was only re-asked on the slower rePollAllPending backoff. avalanchego never silently drops a rebuild — its repoll re-queries the still-processing preferred block until it decides (quiesce at NumProcessing==0). Mirror that on the build path: re-solicit an UNDECIDED OWN proposal's votes (idempotent; own-only; never a gossiped block, which keeps the rePoll cap). Pure liveness retry — finality is still the a-of-K cert, so it can never change WHICH block finalizes. Adds a REAL multi-node consensus simulation: N independent *Runtime engines wired through an in-process gossip bus carrying the production topology (block gossip -> HandleIncomingBlock, signed vote broadcast -> HandleIncomingVote, a-of-K cert -> HandleIncomingCert). Nodes run concurrently; finalization is EMERGENT (no synthetic quorum). Distinct down / wedged-present / forked scenarios + safety (sub-quorum never finalizes, forged/divergent rejected, no double-finalize, single head, sustained two-height liveness). BFT >= avalanche: K=5 alpha=4 f=1, the 4 healthy validators the exact zero-margin quorum. fails-before/passes-after proven: build-path re-solicit (RED: RequestVotes flat), execution-Verify binding (RED: forked block finalizes on all 5), threshold boundary (DownProposer 4->final vs SubQuorum 3->never via the 2/3-by-stake gate). Full engine/chain suite green under -race, 5x no flake. Co-authored-by: Hanzo Dev <[email protected]> |
||
|
zeekayandHanzo Dev
|
0392abe0d7 |
fix(engine/chain): bound the build-rebuild storm + restore re-poll on the recovery test
Builds on the already-landed liveness fix (rePollAllPending no longer abandons an
undecided OWN proposal — the down/wedged/forked-proposer halt) with the second
half of avalanchego's deliver() contract and the test reconciliation it requires.
STORM BOUND (FIX 2): after building its own proposal the engine now advances the
VM's build preference to the just-built tip (t.vm.SetPreference(PreferredBuildTip)),
exactly as avalanchego deliver() does after adding to consensus, and as the receive
side already does in followVerifiedBlock. Without it the proposervm WaitForEvent
kept signalling "build THIS height" while the block awaited votes, so the node
rebuilt one height hundreds of times (the mainnet 511-rebuild-in-4-min spin). It is
a build hint only — it never changes WHICH block finalizes or WHEN; finality is
still the verified alpha-of-K cert.
TEST RECONCILIATION: TestRecovery_StuckBlockAbandonedAfterCap asserted the OLD
(buggy) "own proposal abandoned after the cap" behavior, so it failed once the
abandon was gated to non-own blocks. Repurposed to
TestRecovery_OwnProposalNeverAbandoned_GossipedBlockAbandoned: an own proposal is
NEVER abandoned (re-solicited past the cap) while a gossiped block STILL abandons
(the spam guard for a block whose voters are behind its parent). The exponential-
backoff test's bound is reframed from the attempt cap to the geometric backoff
(the true storm-safety invariant, independent of the abandon).
NEW TESTS (fails-before/passes-after, real cert path, no mocks of the unit under
test):
- TestProposerStorm_AdvancesBuildPreferenceAfterOwnBuild — the storm bound.
- TestProposerSafety_PersistentRePoll_NoDoubleFinalizeAtHeight — proves the
liveness retry opens NO safety hole: a conflicting sibling with its own full
alpha-of-K quorum is REFUSED at an already-finalized height (one block per
height; persistent re-solicitation cannot fork).
Full engine/chain suite green under -race -count.
Co-authored-by: Hanzo Dev <[email protected]>
|
||
|
zeekayandHanzo Dev
|
f09d8a1b3e |
deps: pulsar v1.9.0 pkg/ layout + threshold v1.12.0 dealerless daemon
Co-authored-by: Hanzo Dev <[email protected]> |
||
|
zeekayandHanzo Dev
|
6fd4f3d858 |
fix(engine/chain): RED round — verify-before-slash (H1), fetch deferred valid cert (M1), roll-discipline doc (M2)
H1 (HIGH, false-slash): HandleIncomingCert recorded slashing DoubleVote evidence on the height-gate branch BEFORE any signature verification. A forged cert (junk signatures naming honest validators, a random different canonical, height <= finalized) jailed honest validators below quorum -> re-halt. FIX: a conflicting cert is fork EVIDENCE only after it passes the FULL predicate. Extracted Transitive.verifyCert (the single signature+active-set+quorum gate, stake-weighted when wired) and run it BEFORE reportCertEquivocation; the finalize path reuses the same helper. A forged cert now slashes NOBODY and finalizes nothing. Regression: TestE2E_ForgedConflictingCertSlashesNobody (zero records, nobody jailed); the genuine canonical fork still records evidence (TestE2E_GenuineForkCertStillSlashes). M1 (liveness/eclipse): a VERIFIED finality proof for a block we do not track was silently dropped, so a targeted-eclipsed follower (adopted a losing sibling envelope) stalled on finalization forever. FIX: on a verified-but-untracked cert, trigger a throttled best-effort catch-up fetch for the certified block (requestCatchup, claimCatchupLocked rate-limited). The fetch is gated on verification, so a forged cert cannot amplify fetches. Regression: TestE2E_DeferredValidCertTriggersFetch (valid cert fetches the winning block; forged cert triggers no fetch). M2 (roll discipline, doc): documented on QuorumCertVersion that the version is fail-SAFE (cross-version certs fail closed, no fork) but NOT fail-live across the skew — cut >= alpha validators to v3 atomically per net, and the v2 baseline must already carry the converge-on-A recovery so the mixed window is not halt-prone. Full proof suite re-run green: 6-row matrix + invariant + inversions + H1/M1 regressions (337 tests), go vet clean, staticcheck adds zero findings, -race clean, both fuzzers clean (codec 461k execs, finalize-decision 81k execs, 0 crashes). No deploy/push/merge/tag — back to RED for re-review. Co-authored-by: Hanzo Dev <[email protected]> |
||
|
zeekayandHanzo Dev
|
8793eb2d71 |
fix(engine/chain): QCv2 canonical-commitment finality + seed-hint semantics (incident 1082814)
Durable fix for the finality-equivocation bug class. Root cause: finality was
defined over the OUTER proposervm envelope id, and the in-memory finality ledger
was SEEDED from vm.LastAccepted on boot. Two outer envelopes wrapping the SAME
inner execution block looked like a fork -> fatal EQUIVOCATION (reportCertEquivocation
-> Logger.Crit -> Fatal -> os.Exit), halting mainnet 43h. They were duplicates.
Two layers, shipped together (atomic roll, all validators upgrade at once):
PART A - seed semantics. ChainConsensus.SyncState (the vm.LastAccepted reconcile)
now writes a NON-AUTHORITATIVE recovery HINT, never certified finality.
FinalityLedger separates a certified frontier (advanced ONLY by a verified QC or
bootstrap frontier-trust) from a hint anchor. GetFinalizedHeight is certified-only;
GetFinalizedTip/BuildAnchor return the higher of {certified tip, hint} for the VM
to build on. A locally-seeded sibling can never manufacture a finalized entry, so
it can never equivocate. The equal-height "different block => equivocation" check
in SyncState was the bug and is deleted.
PART B - QCv2 canonical cert. Finality is keyed on the inner EXECUTION commitment,
not the envelope. VotePosition carries both outer transport ids {BlockID, ParentID}
and canonical {CanonicalID, ParentCanonicalID, ExecutionStateRoot, PayloadRoot}.
byHeight indexes the CANONICAL id. Two certs at one height conflict IFF canonical
differs; same canonical / different envelope = duplicate alias (idempotent no-op).
CanonicalVoteMessage (tag LUX/chain/vote/v2, QuorumCertVersion=3) binds the
canonical identity and EXCLUDES the envelope, so validators that executed the same
inner block sign byte-identical messages even under different envelopes -> their
votes interoperate. The non-wrapped degrade (Empty canonical => bind BlockID) is
resolved in ONE place (canonicalVoteMessageFor). Canonical id is plumbed from
proposervm via the optional canonicalCommitter block iface + setCanonicalFromVM /
canonicalIDOf (falls back to b.ID for a bare block).
HARD GATE: the only writers of certified finalizedTip/byHeight are the cert fold
(Finalize, via ApplyCert/FinalizeBranch) and bootstrap frontier-trust
(AcceptBootstrapBlock). SyncState writes only the hint; preferredTip/build anchor
are build-only, never finality.
Proof suite (all green): 6-row regression matrix + invariant (incident_1082814_test.go),
canonical inversions (cert_canonical_inversion_test.go), two fuzz targets
(cert_fuzz_test.go: codec round-trip/malformed + finalize-decision invariant),
-race clean, go vet clean, staticcheck adds zero findings. 335 engine/chain tests
pass. Pre-fix repro captured (seed A + cert B at one height returned
ErrHeightAlreadyFinalized; now finalizes the canonical block, no equivocation).
No deploy/push/merge/tag — review-pending.
Co-authored-by: Hanzo Dev <[email protected]>
|
||
|
zeekay
|
8067e0da05 |
fix(engine/chain): converge competing siblings — no down-leader halt
When a slot-leader proposer is down, each healthy validator builds its OWN block at the next height (own clock -> own blkID). The alpha-of-K finality cert needs alpha votes on ONE block, but the votes split across the competing siblings, no cert assembles, and the chain HALTS the moment one proposer is down (observed: devnet stuck at height 36 with 4/5 validators healthy). Root cause: the VM kept building on the last FINALIZED block, so every node re-proposed the same height instead of building on a verified-but-unfinalized sibling. There was no convergence onto one build target. Fix (matches avalanchego snowman, which steers the VM to its preferred non-finalized tip via SetPreference after every poll): - Add ChainConsensus.PreferredBuildTip / Transitive.PreferredBuildTip — the DETERMINISTIC build target: descend from the finalized tip through verified, non-rejected children, lowest-ID per level, to the deepest verified tip. Every node with the same block tree picks the SAME chain. - followVerifiedBlock steers the inner VM to PreferredBuildTip on adopt, so validators build H+1 on top of one verified block at H and CONVERGE; the alpha-of-K votes then land on a single block and finalize. Decomplected from the finality Preference (UNCHANGED — still the cert tip; the conformance suite pins that contract). Build hint only: finality stays governed by the alpha-of-K cert on c.ledger, so this cannot affect safety, only liveness. Tests: TestPreference_ConvergesSiblings_NoDownLeaderHalt (deterministic lowest-ID build tip across competing siblings, deepest-verified follow, rejected-sibling skip), TestPreference_NoChildren_ReturnsFinalizedTip (no regression). Full engine/... suite green; nil-VM path guarded. |
||
|
zeekay
|
3381f74b26 |
fix(chain): close RED LOW-1 (cert-trio consistency) + LOW-2 (empty-reset desync)
LOW-1: ApplyCert's finalize Cert took Block from the function arg but Parent/Height
from the verified position — a future caller pairing a mismatched blockID would
finalize the wrong block and prune the real siblings. Now built ENTIRELY from the
α-attested position {BlockID,ParentID,Height}, with a fail-closed guard if a caller
ever passes blockID != pos.BlockID.
LOW-2: SyncState's empty reset (Empty,0) cleared ONLY the tip, leaving height/byHeight/
set stale — a tip-vs-height desync that wedged every future cert in pathFromTip (it
seeks the now-Empty tip forever). Now a whole-value reset to a clean genesis ledger:
the next cert seeds afresh, no wedge. Preserves the teardown semantic the existing
tests assert (tip cleared) while removing the desync — better than refusing the reset.
Tests: TestSyncState_EmptyResetYieldsCleanGenesis proves clean-genesis + a post-reset
finalize succeeds (no wedge). Full suite green CGO_ENABLED=0 -count=1.
|
||
|
zeekay
|
dd417e6471 |
fix(chain): bound byHeight to an equivocation window — clone() O(window) not O(height)
RED HIGH-1: the pure fold's clone() deep-copied the never-pruned byHeight map on every finalize. byHeight is append-only, so this was O(chain height) per finalize under c.mu, and O(N²) on bootstrap replay — at ~10^6 heights, 86.7ms + 100MB per block (measured), which re-introduces the finality stall this branch exists to fix. Fix: pruneBelowWindow drops index entries older than equivocationWindow (1024, matching slashingRetentionHeights) below the tip. Equivocation is only actionable near the tip; an older height is refused by the monotonic guard (cert.Height <= led.height) without the index, so dropping ancient entries is safety-neutral. Proof: BenchmarkFinalize_FlatCost — FLAT 38µs/98KB/9-allocs at tip heights 4096 / 262144 / 1048576 (was 86.7ms/100MB at 10^6). TestLedger_WindowPrune asserts ancient heights pruned, in-window retained, byHeight bounded, AND a stale cert at a pruned height still refused (ErrNonMonotonicFinalizedHeight — no double-finalize). Decomplect intact: still a pure value-fold, input ledger untouched. Green: CGO_ENABLED=0 go test -count=1 ./engine/chain/ ok. |
||
|
zeekay
|
4af8ba432b |
chore(chain): rip AI-slop filenames — avalanche-clean naming
Renamed 28 files to short, single-concept names matching avalanchego's
convention; dropped the noise tokens quorum_/red_/zz_/_gate_/_guard_/
_regression_/_harness_/_probe_/_addition_/_propagation_/_tracking_/
_convergence_/_edge_cases_/configurations. Source: cert.go cert_codec.go
quasar.go topology.go guard.go (was quorum_*). Tests: accept/block/catchup/
params/errors/fetch/poll/proposer/dynamic/finality/height/stake/status/sync/
reorg/weighted/probe + epoch_{binding,receive,finality}_test.go + helpers_test.go.
Stale file-header references updated to the new names. Pure cosmetic — same
package, no symbol changes.
Green: CGO_ENABLED=0 go test -count=1 ./engine/chain/ ok (27.4s).
|
||
|
zeekay
|
b6ea2eee85 |
feat(chain): topological.go — preference layer mirrors avalanchego Topological
Complete the Stage-2 cut-over (finish the half-applied move): the preference layer — Block tree node, AddBlock/ProcessVote/Poll, IsAccepted/IsRejected, Preference/ForcePreference, GetBlock/EpochHeightOf, and the read-only Ancestry view the pure fold reads — now lives in topological.go, each method mapped to its avalanchego snow/consensus/snowman/topological.go equivalent. consensus.go keeps only the engine shell (ApplyCert/applyCertLocked/applyPlan, SyncState, Stats, getters); the duplicated declarations + dead context/engine imports are removed. Decomplect made honest: finality is a single immutable value (c.ledger), replaced WHOLE, never field-poked — no markFinalized. Exactly two whole-value replacements: the cert fold (applyCertLocked, production) and SyncState (the out-of-band import reconcile). Comments corrected from the overclaimed 'exactly one line' to the true two-writer invariant. Green: CGO_ENABLED=0 go test ./engine/chain/... ok (27.5s); build + vet clean. |
||
|
zeekay
|
2cec2ad486 |
feat(chain): decomplect finality into a pure (Ledger,Cert,DAG)→(Ledger',Plan) fold; markFinalized unwriteable
Split the Snowman Preference state (DAG/votes/β, sibling-tolerant) from the
FinalityLedger (committed append-only prefix, cert-driven, immutable). They now
share NO mutable surface, so `accept = markFinalized(...)` is structurally
unwriteable: the ONLY thing that advances finality is replacing the whole ledger
VALUE after a pure fold.
ledger.go (functional core, zero engine/VM):
- FinalityLedger: immutable VALUE (tip/height/set + per-height equivocation
index). Projections Tip/Height/At; never mutated in place.
- Ancestry: read-only DAG view (Parent/Children) the fold reads; never mutates.
- Cert: minimal finality subject, decoupled from the wire VerifiedQuorumCert.
- Plan{Accept,Reject}: avalanchego acceptPreferredChild + rejectTransitively.
- Finalize(led,cert,dag)->(led',plan,err): THE finality function — a pure fold.
Ported verbatim from finalizeBranchLocked (same error semantics): idempotent/
equivocation, seed, monotonic, contiguous-ancestry walk, losing-subtree prune.
Builds the next ledger by COPY — never mutates the input map.
consensus.go (engine shell):
- 4 braided fields (finalizedTip/Height/HeightSet/ByHeight) -> one `ledger
FinalityLedger` value + a decomplected `preference` (pre-finalize build tip,
avalanchego keeps preference separate from lastAcceptedID).
- ApplyCert is the canonical finalize; applyCertLocked is the SOLE finality
writer: `led,plan,err := Finalize(...); c.ledger = led; applyPlan(plan)`.
`grep "c.ledger ="` => applyCertLocked + SyncState's two reconcile assigns;
NO field poke anywhere.
- DELETED recordFinalizedLocked / finalizeBranchLocked / pathFromTipLocked /
collectLosingSubtreesLocked / finalizeStep — no method mutates finality.
- FinalizeBranch kept as a thin (target,height,parent)->Cert wrapper for the
bootstrap/catch-up and safety-test call sites.
engine.go: acceptWithCertCore finalizes via the canonical ApplyCert(Cert{...});
applyBranchFinalization takes Plan (Accept/Reject).
Tests: ledger_test.go proves the fold with a plain-map Ancestry and ZERO engine
(sibling reorg, multi-step descendant, idempotent, equivocation, conflict, gap,
behind-fetch DEFER, and PURE-no-input-mutation). Existing suite migrated to
Plan.Accept/Reject and the public getters.
CGO_ENABLED=0 GOWORK=off go test ./engine/chain/... : ok (27.6s). go vet: clean.
|
||
|
zeekay
|
53c7988304 |
test(chain): port avalanchego Snowman consensus conformance suite onto ChainConsensus
Faithfully ports the ~30 factory-driven conformance funcs from avalanchego snow/consensus/snowman/consensus_test.go onto our sibling-tolerant ChainConsensus API, proving the just-landed cert-driven reorg finality (FinalizeBranch = acceptPreferredChild + rejectTransitively) conforms to the proven Snowman accept/reject/reorg semantics. Each ava RecordPoll-accept/reject scenario maps to a FinalizeBranch whose BranchFinalization plan (Accepted path ascending, Pruned losing subtrees) is asserted directly, plus IsAccepted/GetFinalizedTip/ GetFinalizedHeight/FinalizedBlockAtHeight. Deterministic by construction: no assertion touches the FPC-seeded (crypto/rand) driver decision; the randomized-consistency fuzz uses an explicit math/rand seed. Additive only — no production logic changed. Highest-value reorg proofs included: AcceptAndReject, RejectTransitively, TransitiveVoting, ChangePreferredChain (both selections), SplitVoteNoChange (beta- confidence + premature-accept regression guard), RandomizedConsistency. Documented faithful adaptations (safety never weakened): - AddOnUnknownParent: AddBlock is tracking-only (decomplected); the unknown-parent safety is enforced at finalize (ErrAncestorNotTracked). - Error-on-Accept/Reject: asserted as the explicit plan Accept/Reject sets (stronger than ava's indirect error probe); VM-side error propagation remains covered by error_propagation_test.go. - Metrics*ErrorTest: skipped (no ChainConsensus-owned metrics registry). go test ./engine/chain/ -count=1: ok; -race: ok. |
||
|
zeekay
|
a93027c81e |
feat(chain): sibling-tolerant finality — cert-driven reorg replaces the per-height admission gate (v1.30.0)
THE BUG (fresh-net deadlock). The engine forked avalanchego's Snowman model by adding a hard per-height finality gate (markFinalizedLocked invariant (c): a finalized block's parent MUST equal the current finalized tip) AND never reorged. When a cert finalized one child of a parent while production/preference had explored a SIBLING and built on it, the winner finalized but the losing sibling subtree was NEVER pruned and preference never reorged — so production ran away on the losing branch and every cert for it hit "parent != finalized tip" forever (devnet: finalized stuck at height 1, blocks past 10, every cert REFUSED). A refusal cannot reorg. THE FIX (avalanchego-shaped). Siblings are TRACKED (admission already tolerated them); when a cert SELECTS one, FinalizeBranch walks the certified branch from the finalized tip, finalizes the whole contiguous path, and PRUNES the losing-sibling subtrees — exactly topological.go's acceptPreferredChild (accept the path) + rejectTransitively (prune the losers). The single-non-branching-chain property is now the OUTPUT of the reorg, not a precondition refused at admission. The reorg IS the recovery. - consensus.go: markFinalizedLocked → FinalizeBranch (branch walk + reorg/prune plan). KEPT as α-of-K-cert safety: one-finalized-block-per-height (ErrHeightAlreadyFinalized, equivocation evidence) and the conflicting/losing-branch refusal (renamed ErrParentNotFinalizedTip → ErrConflictsWithFinalizedBranch; now a branch check, not a blanket non-tip-parent refusal). Added ErrAncestorNotTracked (behind-node DEFER). - Decomplected the count path: ProcessVote/Poll set block.accepted as a pure LIVENESS trigger; the per-height ledger is advanced ONLY by FinalizeBranch (the cert path). - engine.go: acceptWithCertCore is the SOLE finalizer — calls FinalizeBranch and applies the plan (VM.Accept the path ascending, VM.Reject the pruned losers, SetPreference to the certified tip). New applyBranchFinalization helper. - Removed dead one-off entrypoints AcceptViaCert / ForceAccept / consensus.AcceptBootstrapBlock; HandleIncomingCert + the K==1 path + Runtime.AcceptBootstrapBlock all route through the one finalize. Simplified ForcePreference (the gate-(c) desync defense is now moot). Regression (RED→GREEN): TestSiblingReorg_CertPrunesLoserBranchAndContinues — two children at one height from a finalized parent, cert one, winner descendants accepted, loser subtree rejected, no permanent refusal, chain continues. Plus FinalizeBranch unit tests (multi-step finalize, conflicting-branch refusal, missing-ancestor defer) and the decomplected count path. Full engine/chain suite green incl -race; CGO_ENABLED=0. |
||
|
zeekay
|
c57928817b |
feat(chain-bootstrap): FrontierCaughtUp — fix fresh-net frontier-naming (decomplect caught-up from named)
A node whose own stake makes the PEER-ONLY responder weight fall below the
stake-majority NAMING floor (a heavy validator, a small beacon set, or skewed
stake) could NEVER name a frontier on a FRESH net even with the WHOLE validator
set connected — it hung in FrontierConnecting forever ("bootstrap waiting for
beacon connectivity before naming the frontier"). The node is itself a beacon and
knows its own accepted tip; under FULL beacon connectivity (no eclipse can hide
an ahead-tip) the full set plus itself proves nothing is ahead -> caught up.
Add FrontierCaughtUp: the BlockSource PROVED the node is at/above the frontier
even though no 2/3 quorum NAMED a tip. Decomplects "a quorum named a tip I hold"
(FrontierNamed -> still drives descend+execute) from "I proved nothing is ahead"
(FrontierCaughtUp -> done), which the pre-fix code conflated by returning
FrontierNamed(ownTip). syncOnce maps FrontierCaughtUp -> passCaughtUp.
Safety: returned ONLY after proving caught-up under FULL connectivity, so an
eclipse suppressing any beacon breaks full-set and falls back to the
partition-capture floor (fail safe) — the stale-go-live guard holds. The
luxfi/node FrontierTip consumes this.
Tests: TestLoop_FreshNet_FullSetConnected_CaughtUpAtGenesis_Completes,
TestLoop_FreshNet_ConnectingThenCaughtUp_WaitsThenCompletes.
go.sum: refresh luxfi/[email protected] (upstream re-tag).
|
||
|
zeekay
|
c983a360de |
fix(chain-bootstrap): caught-up by ACCEPTANCE not store presence + FinalizedLedger accessors
Has→Accepted: a stored-but-unaccepted named frontier (block in the VM store via gossip but lastAccepted below it) no longer short-circuits to caught-up — it descends and ACCEPTS through the cert-gated per-height guard. Add read-only Runtime.FinalizedLedger()/FinalizedBlockAtHeight() over the existing per-height finalized ledger so the node bootstrap loop recognizes convergence off the source that ADVANCES (markFinalizedLocked) rather than the frozen VM.LastAccepted cache. Fixes the mainnet luxd-2 behind-node freeze (1082780→1082796). C1 / VerifyWeighted / per-height guard / fail-closed-post-bootstrap intact. blue→red→blue→red SHIP (0 crit/high/med). Diagnosed from production ground truth. |
||
|
zeekay
|
4f385f0fbb |
fix(chain-bootstrap): bounded NoQuorum retry so a transient live-frontier split converges (F2) + fail-safe undefined-status default (F4)
F2 (MEDIUM): a terminal FrontierNoQuorum (return ErrNoBeaconQuorum on the FIRST no-quorum round) transiently bricked a HEALTHY stale node. On a live chain the beacon-named frontier is a MOVING TARGET: a single GetAcceptedFrontier tally at the instant connectivity crosses the 2/3 floor can catch honest beacons momentarily split across ADJACENT finalized tips (finality-propagation skew) -> one-shot NoQuorum -> immediate terminal failure -> only recovery was the 5-min watchdog -> restart. Run now RETRIES NoQuorum a bounded number of rounds (Config.MaxNoQuorumRounds, default 10) before terminally failing safe: a transient split converges on a later round; a PERSISTENT no-quorum (genuine eclipse / >=1/3 Byzantine beacon stake) still fails safe (ErrNoBeaconQuorum) after the bound, within a sane window (~40s, << the 5-min watchdog). The counter resets the instant a frontier IS named. C1 unchanged: the 2/3-stake frontier-naming gate, total-stake denominator, and content-addressed descent are untouched — only the NoQuorum RETRY behavior changed. F4 (INFO): FrontierStatus zero value was FrontierNamed (permissive). Introduced FrontierInvalid=0 (meaningless by construction) and made FrontierNamed the SOLE permissive case; syncOnce's switch now has a fail-safe default that maps any undefined/out-of-range status to bounded-retry-then-fail-safe. A future bug that returns a zero status can never false-complete a node at its stale height. Tests: TestRED_TransientNoQuorumOnHealthyNodeMustConverge (transient -> converge), TestRED_PersistentNoQuorumStillFailsSafe + TestRED_NoQuorumBoundedRetryCount (persistent/boundary -> ErrNoBeaconQuorum), TestRED_UndefinedFrontierStatusFailsSafe (F4). Forged PoCs still finalize ZERO; deep-gap, weak-subjectivity, single-node, and the clean stale-convergence case all still green. |
||
|
zeekay
|
83ee943451 |
fix(chain-bootstrap): wait for beacon connectivity before concluding caught-up — close the stale-node convergence bug (mainnet canary)
THE CANARY (luxd-2, real mainnet): a STALE node at C-Chain height 1082780,
producers at 1082797 (gap-17, within the 64Ki window — should converge), rolled
to the C1-fixed bootstrap, logged "bootstrap complete — reached the network
frontier" AT ITS STALE HEIGHT, *then* the beacons connected. It entered live
consensus at 1082780 and never converged (cert-catchup cannot pull an
already-finalized gap). Fail-safe (adopted nothing wrong) but did NOT converge.
ROOT CAUSE: syncOnce read FrontierTip's single ok=false as caughtUp=true. But at
boot FrontierTip runs BEFORE any beacon has connected, so it returns ok=false
because it cannot form a ⅔-stake quorum from ZERO connected beacons — not because
"nothing is ahead". The two meanings of ok=false were braided.
DECOMPLECT (Hickey): FrontierTip now returns a FrontierStatus that names the THREE
distinct reasons a tip may not be named, which the loop treats differently:
- FrontierNoBeacons — no beacon set (single-node/dev/--skip-bootstrap): genuinely
caught up, COMPLETE (never hang a legit single node).
- FrontierConnecting — beacons configured but not enough stake CONNECTED yet to
form a ⅔ quorum (the boot race): NOT caught up — WAIT,
bounded by Config.ConnectDeadline (default 3m, < the node's
5m no-progress watchdog), then fail SAFE
(ErrBeaconsUnreachable).
- FrontierNoQuorum — enough connected to ask, but no ⅔-by-stake agreement
(eclipse/partition): fail SAFE (ErrNoBeaconQuorum), NOT
caught up at the stale height. (folds in red's LOW —
eclipsed node fails safe, never false-completes.)
- FrontierNamed — ⅔-stake beacon quorum agreed (C1 gate, UNCHANGED): sync.
syncOnce returns a passOutcome (passCaughtUp/passConnecting/passNoQuorum/passWorking)
so Run can WAIT through connecting passes and fail safe on no-quorum instead of
concluding caught-up. A node that has not reached the beacon-named frontier can no
longer return nil from Run — so the node-side BootstrapComplete/monitorBootstrap
gate stays consistent automatically.
PRESERVED unchanged: the C1 forged-chain defenses (beacon ⅔-stake-quorum frontier
naming + content-addressed descent), the empty-node deep-gap fail-safe
(ErrGapTooLarge), the weak-subjectivity checkpoint, and single-node completion.
The FrontierNamed path + descent + execute is byte-identical; only the bool→status
mapping in front of it is new.
Tests (engine/chain/bootstrap): reproduces the canary
(TestLoop_StaleNodeConvergesAfterDelayedBeaconConnect — beacons connect after a
delay; the node WAITS, does not false-complete at the stale height, then names the
⅔ frontier and converges), plus genuinely-caught-up completes promptly,
beacons-never-connect fails safe (ErrBeaconsUnreachable), beacons-connected-no-quorum
fails safe (ErrNoBeaconQuorum), and single-node (no beacons) completes. All existing
C1 forged-rejection, deep-gap fail-safe, weak-subjectivity, and stall tests stay green.
|
||
|
zeekay
|
2ce3862a7e |
test(chain-bootstrap): deep-gap fails safe with NO false-complete (empty-node mainnet case)
Explicit confirmation that the C1 fix preserves the state-sync boundary: an empty node whose gap exceeds the in-memory window returns ErrGapTooLarge and finalizes NOTHING, so the caller leaves it un-bootstrapped (state-sync first). The STALE node (small gap) still converges via TestLoop_PartialNodeConverges. |
||
|
zeekay
|
43c5d1c611 |
fix(chain-bootstrap): content-addressed descent + weak-subjectivity + M2/M3 — close the forged-ancestry half of C1
The frontier-trust bootstrap loop accepted any Verify-passing, locally-contiguous
block a peer served during the descent. Even under an honest frontier a malicious
Ancestors peer could inject a forged-but-valid sidechain that connects to the local
tip but never reaches the agreed frontier — finalizing forged blocks and bricking
the node against the canonical chain (red: 40 forged blocks finalized).
- Content-addressed descent (syncOnce): walk the parent chain DOWN from the
α-agreed tip, buffering ONLY blocks whose id is on that path. A forged block is
off-path (its id != the want derived from tipID's parent chain) and is ignored;
a batch lacking the requested block is abandoned (re-sample). The executed chain
now provably REACHES the agreed frontier — the per-height guard alone enforced
only LOCAL parent linkage, not that the top equals the agreed tip.
- Weak-subjectivity checkpoint (optional, Config.WeakSubjectivity{ID,Height}): the
α-agreed frontier must descend from an operator-pinned recent finalized block —
defense-in-depth for the empty-genesis case.
- M2: AcceptBootstrapBlock binds the FIRST block to the VM's actual last-accepted
when the finalized-height tracker is unset (no seeding finality at an arbitrary
height/parent off an empty tracker).
- M3: MaxBufferBytes byte budget on the descent buffer alongside the 64Ki block count.
Frontier NAMING (a forged frontier can never be named — beacon + α-weighted-stake
quorum) is the node-side half, landing next. Tests: forged-ancestry-rejected,
honest-syncs-only-canonical, weak-subjectivity refuse/accept, byte-budget, M2 anchor.
All existing engine/chain + bootstrap tests stay green.
|
||
|
zeekay
|
e66299143a |
feat(chain): real bootstrap accept path — empty/behind nodes sync genesis->tip by fetch+execute
The chain bootstrapper was a 56-line no-op stub: Start() set started=true and
fetched nothing, so an empty or behind node declared itself 'bootstrapped' at its
LOCAL last-accepted (genesis if the DB is empty) and never synced to the network
tip. Port avalanche's snowman bootstrapper (fetch-from-frontier + re-execute) into
the Lux split architecture.
THE MISSING PRIMITIVE — a third, decomplected accept authority:
- Runtime.AcceptBootstrapBlock (bootstrap_accept.go): parse -> contiguity ->
local Verify (re-execute) -> ledger commit -> VM.Accept -> SetPreference. It
finalizes a fetched block on FRONTIER-TRUST (no vote, no cert), gated to the
bootstrap phase (InBootstrapPhase) and FAIL-CLOSED once live (FinishBootstrap)
so it can NEVER bypass the cert-gate during live operation.
- ChainConsensus.AcceptBootstrapBlock (consensus.go): the frontier-trust caller of
the ONE per-height guard (markFinalizedLocked), sibling to AcceptViaCert. So a
bootstrap accept obeys the IDENTICAL single-non-branching-chain invariants
(one block per height, height advances by exactly one, parent == finalized tip)
— a malicious peer cannot feed a gapped/forked/invalid history.
- Transitive.bootstrapPhase + InBootstrapPhase()/FinishBootstrap(): the SOLE gate
on the bootstrap authority; the boundary where bootstrap ends and the cert-gated
live path begins.
THE LOOP (bootstrap/bootstrapper.go) replaces the no-op stub: a cohesive,
transport-agnostic fetch+execute driver (discover frontier -> descend+buffer the
ancestry top-down -> execute oldest-first -> repeat to the tip), bounded
(MaxBuffer/maxDescentRounds), with dead-peer handling (re-sample, no hot loop) and a
real stall failure surfaced (never masked as 'ready'). Drives against a narrow
BlockSource the node implements over its GetAcceptedFrontier/GetAncestors wire.
Tests (all green): empty node genesis->50, partial M->N, reject-invalid (no advance
past an unverifiable block), fail-closed-once-live, out-of-order refused; loop:
one-batch, multi-batch descent (gap>window), partial, invalid-halts, no-peer/dead-peer.
Existing cert-catchup + accept-cert-gate + op-table suites stay green.
|
||
|
zeekay
|
f2725f01a6 |
feat(chain): cert-carrying catch-up so a behind validator converges without restart
A validator stranded below the finalized frontier could not recover at runtime:
producers do not re-gossip already-finalized blocks, and the network will not
re-vote a decided height, so re-entering the VOTING path never reaches quorum.
Add the CERT path for catch-up, reusing the audited finality predicate:
- storeServedCertLocked / CertForBlock: the SOLE finalizer (acceptWithCertCore)
now retains each finalized block's marshaled cert in a bounded (maxServedCerts)
sliding window, so a node can SERVE the proof to a catching-up peer.
- AcceptCatchupBlock(blockBytes, certBytes): parse -> local Verify -> track
(no vote) -> finalize through HandleIncomingCert (the SAME VerifyWeighted /
alpha-floor / per-height guard live finality uses). A forged/sub-quorum/
below-floor/out-of-order cert is rejected with identical rigor; a node cannot
be force-fed a chain. Oldest-first parent order is REQUIRED and ENFORCED by
the contiguity guard.
Does NOT weaken VerifyWeighted, the 2/3-stake cert, or 'no VerifiedQuorumCert,
no finality' (accept_cert_gate_test.go stays green).
|
||
|
zeekay
|
2df2fb71de |
consensus(chain): dynamically clamp K/α to live validator count
A static preset (TestnetParams K=11/α=8) assigned to a network smaller than the preset wedges finality: the α-of-K quorum cert demands more affirmative votes than there are validators, so every block verifies but none finalizes. NewRuntime now scales an oversized committee down to the live set and derives α = ⌊2K/3⌋+1 — the BFT supermajority every preset already encodes (K4→α3, K11→α8, K20→α14, K21→α15) — at the single chokepoint where the validator count is known. A preset that already fits is untouched. The ⅔-stake cert still layers weighted safety on top, so a smaller committee never weakens the supermajority. One mechanism: any validator count boots a satisfiable, BFT-valid quorum without per-network tuning to drift. |
||
|
zeekay
|
1232e1226a |
test(chain): red integration probes — low-stake coalition via live+gossip vote path does not finalize
Independent Red adversarial probes for the integration/finality-fix-complete seam (L-1 buffer + 6cf41ba98 Gossip routing): feed a count>=alpha, stake<2/3 coalition through the LIVE vote channel and the Gossip entry into a stake-weighted engine and assert assembleCertLocked's VerifyWeighted gate refuses to finalize. TestRED_INT_LowStakeCoalition_LiveVotePath/_GossipVoteEntry/_DoubleCount_SelfLowStake all green under -count=1. Drives only public/engine API. |
||
|
zeekay
|
fe90208c9f |
consensus(chain): L-1 — dedup buffered votes by NodeID + bound catchupRequested (fail-closed)
bufferVoteLocked now dedups by (BlockID, NodeID): at most one parked vote per validator per block (a repeat from the same NodeID REPLACES, never appends), the dual of certVotes' NodeID keying. Closes single-Byzantine-ID buffer crowd-out — one NodeID can occupy at most one slot per block, so it cannot flood maxBufferedVotesPerBlock junk entries and crowd genuine validators out of the buffer fast-path. catchupRequested gains a hard cap (maxCatchupRequested=4096) + TTL sweep (catchupRequestTTL=30s) + reclaim-on-known at the tracked/decided early returns and the accept/reject/sync sites — fail-closed against a forged-ID flood whose entries layer-1 reclaim never fires for. The fetch is GATED on buffer acceptance: a vote the buffer refused fires NO fetch (no amplification), giving a bounded global fetch ceiling of min(maxBufferedVoteBlocks, maxCatchupRequested). maxBufferedVotesPerBlock raised 256->512 to cover the 500-validator tokenomics tier (alpha~334) so the buffered fast-path never silently drops genuine votes. Red probes (fetch_on_unknown_vote_red_test.go) all green under -race: TestRED_BufferedVote_DedupByNodeID, TestRED_CatchupRequested_UnboundedUnderForgedVoteIDs, TestRED_FetchStorm_NoGlobalRateLimit, TestRED_ProcessPendingFinalizesSubTwoThirdsStake (safety: no finalize without verified 2/3-stake cert preserved). |
||
|
zeekay
|
1d7feef23f |
consensus(chain): fetch-on-unknown-vote — buffer votes for untracked blocks, self-heal the write-path wedge
THE BUG (shared engine, hardens C-Chain too; surfaced on D-Chain dexvm): handleVote DROPPED a vote whose block this node did not yet track. Because a block's votes are solicited exactly once, the drop was permanent: a follower that received the vote-gossip WITHOUT the block bytes (the gossip race) could never reach the alpha-of-K quorum, the block never Accepted, and a write (dex submitTx) blocked forever on the accept channel. Reads bypass consensus, so only writes wedged. THE FIX (one mechanism, reuses the catch-up seam): - handleVote no longer drops a vote for an untracked block. It BUFFERS the vote (bounded) and asks the SAME catch-up transport the missing-PARENT path uses to FETCH the missing block. The fetched block arrives via HandleIncomingBlock, lands at a tracking site, and drainBufferedVotes REPLAYS the parked votes through the normal channel path — where each is signature-verified exactly as a live vote. Buffering never bypasses VerifyVote: a forged/unsigned parked vote costs one map slot and is dropped on replay. - Runtime.requestCatchup is the ONE transport (gate via claimCatchupLocked, then RequestAncestors with the networkID the engine does not hold). It serves BOTH the missing-parent trigger (followVerifiedBlock) and the missing-bytes trigger (engine requestMissing hook). The engine fires it AFTER releasing t.mu (the gate re-locks; firing under the lock would deadlock — non-reentrant RWMutex). - Bounds (fail-closed): maxBufferedVotesPerBlock=256, maxBufferedVoteBlocks=1024. A vote past a cap is dropped, never evicting an existing entry. The buffer is drained on track and deleted on decide (accept/reject/sync) and on a late vote for an already-finalized block — it cannot leak. Completes the prior in-progress catch-up scaffolding: defines the dangling Runtime.requestCatchup and the NetworkConfig.Catchup field it needs (the package did not build before this), wires WithCatchup + requestMissing in NewRuntime. Tests (engine/chain, -race, all green incl. full existing suite): - TestVoteForUnknownBlock_FetchesThenFinalizes: votes-before-block trigger a fetch, replay on delivery, and finalize — the wedge made impossible. - TestBufferedVotes_Bounded: floods prove the map never exceeds either cap. - TestBufferedVote_StillSignatureGated: bad-sig parked votes never finalize; the same block finalizes once genuine alpha votes arrive. - recovery_test.go (parent-path self-heal) now builds and passes on this branch. GUARDRAILS: VerifyWeighted / 2/3-stake cert untouched; signature gate at handleVote unchanged (replay verifies). GOWORK=off. |
||
|
zeekay
|
d24ee92d1b |
consensus(finality): collapse acceptance onto one cert-gated finalize path
THE WEDGE: finality could be triggered by a raw α-of-K vote COUNT on three ungated paths (processPendingBlocks, DrainAccepted, and the K==1 short-circuit), none of which checked the stake-weighted ⅔ predicate. On a skewed-stake set an α-count quorum that did NOT hold ⅔ of stake could finalize — a safety break that also manifested as the block-height wedge (a block "accepted" by count but not by a verifiable cert, leaving the chain unable to make verifiable progress). THE FIX (one rule, one place): no VerifiedQuorumCert, no finality. - verified_cert.go: introduce VerifiedQuorumCert — the unforgeable finality authority token. Its only field is unexported, so no other file can mint a non-zero one with a struct literal. The sole multi-validator producer is BuildVerifiedQuorumCert, which runs the FULL predicate (VerifyWeighted, strict >⅔ of stake when a StakeSource is wired; count-only Verify on equal-stake chains) before wrapping. A raw count can never produce this token. - AcceptWithCert is now the SOLE finalizer. It takes a VerifiedQuorumCert by value and refuses the zero value (ErrNoVerifiedQC). It is impossible to call the finalize body without first clearing the predicate. The old finalizePendingLocked body is unchanged — only its reachability is gated. - Every trigger (vote arrived, re-poll, pending-queue change, build/verify, poll timeout, gossiped-in cert) routes to TryAccept, which obtains a cert or returns ErrNoVerifiedQC and lets the trigger retry on its next tick. The incoming-cert path (quorum_topology.go) re-verifies then promotes via wrapVerifiedCert → AcceptWithCert — one finalization road for everything. - Remove the 3 ungated finalize sites: processPendingBlocks and DrainAccepted no longer finalize on count; they signal liveness only. The K==1 path finalizes through BuildSingleValidatorCert (the 1-of-1 quorum), still cert-gated. - Rename the count helper HasEnoughResponses -> HasEnoughResponsesForRetry to make its role unambiguous at every call site: it is a LIVENESS retry signal, never an acceptance authority (structural form of HIGH-3). Tests: accept_cert_gate_test.go proves AcceptWithCert(zero cert)==ErrNoVerifiedQC, that BuildVerifiedQuorumCert refuses sub-quorum / forged-signature vote sets, and that a real ⅔ quorum finalizes. red_processpending_stake_bypass_test.go + config/red_floor_differential_test.go pin the skewed-stake REJECT (the wedge) and the liveness path. Green under -race. |