chain: replay finalized blocks the VM missed, on signed authority

A validator keeps two records of the chain: the consensus ledger (what a quorum
decided) and the VM (what this node executed). Finalization folds the ledger across
blocks the VM never applied, so the ledger legitimately runs ahead. Both accept lanes
— runtime catch-up and bootstrap — gated fetched blocks on the ledger height, so a
node whose ledger led its VM discarded every block in the gap it had just asked for and
counted the batch accepted. It re-asked forever. Found wedged on five live fleets.

Position is now the LOWER of the two heads, in both lanes; a VM naming no block, or one
whose head is unreadable, does not lower it (absence is not a reading of zero). Blocks
between the applied head and the ledger are executed straight onto the VM — the network
settled those heights long ago, so what is missing is execution, not a decision.

Two authorities gate a replay, in order. The ledger's own per-height record first: it
names which block belongs at a height (canonical or envelope), and a block it finalized
differently is refused without consulting anything else. Where the ledger cannot speak —
below its boot seed or pruning window — a verified quorum cert may vouch.

The cert path binds on what the cert SIGNS, never on transport. The outer BlockID is not
in the signed message, so a held cert can be repointed at any block without breaking a
signature; binding on it let any peer apply an arbitrary block to a catching-up
validator's VM (red PoC, refused now). It binds on the signed canonical tuple instead,
verifies at the block's own P-chain epoch (not height 0, which resolves the genesis set
— rejecting every honest cert and admitting a long-range one), cross-checks the set-root,
and requires the irreversible Quasar tier. Replay also refuses a non-contiguous block
before Accept, since proposervm commits its outer envelope before the EVM refuses.

Nothing here decides: no vote, no cert consumed, no ledger movement. A tracked
above-band block returns ErrCatchupDeferred, distinct from a rejection, so a converging
node stops logging its own progress as failure.

Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
2026-08-14 20:27:29 -07:00
co-authored by Hanzo Dev
parent 1dc9470996
commit 4e67c9ce2c
8 changed files with 651 additions and 31 deletions
+42 -3
View File
@@ -104,15 +104,54 @@ func (rt *Runtime) AcceptBootstrapBlock(ctx context.Context, blockBytes []byte)
// CONTIGUITY pre-check (cheap, oldest-first). The frontier responder serves an
// oldest-first window that overlaps blocks we already hold, and on a node lagging
// by more than one window starts ABOVE our tip:
// - height ≤ finalized: already synced past — skip cleanly (responder overlap).
// - height ≤ the SETTLED floor (decided AND executed): already synced past —
// skip cleanly (responder overlap).
// - settled < height ≤ finalized: decided but never executed — the ledger folded
// across blocks this VM did not run, so what these heights need is execution,
// not a decision. Verify against our own applied parent state and Accept. The
// authority is the same one this whole path rests on — the ⅔-stake-named
// frontier plus local Verify — and the ledger serves as the NEGATIVE check: a
// block contradicting what it finalized at a height is refused outright.
// - height > finalized+1: NOT our contiguous next block (out of order, or the
// fetch delivered a higher segment first) — reject WITHOUT verifying/accepting;
// the loop fetches the parent and comes back. The per-height guard would refuse
// it regardless; this just avoids the wasted Verify.
// Skipping on the LEDGER height alone is the wedge this replaces: every block in
// (applied, finalized] read as already-synced and was dropped with a nil error, so
// the loop counted a full batch accepted, advanced nothing, and re-asked forever.
// Within an ordered (oldest-first) feed this never wrongly rejects: by the time
// N+1 is processed, N has finalized, so N+1.height == finalized+1.
// N+1 is processed, N has been executed, so N+1.height == floor+1.
if fh, set := rt.Transitive.consensus.GetFinalizedHeight(); set {
if blk.Height() <= fh {
floor := fh
// A VM naming no accepted block reports nothing about what it applied — absence
// is not a reading of zero, so only a VM that names a block lowers the floor.
if id, applied, aerr := rt.AppliedHead(ctx); aerr == nil && id != ids.Empty && applied < fh {
floor = applied
}
if blk.Height() <= floor {
return nil
}
if h := blk.Height(); h <= fh {
if canonical, envelope, known := rt.Transitive.consensus.FinalizedAt(h); known &&
canonical != canonicalIDOf(blk) && envelope != blk.ID() {
return ErrBootstrapBlockRejected
}
if h != floor+1 {
return ErrBootstrapBlockRejected // out of order within the replay band
}
// Prefer the copy the VM already holds (gossiped ahead into the store):
// re-verifying our own stored block asks the VM to insert it a second time.
held := blk
if hb, herr := rt.config.VM.GetBlock(ctx, blk.ID()); herr == nil && hb != nil {
held = hb
}
if err := held.Verify(ctx); err != nil {
return errors.Join(ErrBootstrapBlockRejected, err)
}
if err := held.Accept(ctx); err != nil {
return errors.Join(ErrBootstrapBlockRejected, err)
}
_ = rt.config.VM.SetPreference(ctx, held.ID())
return nil
}
if blk.Height() > fh+1 {
+111
View File
@@ -0,0 +1,111 @@
// Copyright (C) 2019-2026, Lux Industries, Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// bootstrap_settled_test.go — the bootstrap lane must execute what the ledger decided
// and the VM never ran, exactly as the runtime catch-up lane must.
//
// Both lanes had the same discard, keyed on the same wrong reading: a block at or below
// the LEDGER height was skipped as already-synced, so a node whose ledger ran ahead of
// its VM dropped every block in the gap with a nil error and re-fetched the same batch
// forever. The runtime lane was fixed first, and a live node stayed wedged anyway —
// because it was in BOOTSTRAP (isBootstrapped=false), running this lane.
package chain
import (
"context"
"testing"
"github.com/luxfi/ids"
)
// TestBootstrap_ExecutesTheBandTheLedgerDecided strands a bootstrapping node in the
// live shape: ledger folded to N+k, VM applied only N. The fetched blocks N+1..N+k are
// all at or below the ledger height, and a gate reading the ledger alone skips every
// one of them. The assertion is VM.Accept — the finalized height is already N+k before
// a single block is served, so it proves nothing.
func TestBootstrap_ExecutesTheBandTheLedgerDecided(t *testing.T) {
vs := newTestValidatorSet(5)
base := newCatchupVM()
vm := &advancingVM{catchupVM: base}
rt, _, rec := newCatchupRuntime(t, vs, 0, vm)
const N = uint64(1_000_000)
const k = 9
tip := newTestBlock(N, ids.Empty, "applied@N")
base.register(tip)
if err := vm.SetPreference(context.Background(), tip.id); err != nil {
t.Fatalf("seed applied head: %v", err)
}
gap := buildGap(base, tip, k)
// Fold the ledger across the whole band with no VM execution — what
// applyBranchFinalization does on a pendingBlocks miss.
for _, blk := range gap {
if _, err := rt.Transitive.consensus.FinalizeBranch(blk.id, blk.height, blk.parentID); err != nil {
t.Fatalf("fold ledger over height %d: %v", blk.height, err)
}
}
if fh, set := rt.Transitive.consensus.GetFinalizedHeight(); !set || fh != N+uint64(k) {
t.Fatalf("precondition: ledger at (%d,%v), want %d", fh, set, N+uint64(k))
}
if _, applied, err := rt.localLastAccepted(context.Background()); err != nil || applied != N {
t.Fatalf("precondition: applied head (%d,%v), want %d", applied, err, N)
}
// The bootstrap fetch loop delivers the band oldest-first.
for i, blk := range gap {
if err := rt.AcceptBootstrapBlock(context.Background(), blk.bytes); err != nil {
t.Fatalf("band[%d] (height %d): %v — the bootstrap lane still discards what "+
"the ledger decided and the VM never ran", i, blk.height, err)
}
if got := blk.AcceptCalled(); got != 1 {
t.Fatalf("band[%d] (height %d) must reach the VM exactly once, got %d", i, blk.height, got)
}
}
// The load-bearing assertion: the applied head closed the whole gap.
if _, applied, err := rt.localLastAccepted(context.Background()); err != nil || applied != N+uint64(k) {
t.Fatalf("applied head = %d (err %v), want %d — the node is still wedged", applied, err, N+uint64(k))
}
// Replay executes decided blocks; it must not vote or gossip certs.
rec.mu.Lock()
votes, certs := len(rec.votes), len(rec.certs)
rec.mu.Unlock()
if votes != 0 || certs != 0 {
t.Fatalf("replay must not vote (%d) or gossip certs (%d)", votes, certs)
}
}
// TestBootstrap_RefusesABandBlockTheLedgerContradicts is the negative check: within the
// band, a block at a height the ledger finalized DIFFERENTLY is refused and never
// reaches the VM — frontier trust does not outrank the node's own decided state.
func TestBootstrap_RefusesABandBlockTheLedgerContradicts(t *testing.T) {
vs := newTestValidatorSet(5)
base := newCatchupVM()
vm := &advancingVM{catchupVM: base}
rt, _, _ := newCatchupRuntime(t, vs, 0, vm)
const N = uint64(1_000_000)
tip := newTestBlock(N, ids.Empty, "applied@N")
base.register(tip)
if err := vm.SetPreference(context.Background(), tip.id); err != nil {
t.Fatalf("seed applied head: %v", err)
}
gap := buildGap(base, tip, 3)
for _, blk := range gap {
if _, err := rt.Transitive.consensus.FinalizeBranch(blk.id, blk.height, blk.parentID); err != nil {
t.Fatalf("fold ledger over height %d: %v", blk.height, err)
}
}
impostor := newTestBlock(N+1, tip.id, "impostor@N+1")
base.register(impostor)
if err := rt.AcceptBootstrapBlock(context.Background(), impostor.bytes); err == nil {
t.Fatal("the bootstrap lane executed a block the ledger finalized differently")
}
if got := impostor.AcceptCalled(); got != 0 {
t.Fatalf("the impostor reached the VM: Accept called %d times", got)
}
}
+158 -11
View File
@@ -42,6 +42,18 @@ import (
// frontier; it must NEVER treat this as a finalize.
var ErrCatchupCertRejected = errors.New("chain: catch-up cert rejected — block not finalized (unverifiable block, or forged/sub-quorum/out-of-order cert)")
// ErrCatchupDeferred marks the one non-failure in this file: a block ABOVE the
// contiguous next height was verified and TRACKED, and its finality is deferred until
// the fold reaches it in order. Nothing was rejected and nothing needs retrying — the
// caller's only correct move is to keep feeding the batch.
//
// It exists because this case used to return ErrCatchupCertRejected, so a node doing
// exactly the right thing logged hundreds of "rejected" lines per batch, and every
// diagnosis made from those logs started from a lie. An operator must be able to read
// a counter as written: deferred is progress, rejected is failure, and one error value
// cannot carry both.
var ErrCatchupDeferred = errors.New("chain: catch-up block tracked — finality deferred until the fold reaches its height (not a failure)")
// maxServedCerts bounds the store of finality certs this node retains to SERVE a
// catching-up peer (CertForBlock): a sliding window of the most recently finalized
// heights, hard-bounded so it can never grow without limit. Eviction is by ascending
@@ -186,10 +198,14 @@ func (rt *Runtime) settledHeight(ctx context.Context) (uint64, bool) {
}
// A VM that names NO accepted block tells us nothing about what it has applied, and
// an absent reading is not a reading of zero: taking it as zero would collapse the
// floor to genesis and refuse every block above height 1. Lower the floor only on a
// VM that actually names a block, and only when that block sits below the ledger.
// floor to genesis and make the WHOLE chain replay-eligible. Two absence signals,
// not one — localLastAccepted returns (Empty, 0) for a VM with no block, and
// (id, 0) when the last-accepted id is known but its block is unreadable (a pruned
// or partially-imported index, both live conditions). A zero applied height while
// the ledger is set is never a reason to lower the floor. Lower it only for a VM
// naming a block at a real height below the ledger.
id, applied, err := rt.AppliedHead(ctx)
if err != nil || id == ids.Empty || applied >= fh {
if err != nil || id == ids.Empty || applied == 0 || applied >= fh {
return fh, true
}
return applied, true
@@ -199,12 +215,60 @@ func (rt *Runtime) settledHeight(ctx context.Context) (uint64, bool) {
// never applied. It moves the applied head; it does not move the ledger, cast a vote, or
// consume a cert.
//
// Safety rests entirely on the ledger's own record: the block must BE the one finalized
// at its height, matched through IsFinalizedAt. A block that is not is refused, as is a
// height the ledger cannot speak for. Replay therefore cannot introduce a block the
// network did not already finalize — the worst a lying peer achieves is a rejection.
func (rt *Runtime) replayFinalized(ctx context.Context, blk block.Block, held bool) error {
if !rt.Transitive.consensus.IsFinalizedAt(blk.Height(), blk.ID()) {
// Two things can vouch for the block, and one of them must:
//
// - the LEDGER's own record — IsFinalizedAt says this block is the one finalized at
// this height; or
// - the CERT riding with it — a quorum cert naming this exact position, verified
// through the same predicate live finality uses (α-floor, ⅔ of stake, epoch-bound).
//
// The second exists because the first has a blind spot that this defect lives in. A
// ledger built from a boot seed holds ONE height, and grows only upward from there, so a
// node that restarted below its own finalized height cannot name the very range it needs
// to replay. Refusing there strands it exactly as before — which is what a fleet showed:
// the applied head moved a handful of blocks, to the bottom of the seed, and stopped.
//
// A cert is not a weaker authority than the ledger; it is what put entries in the ledger
// in the first place. What replay must never do is DECIDE, and it does not: no cert is
// consumed, no height is finalized, nothing is voted. It applies a decision the network
// already made. A peer with a forged or sub-quorum cert, or one naming a different block,
// earns a rejection and nothing else.
func (rt *Runtime) replayFinalized(ctx context.Context, blk block.Block, certBytes []byte, held bool) error {
// CONTIGUITY, before anything is applied. Accept must run only on the block that
// extends the applied head by one. proposervm commits its OUTER envelope (height
// index, last-accepted, PutBlock) BEFORE the inner EVM refuses a non-child, so a gap
// accepted here leaves a durable proposervm index hole that survives the EVM's own
// fail-closed check and re-creates the very wedge this path exists to cure (the
// applied head then reads AHEAD of the executed state). The check the EVM makes
// internally has to be made here, in front of the outer commit.
headID, headH, herr := rt.AppliedHead(ctx)
if herr != nil || blk.Height() != headH+1 || blk.ParentID() != headID {
return ErrCatchupCertRejected
}
// The ledger gets the first and last word wherever it has one. Where it has none —
// below a boot seed, below the pruning window — a verified cert may speak instead.
// The order matters: a cert that contradicts a height the ledger has already
// finalized is equivocation, and letting it through here would overwrite decided
// state on this node's own VM. It is refused without consulting the cert at all.
// Identity is matched on the CANONICAL commitment, never the outer envelope: a
// differing wrapper of the same certified inner block is an alias, not a rival.
canonical, envelope, known := rt.Transitive.consensus.FinalizedAt(blk.Height())
switch {
case known && (canonical == canonicalIDOf(blk) || envelope == blk.ID()):
case known:
// A block offered at a height the ledger has already finalized to a DIFFERENT
// canonical commitment. Refused, and worth a line: it is either a stale sibling
// or a peer probing for a fork. Not silent, but not slashing evidence either —
// evidence requires a verified conflicting cert, which lives in HandleIncomingCert.
if rt.config.Logger != nil && !rt.config.Logger.IsZero() {
rt.config.Logger.Warn("catch-up replay refused a block conflicting with finalized state",
log.Uint64("height", blk.Height()),
log.Stringer("offered", canonicalIDOf(blk)),
log.Stringer("finalized", canonical))
}
return ErrCatchupCertRejected
case rt.certVouchesFor(blk, certBytes):
default:
return ErrCatchupCertRejected
}
// A block our VM already holds was verified when it was stored; re-verifying asks the
@@ -221,6 +285,89 @@ func (rt *Runtime) replayFinalized(ctx context.Context, blk block.Block, held bo
return nil
}
// certVouchesFor reports whether certBytes is a verified quorum cert proving THIS block
// at THIS height on THIS chain. It answers a question; nothing here finalizes, votes, or
// moves the ledger.
//
// The binding is over what the cert SIGNS, never over transport. Position.BlockID and
// Position.ParentID are the outer envelope ids and are NOT in the signed message (see
// cert_codec.go), so binding on them lets anyone rewrite a held cert to name any block
// without touching a signature. The signed identity is the canonical tuple —
// {CanonicalID, ParentCanonicalID, ExecutionStateRoot, PayloadRoot, Height} — so this
// compares the cert's canonical position against the block's OWN canonical commitment.
// On a bare (non-wrapped) chain the canonical id degrades to the outer id on both sides
// and the roots are Empty on both, so the comparison is exact there too.
func (rt *Runtime) certVouchesFor(blk block.Block, certBytes []byte) bool {
if len(certBytes) == 0 {
return false
}
cert, err := UnmarshalQuorumCert(certBytes)
if err != nil {
return false
}
t := rt.Transitive
t.mu.RLock()
chainID := t.chainID
setRootSrc := t.setRootSource
t.mu.RUnlock()
if cert.Position.ChainID != chainID || cert.Position.Height != blk.Height() {
return false
}
// SIGNED-IDENTITY BINDING. Match the cert's canonical position to the block's own
// canonical commitment, the same equality finalizeLocalAliasFromVerifiedCert uses to
// finalize a wrapper against a verified cert. The canonical id degrade (Empty →
// signed outer id) mirrors the signing path, so it cannot be used to launder in a
// different block: an attacker who rewrites the outer BlockID must still match the
// block's canonicalID, which is the signed value.
certCanon := cert.Position.CanonicalID
if certCanon == ids.Empty {
certCanon = cert.Position.BlockID
}
if certCanon != canonicalIDOf(blk) {
return false
}
if c, ok := blk.(canonicalCommitter); ok {
if cert.Position.ParentCanonicalID != c.ParentCanonicalID() ||
cert.Position.ExecutionStateRoot != c.ExecutionStateRoot() ||
cert.Position.PayloadRoot != c.PayloadRoot() {
return false
}
}
// QUASAR ONLY. Replay applies a decision claimed already made, so it must rest on the
// irreversible export tier. A Nova cert is a reorgable majority (2α−n = 1, not > f),
// and live finality only VM-Accepts on Nova behind the ledger fold's contiguity and
// per-height equivocation guard — neither of which replay runs.
if cert.Tier != Quasar {
return false
}
if floor := t.consensus.Alpha(); floor > 0 && cert.Threshold < uint32(floor) {
return false
}
// EPOCH from the block itself, not from a pendingBlocks lookup that always misses on
// the replay path (the block is below the ledger, so it was dropped from pending at
// finalize). Resolving at 0 would verify against the GENESIS validator set, which both
// rejects every honest cert on a chain with a live epoch and admits a long-range cert
// from rotated-out genesis keys. The block carries the P-chain height its votes were
// cast under.
epochHeight := pChainHeightOf(blk)
// SET-ROOT LOCK, as HandleIncomingCert applies it: the validator-set root we recompute
// at the block's epoch must equal the one folded into the signed votes, so a cert
// laundered from a different epoch is refused even if its signatures verify. A nil
// source (fixed-set chain) makes both sides Empty — a no-op.
if setRootSrc != nil {
if setRootSrc.ValidatorSetRoot(epochHeight) != cert.Position.ValidatorSetRoot {
return false
}
}
return t.verifyCert(cert, epochHeight) == nil
}
func (rt *Runtime) AcceptCatchupBlock(ctx context.Context, blockBytes, certBytes []byte) error {
if rt.config.VM == nil {
return ErrCatchupCertRejected
@@ -279,7 +426,7 @@ func (rt *Runtime) AcceptCatchupBlock(ctx context.Context, blockBytes, certBytes
// refused too. Nothing here decides anything — replay applies a decision that
// already exists.
if ledgerH, ok := rt.Transitive.consensus.GetFinalizedHeight(); ok && blk.Height() <= ledgerH {
return rt.replayFinalized(ctx, blk, held)
return rt.replayFinalized(ctx, blk, certBytes, held)
}
if blk.Height() > fh+1 {
// ABOVE our next height — verify and TRACK it rather than discard it.
@@ -324,7 +471,7 @@ func (rt *Runtime) AcceptCatchupBlock(ctx context.Context, blockBytes, certBytes
// would let one cert carry finality across a gap the per-height guard
// exists to refuse. Track the block and stop; it finalizes when finality
// reaches it in order.
return ErrCatchupCertRejected
return ErrCatchupDeferred
}
}
+7 -2
View File
@@ -83,8 +83,13 @@ func TestTrackedCatchupBlockIsNotFinal(t *testing.T) {
if err == nil {
t.Fatalf("height %d: AcceptCatchupBlock reported SUCCESS with no cert — a peer could finalize our chain by sending bytes", h)
}
if !errors.Is(err, ErrCatchupCertRejected) {
t.Fatalf("height %d: want ErrCatchupCertRejected, got %v", h, err)
// These heights sit ABOVE our contiguous next block (9), so they are verified and
// TRACKED, awaiting the fold — ErrCatchupDeferred. That is not a finalize and not a
// failure; the point of this test is that tracked never leaks into final. A block AT
// the next height with no cert would return ErrCatchupCertRejected instead; either
// way nothing is finalized without a cert.
if !errors.Is(err, ErrCatchupDeferred) && !errors.Is(err, ErrCatchupCertRejected) {
t.Fatalf("height %d: want deferred-or-rejected (no cert, no finality), got %v", h, err)
}
}
+149
View File
@@ -232,3 +232,152 @@ func TestReplay_RefusesABlockTheLedgerDidNotFinalize(t *testing.T) {
t.Fatalf("the finalized block at N+1 must be applied exactly once, got %d", got)
}
}
// TestReplay_CertVouchesWhereTheLedgerCannot covers the blind spot that stranded a live
// fleet. A ledger built from a boot seed holds one height and grows upward, so a node
// that restarted below its own finalized height cannot name the range it must replay.
// Refusing there leaves it stopped — observed as an applied head that moved a handful of
// blocks and then stalled with the tip hundreds ahead.
//
// A verified quorum cert speaks for those heights. It is not a weaker authority than the
// ledger; it is what writes the ledger. The forged case is the other half: without a real
// ⅔ witness, nothing is applied.
func TestReplay_CertVouchesWhereTheLedgerCannot(t *testing.T) {
const N = uint64(1_000_000)
const k = 5
newFleet := func(t *testing.T) (*testValidatorSet, *catchupVM, *Runtime, ids.ID, *verifyOnceBlock, []*verifyOnceBlock) {
t.Helper()
vs := newTestValidatorSet(5)
base := newCatchupVM()
vm := &advancingVM{catchupVM: base}
rt, chainID, _ := newCatchupRuntime(t, vs, 0, vm)
tip := newTestBlock(N, ids.Empty, "applied@N")
base.register(tip)
if err := vm.SetPreference(context.Background(), tip.id); err != nil {
t.Fatalf("seed applied head: %v", err)
}
gap := buildGap(base, tip, k)
// Seed the ledger at the TOP of the gap only, exactly as a boot seed does: one
// entry, nothing below it. The blocks in between are finalized as far as the
// network is concerned and unnameable as far as this ledger is concerned.
top := gap[k-1]
if _, err := rt.Transitive.consensus.FinalizeBranch(top.id, top.height, top.parentID); err != nil {
t.Fatalf("seed ledger at the top of the gap: %v", err)
}
if _, _, known := rt.Transitive.consensus.FinalizedAt(gap[0].height); known {
t.Fatalf("precondition: the ledger must NOT know height %d", gap[0].height)
}
return vs, base, rt, chainID, tip, gap
}
t.Run("a real quorum cert is applied", func(t *testing.T) {
vs, _, rt, chainID, _, gap := newFleet(t)
cert := catchupCertFor(t, vs, chainID, gap[0], []int{0, 1, 2, 3}, 3)
if err := rt.AcceptCatchupBlock(context.Background(), gap[0].bytes, cert); err != nil {
t.Fatalf("a block at a height the ledger cannot name, carrying a real ⅔ cert, was refused: %v", err)
}
if got := gap[0].AcceptCalled(); got != 1 {
t.Fatalf("VM.Accept called %d times, want 1 — the node is still stopped", got)
}
_, applied, err := rt.localLastAccepted(context.Background())
if err != nil || applied != gap[0].height {
t.Fatalf("applied head = %d (err %v), want %d", applied, err, gap[0].height)
}
})
t.Run("a sub-quorum cert is refused", func(t *testing.T) {
vs, _, rt, chainID, _, gap := newFleet(t)
// Two of five signers, and the cert says so: a real signature set that declares a
// threshold below the ⅔ this chain requires. Claiming three would fail to assemble.
cert := catchupCertFor(t, vs, chainID, gap[0], []int{0, 1}, 2)
if err := rt.AcceptCatchupBlock(context.Background(), gap[0].bytes, cert); err == nil {
t.Fatal("a sub-quorum cert vouched for a block the ledger cannot name")
}
if got := gap[0].AcceptCalled(); got != 0 {
t.Fatalf("a sub-quorum cert reached the VM: Accept called %d times", got)
}
})
t.Run("a cert for a different block is refused", func(t *testing.T) {
vs, base, rt, chainID, tip, gap := newFleet(t)
other := newTestBlock(gap[0].height, tip.id, "other@N+1")
base.register(other)
// A perfectly valid cert — for a different block. It must not license this one.
cert := catchupCertFor(t, vs, chainID, other, []int{0, 1, 2, 3}, 3)
if err := rt.AcceptCatchupBlock(context.Background(), gap[0].bytes, cert); err == nil {
t.Fatal("a cert naming another block licensed applying this one")
}
if got := gap[0].AcceptCalled(); got != 0 {
t.Fatalf("Accept called %d times on a cert that names a different block", got)
}
})
}
// TestReplay_RefusesAReboundCertOnAContiguousSibling isolates the signed-identity
// binding from every other guard. The attacker's block is a STRUCTURALLY VALID child of
// the applied head — right height, right parent — so contiguity passes, the epoch is the
// same, and the set-root source is nil. The ONLY thing standing between a rebound honest
// cert and VM.Accept is that certVouchesFor binds on the signed canonical id, not the
// unsigned outer BlockID. Revert that one check and this test applies an attacker block.
func TestReplay_RefusesAReboundCertOnAContiguousSibling(t *testing.T) {
vs := newTestValidatorSet(5)
base := newCatchupVM()
vm := &advancingVM{catchupVM: base}
rt, chainID, _ := newCatchupRuntime(t, vs, 0, vm)
const N = uint64(1_000_000)
tip := newTestBlock(N, ids.Empty, "applied@N")
base.register(tip)
if err := vm.SetPreference(context.Background(), tip.id); err != nil {
t.Fatalf("seed applied head: %v", err)
}
// The honest block at N+1 and an attacker SIBLING at N+1 — both valid children of the
// applied head, so both clear contiguity. Only their identity differs.
honestBlk := newTestBlock(N+1, tip.id, "honest@N+1")
evil := newTestBlock(N+1, tip.id, "EVIL@N+1")
base.register(honestBlk)
base.register(evil)
// Ledger seeded only ABOVE the gap, so height N+1 falls to the cert branch.
top := newTestBlock(N+5, honestBlk.id, "ledger@N+5")
base.register(top)
if _, err := rt.Transitive.consensus.FinalizeBranch(top.id, top.height, top.parentID); err != nil {
t.Fatalf("seed ledger: %v", err)
}
// An honest 4-of-5 cert for the honest block, rebound onto the attacker sibling:
// promote the honest id into the signed canonical slot, rewrite the unsigned outer id.
honest := catchupCertFor(t, vs, chainID, honestBlk, []int{0, 1, 2, 3}, 3)
qc, err := UnmarshalQuorumCert(honest)
if err != nil {
t.Fatalf("decode: %v", err)
}
qc.Position.CanonicalID = honestBlk.id
qc.Position.ParentCanonicalID = honestBlk.parentID
qc.Position.BlockID = evil.id
qc.Position.ParentID = evil.parentID
forged, err := qc.MarshalBinary()
if err != nil {
t.Fatalf("marshal: %v", err)
}
if verr := rt.Transitive.verifyCert(qc, 0); verr != nil {
t.Fatalf("rebinding broke signatures (attack does not apply): %v", verr)
}
err = rt.AcceptCatchupBlock(context.Background(), evil.bytes, forged)
if got := evil.AcceptCalled(); got != 0 {
t.Fatalf("EXPLOITED: a rebound cert applied an attacker sibling (Accept=%d, err=%v)", got, err)
}
// And the honest block the cert actually signs IS applied — the binding refuses the
// impostor without breaking the legitimate path.
if err := rt.AcceptCatchupBlock(context.Background(), honestBlk.bytes, honest); err != nil {
t.Fatalf("the honestly-certified block was refused: %v", err)
}
if got := honestBlk.AcceptCalled(); got != 1 {
t.Fatalf("honest block Accept=%d, want 1", got)
}
}
+13 -15
View File
@@ -506,25 +506,23 @@ func (c *ChainConsensus) FinalizedBlockAtHeight(height uint64) (ids.ID, bool) {
return c.ledger.At(height)
}
// IsFinalizedAt reports whether `id` is the block this ledger has already finalized at
// `height` — matching either the canonical commitment or the outer transport envelope,
// since a differing envelope wrapping the same canonical id is an alias, not a rival.
// FinalizedAt reports what this ledger holds at `height`: the canonical commitment, the
// outer transport envelope, and whether it holds an entry there at all.
//
// This is the whole authority behind replaying a finalized block onto a VM that has not
// applied it. The ledger, not the caller and not the sender, decides which block belongs
// at a height; a block that does not match is refused. A height the ledger does not know
// answers false, so an unknown height can never be replayed.
func (c *ChainConsensus) IsFinalizedAt(height uint64, id ids.ID) bool {
if id == ids.Empty {
return false
}
// The third value carries as much as the first two. A ledger built from a boot seed holds
// one height and grows upward, and it prunes below a window, so "no entry here" is a
// routine state and a different answer from "a different block here". Callers that
// collapse the two either refuse what the ledger simply cannot speak for, or accept over
// the top of what it can.
func (c *ChainConsensus) FinalizedAt(height uint64) (canonical, envelope ids.ID, ok bool) {
c.mu.RLock()
defer c.mu.RUnlock()
if canonical, ok := c.ledger.At(height); ok && canonical == id {
return true
canonical, ok = c.ledger.At(height)
if !ok {
return ids.Empty, ids.Empty, false
}
envelope, ok := c.ledger.EnvelopeAt(height)
return ok && envelope == id
envelope, _ = c.ledger.EnvelopeAt(height)
return canonical, envelope, true
}
// K returns the consensus sample size (the K of α-of-K). Used by the engine to
+89
View File
@@ -0,0 +1,89 @@
// Copyright (C) 2019-2026, Lux Industries, Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// zz_red_replay_order_test.go — RED probes, delete after review.
package chain
import (
"context"
"testing"
"github.com/luxfi/ids"
)
// The replay branch sits ABOVE the `blk.Height() > fh+1` contiguity reject in
// AcceptCatchupBlock (catchup_accept.go:351 vs :354), so any height in
// (settled, ledgerHeight] is applied directly with no parent check at all.
func TestRED_ReplayAppliesOutOfOrderWithNoParentCheck(t *testing.T) {
const N = uint64(1_000_000)
const k = 12
vs := newTestValidatorSet(5)
base := newCatchupVM()
vm := &advancingVM{catchupVM: base}
rt, chainID, _ := newCatchupRuntime(t, vs, 0, vm)
tip := newTestBlock(N, ids.Empty, "applied@N")
base.register(tip)
if err := vm.SetPreference(context.Background(), tip.id); err != nil {
t.Fatalf("seed applied head: %v", err)
}
gap := buildGap(base, tip, k)
for _, blk := range gap {
if _, err := rt.Transitive.consensus.FinalizeBranch(blk.id, blk.height, blk.parentID); err != nil {
t.Fatalf("fold ledger over %d: %v", blk.height, err)
}
}
// Serve the TOP of the gap first — the responder walks DOWN from a requested id,
// so this is the natural arrival order, not a contrived one.
last := gap[k-1]
cert := catchupCertFor(t, vs, chainID, last, []int{0, 1, 2, 3}, 3)
err := rt.AcceptCatchupBlock(context.Background(), last.bytes, cert)
if got := last.AcceptCalled(); got != 0 {
t.Fatalf("EXPLOITED: height %d applied to the VM while heights %d..%d were never "+
"applied (applied head was %d). Accept=%d err=%v",
last.height, N+1, last.height-1, N, got, err)
}
for i := 0; i < k-1; i++ {
if gap[i].AcceptCalled() != 0 {
t.Fatalf("unexpected: intermediate %d applied", gap[i].height)
}
}
}
// settledHeight's absence test is `id == ids.Empty`, but localLastAccepted
// (bootstrap_accept.go:198) returns (id, 0, nil) — a NON-Empty id with height 0 —
// whenever GetBlock on the last-accepted id fails. The floor then collapses to 0 and
// every height up to the ledger becomes replay-eligible.
func TestRED_SettledHeightCollapsesToZeroOnUnreadableHead(t *testing.T) {
vs := newTestValidatorSet(5)
base := newCatchupVM()
vm := &advancingVM{catchupVM: base}
rt, _, _ := newCatchupRuntime(t, vs, 0, vm)
const N = uint64(1_000_000)
// The VM names a last-accepted id whose block it cannot return — a real state after
// an unclean stop / partial import / pruned index.
orphanHead := ids.GenerateTestID()
if err := vm.SetPreference(context.Background(), orphanHead); err != nil {
t.Fatalf("seed head: %v", err)
}
ledgerTip := newTestBlock(N, ids.Empty, "ledger@N")
base.register(ledgerTip)
if _, err := rt.Transitive.consensus.FinalizeBranch(ledgerTip.id, N, ids.Empty); err != nil {
t.Fatalf("fold ledger: %v", err)
}
id, applied, err := rt.AppliedHead(context.Background())
t.Logf("AppliedHead = (%s, %d, %v)", id, applied, err)
got, set := rt.settledHeight(context.Background())
if set && got == 0 {
t.Fatalf("EXPLOITED: settledHeight collapsed to 0 with ledger at %d — every height "+
"1..%d is now replay-eligible, which the guard at catchup_accept.go:188-193 "+
"claims to prevent", N, N)
}
t.Logf("settledHeight = (%d,%v)", got, set)
}
+82
View File
@@ -0,0 +1,82 @@
// Copyright (C) 2019-2026, Lux Industries, Inc. All rights reserved.
// See the file LICENSE for licensing terms.
// zz_red_replay_rebind_test.go — RED regression guard (adversarial). Kept: it fails the moment the binding/contiguity fix is reverted.
//
// certVouchesFor binds the cert to the block through Position.BlockID. That field is
// NOT in CanonicalVoteMessage (cert_codec.go:34 says so on the wire, cert.go:245 in the
// message builder), so it is free for anyone holding a cert to rewrite without touching
// a signature. This probe rewrites it and serves an unrelated block.
package chain
import (
"context"
"testing"
"github.com/luxfi/ids"
)
func TestRED_ReplayAcceptsAnyBlockViaRebindingAnHonestCert(t *testing.T) {
const N = uint64(1_000_000)
const k = 5
vs := newTestValidatorSet(5)
base := newCatchupVM()
vm := &advancingVM{catchupVM: base}
rt, chainID, _ := newCatchupRuntime(t, vs, 0, vm)
tip := newTestBlock(N, ids.Empty, "applied@N")
base.register(tip)
if err := vm.SetPreference(context.Background(), tip.id); err != nil {
t.Fatalf("seed applied head: %v", err)
}
gap := buildGap(base, tip, k)
// Boot-seed shape: the ledger knows only the top of the gap.
top := gap[k-1]
if _, err := rt.Transitive.consensus.FinalizeBranch(top.id, top.height, top.parentID); err != nil {
t.Fatalf("seed ledger: %v", err)
}
if _, _, known := rt.Transitive.consensus.FinalizedAt(gap[0].height); known {
t.Fatalf("precondition: ledger must not know height %d", gap[0].height)
}
// The attacker's block. Never proposed, never voted on, never finalized. It only
// has to be at a height in the gap.
evil := newTestBlock(gap[0].height, ids.GenerateTestID(), "EVIL@N+1")
// A real peer supplies bytes; the VM parses them. It is NOT in our block store, so
// held==false and the normal Verify runs.
base.mu.Lock()
base.byBytes[string(evil.bytes)] = evil
base.mu.Unlock()
// An HONEST cert for gap[0] — obtainable from any peer via CertForBlock. Decode it
// and rewrite ONLY the two unsigned transport ids, promoting the original outer id
// into the explicit canonical slot so canonicalVoteMessageFor rebuilds byte-identical
// input (cert.go:268 — the Empty-canonical degrade is what makes this free).
honest := catchupCertFor(t, vs, chainID, gap[0], []int{0, 1, 2, 3}, 3)
qc, err := UnmarshalQuorumCert(honest)
if err != nil {
t.Fatalf("decode honest cert: %v", err)
}
qc.Position.CanonicalID = gap[0].id // was Empty (degraded to BlockID)
qc.Position.ParentCanonicalID = gap[0].parentID
qc.Position.BlockID = evil.id // <- unsigned
qc.Position.ParentID = evil.parentID // <- unsigned
forged, err := qc.MarshalBinary()
if err != nil {
t.Fatalf("marshal forged cert: %v", err)
}
// Sanity: the forged cert still verifies as a real 4-of-5 quorum cert.
if verr := rt.Transitive.verifyCert(qc, 0); verr != nil {
t.Fatalf("rebinding broke the signatures (attack does not apply): %v", verr)
}
err = rt.AcceptCatchupBlock(context.Background(), evil.bytes, forged)
if got := evil.AcceptCalled(); got != 0 {
t.Fatalf("EXPLOITED: an uncertified attacker block was applied to the VM "+
"(Accept called %d times, AcceptCatchupBlock err=%v)", got, err)
}
t.Logf("refused: %v", err)
}