TestReplicationRequestTimeoutCancels hung indefinitely, burning the whole
per-package timeout so no run of the suite could go green. Two defects, both in
the test, and the first turns the second into a hang instead of a failure.
failOnReplicationRequest ran require.Nil on the LAGGING NODE'S OWN send
goroutine, inside Comm.Send. require's FailNow calls runtime.Goexit, which kills
the goroutine it runs on — not the test. That goroutine is the node's message
pump, so once it died the node stopped draining its ingress channel, never saw
the finalization for the last round, and the test waited on a commit that could
no longer happen. A goroutine dump shows three pumps for four nodes; instrumenting
the exit prints the lagging node leaving with stopping=false, alone. assert
records through t.Errorf instead and leaves the pump alive.
The predicate was also stronger than the property. It rejected EVERY replication
request, but one is ordinary here: when a notarization arrives before the block
it refers to has finished verifying, the round is not in e.rounds yet, so the node
requests the block digest and the reply walks it forward through the replication
path — a request for round startSeq+1, after the tick, for work it has not done
yet. Cancellation itself was never broken: the timeout resend path logs nothing in
a failing run, and rounds and sequences 0..8 were all cleared before the tick.
The bound is now the highest cancelled id, which is what the test is named for.
Predates the fork — the predicate is byte-identical in the commit that added it,
and the hang reproduces on a pristine upstream checkout at that commit and at
every point tested since. Load-sensitive, not deterministic: it widens with the
verification-versus-notarization window, which is why one machine reads 3/3 and
another 7/20.
Measured here: 6/6 hangs before, 12/12 clean after, and the assertion keeps its
teeth — breaking the cancel path fails 3/3 in 0.13s with "0 is not greater than 8:
re-sent a cancelled replication request", where it used to hang for 30.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
zeekayandhanzo-dev
d5ebf8d37c