mirror of
https://github.com/hanzoai/datastore.git
synced 2026-09-26 13:55:07 +00:00
test_lost_part_other_replica (and the sibling same_replica and mutation
cases) drain the replication queue with a tight loop of 10 one-second
polls after lost-part recovery. Recovery produces queue entries that fail
transiently: a GET_PART for the recovered part can hit
PART_IS_TEMPORARILY_LOCKED while the outdated duplicate is being cleaned
up, and the dependent MERGE_PARTS hits NO_REPLICA_HAS_PART until the
fetch lands. Each failure increments num_tries, and
ReplicatedMergeTreeQueue::getPostponeTimeMsForEntry parks the entry for
1 << num_tries ms, capped at max_postpone_time_for_failed_replicated_
{fetches,merges}_ms (default 60000). After ~14 retries the backoff
exceeds the 10s drain window, so the queue is still non-empty when the
loop ends and the test fails with "Still have something in replication
queue" (observed in CI with num_postponed=156 and a 29s backoff).
Disable the fetch and merge backoff in these tests by setting
max_postpone_time_for_failed_replicated_fetches_ms and
max_postpone_time_for_failed_replicated_merges_ms to 0, mirroring the
existing max_postpone_time_for_failed_mutations_ms=0 in
test_lost_part_mutation. Entries then retry immediately and the queue
drains within the window. test_lost_last_part is left unchanged: it
tolerates pending entries and waits up to 50s.
Co-Authored-By: Claude Opus 4.8 <[email protected]>