papers/lux-distributed-systems-testing/sections/10-how-to-find-bugs-that-only-appear-under-.tex

\section{How to Find Bugs That Only Appear Under Network Partition}


This is the most practically useful section. Here is the methodology:

\subsection{Step 1: Identify the Invariant}

Before injecting faults, define what ``correct'' means. For consensus:
\begin{itemize}[nosep]
  \item \textbf{Safety}: No two honest nodes finalize conflicting blocks at the same height.
  \item \textbf{Liveness}: Every submitted transaction is eventually finalized.
  \item \textbf{Consistency}: After partition heals, all nodes converge to the same state.
\end{itemize}

\subsection{Step 2: Design the Partition}

Common partition topologies:
\begin{itemize}[nosep]
  \item \textbf{Minority isolation}: 1 node isolated from 4 (minority should not finalize).
  \item \textbf{Majority split}: 3 vs.\ 2 (majority should continue, minority should stall).
  \item \textbf{Ring partition}: Each node can talk to its neighbors but not distant nodes
    (tests message propagation assumptions).
  \item \textbf{Asymmetric}: A can send to B but not receive (tests connection
    directionality assumptions).
\end{itemize}

\subsection{Step 3: Run Workload During Partition}

Submit transactions to nodes on both sides of the partition. Record all operations with
precise timestamps.

\subsection{Step 4: Heal and Verify}

Remove the partition. Wait for convergence (how long? this is itself a property to test).
Verify:
\begin{itemize}[nosep]
  \item All transactions submitted to the majority partition are finalized.
  \item No conflicting blocks exist at any height.
  \item The minority partition caught up to the majority's state.
  \item No transactions were lost or duplicated.
\end{itemize}

\subsection{Step 5: Vary Timing}

The same partition with different timing (partition during proposal, during voting,
during commit) may expose different bugs. Run the scenario many times with randomized
timing.

\begin{lstlisting}[language=Go,caption={Randomized partition timing}]
func TestPartitionAtRandomPhase(t *testing.T) {
    for trial := 0; trial < 100; trial++ {
        cluster := StartCluster(5)

        // Random delay before partition
        delay := time.Duration(rand.Intn(3000)) * time.Millisecond
        time.Sleep(delay)

        // Partition
        nemesis := RandomPartition(5)
        nemesis.Inject(cluster)

        // Hold partition for random duration
        hold := time.Duration(1000+rand.Intn(5000)) * time.Millisecond
        time.Sleep(hold)

        // Heal
        nemesis.Recover(cluster)

        // Verify
        require.Eventually(t, func() bool {
            return cluster.AllConverged()
        }, 30*time.Second, 100*time.Millisecond,
            "trial %d: nodes did not converge after partition heal", trial)

        require.True(t, cluster.NoForks(),
            "trial %d: fork detected after partition", trial)

        cluster.Stop()
    }
}
\end{lstlisting}

%% ============================================================