papers/evmgpu-benchmark/sections/16-replication.tex

\section{Replication: the committed kernel, re-measured}
\label{sec:replication}

The figures in Section~\ref{sec:production} were produced by a harness that was
never committed---\texttt{bench\_dex\_multi.cu} was an empty file in
\texttt{gpu-kernels}---so the configuration behind them is not recoverable from
the tree. This section records a re-run against the kernel that \emph{is}
committed, so that a number in this paper has a procedure attached to it.

\subsection{Method}

The harness drives \verb|luxgpu_dex_match_batch_kernel| directly: one thread per
book, over the same \verb|match_one_book| body that the single-order precompile
path and the CPU oracle share. The byte-parity gate
(\texttt{tools/kat/test\_dex\_oracle}) passes 34/34 on the same build, so the
kernel under measurement is the kernel whose output is checked against the
oracle. Each book is seeded with \texttt{depth} resting ask levels of quantity
10; each pass sends one crossing bid of quantity 1 per book.

Two properties of the workload decide whether such a rate means anything, and
both are reported alongside it.

\paragraph{Fills per taker.} A book holds $\texttt{depth} \times \texttt{qty}$
units. Every taker after those are gone matches nothing, and matching nothing is
fast: at \num{65536} books and depth 32, the measured rate rises by
$3.7\times$ once the books are exhausted, while fills fall to $0.00$ per taker.
A throughput figure with no fills under it is timing an empty loop, so the
harness prints the fill count beside the rate. Every row below carries
$1.000$ fills per taker.

\paragraph{Arena stride.} \verb|kMaxLevels| sizes the book arena regardless of
the depth a workload uses, and that size is the stride between adjacent threads.
At 1024 levels neighbouring threads sit \SI{128}{\kibi\byte} apart and each
thread's read becomes its own memory transaction; the rate then \emph{falls}
with scale, from \SI{41.9}{\mega\text{ord}/\second} at \num{4096} books to
\SI{6.6}{\mega\text{ord}/\second} at \num{131072}. Sized to the workload at 32
levels it \emph{rises} with scale, and a million books occupy
\SI{4.01}{\gibi\byte} rather than \SI{131}{\gibi\byte}. It is now a build-time
parameter (\verb|-DLUX_DEX_MAX_LEVELS|).

\subsection{Measurements}

Depth 32, 32 takers per book, 32-level arenas, fills $1.000$ per taker.

\begin{table}[h]
\centering
\begin{tabular}{llrrr}
\toprule
Host & Accelerator & Books & As found & Limb-loaded \\
\midrule
evo   & AMD Radeon 8060S (gfx1151, HIP) & \num{1048576} & \SI{17.9}{\mega\text{ord}/\second} & \SI{74.3}{\mega\text{ord}/\second} \\
evo   & AMD Radeon 8060S                & \num{262144}  & \SI{12.8}{\mega\text{ord}/\second} & \SI{61.6}{\mega\text{ord}/\second} \\
spark & NVIDIA GB10 (CUDA 13.0)         & \num{1048576} & \SI{36.0}{\mega\text{ord}/\second} & \SI{46.3}{\mega\text{ord}/\second} \\
spark & NVIDIA GB10                     & \num{65536}   & \SI{33.5}{\mega\text{ord}/\second} & \SI{46.6}{\mega\text{ord}/\second} \\
\bottomrule
\end{tabular}
\caption{Batched deterministic CLOB matcher, 2026-09-01. ``As found'' is the
committed kernel; ``limb-loaded'' is the same kernel after the
\texttt{uint256} helpers stopped walking their operands a byte at a time. The
byte-parity oracle passes 34/34 on both.}
\label{tab:replication-dex}
\end{table}

\subsection{Where the first factor of four went}

The matcher's \texttt{uint256} helpers read their operands one byte at a time,
and in the batched path those operands are in global memory one arena-stride
apart per thread. A single compare was 32 uncoalesced transactions and
\texttt{u256\_mul} was 128, rebuilding every 64-bit limb from eight shifts and an
or. Reading four limbs instead---the arena is 8-byte aligned and each 32-byte
field sits on a multiple of 8, and big-endian lexicographic order is numeric limb
order, so the results are unchanged---takes gfx1151 from
\SI{17.9}{\mega\text{ord}/\second} to \SI{74.3}{\mega\text{ord}/\second}.

Two alignment details carried most of the remainder. The calldata's two
\texttt{uint256} fields begin at byte 1 and byte 33, one past an 8-byte boundary,
so reading them in place fell back to byte loads and returned nothing for the
wide-load change; staging them into aligned locals once recovers it. The working
temporaries needed the same treatment.

The gain is uneven across vendors---$4.15\times$ on gfx1151, $1.29\times$ on
GB10---which is itself informative: the AMD part was paying far more for the
uncoalesced byte traffic than the Grace-Blackwell memory system was.

For scale on the same host: the C++ CPU engine (\texttt{luxcpp/dex}) sustains
\SI{65.9}{\mega\text{orders}/\second} across 32 threads on evo, and the
single-order GPU path costs \SI{0.074}{\milli\second} per call---launch-latency
bound, which is the measurement that makes determinism look expensive and is
the one Section~\ref{sec:accounting-dex-honesty} sets aside.

Companion primitives on evo, same build: TFHE programmable bootstrap
\SI{62.5}{PBS/\second}; BLS12-381 pairing \num{12.5} operations per second;
Poseidon2 \SI{12.15}{\mega\text{ops}/\second}; C-Chain conflict detection
\SI{24.26}{\nano\second} per transaction pair.

\subsection{The gap this leaves}

At the workload stated in Section~\ref{sec:production} the committed kernel
returns \SI{46.3}{\mega\text{ord}/\second} on GB10 against the
\SI{9.13}{\giga\text{ord}/\second} recorded there, and the two accelerators
disagree with the recorded figures by a consistent factor. Per-thread cost is
roughly flat in the number of books, which is the signature of a per-thread cost
that dominates rather than a bandwidth ceiling: the matcher's \verb|uint256|
arithmetic runs byte-wise over big-endian arrays, and VWAP is computed by Knuth
Algorithm D per fill.

The honest reading is that the committed kernel and the kernel behind
Section~\ref{sec:production} are not the same artifact, and the harness that
would settle it was not preserved. Until it is, Table~\ref{tab:replication-dex}
is the rate this repository can reproduce on demand, and the earlier figures
stand as recorded but unreproduced.