mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-29 07:34:31 +00:00
zeekay
ff05c1b547
Decode was measured stochastic vs llama greedy (an unstated control). Corrected greedy: Metal 0.72->0.92, ROCm 0.72->0.85, Vulkan 0.52->0.60, CUDA 0.84 (device sampler, tax~0). Adds the erratum, makes sampling parity a required protocol control, revises the abstract/ BW/model-size/conclusion, updates the roofline (Vulkan 120 vs 199 GB/s, measured 213 peak). The correction increases the decode standing and the paper's credibility.
13 lines
691 B
TeX
13 lines
691 B
TeX
\begin{tabular}{@{}llrrrrl@{}}
|
|
\toprule
|
|
backend & phase & $n$ & hanzo t/s & llama t/s & ratio & verdict \\
|
|
\midrule
|
|
Metal & prefill & 2048 & 1221.4$\pm$48.9 & 1223.8$\pm$23.6 & 1.00$\pm$0.04 & \textsc{parity} \\
|
|
Metal & prefill & 512 & 1411.0$\pm$7.5 & 1628.5$\pm$22.2 & 0.87$\pm$0.01 & \textsc{loss} \\
|
|
Metal & decode & 128 & 128.2$\pm$0.8 & 136.6$\pm$2.1 & 0.94$\pm$0.02 & \textsc{loss} \\
|
|
CUDA & prefill & 2048 & 3774.4$\pm$89.0 & 3592.5$\pm$79.2 & 1.05$\pm$0.03 & \textsc{win} \\
|
|
CUDA & prefill & 512 & 3857.6$\pm$336.4 & 3648.2$\pm$232.9 & 1.06$\pm$0.11 & \textsc{parity} \\
|
|
CUDA & decode & 128 & 47.3$\pm$3.9 & 53.7$\pm$0.9 & 0.88$\pm$0.07 & \textsc{loss} \\
|
|
\bottomrule
|
|
\end{tabular}
|