mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-29 07:34:31 +00:00
zeekay
ff05c1b547
Decode was measured stochastic vs llama greedy (an unstated control). Corrected greedy: Metal 0.72->0.92, ROCm 0.72->0.85, Vulkan 0.52->0.60, CUDA 0.84 (device sampler, tax~0). Adds the erratum, makes sampling parity a required protocol control, revises the abstract/ BW/model-size/conclusion, updates the roofline (Vulkan 120 vs 199 GB/s, measured 213 peak). The correction increases the decode standing and the paper's credibility.
27 lines
1.9 KiB
TeX
27 lines
1.9 KiB
TeX
\begin{tabular}{@{}llrrrrl@{}}
|
|
\toprule
|
|
backend & phase & $n$ & hanzo t/s & llama t/s & ratio & verdict \\
|
|
\midrule
|
|
ROCm & prefill & 2048 & 3506.1$\pm$47.0 & 4551.3$\pm$20.1 & 0.77$\pm$0.01 & \textsc{loss} \\
|
|
ROCm & prefill & 4096 & 2840.8$\pm$93.0 & 4103.3$\pm$16.0 & 0.69$\pm$0.02 & \textsc{loss} \\
|
|
ROCm & prefill & 500 & 3603.8$\pm$19.4 & 4678.0$\pm$186.3 & 0.77$\pm$0.03 & \textsc{loss} \\
|
|
ROCm & prefill & 512 & 3693.6$\pm$14.0 & 4742.5$\pm$214.0 & 0.78$\pm$0.04 & \textsc{loss} \\
|
|
ROCm & decode & 128 & 113.7$\pm$1.1 & 133.8$\pm$0.7 & 0.85$\pm$0.01 & \textsc{loss} \\
|
|
Vulkan & prefill & 2048 & 1475.0$\pm$37.6 & 4256.8$\pm$28.9 & 0.35$\pm$0.01 & \textsc{loss} \\
|
|
Vulkan & prefill & 4096 & 922.8$\pm$33.3 & 3850.3$\pm$84.0 & 0.24$\pm$0.01 & \textsc{loss} \\
|
|
Vulkan & prefill & 500 & 1098.9$\pm$241.4 & 4449.6$\pm$33.6 & 0.25$\pm$0.05 & \textsc{loss} \\
|
|
Vulkan & prefill & 512 & 3042.3$\pm$10.9 & 4547.7$\pm$49.4 & 0.67$\pm$0.01 & \textsc{loss} \\
|
|
Vulkan & decode & 128 & 93.9$\pm$1.7 & 155.4$\pm$0.1 & 0.60$\pm$0.01 & \textsc{loss} \\
|
|
CUDA & prefill & 2048 & 7285.6$\pm$745.2 & 7326.8$\pm$394.0 & 0.99$\pm$0.11 & \textsc{parity} \\
|
|
CUDA & prefill & 4096 & 7474.8$\pm$217.1 & 6871.3$\pm$125.0 & 1.09$\pm$0.04 & \textsc{win} \\
|
|
CUDA & prefill & 500 & 7961.3$\pm$228.7 & 7250.2$\pm$208.5 & 1.10$\pm$0.04 & \textsc{win} \\
|
|
CUDA & prefill & 512 & 7873.6$\pm$608.2 & 6190.0$\pm$1081.7 & 1.27$\pm$0.24 & \textsc{win} \\
|
|
CUDA & decode & 128 & 86.2$\pm$1.0 & 102.8$\pm$1.1 & 0.84$\pm$0.01 & \textsc{loss} \\
|
|
Metal & prefill & 2048 & 3301.5$\pm$4.3 & 3696.9$\pm$111.4 & 0.89$\pm$0.03 & \textsc{loss} \\
|
|
Metal & prefill & 4096 & 2767.3$\pm$113.6 & 2863.0$\pm$69.4 & 0.97$\pm$0.05 & \textsc{parity} \\
|
|
Metal & prefill & 500 & 3345.3$\pm$9.4 & 3988.9$\pm$14.8 & 0.84$\pm$0.00 & \textsc{loss} \\
|
|
Metal & prefill & 512 & 3386.0$\pm$11.6 & 4097.5$\pm$39.3 & 0.83$\pm$0.01 & \textsc{loss} \\
|
|
Metal & decode & 128 & 233.4$\pm$1.5 & 253.8$\pm$1.0 & 0.92$\pm$0.01 & \textsc{loss} \\
|
|
\bottomrule
|
|
\end{tabular}
|