mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-22 17:20:17 +00:00
papers: kernel DSL migration + kernel-optimization catalog (incl. the flash-attention repetition bifurcation boundary) + cross-backend correctness finding
- hanzo-kernel-dsl: one-source #[kernel] DSL, 2 kernels drained to production, perf-gated migration - hanzo-kernel-optimization: cross-backend kept-levers + dead-ends catalog + the flash-attn P*V bifurcation boundary (fusion & f32-P*V mutually exclusive on tensor cores -> eager is optimal) - cross-backend-inference: the model-level correctness lesson (one numerically-vetted impl per op)
This commit is contained in:
Binary file not shown.
@@ -1,5 +1,5 @@
|
||||
\documentclass[11pt]{article}
|
||||
\usepackage{shared/hanzocover}
|
||||
\usepackage[margin=1in]{geometry}
|
||||
\usepackage{booktabs}
|
||||
\usepackage{amsmath}
|
||||
\usepackage{graphicx}
|
||||
@@ -8,17 +8,18 @@
|
||||
\hypersetup{colorlinks=true, linkcolor=blue, urlcolor=blue, citecolor=blue}
|
||||
\usepackage{enumitem}
|
||||
|
||||
\title{Edge LLM Inference Across Commodity Accelerators: Measured AMD, NVIDIA, and Apple Performance, and hanzo-engine at llama.cpp Decode Parity on Every Backend}
|
||||
\author{Hanzo Dev \\ \texttt{[email protected]} \\ \textit{Hanzo AI}}
|
||||
\date{June 2026}
|
||||
\title{\textbf{Edge LLM Inference Across Commodity Accelerators:\\
|
||||
Measured AMD, NVIDIA, and Apple Performance, and\\
|
||||
hanzo-engine at llama.cpp Decode Parity on Every Backend}}
|
||||
\author{Hanzo AI --- hanzo-engine technical white paper, 05}
|
||||
\date{Measurement pass: 2026-06-17}
|
||||
|
||||
\begin{document}
|
||||
|
||||
\hanzocoverpage
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
This paper reports \emph{measured} single-GPU inference performance for one model
|
||||
(\textbf{an 8B dense transformer at Q8\_0}) across three commodity edge accelerators --- AMD Ryzen AI Max+ 395
|
||||
(\textbf{Qwen3-8B-Q8\_0}) across three commodity edge accelerators --- AMD Ryzen AI Max+ 395
|
||||
``Strix Halo'' (Radeon 8060S iGPU, gfx1151), NVIDIA GB10 ``Grace Blackwell'' (sm\_121), and
|
||||
Apple M4 Max --- comparing \textbf{hanzo-engine} against \texttt{llama.cpp} on each, same model,
|
||||
same metric (512-token prefill throughput \texttt{pp512}; single-stream decode throughput).
|
||||
@@ -53,7 +54,7 @@ win one regime and lose the other; this paper shows exactly that.
|
||||
\end{itemize}
|
||||
|
||||
\section{Method}
|
||||
One model throughout: \textbf{an 8B dense transformer at Q8\_0} (36 layers, $n_\text{embd}$ 4096, $n_\text{ff}$ 12288,
|
||||
One model throughout: \textbf{Qwen3-8B-Q8\_0} (36 layers, $n_\text{embd}$ 4096, $n_\text{ff}$ 12288,
|
||||
32 query / 8 KV heads, head dim 128; $\approx$8.11 GiB resident). The 8-bit quant is the honest
|
||||
``no accuracy excuse'' point --- it is the largest of the common quants, so decode is
|
||||
bandwidth-bound and the comparison is purely about kernel/streaming efficiency, not about a
|
||||
@@ -76,7 +77,7 @@ cheaper format hiding a slower engine.
|
||||
\section{Results}
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{The 8B model at Q8\_0, single GPU. Prefill = \texttt{pp512} t/s; Decode = single-stream t/s.
|
||||
\caption{Qwen3-8B-Q8\_0, single GPU. Prefill = \texttt{pp512} t/s; Decode = single-stream t/s.
|
||||
AMD from paper~01 (2026-06-08); NVIDIA + Apple measured 2026-06-17.}
|
||||
\begin{tabular}{l rr c rr c}
|
||||
\toprule
|
||||
@@ -116,9 +117,25 @@ Metal (0.91$\times$) and largest on NVIDIA (0.76$\times$), where \texttt{llama.c
|
||||
is most heavily tuned. The prefill gap is real engineering work (matrix-core GEMM scheduling),
|
||||
not a rounding error --- the same honest conclusion paper~01 reached on AMD.
|
||||
|
||||
\paragraph{4. A cross-backend \emph{correctness} finding: attention kernel choice, not just speed.}
|
||||
Speed parity is not the only cross-backend variable --- numerical \emph{behaviour} is too, and it
|
||||
bites hardest where the kernels differ most. Qwen3-8B-Q4\_K\_M generates coherently on Metal and
|
||||
under an eager attention on CUDA, but \emph{collapses into token repetition} under CUDA
|
||||
flash-attention-2 (byte-identical weights, same sampler; only the attention kernel differs). We
|
||||
traced it to the reduced-precision $P\!\cdot\!V$ tensor-core matmul and, by an exhaustive
|
||||
proved-by-build precision ladder (bf16 $\to$ f16 $\to$ f32-class split-$P$, each verified on the
|
||||
running model), showed that raising precision only \emph{moves} the repetition attractor rather than
|
||||
clearing it: the fused low-precision path and a numerically-f32 path are mutually exclusive on tensor
|
||||
cores, so the stable attention \emph{is} the eager (CUDA-core, f32) kernel. The shipped fix routes
|
||||
bf16 CUDA attention to eager --- coherent in production, with the only cost being long-context
|
||||
prefill (short prompts and all decode are unaffected). The lesson generalizes the paper's thesis:
|
||||
a multi-backend engine must hold one \emph{numerically-vetted} implementation per operation, because
|
||||
``the same op, a different kernel per backend'' is a correctness hazard, not merely a perf one. (The
|
||||
full bisection is paper~06, \S\,``A correctness boundary''.)
|
||||
|
||||
\section{The edge/commodity angle: smaller models}
|
||||
For latency-sensitive on-device use the 4B tier is the sweet spot. Measured \texttt{llama.cpp}
|
||||
(2026-06-17), a 4B dense transformer at Q8\_0:
|
||||
(2026-06-17), Qwen3-4B-Q8\_0:
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\begin{tabular}{l rr}
|
||||
@@ -157,7 +174,7 @@ what is roadmap.
|
||||
\end{itemize}
|
||||
|
||||
\section{What this paper does \emph{not} claim}
|
||||
Single model (the 8B model at Q8\_0) at a single quant; single-stream decode (no concurrency/batching
|
||||
Single model (Qwen3-8B-Q8\_0) at a single quant; single-stream decode (no concurrency/batching
|
||||
sweep); \texttt{pp512}/128-token decode only (no long-context or KV-pressure regime); AMD numbers
|
||||
carried from paper~01 rather than re-measured on this pass; and \texttt{llama.cpp} taken as the
|
||||
stock build (not re-tuned per box). Accuracy is not evaluated here --- Q8\_0 is bit-faithful to
|
||||
@@ -212,7 +229,8 @@ native Windows + ROCm 7.1); NVIDIA GB10 (Grace-Blackwell, sm\_121, CUDA 13, 121
|
||||
aarch64); Apple M4 Max (137 GB unified, Metal / \texttt{BLAS,MTL}).\\
|
||||
\textbf{Software.} \texttt{llama.cpp} stock HIP/CUDA/Metal builds; hanzo-engine built \texttt{--release}
|
||||
per box (git \texttt{8c50abb32}). Measurement passes: AMD 2026-06-08 (paper~01), NVIDIA + Apple 2026-06-17.\\
|
||||
\textbf{Models.} an 8B dense transformer at Q8\_0 (8.11 GiB) and a 4B dense transformer at Q8\_0 (3.98 GiB), GGUF.\\
|
||||
\textbf{Models.} \texttt{Qwen3-8B-Q8\_0} (8.11 GiB) and \texttt{Qwen3-4B-Q8\_0} (3.98 GiB), GGUF, from the
|
||||
public Qwen3 releases.\\
|
||||
\textbf{Commands.} Baseline: \texttt{llama-bench -m <model>.gguf -ngl 99 -p 512 -n 128}. hanzo:
|
||||
\texttt{hanzo bench auto -m <dir> --format gguf -f <model>.gguf} (512-token prefill, 128-token decode,
|
||||
1 warmup + 3 timed iterations).\\
|
||||
|
||||
Binary file not shown.
@@ -0,0 +1,379 @@
|
||||
\documentclass[11pt]{article}
|
||||
\usepackage[margin=1in]{geometry}
|
||||
\usepackage{booktabs}
|
||||
\usepackage{amsmath}
|
||||
\usepackage{amssymb}
|
||||
\usepackage{graphicx}
|
||||
\usepackage{xcolor}
|
||||
\usepackage{hyperref}
|
||||
\hypersetup{colorlinks=true, linkcolor=blue, urlcolor=blue, citecolor=blue}
|
||||
\usepackage{enumitem}
|
||||
\usepackage{listings}
|
||||
\usepackage{array}
|
||||
|
||||
\definecolor{kw}{rgb}{0.13,0.29,0.53}
|
||||
\definecolor{cmt}{rgb}{0.40,0.40,0.40}
|
||||
\definecolor{str}{rgb}{0.16,0.50,0.16}
|
||||
\lstset{
|
||||
basicstyle=\ttfamily\footnotesize,
|
||||
keywordstyle=\color{kw}\bfseries,
|
||||
commentstyle=\color{cmt}\itshape,
|
||||
stringstyle=\color{str},
|
||||
showstringspaces=false,
|
||||
breaklines=true,
|
||||
columns=fullflexible,
|
||||
frame=single,
|
||||
framesep=4pt,
|
||||
}
|
||||
\lstdefinelanguage{Rust}{
|
||||
morekeywords={fn,let,mut,pub,struct,impl,trait,enum,match,if,else,for,while,loop,return,use,mod,crate,self,super,async,await,where,type,const,static,unsafe,ref,move,dyn,as,true,false},
|
||||
morecomment=[l]{//}, morecomment=[s]{/*}{*/}, morestring=[b]", sensitive=true,
|
||||
}
|
||||
|
||||
\newcommand{\x}{\ensuremath{\times}}
|
||||
\newcommand{\dpa}{\texttt{dp4a}}
|
||||
|
||||
\title{\textbf{One Source, Every Backend: A Kernel DSL Migration\\for the Hanzo ML Stack}}
|
||||
\author{Hanzo AI --- ML Systems}
|
||||
\date{Paper 07 --- companion to Paper 06 (\emph{When Does a Kernel Win Surface?})}
|
||||
|
||||
\begin{document}
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
Hanzo ML today carries \textbf{52{,}700 lines of hand-written GPU kernel code across four
|
||||
shading languages} --- CUDA C++, Metal Shading Language, HIP C++, and GLSL. The same
|
||||
$\sim$20 operations, specialized over $\sim$22 GGUF quantization formats, are re-implemented
|
||||
once per backend. That Cartesian product is not just a maintenance cost: it is where the
|
||||
correctness bugs breed, because \emph{one operation has four implementations and therefore four
|
||||
numeric behaviors}. This paper argues for collapsing the product into a sum via a single-source
|
||||
kernel DSL, presents a \emph{first-party facade} design (\texttt{hanzo-kernel}) over the CubeCL
|
||||
lowering engine, audits that the DSL can express every performance primitive our hand-tuned
|
||||
kernels rely on, and reports a working proof: the same Rust kernel source, compiled unchanged,
|
||||
runs numerically correct on CPU, Vulkan (AMD gfx1151), and Metal (Apple M4~Max). Crucially, the
|
||||
portability costs nothing: a tuned single-source \dpa{} matvec measures \textbf{196~GB/s on gfx1151,
|
||||
118\% of the hand-tuned kernel it would replace}, bit-exact --- the DSL reaches the same hardware
|
||||
integer-dot instruction and, with standard coalescing, \emph{beats} the hand-written code. The
|
||||
migration is incremental and perf-gated: a kernel is retired only when its DSL twin is both
|
||||
bit-exact and at least as fast, so the engine is faster-or-equal at every step. We further show the
|
||||
DSL plugs in as a \emph{code generator} --- its emitted SPIR-V is byte-compatible with the engine's
|
||||
existing dispatch --- so no second GPU runtime is introduced. CUDA and Metal build integration
|
||||
required resolving a toolchain issue (documented) but is not a fundamental blocker.
|
||||
\end{abstract}
|
||||
|
||||
\section{The problem: a hand-written Cartesian product}
|
||||
|
||||
Table~\ref{tab:dup} is the current kernel layer, measured. Four backends, four languages, the
|
||||
same arithmetic written four times. The bulk is the quantized matrix--vector and matrix--matrix
|
||||
family, repeated across the GGUF format zoo.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\begin{tabular}{llrr}
|
||||
\toprule
|
||||
\textbf{Backend} & \textbf{Language} & \textbf{Files} & \textbf{Lines} \\
|
||||
\midrule
|
||||
CUDA & CUDA C++ & 64 & 21{,}300 \\
|
||||
Metal & Metal Shading Language & 17 & 17{,}000 \\
|
||||
ROCm & HIP C++ & 19 & 7{,}300 \\
|
||||
Vulkan & GLSL $\rightarrow$ SPIR-V & 124 & 7{,}100 \\
|
||||
\midrule
|
||||
\textbf{Total} & 4 languages & 224 & \textbf{52{,}700} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{The hand-written kernel duplication, measured across the four backends.}
|
||||
\label{tab:dup}
|
||||
\end{table}
|
||||
|
||||
\subsection{The product is where bugs breed}
|
||||
|
||||
Three defects diagnosed during a single debugging session make the argument concretely. Each is a
|
||||
direct consequence of one operation having multiple independent implementations.
|
||||
|
||||
\begin{itemize}[leftmargin=1.4em]
|
||||
\item \textbf{The 8B repetition-collapse.} Qwen3-8B-Q4\_K\_M generates coherently on Metal
|
||||
(``The capital of France is \textbf{Paris}'') but collapses into a repeated-token degeneration
|
||||
on CUDA and Vulkan during long generation. The model file is byte-identical across all three
|
||||
boxes (SHA-256 verified); the fault is that flash attention's accumulated numeric behavior
|
||||
differs from Metal's, and only the CUDA/Vulkan implementations drift. \emph{Same operation, three
|
||||
implementations, three numeric behaviors.}
|
||||
\item \textbf{The Vulkan 8B garble.} A distinct large-$k$ decode defect that produced garbage on
|
||||
the Vulkan backend but nowhere else.
|
||||
\item \textbf{The flash \texttt{.contiguous()} defect.} A missing contiguity materialization in the
|
||||
CUDA flash-attention dispatch, silently correct at sequence length~1 (decode) and garbage at
|
||||
sequence length${>}1$ (prefill), fixed once but per-backend.
|
||||
\end{itemize}
|
||||
|
||||
None of these can occur when there is exactly one implementation. Uniform numeric behavior is a
|
||||
\emph{structural} property of a single-source kernel, not something achieved by testing four
|
||||
copies into agreement.
|
||||
|
||||
\section{The decomplect: three orthogonal axes}
|
||||
|
||||
The kernel code braids three independent concerns into one hand-multiplied product. Pulled apart,
|
||||
they compose, and each is written once:
|
||||
|
||||
\begin{center}
|
||||
\begin{tabular}{lll}
|
||||
\toprule
|
||||
\textbf{Axis} & \textbf{Count} & \textbf{What it is} \\
|
||||
\midrule
|
||||
Ops & $\sim$20 & the math: matvec, matmul, flash-attn, rmsnorm, rope, softmax \\
|
||||
Quant types & $\sim$22 & the bit layout: Q4\_K, Q6\_K, Q8\_0, the IQ codebooks \\
|
||||
Backends & 4 & the lowering: threads, memory, tensor cores \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{center}
|
||||
|
||||
The transformation is from a product to a sum:
|
||||
\[
|
||||
\underbrace{\text{backends} \x \text{types} \x \text{ops}}_{\text{hand-written today}}
|
||||
\;\longrightarrow\;
|
||||
\underbrace{\text{backends} + \text{types} + \text{ops}}_{\text{one source, lowered}}.
|
||||
\]
|
||||
|
||||
Crucially, two of the three axes are \emph{already} decomplected on ROCm. Its kernels are written
|
||||
as \texttt{qmatvec\_core<WTYPE>} --- one contraction, the quantization type injected as a
|
||||
compile-time template parameter. The DSL is exactly that pattern with the backend as the
|
||||
\emph{other} compile-time axis. We are generalizing a decomplect we already trust in production,
|
||||
not inventing an untested one.
|
||||
|
||||
\section{The engine and the facade}
|
||||
|
||||
We adopt CubeCL as the lowering engine and wrap it in a first-party facade crate,
|
||||
\texttt{hanzo-kernel}. Kernel source names only \texttt{hanzo\_kernel::*}; the string
|
||||
\texttt{cubecl} appears in exactly one line of one \texttt{Cargo.toml}. This is ``values, not
|
||||
places'': CubeCL is the implementation \emph{value}; \texttt{hanzo\_kernel} is the stable
|
||||
\emph{namespace} we build against. The engine can be upgraded --- or forked and vendored, should
|
||||
we ever need deep control --- without editing a single kernel.
|
||||
|
||||
\subsection{Capability audit: the DSL is a superset of what we hand-write}
|
||||
|
||||
The central risk is that a portable DSL cannot express the primitives our hand-tuned kernels
|
||||
depend on, forcing a performance downgrade. We audited the CubeCL frontend and IR directly. Every
|
||||
primitive is present (Table~\ref{tab:prims}).
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\begin{tabular}{lll}
|
||||
\toprule
|
||||
\textbf{Primitive} & \textbf{DSL surface} & \textbf{Lowers to} \\
|
||||
\midrule
|
||||
int8 4-way dot (\dpa) & \texttt{Line<i8>.dot} / IR \texttt{Arithmetic::Dot} & \texttt{dp4a} / \texttt{OpSDotAccSat} \\
|
||||
tensor cores & \texttt{cmma} (cooperative matrix) & WMMA / coopmat / simdgroup \\
|
||||
shared memory & \texttt{SharedMemory<E>} & \texttt{\_\_shared\_\_} / \texttt{shared} / threadgroup \\
|
||||
subgroup reduce/shuffle & \texttt{plane\_broadcast}, \texttt{plane\_shuffle\_xor}, \ldots & warp / subgroup / SIMD-group ops \\
|
||||
atomics, barriers & \texttt{Atomic<E>}, \texttt{Barrier} & native atomics / \texttt{\_\_syncthreads} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{CubeCL exposes every hand-tuned performance primitive; each lowers to the native
|
||||
instruction per backend. Verified by reading the CubeCL 0.10 source.}
|
||||
\label{tab:prims}
|
||||
\end{table}
|
||||
|
||||
Because the DSL emits the same instructions, nothing forces a downgrade. The DSL is a superset of
|
||||
what we express by hand.
|
||||
|
||||
\subsection{A kernel, written once}
|
||||
|
||||
Listing~\ref{lst:k} is a quantized matrix--vector kernel in the facade. There is no backend in it;
|
||||
the decode and the contraction are the arithmetic each backend hand-writes today.
|
||||
|
||||
\begin{lstlisting}[language=Rust,caption={A quant matvec in \texttt{hanzo-kernel} --- one source, every backend.},label={lst:k}]
|
||||
use hanzo_kernel::prelude::*;
|
||||
|
||||
#[cube(launch_unchecked)]
|
||||
pub fn matvec_q8<F: Float>(wd: &Array<F>, wq: &Array<i32>, x: &Array<F>,
|
||||
out: &mut Array<F>, #[comptime] k: usize) {
|
||||
let row = ABSOLUTE_POS;
|
||||
if row < out.len() {
|
||||
let nb = k / 32; let wbase = row * k; let dbase = row * nb;
|
||||
let mut acc = F::new(0.0);
|
||||
for i in 0..k {
|
||||
let d = wd[dbase + i / 32];
|
||||
acc += d * F::cast_from(wq[wbase + i]) * x[i];
|
||||
}
|
||||
out[row] = acc;
|
||||
}
|
||||
}
|
||||
\end{lstlisting}
|
||||
|
||||
\section{The proof: correct on three backends from one source}
|
||||
|
||||
The kernel above, compiled unchanged, was run against a pure-Rust oracle (\texttt{matvec\_q8\_ref})
|
||||
on three targets. Table~\ref{tab:proof} reports the maximum relative error and a kernel-only
|
||||
throughput figure ($4096\x4096$, weights represented as widened \texttt{i32}). The CPU backend
|
||||
matches bit-for-bit (in-order accumulation); the GPU backends differ only by f32 re-association
|
||||
noise, confirmed by a small-$k$ control whose error does not grow with $k$ --- the signature of
|
||||
per-element rounding, not a decode defect.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\begin{tabular}{llrr}
|
||||
\toprule
|
||||
\textbf{Backend} & \textbf{Hardware} & \textbf{max rel. err} & \textbf{kernel-only} \\
|
||||
\midrule
|
||||
CPU & (reference) & $0.0$ & --- \\
|
||||
Vulkan & AMD gfx1151 (Radeon 8060S) & $6.6\times10^{-4}$ & 19 GB/s \\
|
||||
Metal & Apple M4~Max & $1.8\times10^{-3}$ & 176 GB/s \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Same \texttt{matvec\_q8} source, three backends, all numerically correct. The throughput
|
||||
gap reflects hardware and driver, not the source: the untuned scalar kernel over \texttt{i32}-widened
|
||||
weights ($4\x$ the bytes of packed int8) is memory-bound, and Metal's higher bandwidth shows through.}
|
||||
\label{tab:proof}
|
||||
\end{table}
|
||||
|
||||
The throughput numbers are of the \emph{untuned scalar} kernel and are not the performance target;
|
||||
they establish correctness and portability. The performance gate is the \dpa{}-tuned kernel
|
||||
(Section~\ref{sec:migrate}).
|
||||
|
||||
\section{Migration: perf-gated, incremental, reversible}
|
||||
\label{sec:migrate}
|
||||
|
||||
The migration never risks a performance regression, because it is bench-gated at every step.
|
||||
|
||||
\begin{enumerate}[leftmargin=1.6em]
|
||||
\item \textbf{Coverage first.} The DSL fills the entire $22\,\text{types}\x4\,\text{backends}$
|
||||
matrix from one source. Many cells are currently missing or buggy; the DSL makes them uniform.
|
||||
\item \textbf{The gate.} A hand-tuned kernel is retired for its DSL twin \emph{only} when the twin
|
||||
is (a) bit-exact against the same oracle and (b) within performance noise on a kernel-only
|
||||
benchmark. The number decides, per quant type, per backend.
|
||||
\item \textbf{Peak stays.} Where a hand-tuned kernel still wins, it remains the specialized fast
|
||||
path and the DSL is the portable fallback for the backends that lack a tuned version. We only ever
|
||||
\emph{add} coverage and correctness.
|
||||
\item \textbf{Delete on parity.} As each DSL twin passes the gate, its four hand-written copies are
|
||||
deleted. The $52{,}700$ lines collapse toward op-specs plus type-specs, and we gain autotuning and
|
||||
new targets (WebGPU, DX12) for free.
|
||||
\end{enumerate}
|
||||
|
||||
Order of attack: the matvec/matmul family first --- it is the bulk of Table~\ref{tab:dup} and the
|
||||
home of the 8B collapse --- then attention, norms, and RoPE. When attention is a single
|
||||
implementation, the 8B repetition-collapse is not a bug to fix but a state that cannot be reached.
|
||||
|
||||
\subsection{The coverage matrix (grounded, not aspirational)}
|
||||
The migration surface is not uniform, and the DSL's value is highest exactly where the hand-written
|
||||
coverage is thinnest. The real state per backend, for the quant-matvec family:
|
||||
|
||||
\begin{center}\small
|
||||
\begin{tabular}{@{}lll@{}}
|
||||
\toprule
|
||||
Backend & Quant coverage & Structure \\
|
||||
\midrule
|
||||
ROCm & $\sim$25 types (Q/K/IQ/TQ/MXFP4) & \emph{decomplected}: one \texttt{qmatvec\_core<WTYPE>} core, type as a template param \\
|
||||
CUDA & per-type \texttt{mmq\_instance\_*.cu} & one hand-written kernel \emph{file per type} --- heavy duplication \\
|
||||
Vulkan& per-op/per-type \texttt{.comp} & the most sprawl (124 shaders); several per-type matvec variants \\
|
||||
Metal & 17 \texttt{.metal} files & its own dialect again \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{center}
|
||||
|
||||
ROCm already proves the target: \texttt{qmatvec\_core<WTYPE>} is one contraction with the quant type
|
||||
injected as a comptime template parameter. The DSL is precisely that decomplect with \texttt{BACKEND}
|
||||
as the second comptime axis --- so the migration is \emph{generalizing a pattern we already ship and
|
||||
trust}, not inventing one. A new quant type becomes one \texttt{\#[cube]} decode function, lowered to
|
||||
all four backends, instead of four hand-written kernels.
|
||||
|
||||
\subsection{The dispatch registry: how retirement is incremental}
|
||||
Retirement is never all-or-nothing. A per-$(\text{op},\text{type},\text{backend})$ registry routes
|
||||
each call to either the DSL kernel or the surviving hand-tuned one, keyed by the gate result. A
|
||||
quant type on a backend flips to the DSL the moment its twin passes bit-exactness and the perf gate;
|
||||
until then it keeps the hand-tuned kernel (or, for a cell that was simply \emph{missing}, gains DSL
|
||||
coverage immediately). The registry is the seam that lets the $52{,}700$-line product drain to a sum
|
||||
one measured cell at a time, with a working, faster-or-equal engine at every intermediate state.
|
||||
|
||||
\section{Integration architecture: the DSL is a code generator, not a runtime}
|
||||
\label{sec:integration}
|
||||
|
||||
The obvious fear is that adopting the DSL means running a second GPU runtime alongside the engine's
|
||||
existing one (ash-Vulkan, cudarc-CUDA, Metal), with a second device and cross-runtime data copies.
|
||||
It does not. The engine's Vulkan path is already just \texttt{include\_bytes!(<name>.spv)} $\to$
|
||||
\texttt{create\_shader\_module} $\to$ dispatch --- it runs \emph{SPIR-V bytes} and does not care where
|
||||
they came from. And CubeCL \emph{emits} exactly those bytes: compiling a \texttt{\#[kernel]} through
|
||||
\texttt{cubecl-spirv} yields a standard compute module. Dumped and inspected, the generated SPIR-V for
|
||||
the RMSNorm kernel is
|
||||
\begin{quote}\small\ttfamily
|
||||
OpEntryPoint GLCompute "rms\_norm\_f\_f32" \dots\\
|
||||
OpDecorate \%5 Binding 0 \quad OpDecorate \%8 Binding 1 \quad OpDecorate \%11 Binding 2
|
||||
\end{quote}
|
||||
--- a \texttt{GLCompute} entry point with buffers on descriptor bindings $0,1,2$: byte-for-byte the
|
||||
shape the engine's hand-written \texttt{.comp}$\to$\texttt{.spv} shaders already take. So the DSL plugs
|
||||
in as a \emph{code generator}: \texttt{\#[kernel]} $\to$ SPIR-V for Vulkan, CUDA source/PTX for CUDA
|
||||
(loaded by cudarc like the engine's own kernels), MSL for Metal --- and each backend's \emph{existing}
|
||||
dispatch/device/buffer machinery runs it \emph{unchanged}. There is no dual runtime, no data copy, and
|
||||
no backend rewrite; the $52{,}700$ lines drain by swapping kernel \emph{bytes}, not plumbing. The one
|
||||
bounded integration detail is aligning CubeCL's binding/push-constant layout with the engine's dispatch
|
||||
convention (both use the standard buffers-at-$0,1,2$ layout, so this is a mapping, not a redesign).
|
||||
|
||||
\section{Integration status and open work}
|
||||
|
||||
Honesty about what is proven versus what remains:
|
||||
|
||||
\begin{itemize}[leftmargin=1.4em]
|
||||
\item \textbf{Proven.} Concept and correctness on CPU, Vulkan, and Metal from one source;
|
||||
capability audit; the first-party facade; the perf-gated migration policy.
|
||||
\item \textbf{Toolchain resolved.} The CubeCL \texttt{cpu} feature transitively pulls
|
||||
\texttt{cubecl-cpu}, whose \texttt{tracel-llvm-bundler} build-dependency downloads a prebuilt
|
||||
LLVM/MLIR archive from GitHub at build time; that download failed on the CUDA host and the Metal
|
||||
host, and the \texttt{cpu} feature was the sole cause. The GPU backends do not need it:
|
||||
\texttt{cubecl-cuda} compiles through \texttt{cudarc}/NVRTC and \texttt{cubecl-wgpu} through
|
||||
naga. \textbf{Fix:} build GPU targets without the \texttt{cpu} feature (e.g.
|
||||
\texttt{-{}-features metal}); use the pure-Rust reference as the correctness oracle rather than
|
||||
the MLIR CPU runtime. With this, the Metal target built and ran clean.
|
||||
\item \textbf{Confirmed since.} (i) The full Q4\_K decode runs \emph{in-kernel} in the DSL (6-bit
|
||||
scale/min unpack + 4-bit extract via bit-ops), bit-exact on CPU and Vulkan. (ii) The \dpa{}
|
||||
\emph{hardware lowering} is verified at the binary level: \texttt{cubecl-spirv}'s
|
||||
\texttt{Arithmetic::Dot} emits \texttt{OpSDot}/\texttt{OpUDot}/\texttt{OpSUDot} with
|
||||
\texttt{Capability::DotProduct} (\texttt{SPV\_KHR\_integer\_dot\_product}), dumped and confirmed as a
|
||||
single \texttt{OpSDot} --- the same instruction the hand-tuned kernels use, $2.35\times$ a scalar
|
||||
mul-add on identical data.
|
||||
\item \textbf{The perf question is settled: the portable DSL \emph{matches and beats} hand-tuned.}
|
||||
A tuned \dpa{} Q8 matvec --- \texttt{Vector<i8,4>}$\to$\texttt{OpSDot}, block-per-row \emph{coalesced}
|
||||
256-byte reads, shared-memory tree reduction --- measures \textbf{196~GB/s} on evo (gfx1151, Vulkan,
|
||||
$8192^2$ cache-busting), \textbf{118\% of the $\sim$166~GB/s hand-tuned \dpa{} target} and $\sim$77\%
|
||||
of the $\sim$256~GB/s DRAM roofline, bit-exact against the CPU oracle. The naive one-thread-per-row
|
||||
kernel that started at $\sim$17~GB/s was memory-\emph{uncoalesced}, not DSL-limited; the fix was two
|
||||
standard decomplected steps (coalescing + reduction), both expressed in the DSL. The migration does
|
||||
not trade performance for portability.
|
||||
\item \textbf{Cross-backend, confirmed on real hardware.} The op library now spans matvec (\dpa{}),
|
||||
Q4\_K, \texttt{rms\_norm}, \texttt{layer\_norm}, \texttt{rope} (both conventions), and \texttt{sdpa}
|
||||
attention (online-softmax) --- \emph{all bit-exact from one source on both Vulkan (gfx1151) and Metal
|
||||
(M4~Max)}: e.g. matvec 196~GB/s Vulkan / 206~GB/s Metal, \texttt{rms\_norm}/\texttt{rope} to $\sim\!10^{-6}$,
|
||||
\texttt{sdpa} to $\sim\!10^{-4}$. \texttt{cubecl-cuda} additionally builds clean on the Blackwell (GB10)
|
||||
host. The \texttt{sdpa} single-source is the structural cure for the repetition-collapse: one attention
|
||||
implementation means the flash-vs-eager-vs-Metal numeric fork cannot occur.
|
||||
\item \textbf{Integration, landed --- and kernels retired in production.} The drop-in dispatch is no
|
||||
longer hypothetical. Two hand-written Vulkan kernels have been \emph{drained}: \texttt{rms\_norm} (the
|
||||
DSL block-per-row kernel, \textbf{$10.6\times$} the naive hand-written \texttt{.comp} it replaced) and
|
||||
\texttt{softmax} (351~GB/s), both now the production kernel with their \texttt{.comp} \emph{deleted} on
|
||||
\texttt{main}. Each dispatches through the engine's real Vulkan pipeline bit-exact; a reusable emitter
|
||||
(\texttt{tools/dsl/spv\_to\_ml.sh}) automates the two codegen steps (entry $\to$ \texttt{main}, strip the
|
||||
unused info interface), and a matching \texttt{MetalDevice::compile\_msl} primitive runs a DSL kernel
|
||||
through the engine's \emph{real Metal} pipeline bit-exact ($2.35\mathrm{e}{-}7$). The migration is a
|
||||
codegen loop, proven on two backends.
|
||||
\item \textbf{The discipline that matters: retire the naive, keep the optimal.} The gate did its job in
|
||||
both directions. \texttt{rms\_norm}/\texttt{softmax} were \emph{naive} hand-written kernels, so the DSL
|
||||
drained them for a large win. But the Vulkan quant matvec, Metal's \texttt{simd\_sum} reductions, and the
|
||||
fused \texttt{add\_rmsnorm}/\texttt{rope\_norm} are \emph{already optimal}: agents built bit-exact DSL
|
||||
twins, measured, and \emph{correctly declined to swap} --- a uniformity-only change with zero speed and
|
||||
real regression risk is not a migration, it is churn. ``One source'' is the goal; a kernel is retired
|
||||
only when the DSL twin is bit-exact \emph{and} at least as fast.
|
||||
\item \textbf{Open (bounded, non-fundamental).} A \emph{tuned} tensor-core (\texttt{cmma}) attention and
|
||||
GEMM to reach flash-attention prefill parity --- the DSL \texttt{cmma} frontend is proven on Blackwell
|
||||
($1.18\mathrm{e}{-}7$), but the tuned kernels the ecosystem ships (\texttt{cubecl-matmul}/%
|
||||
\texttt{-attention}) are now internal to Burn, so this is a dedicated hand-roll cycle. Q6\_K decode;
|
||||
the scalar-buffer bind for fully-dynamic shapes; and wiring the production Metal/CUDA dispatch to more
|
||||
DSL kernels. All engineering, none fundamental.
|
||||
\end{itemize}
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
The kernel layer is a Cartesian product of backends, quant types, and operations, hand-written in
|
||||
four languages, and it is the breeding ground for the exact class of divergence bugs we spend the
|
||||
most time chasing. A single-source DSL turns the product into a sum and makes those bugs
|
||||
structurally impossible while preserving --- via a perf-gated migration and a superset of
|
||||
primitives --- every hand-tuned kernel that still wins. The scariest unknowns are now retired: the
|
||||
DSL expresses what we need, and the same source runs correct on three backends. What remains is a
|
||||
disciplined, reversible migration.
|
||||
|
||||
\end{document}
|
||||
Binary file not shown.
@@ -0,0 +1,608 @@
|
||||
\documentclass[11pt]{article}
|
||||
\usepackage[margin=1in]{geometry}
|
||||
\usepackage{booktabs}
|
||||
\usepackage{amsmath}
|
||||
\usepackage{amssymb}
|
||||
\usepackage{graphicx}
|
||||
\usepackage{xcolor}
|
||||
\usepackage{hyperref}
|
||||
\hypersetup{colorlinks=true, linkcolor=blue, urlcolor=blue, citecolor=blue}
|
||||
\usepackage{enumitem}
|
||||
\usepackage{listings}
|
||||
\usepackage{array}
|
||||
|
||||
\definecolor{kw}{rgb}{0.13,0.29,0.53}
|
||||
\definecolor{cmt}{rgb}{0.40,0.40,0.40}
|
||||
\definecolor{str}{rgb}{0.16,0.50,0.16}
|
||||
\lstset{
|
||||
basicstyle=\ttfamily\footnotesize,
|
||||
keywordstyle=\color{kw}\bfseries,
|
||||
commentstyle=\color{cmt}\itshape,
|
||||
stringstyle=\color{str},
|
||||
showstringspaces=false,
|
||||
breaklines=true,
|
||||
columns=fullflexible,
|
||||
frame=single,
|
||||
framesep=4pt,
|
||||
}
|
||||
\lstdefinelanguage{Rust}{
|
||||
morekeywords={fn,let,mut,pub,struct,impl,trait,enum,match,if,else,for,while,loop,return,use,mod,crate,self,super,async,await,where,type,const,static,unsafe,ref,move,dyn,as,true,false},
|
||||
morecomment=[l]{//}, morecomment=[s]{/*}{*/}, morestring=[b]", sensitive=true,
|
||||
}
|
||||
\lstdefinelanguage{CUDAC}{
|
||||
morekeywords={int,float,half,char,void,for,if,else,return,const,struct,template,typename,constexpr,__global__,__device__,__constant__,__shared__,if,unsigned,auto,using},
|
||||
morecomment=[l]{//}, morecomment=[s]{/*}{*/}, morestring=[b]", sensitive=true,
|
||||
}
|
||||
|
||||
\newcommand{\x}{\ensuremath{\times}}
|
||||
\newcommand{\dpa}{\texttt{dp4a}}
|
||||
|
||||
\title{\textbf{When Does a Kernel Win Surface? A Cross-Backend Catalog of\\
|
||||
Proven Levers, Measured Dead-Ends, and a Bottleneck Taxonomy for\\
|
||||
Quantized LLM Decode and Prefill}}
|
||||
\author{Hanzo AI --- hanzo-engine / hanzo-ml technical white paper, 06}
|
||||
\date{Consolidation pass: kernel work measured 2026-06}
|
||||
|
||||
\begin{document}
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
Companion paper~05 established the \emph{model-level} result: hanzo-engine reaches
|
||||
\texttt{llama.cpp} decode parity (0.98--1.04\x) and a bounded prefill gap (0.76--0.91\x) on
|
||||
AMD, NVIDIA, and Apple from one multi-backend build. This paper documents the
|
||||
\emph{kernel-level} engineering underneath that result --- the optimizations we wrote,
|
||||
measured, and \emph{kept} in hanzo-ml/hanzo-engine, and, just as importantly, the ones we
|
||||
wrote, measured, and \emph{reverted}. We organize roughly thirty controlled experiments
|
||||
(ROCm/gfx1151, CUDA/GB10, Vulkan/gfx1151, Metal/M4~Max) into a four-regime
|
||||
\textbf{bottleneck taxonomy} --- bandwidth-roofline decode, compute-bound prefill,
|
||||
host/launch-bound decode, and GPU-busy-bound decode --- and show that the taxonomy
|
||||
\emph{predicts} whether a given kernel win surfaces end-to-end. The strongest evidence for the
|
||||
taxonomy is that the \emph{same} optimization is a real lever in one (platform, regime) and a
|
||||
measured non-lever in another: int8 \dpa{} lifts ROCm i-quant decode 2.27\x{} (compute-bound) but
|
||||
is end-to-end flat on GB10 (host-bound); a fused router gives ROCm $+13\%$ but CUDA $0\%$ (no
|
||||
expensive sort to eliminate); a fused-coopmat tensor-core prefill is correct but $3.8\x$ slower
|
||||
than int8-ALU \dpa{} on gfx1151 (decode-bound). The governing rule, stated once and tested
|
||||
repeatedly: \emph{a fusion or kernel change helps end-to-end only if it eliminates work in the
|
||||
binding resource; cutting launches or FLOPs in a non-binding resource is flat at best and
|
||||
negative if it injects redundant work.} Every win we report is gated bit-exact against a CPU
|
||||
reference; every A/B is device-map-controlled or kernel-isolated. We close with the one frontier
|
||||
this program has fully characterized but not yet closed --- Vulkan \emph{decode}, which we show
|
||||
by profiling is op-count-serialization-bound (1336 dispatches / 903 global barriers / 143~ms
|
||||
GPU fence-wait per token, with CPU recording effectively free) --- and the fusion roadmap that
|
||||
the ROCm path already validates.
|
||||
\end{abstract}
|
||||
|
||||
\tableofcontents
|
||||
|
||||
\section{Introduction: one engine, every backend, and a measurement question}
|
||||
hanzo-engine runs CUDA, ROCm, Metal, Vulkan, and CPU behind one model loader; hanzo-ml is the
|
||||
tensor/kernel layer beneath it. The portability thesis is only credible if the \emph{same} code
|
||||
is competitive with the hand-tuned, single-vendor baseline (\texttt{llama.cpp}) on each backend.
|
||||
Paper~05 showed it is, at the model level. The engineering question this paper answers is the one
|
||||
that actually consumes the work: \textbf{given a kernel that is provably faster in isolation, will
|
||||
it make the end-to-end token rate go up?}
|
||||
|
||||
The naive assumption --- faster kernel $\Rightarrow$ faster model --- is wrong often enough to be
|
||||
the central hazard of inference optimization. A 9\x{} matmul can be flat end-to-end; a $5\x$ i-quant
|
||||
kernel can fully surface on one GPU and vanish on another; cutting half the launches can lose
|
||||
$22\%$. The difference is always \emph{which resource is binding}. We therefore present the work
|
||||
not as a list of tricks but as a taxonomy that tells you, before you write the kernel, whether the
|
||||
win can possibly surface --- and we validate it on the cases where it predicted ``no'' and we
|
||||
measured ``no.''
|
||||
|
||||
\paragraph{What this paper contributes.}
|
||||
\begin{enumerate}[noitemsep]
|
||||
\item A \textbf{bottleneck taxonomy} (\S\ref{sec:taxonomy}) for single-stream quantized
|
||||
inference, with a falsifiable surfacing rule.
|
||||
\item A \textbf{catalog of kept levers} (\S\ref{sec:rocm}--\ref{sec:vulkan}) per backend, each
|
||||
with its bit-exact gate and a device-map-controlled or kernel-isolated A/B.
|
||||
\item A \textbf{catalog of measured dead-ends} (\S\ref{sec:deadends}) --- the negatives are the
|
||||
contribution: each one is explained by, and so confirms, the taxonomy.
|
||||
\item The \textbf{Vulkan-decode frontier} (\S\ref{sec:frontier}): a profile-backed diagnosis and
|
||||
the fusion roadmap that ROCm has already walked.
|
||||
\end{enumerate}
|
||||
|
||||
\section{Architecture: three orthogonal axes and a bit-exact invariant}
|
||||
\label{sec:arch}
|
||||
The kernel surface is a product of three axes that we keep deliberately orthogonal, because
|
||||
collapsing them is what produces an unmaintainable $N\times M\times K$ explosion of hand-written
|
||||
kernels:
|
||||
\begin{itemize}[noitemsep]
|
||||
\item \textbf{Quant format} (the \emph{value}): the GGUF block layout --- how 1--8 bits/weight
|
||||
plus per-block scales reconstruct a float. 22 families are wired
|
||||
(Q8\_0/Q4\_0/Q4\_K/Q5\_K/Q6\_K/Q2\_K/Q3\_K, the IQ1/IQ2/IQ3 codebook quants, the TQ
|
||||
ternary quants, etc.).
|
||||
\item \textbf{Contraction} (the \emph{operation}): mat-vec (decode, $M{=}1$), GEMM (prefill,
|
||||
$M{>}1$), and their indexed-MoE twins (gather/scatter over routed experts).
|
||||
\item \textbf{Backend} (the \emph{functor}): how a contraction lowers to ROCm/CUDA/Metal/Vulkan
|
||||
instructions and memory.
|
||||
\end{itemize}
|
||||
On ROCm this is already decomplected to one generic contraction with the type injected as a
|
||||
trait: \texttt{qmatvec\_core<WTYPE,XT>} (decode) and \texttt{qmmq\_core<WTYPE,MOE,NWAVE\_M>}
|
||||
(prefill), each instantiated per format, with the MoE variant differing only by an
|
||||
\texttt{if constexpr (MOE)} gather/scatter so the dense path codegens byte-identically. The same
|
||||
22 formats decode through one core; the int8-\dpa{} fast path is a second core selected by a single
|
||||
trait predicate \texttt{RocmQuantType::dp4a\_active()}. The forward graph never branches on
|
||||
``is this Q4\_K'' --- dispatch is by kernel-name string, so adding a format is additive.
|
||||
|
||||
\paragraph{The invariant that makes aggressive kernels safe.} Every kernel --- every \dpa{}
|
||||
reinterpretation, every fusion, every tiling --- is gated against a CPU \texttt{to\_float} oracle
|
||||
to \texttt{nbad=0} (bit-exact, modulo documented f32 reorder / f16-rounding tolerances). This is
|
||||
not a test suite bolted on after; it is the precondition for shipping a kernel at all. The reason
|
||||
the catalog below can be this aggressive (reinterpreting a 6-bit asymmetric quant as signed int8,
|
||||
folding a residual-add into an RMS reduction, quantizing activations to int8 mid-GEMM) is that
|
||||
each transform is proven equivalent before it is allowed to matter for performance. The oracles
|
||||
are named inline (\texttt{qmatvec\_unified\_numeric}, \texttt{moe\_qmmq\_numeric},
|
||||
\texttt{add\_rmsnorm\_numeric}, \texttt{qk\_rms\_norm\_rope\_numeric}, \texttt{moe\_route\_numeric},
|
||||
\dots) and are the reusable gate for every follow-up.
|
||||
|
||||
\section{A bottleneck taxonomy for single-stream quantized inference}
|
||||
\label{sec:taxonomy}
|
||||
Single-stream decode and prefill bind on different resources, and within decode the binding
|
||||
resource differs by platform and by whether HIP/CUDA graphs are active. We name four regimes.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{The four regimes. ``Surfacing condition'' is what a kernel change must do to move the
|
||||
end-to-end token rate.}
|
||||
\small
|
||||
\begin{tabular}{>{\raggedright\arraybackslash}p{2.8cm} >{\raggedright\arraybackslash}p{4.3cm} >{\raggedright\arraybackslash}p{6.0cm}}
|
||||
\toprule
|
||||
Regime & Binding resource & Surfacing condition \\
|
||||
\midrule
|
||||
\textbf{Bandwidth-roofline decode} & weight bytes / memory BW & Only fewer weight bytes moved (a smaller quant). Kernel arithmetic changes are flat once at roofline. \\
|
||||
\textbf{Compute-bound prefill} & matrix/tensor-core FLOPs & Higher effective FLOP/s: better tiling, tensor cores, higher M-tile fill. \\
|
||||
\textbf{Host/launch-bound decode} (eager, or graph-less) & per-token CPU dispatch / kernel launch & Fewer launches, or graph capture that hides launch latency. GPU-time cuts are flat. \\
|
||||
\textbf{GPU-busy-bound decode} (graphs on) & sum of small-kernel GPU time & Eliminating GPU work (a true fusion). Repackaging the same work is flat. \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\paragraph{The surfacing rule (falsifiable).}
|
||||
\begin{quote}\itshape
|
||||
A kernel change improves the end-to-end token rate only if it reduces consumption of the
|
||||
\emph{binding} resource. Reducing a non-binding resource is flat; reducing a non-binding resource
|
||||
\emph{while adding} work to the binding one is negative.
|
||||
\end{quote}
|
||||
Three corollaries drive the rest of the paper. (i) On a bandwidth-roofline decode, the matvec is
|
||||
already at the memory ceiling, so faster arithmetic is flat --- the only decode lever is reducing
|
||||
\emph{kernel count} (in the launch- or busy-bound sub-regimes), not making any one kernel faster.
|
||||
(ii) ``Fewer launches'' helps only in the launch/busy-bound regimes, and only if the launch you
|
||||
removed was not already free (under graphs it is). (iii) The same kernel can flip regime across
|
||||
platforms: a GPU with more launch overhead relative to memory bandwidth is launch-bound where
|
||||
another is bandwidth-bound, so a launch-cut surfaces on one and not the other.
|
||||
|
||||
\section{Kept levers: ROCm (gfx1151, Strix Halo)}
|
||||
\label{sec:rocm}
|
||||
ROCm is the most mature backend and the one where the taxonomy was first forced on us. All decode
|
||||
A/Bs use the device-map pin \texttt{-n 0:48} (the auto-mapper otherwise mis-reads unified memory
|
||||
and offloads $\sim 1/3$ of layers to CPU, which silently corrupts the comparison --- a measurement
|
||||
hazard worth more than any single lever).
|
||||
|
||||
\subsection{Unified \dpa{} decode cores (the K-quant and i-quant families)}
|
||||
The decode matvec for K-quants reinterprets the block as int8 and uses the RDNA3.5 dot-product
|
||||
builtin (\texttt{\_\_builtin\_amdgcn\_sudot4}; the plain \texttt{sdot4} does not compile on gfx1151,
|
||||
which lacks the dot1-insts feature). The win is per-format and depends entirely on the regime:
|
||||
|
||||
\begin{itemize}[noitemsep]
|
||||
\item \textbf{Q6\_K \dpa{}}: a $4.9\x$ \emph{kernel} win (260$\to$53\,$\mu$s per call) by reading
|
||||
the 16-element scale group as four packed int reads and reconstructing 4 weights/int, one
|
||||
branch per group not per element. The q6 centering ($q-32$) is handled as a bias term
|
||||
$\sum(q-32)u = \dpa(q,u) - 32\,\dpa(\mathbf{1},u)$ because gfx1151 has no per-byte
|
||||
\texttt{\_\_vsubss4}. Total GPU kernel time $-48\%$. \textbf{Model-level: $+17\%$}
|
||||
($28.2\to33.0$ T/s) --- it surfaces because eager MoE decode here is partly compute-bound
|
||||
on the slow Q6\_K (output.weight + attn\_v + ffn\_down experts $=37.7\%$ of GPU work).
|
||||
\item \textbf{IQ codebook \dpa{} (IQ2\_XXS/XS/S, IQ3\_XXS/S, IQ1\_S/M)}: $2.27\x$ \emph{end-to-end}
|
||||
($30.4\to 67.7$ T/s on Qwen3-30B-A3B-IQ3\_XXS), \emph{beating} \texttt{llama.cpp}-HIP
|
||||
(67.6). The grid coords land in int8 and \dpa{} against the once-quantized activation; signs
|
||||
without \texttt{\_\_vsub4} use $\sum(-1)^s g\,u = \dpa(g,u) - 2\,\dpa(g\,\&\,\text{mask}(s),u)$.
|
||||
Unlike Q6\_K, this win fully surfaces \emph{and} is graph-flat --- i-quant decode was
|
||||
\emph{compute-bound} (per-element f32 dequant + MAC), so the kernel \emph{was} the wall.
|
||||
This is the textbook ``compute-bound decode'' case of the taxonomy.
|
||||
\end{itemize}
|
||||
|
||||
\noindent The contrast between Q6\_K (kernel $4.9\x$, model $+17\%$) and IQ3 (kernel $2.27\x$, model
|
||||
$2.27\x$) is the taxonomy in one comparison: the i-quant decode was compute-bound so the kernel win
|
||||
\emph{is} the model win; the K-quant decode is mostly bandwidth-bound so only its compute-bound
|
||||
fraction surfaces.
|
||||
|
||||
\subsection{Capture-clean strided ops $\to$ HIP-graph decode}
|
||||
The single highest-leverage ROCm change was not a matmul. Decode under HIP graphs hides per-launch
|
||||
host latency, but capture kept failing on the first strided op: each \texttt{Map/Reduce/cast/copy}
|
||||
uploaded its tiny dims/strides metadata array host$\to$device every op every token, which is an
|
||||
implicit capture violation (HIP 906). The fix rides the metadata \emph{inline by value} in the
|
||||
kernel param block (\texttt{\#[repr(C)] DimsStrides \{ v: [usize; 32] \}}, 256~B), so zero
|
||||
\texttt{clone\_htod} fires on the strided path. This both enabled graph capture \emph{and}, by
|
||||
removing a blocking H2D from the eager hot path, lifted eager decode from $\sim$11 to $\sim$16--30
|
||||
T/s on its own. It is the enabling lever beneath every launch-bound decode win.
|
||||
|
||||
\subsection{Indexed-MoE decode without a host round-trip}
|
||||
MoE decode batches all routed slots into one launch over \texttt{grid.y}; the expert id is read
|
||||
\emph{on device} (the kernel takes \texttt{ids: \&RocmStorage}, not \texttt{\&[u32]}), so no
|
||||
\texttt{to\_vec1}/\texttt{copy\_from\_host} round-trip fires per token. This eliminated the
|
||||
per-expert host launch loop for mixed-precision GGUF (Qwen3-30B-A3B has Q6\_K ffn\_down experts
|
||||
inside a Q4\_K model), $10.22\to12.07$ T/s.
|
||||
|
||||
\subsection{The MoE graph bug: a position-refresh root-cause, not a buffer bug}
|
||||
MoE decode under graphs produced garbage from $\sim$token~12 (``Let. Let. 1 1 1''). The
|
||||
prior hypothesis (recycled routing buffer) was disproven by an env-gated replay-vs-eager
|
||||
\texttt{max\_abs\_diff} probe: dense paths diffed $0.0$, MoE diffed $4$--$16$ from token~0, and
|
||||
bisecting the MoE forward (even forcing the experts to identity) still diverged --- proving the
|
||||
divergence was in the \emph{shared attention path}. The MoE model baked the decode RoPE position
|
||||
from the host offset at capture and never refreshed it, so every replayed token rotated at the
|
||||
frozen warmup position. The dense path already read RoPE off a stable device tensor
|
||||
(\texttt{metadata.rope\_positions}) refreshed in place each token; the MoE model simply never got
|
||||
that fix. Threading \texttt{positions: \&Tensor} through \texttt{forward\_attn} made MoE graphs
|
||||
replay bit-identically ($340$-token coherent, $\times 3$). The lesson is methodological: a
|
||||
replay-vs-eager numeric probe localizes a graph bug faster than any amount of staring at buffers.
|
||||
|
||||
\subsection{Fused indexed-MoE prefill GEMM: the $2.58\x$ prefill lever}
|
||||
MoE prefill was the \emph{entire} deficit versus \texttt{llama.cpp}-HIP (0.34\x{} while dense was
|
||||
already 1.12\x). Root cause (\texttt{rocprofv3}): routed prefill went through the per-\emph{slot}
|
||||
decode matvec, so every routed token re-read its expert's whole weight and ran \emph{no} matrix
|
||||
cores --- $70.7\%$ of GPU time on MoE matvecs. The fix is an expert-grouped WMMA GEMM
|
||||
(\texttt{qmmq\_core<WTYPE,true>}, the same 16-warp/128$\times$128 int8-WMMA prefill machine with
|
||||
\texttt{if constexpr (MOE)} gather/scatter): tokens are counting-sorted by expert host-side
|
||||
(prefill is never captured, so the ids DtoH is free) into $\le$128-row tiles, and each block stages
|
||||
\emph{one} expert's weight and amortizes it over all that expert's tokens (llama's
|
||||
\texttt{mul\_mat\_id}). \textbf{Result: $366\to944.7$ T/s ($2.58\x$), $0.89\x$ of llama};
|
||||
GPU MoE-matmul time $-78\%$. The A/B (\texttt{HANZO\_MOE\_QMMQ\_FALLBACK=1} $\Rightarrow$ 365.8)
|
||||
confirms the kernel is the \emph{sole} lever.
|
||||
|
||||
\paragraph{Dynamic M-tile (the bandwidth/compute balance point).} At $\sim$64 tokens/expert a
|
||||
128-row WMMA tile is half-empty. Parametrizing the core on the M-wave count
|
||||
(\texttt{qmmq\_core<WTYPE,MOE,NWAVE\_M>}, TILE\_M 128/64/32 with thread-count-agnostic staging
|
||||
loops so the dense path stays byte-identical) and picking TILE\_M by routed fill lifts M-fill
|
||||
$49.9\to68.2\%$ but throughput only $+2.4\%$ --- because the full-prefill MoE GEMM is
|
||||
\emph{weight-bandwidth-bound}, not WMMA-compute-bound, \emph{proven} by the A/B: TILE\_M=32 (even
|
||||
less empty WMMA work) is \emph{slower} because it re-reads each expert weight in two tiles. TILE\_M=64
|
||||
(one tile per $\sim$64-token expert, minimal weight re-reads) is the balance point. The residual gap
|
||||
to llama is irreducible expert-weight fetch, not tile fill. This is a clean instance of the
|
||||
surfacing rule: more compute headroom is flat when bandwidth binds.
|
||||
|
||||
\subsection{Fused MoE routing: \texttt{moe\_route} ($+13\%$)}
|
||||
One kernel (one block/token: shared-mem softmax $\to$ top-$k$ $\to$ normalize) replaces the
|
||||
softmax$\to$sort$\to$narrow$\to$sum$\to$div chain ($\sim$6 launches/layer $\times 48 \approx 240$
|
||||
fewer kernels/token). \textbf{$52.9\to59.9$ T/s ($+13\%$), $0.90\x$ llama}; long-context run-loop
|
||||
$+15\%$. This is the launch/busy-bound decode lever the bandwidth-wall analysis pointed to, and it
|
||||
surfaces on ROCm precisely because the amdgcn \texttt{gpu-sort} it eliminates is disproportionately
|
||||
expensive (a fact that, foreshadowing \S\ref{sec:deadends}, makes the \emph{same} fusion flat on
|
||||
CUDA).
|
||||
|
||||
\subsection{Decode op-fusion that eliminates work ($+2.8\%$ each, stacked)}
|
||||
\texttt{rocprofv3} of the shipped decode showed it is \textbf{GPU-busy-bound} under graphs
|
||||
($12.17$~ms busy $=94.5\%$ of the $12.87$~ms wall), with non-matvec busy time spread across
|
||||
\texttt{quantize\_q8\_1} (378\,$\mu$s), \texttt{rmsnorm} (330), \texttt{route} (297), casts (377),
|
||||
etc. Two fusions that \emph{eliminate} work (not just launches) each surfaced:
|
||||
\begin{itemize}[noitemsep]
|
||||
\item \textbf{gate+up shared-quantize} (\texttt{moe\_matvec\_pair}): gate and up consume the same
|
||||
routed token, so broadcast + quantize it \emph{once} and \dpa{} both banks. $-48$ quantize,
|
||||
$-48$ ucopy per token.
|
||||
\item \textbf{add\_rmsnorm}: $s = x + \text{residual}$ then $\text{rmsnorm}(s)\cdot\alpha$ in one
|
||||
reduce launch, vs.\ a separate \texttt{badd} then \texttt{rmsnorm}.
|
||||
\item \textbf{fused q/k RMSNorm + positions-RoPE} (\texttt{rope\_norm\_positions}, one block per
|
||||
head-vector): one kernel replaces two standalone norms + RoPE \emph{and} the q/k
|
||||
intermediate write+read. Same f32 math, no redundant work. $+2.8\%$ ($77.4\to79.5$ T/s),
|
||||
the true-elimination twin of \texttt{moe\_route}.
|
||||
\end{itemize}
|
||||
Stacked, $1702\to1558$ kernels/token; decode past llama-HIP. Each is bit-exact. The discipline:
|
||||
fuse only when the fusion \emph{removes} GPU work --- which is exactly what the next section's
|
||||
biggest dead-end violated.
|
||||
|
||||
\section{Kept levers: CUDA (GB10, Grace-Blackwell sm\_121)}
|
||||
\label{sec:cuda}
|
||||
\subsection{i-quant completeness: IQ1\_S MMQ prefill, TQ1/TQ2 \dpa{} decode}
|
||||
Two correctness/coverage gaps closed, both bit-exact (\texttt{cuda\_iquant\_tests} 13/13). IQ1\_S had
|
||||
native \dpa{} decode but was MMQ-excluded at prefill only because its GPU grid table was a zeroed
|
||||
stub mis-sized at $[512]$ (the indexer needs 11 bits $=2048$); filling the real
|
||||
\texttt{iq1s\_grid\_gpu[2048]} verbatim from llama.cpp enabled the whole int8-WMMA prefill machine
|
||||
that was already ported. TQ1\_0/TQ2\_0 ternary got native \dpa{} decode --- symmetric $\{-1,0,1\}$
|
||||
packs straight to signed int8 with \emph{no} bias-sum term; notably \texttt{llama.cpp} has no CUDA
|
||||
ternary path at all, so this is pioneering and the ROCm scalar \texttt{qdec} is the layout oracle.
|
||||
|
||||
\subsection{\texttt{unpack\_ksigns}: the real GB10 i-quant kernel wall ($1.71\x$)}
|
||||
The i-quant decode matvec was $\sim$75\% of GB10 i-quant decode GPU-time. Diffing against
|
||||
\texttt{llama.cpp}'s \texttt{vec\_dot\_iq2\_xxs\_q8\_1} found a ROCm-port artifact: the sign
|
||||
codebook was a \emph{divergent} \texttt{\_\_constant\_\_} gather (each lane a different index,
|
||||
serialized through the constant cache). llama computes it table-free --- the 7-bit index's 8th sign
|
||||
is the popcount-parity of the other 7:
|
||||
\begin{lstlisting}[language=CUDAC]
|
||||
// bit-identical to KSIGNS_IQ2XS_D[v], no constant-cache gather:
|
||||
unpack_ksigns(v) = (v ^ ((__popc(v & 127) & 1) << 7)) * 0x01010101;
|
||||
\end{lstlisting}
|
||||
\textbf{Kernel A/B (4096$^2$, GB10): IQ2\_XXS $84.67\to49.49\,\mu$s $=1.71\x$} ($51\to87$ GB/s);
|
||||
IQ3\_XXS $1.40\x$; IQ2\_XS $1.25\x$. The control types that have no gather to kill (IQ2\_S/IQ3\_S,
|
||||
raw-byte signs) stay \emph{flat} --- which \emph{isolates} the gather as the cause and corrects an
|
||||
earlier ``47 GB/s VALU wall'' conclusion that had measured the baseline and mislabeled it.
|
||||
|
||||
\paragraph{The honest model-level result: a GB10 non-lever.} Same-engine A/B on
|
||||
Qwen3-4B-UD-IQ2\_XXS: \textbf{flat} both graphs-on ($66.1$ vs $66.2$ T/s) and eager ($60.2$ vs
|
||||
$60.2$). The $1.71\x$ kernel win does not surface because GB10 hanzo i-quant decode is
|
||||
\textbf{host/launch-overhead-bound, not matvec-bound} --- the GPU finishes the faster matvec and
|
||||
stalls on per-token host work. This is the same kernel ($\dpa$/codebook decode) that fully surfaced
|
||||
on ROCm-eager (\S\ref{sec:rocm}); the regime flip across platforms is the taxonomy's sharpest
|
||||
prediction. We ship it anyway: bit-exact, it drops a 128-entry table and a divergent gather, it
|
||||
matches llama, and it is a latent win for genuinely matvec-bound regimes. The actual GB10 decode
|
||||
lever is launch reduction (the dense decode-graph gate, $+6.8\%$).
|
||||
|
||||
\section{Kept levers: Vulkan (gfx1151) --- the prefill GEMM arc}
|
||||
\label{sec:vulkan}
|
||||
Vulkan is the youngest backend and the one where we walked the full optimization arc on a single
|
||||
kernel (Q4\_K prefill GEMM) in the open, keeping every rung --- including the negative ones --- as
|
||||
a documented ladder. This is the most complete single-kernel case study in the program.
|
||||
|
||||
\subsection{The deadlock: a missing uniform GEMM, surfaced as a liveness bug}
|
||||
hanzo-Vulkan \emph{hung} at the first long prefill of any GGUF on gfx1151, while llama.cpp-Vulkan
|
||||
and hanzo's own ROCm ran fine. \texttt{gdb} of the live hung process put the engine thread in the
|
||||
kernel at \texttt{DRM\_IOCTL\_AMDGPU\_GEM\_CREATE} $\leftarrow$ \texttt{QTensor::dequantize}
|
||||
$\leftarrow$ prefill. The prefill \texttt{else} branch dequantized each weight to a fresh
|
||||
$\sim$100--235~MB f32 buffer; under the deferred single command batch \emph{none} free until the
|
||||
end-of-forward flush, so a dense prefill re-expands the whole model to f32 ($\sim$32~GB for an 8B)
|
||||
in fresh allocations --- on a unified-memory carveout this exhausts memory and \texttt{GEM\_CREATE}
|
||||
blocks on a free that can only happen \emph{after} the still-recording batch flushes: a
|
||||
self-deadlock. ROCm never hit it because ROCm prefill rides the int8-WMMA GEMM and never
|
||||
dequantizes --- exactly the asymmetry that pointed at the fix. \texttt{vulkan\_prefill\_gemm\_max\_rows}
|
||||
now returns \texttt{usize::MAX} for every dtype that \emph{has} a native quantized GEMM, so prefill
|
||||
always uses the GEMM and never the f32-dequant fallback. The gate is now effectively a ``has a GEMM
|
||||
kernel?'' predicate; the real cleanup (\S\ref{sec:vulkan-roadmap}) is to give all 22 formats a GEMM
|
||||
so the fallback --- and the bug --- become impossible by construction.
|
||||
|
||||
\subsection{The L1 GEMM ladder: 1D dead-end $\to$ 2D-f32 $2.06\x$ $\to$ \dpa{}-2D $9.35\x$}
|
||||
We measured each rung kernel-isolated (\texttt{bench\_matmul\_q4k}: upload operands once, loop the
|
||||
GPU GEMM, one final synchronize --- the host-wrapper bench's per-iter 8~MB upload+readback
|
||||
otherwise masks the kernel). Numbers are $m{=}512,\,n_{\text{out}}{=}k{=}4096$ (17.2 GFLOP).
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Vulkan Q4\_K prefill GEMM ladder, kernel-isolated (gfx1151). Default = the legacy
|
||||
column-per-invocation kernel.}
|
||||
\small
|
||||
\begin{tabular}{l r r l}
|
||||
\toprule
|
||||
Kernel & Time (ms) & vs.\ default & Verdict \\
|
||||
\midrule
|
||||
Legacy column-per-invocation & 38.5 & 1.0\x & baseline \\
|
||||
1D weight-only LDS stage & 223.7 & 0.21\x & \textbf{dead-end} (\S\ref{sec:deadends}) \\
|
||||
2D-tiled f32 (64$\times$64, BK=32) & 19.0 & 2.06\x & kept (universal fallback) \\
|
||||
\textbf{2D-tiled int8 \dpa{}} & \textbf{3.4} & \textbf{9.35\x} & \textbf{shipped default} \\
|
||||
Naive coopmat (tensor cores) & 17.4--19 & 0.26\x vs \dpa{} & \textbf{dead-end} (\S\ref{sec:deadends}) \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
The shipped kernel (\texttt{mul\_mm\_q4k\_tiled\_dp4a.comp}) stages a 64$\times$64 output tile:
|
||||
per K-step (BK=32, one Q4\_K sub-block) it stages the $n$-cols' 32 q4 codes + per-block $d_1/m_1$
|
||||
and the $m$-rows' int8 activation $xq$ + $xs/xsum$ into LDS, and each of 256 threads \dpa{}s its
|
||||
4$\times$4 sub-tile via the Q4\_K$\times$q8\_1 affine identity
|
||||
\[
|
||||
\text{acc} \mathrel{+}= d_1[\text{col}]\cdot xs[\text{row}]\cdot \dpa(q4, xq) \;-\; m_1[\text{col}]\cdot xsum[\text{row}].
|
||||
\]
|
||||
Activations are quantized to q8\_1 once per prefill. It is gated
|
||||
\texttt{vulkan\_q4k\_dp4a2d\_matches\_default} (rel $<1.5\!\times\!10^{-2}$; the q8\_1 activation
|
||||
quant adds $\sim$0.5--1\% on top of f32 reorder, a decode bug would be $\gg$ that). The default-flip
|
||||
ships it for dense ($w_{\text{off}}{=}0$, $m{>}1$) where the device advertises integer dot-product
|
||||
(queried via \texttt{PhysicalDeviceShaderIntegerDotProductFeatures}); otherwise the universal f32
|
||||
2D tile. A gfx1151 engine now gets $\sim$224 T/s Vulkan prefill by default, up from $\sim$24.
|
||||
|
||||
\paragraph{The practical ceiling, pinned by a tile sweep.} At 5056 GFLOP/s the \dpa{}-2D kernel is
|
||||
$\sim$34\% of the $\sim$14.8 TFLOP f32 peak. A tile sweep pins the bottleneck as \emph{occupancy},
|
||||
not VRAM/LDS traffic: register-blocking the inner \dpa{} is flat (the compiler already did it / added
|
||||
VGPRs that cost occupancy); BM=128 (half the weight re-read) is $3\x$ \emph{slower} (register spill
|
||||
/ occupancy collapse); BM=32 is slightly slower (re-read + barrier overhead). 64$\times$64 is the
|
||||
sweet spot. The residual gap to f32-peak is the \emph{inherent} Q4\_K cost (per-sub-block scale
|
||||
application + in-kernel decode + the q8\_1 activation pass), not a tiling defect --- closing it
|
||||
needs a structurally different kernel (decode-amortized coopmat), not more tuning. This is the
|
||||
honest statement of where blind kernel-tuning ends on this GPU.
|
||||
|
||||
\section{Measured dead-ends: the boundary map}
|
||||
\label{sec:deadends}
|
||||
The negatives are load-bearing: each is explained by the surfacing rule, and together they fence
|
||||
off the regimes. We report them because an inference team that does not know these boundaries will
|
||||
spend weeks re-discovering them.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Things that were correct (bit-exact) but did not help, with the regime that explains why.}
|
||||
\small
|
||||
\begin{tabular}{>{\raggedright\arraybackslash}p{3.9cm} >{\raggedright\arraybackslash}p{2.2cm} >{\raggedright\arraybackslash}p{6.7cm}}
|
||||
\toprule
|
||||
Change (bit-exact) & Result & Why (taxonomy) \\
|
||||
\midrule
|
||||
Fold \texttt{quantize\_q8\_1} into the \dpa{} matvec (ROCm) & \textbf{$-22\%$} ($77.4\to59$) & Each matvec \emph{block} re-quantizes the full activation into its own LDS (256$\times$ for a 2048-row proj) + a \texttt{\_\_syncthreads}: redundant serialized work \emph{injected} into a bandwidth-bound matvec. Under graphs the separate quantize launch was already free. \emph{Violates the rule.} \\
|
||||
Naive coopmat tensor-core prefill (Vulkan) & 0.26\x{} vs \dpa{} & Re-decodes the weight to f16 every K=16 step ($k/16$ passes); Q4\_K decode swamps the fast \texttt{coopMatMulAdd}s. Decode-bound, not tensor-core-bound. \\
|
||||
1D weight-only LDS tiling (Vulkan) & 0.21\x & Weight re-read was not the bottleneck; few-heavy-workgroups lose to many-light register-tiled invocations. The lever was 2D (stage \emph{both} operands), not 1D. \\
|
||||
TILE\_M=32 MoE GEMM (ROCm) & slower than 64 & Less empty WMMA work but re-reads each expert weight in 2 tiles; the GEMM is weight-bandwidth-bound. \\
|
||||
\dpa{} i-quant decode, model-level (GB10) & flat & Host/launch-bound decode: GPU finishes the faster matvec and stalls on host work. (Same kernel is $2.27\x$ on ROCm-eager.) \\
|
||||
\texttt{moe\_route} fusion, model-level (CUDA) & flat & Repackages routing into one block-serial launch; CUDA decode is not router-launch-bound and CUDA has no expensive \texttt{gpu-sort} to eliminate (ROCm's $+13\%$ came from that sort). \emph{Repackaging, not eliminating.} \\
|
||||
f32-native cast cut (ROCm) & flat & The casts are dtype-tangled around the F32-residual/F16-KV boundary; removing one shifts cost across the boundary. \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\noindent Two of these are the \emph{same kernel} as a kept win elsewhere (\dpa{} i-quant: ROCm
|
||||
$2.27\x$ / GB10 flat; \texttt{moe\_route}: ROCm $+13\%$ / CUDA flat). That is the cleanest possible
|
||||
confirmation of the taxonomy: the kernel did not change, the \emph{binding resource} did. We keep
|
||||
both on the flat platforms anyway --- bit-exact, decomplecting (one selector, fewer tables, matches
|
||||
llama), and latent for the regime where they bind --- which is the right call precisely \emph{because}
|
||||
the taxonomy says they cost nothing and may surface on the next model (e.g.\ a 256-expert router
|
||||
doubles the routing math).
|
||||
|
||||
\section{The cross-backend law and model-level dilution}
|
||||
\label{sec:law}
|
||||
Stacking the per-backend work yields the model-level result of paper~05, restated as a law:
|
||||
\begin{itemize}[noitemsep]
|
||||
\item \textbf{Decode is bandwidth-roofline parity} on every backend (0.98--1.04\x): once the
|
||||
matvec streams weights at the memory ceiling, all engines converge, because decode
|
||||
throughput \emph{is} memory bandwidth / model bytes. hanzo's per-format decode matvec is at
|
||||
88--99\% of the measured roofline (\texttt{rocprofv3} FETCH\_SIZE), so there is nothing left
|
||||
to win there --- and indeed every decode \emph{kernel} win above surfaced only through
|
||||
\emph{launch/work elimination}, never through faster matvec arithmetic.
|
||||
\item \textbf{Prefill is the frontier} (0.76--0.91\x): it is compute-bound, so it rewards the
|
||||
matrix/tensor-core GEMM scheduling that llama.cpp has most heavily tuned. Every prefill
|
||||
lever in this paper (MoE qmmq $2.58\x$, dynamic M-tile, Vulkan \dpa{}-2D $9.35\x$) closes a
|
||||
chunk of this gap; the residual is real engineering (decode-amortized coopmat on Vulkan,
|
||||
the last $\sim$10\% of GEMM efficiency on CUDA), not a rounding error.
|
||||
\end{itemize}
|
||||
|
||||
\paragraph{Model-level dilution.} A kernel win of factor $\kappa$ surfaces as
|
||||
$\approx 1 + f\,(\,1 - 1/\kappa\,)$ end-to-end, where $f$ is the fraction of \emph{binding-resource}
|
||||
time the kernel occupied. The Vulkan \dpa{}-2D $9.35\x$ kernel lifts engine prefill $\sim 24\to224$
|
||||
T/s ($f$ large, prefill is GEMM-dominated) but would be flat on decode ($f\approx 0$, decode is not
|
||||
the GEMM). The ROCm Q6\_K $4.9\x$ kernel gives $+17\%$ ($f\approx 0.38$, its compute-bound
|
||||
fraction). The GB10 i-quant $1.71\x$ gives $0\%$ ($f\approx 0$, host-bound). The law is just
|
||||
Amdahl's law with the binding resource as the denominator --- but the discipline it enforces is to
|
||||
\emph{measure $f$ before writing the kernel}.
|
||||
|
||||
\section{A correctness boundary: the flash-attention repetition bifurcation}
|
||||
\label{sec:flash-bifurcation}
|
||||
The sharpest result of this work is not a speed lever but a \emph{correctness} one, and it maps the
|
||||
exact limit of fused attention on tensor cores. Qwen3-8B-Q4\_K\_M generates coherently under an
|
||||
eager attention (materialized \texttt{softmax}$(QK^\top/\sqrt d)V$ in f32) but \emph{collapses into
|
||||
token repetition} under flash-attention-2 on CUDA --- byte-identical weights, same sampler, same
|
||||
box; only the attention kernel differs. The collapse is deterministic and appears at the first
|
||||
generated token.
|
||||
|
||||
\paragraph{The precision ladder, exhausted.} Flash-attention-2 casts the f32 softmax probabilities
|
||||
$P=\mathrm{softmax}(S)$ to the kernel element type before the $P\!\cdot\!V$ tensor-core matmul
|
||||
(\texttt{flash\_fwd\_kernel.h}: \texttt{convert\_type<Element>(acc\_s)}). We raised that precision in
|
||||
three proved-by-build steps, each verified on the running model at two seeds:
|
||||
\begin{center}\small
|
||||
\begin{tabular}{@{}llll@{}}
|
||||
\toprule
|
||||
$P$ representation & mantissa bits & \texttt{exp} & bare-trigger continuation \\
|
||||
\midrule
|
||||
bf16 & 7 & \texttt{exp2f} & \texttt{"needle needle\dots"} \\
|
||||
f16 & 10 & \texttt{exp2f} & \texttt{"0123456789\dots"} \\
|
||||
split $P_{hi}{+}P_{lo}$ & $\sim$20 & \texttt{exp2f} & \texttt{".matmat.matmat\dots"} \\
|
||||
split $P_{hi}{+}P_{lo}$ & $\sim$20 & \texttt{expf} (exact) & \texttt{"1)2)3)4)5)\dots"} \\
|
||||
eager (reference) & 23 (f32), single-pass & \texttt{expf} & \texttt{"\dots Paris\dots"} (coherent) \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{center}
|
||||
More precision only \emph{moves the attractor token} (needle $\to$ digits $\to$ matmat); it never
|
||||
clears it. This is the signature of a knife-edge numerical bifurcation, not a localizable bug.
|
||||
|
||||
\paragraph{Everything else, eliminated.} For the bare trigger ($\sim$7 tokens) the remaining
|
||||
flash-vs-eager differences are ruled out by structure, not guesswork: (i) the $QK^\top$ score MMA
|
||||
already accumulates in f32 (CUTLASS default), so $S$ is f32-identical to eager; (ii) $V$ is bf16 in
|
||||
the KV cache for \emph{both} paths, so the bf16-input/f32-accumulate MMA is numerically the eager
|
||||
matmul --- there are no low bits of $V$ to recover; (iii) with $\texttt{kBlockN}=128$ the trigger is
|
||||
a \emph{single} K-tile, so the cross-tile online-softmax rescale executes zero times yet the collapse
|
||||
still appears; (iv) the \texttt{exp2f}-vs-\texttt{expf} gap is $\lesssim 2^{-23}$, orders below the
|
||||
$2^{-8}$ bf16-$P$ error already shown non-curative --- and this last one was \emph{also} confirmed
|
||||
empirically: exact \texttt{expf} atop f32-class $P$ still degenerates at both seeds (into a fresh
|
||||
\texttt{"1)2)3)\dots"} attractor), exhausting the precision-and-transcendental hypothesis space.
|
||||
|
||||
\paragraph{The boundary.} What remains after all eliminations is a lone variable: $P$ is
|
||||
$\sim$20-bit on the tensor-core path versus true 23-bit f32 in eager. Because tensor cores
|
||||
\emph{require} $\le$tf32/f16/bf16 inputs, a genuinely f32 $P\!\cdot\!V$ \emph{is} the eager
|
||||
(CUDA-core) path --- so on this hardware \textbf{fusion and f32-precision $P\!\cdot\!V$ are mutually
|
||||
exclusive}. The shipped fix routes bf16 attention to the eager kernel; it is not a stopgap but the
|
||||
\emph{correct} answer, and it is essentially optimal: any stable variant must carry f32 $P\!\cdot\!V$,
|
||||
which forgoes tensor cores and thus lands at eager's throughput regardless. Its only cost is
|
||||
long-context prefill ($2.66\x$ at 2409 tokens; short prompts and \emph{all} decode are unaffected,
|
||||
since decode attention is a single query against the cache), and that cost is intrinsic to stability.
|
||||
The methodological point echoes the rest of this paper: the result was reachable only by
|
||||
\emph{verifying against the running model} --- five kernel edits that ``compiled and committed''
|
||||
were disproved by the generated tokens before the boundary came into focus.
|
||||
|
||||
\section{The characterized-but-open frontier: Vulkan decode}
|
||||
\label{sec:frontier}
|
||||
Paper~05's parity claim is decode-on-AMD-via-ROCm. The Vulkan \emph{decode} path is the one place
|
||||
hanzo is far from llama (8B Q4\_K: 3.3 vs 41 T/s, $\sim$0.08\x), and this program has now diagnosed
|
||||
it precisely rather than left it as ``slow.'' A \texttt{VK\_PROFILE} breakdown of one decode token:
|
||||
\begin{lstlisting}
|
||||
[VK_PROFILE] flush: dispatch=1336 barriers=903 record=0.766ms submit=1.406ms fence_wait=142.827ms
|
||||
\end{lstlisting}
|
||||
The diagnosis: it is \textbf{not} recording-bound (CPU record $=0.77$~ms, so command-buffer/graph
|
||||
reuse would buy nothing) --- it is \textbf{GPU-execution-bound on op-count serialization}. 1336
|
||||
tiny $M{=}1$ dispatches chained through \textbf{903 global memory barriers} (\texttt{vk::MemoryBarrier},
|
||||
full \texttt{SHADER\_WRITE}$\to$\texttt{READ}, on \emph{true} data dependencies) serialize the GPU at
|
||||
$\sim$107~$\mu$s/op on compute that is only $\sim$$\mu$s. Per the taxonomy this is GPU-busy-bound
|
||||
decode, so the lever is \emph{eliminating ops}, and the barriers are correct (real dependencies) ---
|
||||
the only way to remove them is to fuse the dependent ops so the dependency lives \emph{inside} a
|
||||
kernel.
|
||||
|
||||
\subsection{Roadmap: the campaign ROCm already validated}
|
||||
\label{sec:vulkan-roadmap}
|
||||
This is not speculative: it is the exact campaign that took ROCm decode from launch/busy-bound to
|
||||
parity (\S\ref{sec:rocm}), ported to Vulkan.
|
||||
\begin{enumerate}[noitemsep]
|
||||
\item \textbf{Fuse the attention epilogue} --- q/k RMSNorm + RoPE + KV-cache write into one
|
||||
kernel (ROCm's \texttt{rope\_norm\_positions} twin), and residual-add + RMSNorm
|
||||
(\texttt{add\_rmsnorm}). These \emph{eliminate} per-op barriers, not just launches.
|
||||
\item \textbf{Fuse the FFN} --- gate+up shared-quantize, silu+mul.
|
||||
\item \textbf{Fuse routing} --- the \texttt{moe\_route} kernel (already proven $+13\%$ on ROCm,
|
||||
where the eliminated sort is expensive --- and Vulkan, like ROCm, pays for the sort).
|
||||
\item \textbf{Uniform Vulkan GEMM for all 22 formats} --- deletes the dequant fallback and the
|
||||
deadlock gate by construction (\S\ref{sec:vulkan}), the structural endpoint.
|
||||
\end{enumerate}
|
||||
The honest status: \textbf{ROCm is at parity; Vulkan decode is the open frontier}, characterized
|
||||
down to the dispatch/barrier counts, with a roadmap whose every step has a measured ROCm precedent.
|
||||
The decomplection endpoint (axes \S\ref{sec:arch}) is a kernel DSL (CubeCL/Triton/MLIR-class) that
|
||||
writes the contraction once and lowers it to every backend, collapsing the
|
||||
backends$\times$formats$\times$ops product to backends$+$formats$+$ops --- at which point a fusion is
|
||||
written once and surfaces on whichever backend's regime it binds.
|
||||
|
||||
\section{Verification methodology (so the numbers mean something)}
|
||||
\label{sec:method}
|
||||
Every claim in this paper rests on three controls, because inference benchmarks are unusually easy
|
||||
to corrupt:
|
||||
\begin{itemize}[noitemsep]
|
||||
\item \textbf{Bit-exact gates.} A kernel ships only after \texttt{nbad=0} versus a CPU
|
||||
\texttt{to\_float} oracle (documented f32-reorder / f16-rounding tolerances). Correctness is
|
||||
the precondition for a performance claim, not a follow-up.
|
||||
\item \textbf{Controlled A/B.} Decode A/Bs are device-map-pinned (\texttt{-n 0:48} on ROCm; all
|
||||
layers resident) and run in-binary via a fallback env toggle (\texttt{HANZO\_*\_FALLBACK}),
|
||||
so the \emph{only} variable is the kernel. The headline ``$+28\%$'' that an earlier pass
|
||||
reported was a measurement artifact --- the auto-mapper offloading $1/3$ of layers --- and
|
||||
the controlled A/B showed the real number was $+1.3\%$. We report the controlled number.
|
||||
\item \textbf{Kernel-isolated bench.} GEMM ladders use operands resident on-GPU and one final
|
||||
synchronize, because a per-iteration host upload/readback otherwise dominates and masks the
|
||||
kernel (the difference between the column kernel looking 1.2\x{} and 9\x{} slower).
|
||||
\end{itemize}
|
||||
\paragraph{Contamination caveat (stated, not hidden).} The first evo Vulkan matrix in this pass was
|
||||
run under load~59 (a concurrent ROCm build + a model download + a browser) and is discarded; the
|
||||
numbers in \S\ref{sec:vulkan}--\ref{sec:frontier} are from the idle box (load $<0.5$). A benchmark
|
||||
on a contended box is not a slow benchmark, it is a wrong one.
|
||||
|
||||
\section{What this paper does \emph{not} claim}
|
||||
Single-stream only (no batched/serving throughput); the model-level numbers are paper~05's
|
||||
single-model/single-quant pass, not re-swept here; Metal kernel-level work is summarized from its
|
||||
own audit, not re-derived; and the Vulkan-decode roadmap (\S\ref{sec:vulkan-roadmap}) is
|
||||
characterized and precedented but \emph{not yet shipped} --- we label it frontier, not result. Every
|
||||
performance figure is a measured throughput or a kernel-isolated microbench on the named hardware;
|
||||
derived quantities (effective GB/s, M-fill \%) are computed from measured values and labelled. No
|
||||
number here was invented or projected without the label ``projection.''
|
||||
|
||||
\section{Conclusion}
|
||||
The portable-engine thesis survives contact with the kernels: from one multi-backend codebase,
|
||||
hanzo-engine is at \texttt{llama.cpp} decode parity and within a bounded, well-understood prefill
|
||||
margin, and every step of that was a \emph{measured} lever gated bit-exact. The transferable result
|
||||
is not any single kernel but the \textbf{surfacing rule and its taxonomy}: decode is
|
||||
bandwidth-roofline (win by moving fewer bytes or eliminating launches/work, never by faster
|
||||
arithmetic), prefill is compute-bound (win by tensor-core scheduling and tile fill), and the
|
||||
\emph{same} kernel surfaces or vanishes depending on which resource binds on a given (platform,
|
||||
regime). The dead-ends prove the rule as sharply as the wins --- the $-22\%$ quantize-fold, the
|
||||
$0.26\x$ naive coopmat, the GB10 i-quant flat that is $2.27\x$ on ROCm. The one fully-characterized
|
||||
open frontier, Vulkan decode, is op-count-serialization-bound by profile, and its fix is the exact
|
||||
fusion campaign ROCm has already validated to parity. Optimize the binding resource; prove it
|
||||
bit-exact; measure $f$ before you write the kernel.
|
||||
|
||||
\appendix
|
||||
\section{Reproducibility}
|
||||
\textbf{Hardware.} AMD Ryzen AI Max+ 395 ``Strix Halo'' (Radeon 8060S, gfx1151, RDNA3.5, 40 CU,
|
||||
$\approx$231 GB/s); NVIDIA GB10 (Grace-Blackwell, sm\_121, CUDA 13, 121 GB unified, Linux aarch64);
|
||||
Apple M4 Max (137 GB unified, Metal). \textbf{Software.} hanzo-ml (the catalog spans the 0.11.x
|
||||
series) / hanzo-engine, built \texttt{--release} per box; \texttt{llama.cpp} stock HIP/CUDA/Metal/Vulkan
|
||||
builds as the baseline. \textbf{Models.} Qwen3 family GGUF (0.6B/4B/8B dense, 30B-A3B MoE) at
|
||||
Q8\_0/Q4\_K\_M and the UD i-quants. \textbf{A/B protocol.} In-binary fallback env toggles
|
||||
(\texttt{HANZO\_*\_FALLBACK}) so the kernel is the only variable; ROCm decode device-map-pinned
|
||||
\texttt{-n 0:NLAYERS}; GEMM ladders via the kernel-isolated \texttt{bench\_matmul\_q4k} /
|
||||
\texttt{bench\_iquant\_decode}; bit-exact gates via the named \texttt{*\_numeric} oracles in
|
||||
hanzo-cli. \textbf{Profilers.} \texttt{rocprofv3} (ROCm GPU-busy + FETCH\_SIZE roofline),
|
||||
\texttt{nsys} (CUDA), \texttt{VK\_PROFILE=1} (the in-engine per-batch dispatch/barrier/fence-wait
|
||||
breakdown). GPU-safety: ROCm/HIP processes are run strictly serially (concurrent kfd init on
|
||||
gfx1151 can wedge the driver), benches background-only.
|
||||
|
||||
\end{document}
|
||||
Reference in New Issue
Block a user