papers/hanzo-kernel-dsl.tex

\documentclass[11pt]{article}
\input{shared/kernel-preamble.tex}

\title{\textbf{One Source, Every Backend:\\ The \hk{} DSL --- Thesis and Guided Tour}}
\author{Hanzo AI --- ML Systems}
\date{Paper 07 (umbrella) --- companion to Paper 06 (\emph{When Does a Kernel Win Surface?})}

\begin{document}
\maketitle

\begin{abstract}
The kernel layer of a multi-backend LLM engine is a Cartesian product: the same $\sim$20 operations,
specialized over $\sim$22 quantization formats, hand-written once per GPU backend --- five kernel trees
and $\sim$52{,}000 lines, and the breeding ground for the divergence bugs that cost the most to chase,
because one operation with five implementations has five numeric behaviors. This umbrella paper states the
thesis that collapses the product to a sum and gives a guided tour of the six companion papers that make
it concrete. The thesis rests on one idea --- \textbf{fusion is composition}, the functor law made
mechanical --- extended by three mechanisms that each attack one axis of the product: comptime-specialized
autotune (the schedule as a value, not a fork), intrinsic islands (a per-backend idiom held behind a
normative oracle arm), and a CPU runtime that is the bit-exact reference for every backend. We describe the
facade that ties them together (a first-party namespace over the CubeCL lowering engine, plugged in as a
code generator rather than a second runtime), summarize the evidence --- portability that costs nothing, a
shipped migration that improved performance, an honest structural negative result --- and place the work
against Triton, XLA, torch.compile, TVM, Halide, Burn, and Mojo. Each fact in this series lives in exactly
one paper; here we connect them.
\end{abstract}

\section{The thesis: a product that should be a sum}
\label{sec:thesis}

Table~\ref{tab:axes} is the disease. A multi-backend engine braids three independent concerns into one
hand-multiplied product: the operations (the math), the quantization types (the bit layout), and the
backends (the lowering). Written by hand, that is $\sim$20 $\times$ 22 $\times$ 5 kernels; in the Hanzo
repositories it is $52{,}345$ lines across five trees and five shading languages (the ledger and its
audit live in the collapse-method paper, \cite{method}).

\begin{table}[h]
\centering
\begin{tabular}{lll}
\toprule
\textbf{Axis} & \textbf{Count} & \textbf{What it is} \\
\midrule
Ops        & $\sim$20 & the math: matvec, matmul, flash-attn, rmsnorm, rope, softmax, GDN \\
Quant types & $\sim$22 & the bit layout: Q4\_K, Q6\_K, Q8\_0, the IQ codebooks \\
Backends   & 5        & the lowering: threads, memory, tensor cores, per shading language \\
\bottomrule
\end{tabular}
\caption{Three orthogonal axes, multiplied by hand today. The DSL's job is to make each a summand.}
\label{tab:axes}
\end{table}

That product is not merely a maintenance cost. It is where correctness bugs breed: a single quantized
model that generates coherent text on one backend and collapses into token repetition on another, from
byte-identical weights, because the same operation has a different kernel --- and therefore a different
numeric behavior --- per backend (the bug and its cure are the subject of the oracle paper,
\cite{oracle}). Uniform numeric behavior across backends cannot be tested \emph{into} five copies; it is a
structural property of having one implementation. So the goal is to turn the product into a sum:
\[
\underbrace{\text{backends} \x \text{types} \x \text{ops}}_{\text{hand-written today}}
\;\longrightarrow\;
\underbrace{\text{backends} + \text{types} + \text{ops}}_{\text{one source, lowered}}.
\]
This is a decomplect we already trust in miniature: the ROCm tree writes \code{qmatvec\_core<WTYPE>} --- one
contraction, the quant type injected as a compile-time parameter. The DSL generalizes exactly that pattern
with the \emph{backend} as the other compile-time axis.

\section{The facade: a namespace, and a code generator}
\label{sec:facade}

\hk{} is a first-party Rust DSL: a kernel is written once, in Rust, and lowered to CUDA, ROCm, Vulkan, and
Metal. It is a \emph{facade} over the CubeCL lowering engine --- kernel source names only
\code{hanzo\_kernel::*}, and the string \code{cubecl} appears in exactly one line of one \code{Cargo.toml}.
That is ``values, not places'': CubeCL is the implementation \emph{value}; \hk{} is the stable
\emph{namespace} we build against, upgradable or forkable without editing a kernel. The authoring surface
uses first-principles names --- \code{\#[kernel]} says what a thing is (a GPU kernel), not \code{cube} (a
brand); \code{Grid} and \code{Block} name the launch shape directly (Listing~\ref{lst:matvec}).

\begin{lstlisting}[language=Rust,caption={A quantized matvec: one source, no backend in it. The decode
(\code{wd*wq}) and the contraction are the arithmetic each backend hand-writes today. This one function
lowers to CUDA, ROCm, Vulkan, Metal, and CPU.},label={lst:matvec}]
use hanzo_kernel::prelude::*;

#[kernel(targets(cuda, metal, vulkan, webgpu, cpu), unchecked)]
pub fn matvec_q8<F: Float>(wd: &Array<F>, wq: &Array<i32>, x: &Array<F>,
                           out: &mut Array<F>, #[comptime] k: usize) {
    let row = ABSOLUTE_POS;
    if row < out.len() {
        let (nb, wbase, dbase) = (k / 32, row * k, row * (k / 32));
        let mut acc = F::new(0.0);
        for i in 0..k { acc += wd[dbase + i / 32] * F::cast_from(wq[wbase + i]) * x[i]; }
        out[row] = acc;
    }
}
\end{lstlisting}

Crucially the DSL is a \emph{code generator}, not a competing GPU runtime. The engine's Vulkan path is
already ``load SPIR-V bytes, create a shader module, dispatch,'' and the DSL \emph{emits} exactly those
bytes: a compiled kernel is a \code{GLCompute} entry point with buffers on descriptor bindings $0,1,2$ ---
byte-for-byte the shape the hand-written \code{.comp}$\to$\code{.spv} shaders already take. So the DSL plugs
into each backend's \emph{existing} dispatch machinery; there is no dual runtime and no cross-runtime copy.
The one bounded integration step is a three-line SPIR-V post-pass (rename the entry point to \code{main};
strip the engine's unused info-uniform; dispatch). The perf primitives the hand-tuned kernels rely on are
all reachable and each lowers to the native instruction --- \code{dot} on a vectorized \code{Line<i8>}
$\to$ \dpa{}, \code{cmma} $\to$ tensor cores, \code{SharedMemory} and the \code{plane\_*} subgroup
reductions --- so portability does not, by construction, force a performance downgrade.

\section{A guided tour of the mechanisms}
\label{sec:tour}

The collapse is not a syntactic retyping of kernels; it follows from one algebraic idea applied three
ways. Each is developed in full in its own companion paper; here is the map.

\paragraph{Fusion is composition (\cite{fuse}).} An elementwise kernel is the lift of a scalar function to
every index, and such lifts obey the functor law $\mathrm{map}(g)\circ\mathrm{map}(f)=\mathrm{map}(g\circ
f)$. Read right-to-left, that law \emph{is} fusion: one kernel computing the composite, no intermediate
array. Reduce and Scan are algebraic \emph{fences} where the pattern changes, decidable from the op's
class rather than a pattern matcher. This is the theory core: the fusibility of two ops is a total function
of their type, one fused-interpreter kernel lowers any composed chain to a single launch, and the same
algebra lifts from a line to an arbitrary DAG --- draining, for instance, a per-layer Gated-DeltaNet chain
from 9 kernel launches to 5, bit-exact.

\paragraph{The schedule is a value (\cite{tune}).} The last place a backend forks a kernel is the schedule
--- tile size, vector width, unroll, block dimension. A fork is exactly what reintroduces the divergence
bug. Autotune folds the fork into the one source by exposing the schedule as compile-time knobs the engine
monomorphizes; a small variant set is timed per device and the winner cached to disk. Because the knobs
never touch the arithmetic, every variant is byte-identical --- the tuner buys speed and can never spend
correctness.

\paragraph{Intrinsic islands (\cite{island}).} When one inner line wants a target-specific instruction, you
do not fork the kernel; you write a scoped, target-gated block \emph{inside} it. Selection is a
compile-time \code{match} on a tag the launch layer fills in, so exactly one arm is lowered per backend.
The \code{default} arm is \emph{normative} --- the CPU always takes it, and every accelerated arm is proven
byte-for-byte equal to it on the CPU oracle before it ships. An escape hatch that cannot escape correctness.

\paragraph{One oracle, every backend (\cite{oracle}).} The foundation under all three: the CPU runtime is
the single bit-exact reference for every backend. Every op's device code mirrors a plain-Rust reference
line-for-line; because the CPU runs pointwise ops in program order, its result is the definition, and a GPU
differs only by bounded reduction-order rounding. The law is enforced nightly by a three-backend GPU CI on
self-hosted NVIDIA, AMD, and Apple machines --- so ``the same op has the same numbers everywhere'' is a
checked invariant, and the repetition collapse becomes an unreachable state rather than a bug to fix.

\paragraph{Where the product won't collapse (\cite{mmq}).} Honesty is part of the thesis. Block-quantized
tensor-core MMQ --- the kernel that governs prefill --- does \emph{not} collapse through the portable API:
its accumulator fragment is opaque, forcing a per-block scale round-trip that caps it near one-tenth of
hand-tuned (proven sync-bound because better tiling is \emph{slower}). The viable route is per-backend
anyway. So MMQ stays hand-written behind exactly the island seam built for it --- a named boundary, not a
quiet exclusion.

\paragraph{Verify, then delete (\cite{method}).} The migration method ties it together: a hand-written
kernel is retired only when its single-source twin is both bit-exact against the oracle and at least as
fast on a kernel-only benchmark, cell by cell through a per-\code{(op,type,backend)} registry, so the
engine is faster-or-equal at every step. Cells have already flipped on \code{main} --- including a Vulkan
\code{add\_rmsnorm} twin that was \emph{7.1$\times$ faster} than the shader it replaced and bit-exact.

\section{Evidence, in one breath}
\label{sec:evidence}

The claims the tour makes are backed by measurement, detailed in the companion papers: the op library is
bit-exact against the CPU oracle and dispatches bit-exact through real Vulkan, Metal, and ROCm runtimes
(rms\_norm on gfx1151 to $2.43\times10^{-7}$; \cite{oracle}); a single-source \dpa{} matvec runs at 118\%
of the hand-tuned kernel it replaces (196~GB/s on gfx1151; \cite{method}); a shipped migration was
7.1$\times$ faster \emph{and} bit-exact (\cite{method}); the autotune variants are byte-identical with a
per-device cached winner (\cite{tune}); and the one op that resists is reported in full as a structural
negative result (\cite{mmq}). Portability costs nothing where it counts, the collapse is real and
reversible, and the boundary is drawn honestly.

\section{Related work}
\label{sec:related}

Portable kernel authoring and kernel fusion are both well-trodden; our specific claim is narrow, and we
state it precisely. \textbf{Triton} gives a tile-level Python DSL, but fusion is the programmer's job or a
larger stack's; it has no notion of fusion as an algebraic law over an op class. \textbf{XLA} fuses HLO
graphs by producer-consumer heuristics and a cost model; \textbf{torch.compile}/TorchInductor captures a
graph and fuses by a scheduler's heuristics --- both fuse by \emph{analysis over a graph}, not by a
predicate that is a function of the op's type. \textbf{Halide} and \textbf{TVM} pioneered separating the
\emph{compute} from the \emph{schedule} and autotuning the latter; this is the nearest neighbor of our
autotune mechanism (\cite{tune}), and we claim no novelty for schedule/compute separation itself --- what
differs is that our schedule knobs are compile-time monomorphizations of one kernel proven byte-identical,
so autotuning cannot perturb numerics. \textbf{Burn}'s \code{burn-fusion} is the closest relative to our
fusion engine: it records an op stream and JITs fused elementwise kernels over the \emph{same} CubeCL
substrate, but it traces Burn tensor ops with its own cut/epilogue policy, coupling to Burn's tensor
abstraction; ours is the functor law over an op \emph{class}, one predicate, on our own tensor type,
composing with our hand-tuned peak kernels. \textbf{Mojo} is a language for kernels, not a
fusion-as-composition model. Across all of these we are unaware of a system that presents kernel fusion
\emph{as} morphism composition --- the functor law as the mechanical rule, Reduce/Scan as the algebraic
fences --- anchored by a bit-exact CPU oracle as the definitional semantics. That framing, and the three
mechanisms it organizes, is the contribution; the substrate (CubeCL) and the individual ideas (portable
IR, autotuning, oracle testing) are prior art we build on and cite.

\section{Conclusion}

The kernel layer is a Cartesian product of backends, quantization types, and operations, hand-written in
five shading languages, and it is the breeding ground for the divergence bugs that cost the most to chase.
\hk{} collapses that product to a sum on one idea --- fusion is composition (\cite{fuse}) --- extended by
three mechanisms that each attack an axis: autotune folds the schedule fork into byte-identical comptime
knobs (\cite{tune}); islands hold the rare per-backend idiom behind a normative, CPU-checkable arm
(\cite{island}); and the CPU runtime is the single bit-exact oracle every backend is measured against,
nightly (\cite{oracle}). Portability costs nothing where it counts, the collapse is real and reversible via
a verify-then-delete method (\cite{method}), and --- stated as plainly as the wins --- one op resists and
stays hand-written behind an island (\cite{mmq}). Read the six companion papers for the depth; this one is
the map.

\begin{thebibliography}{9}
\small
\bibitem{umbrella} Hanzo AI. \emph{One Source, Every Backend: The hanzo-kernel DSL.} Paper 07 (this umbrella).
\bibitem{fuse} Hanzo AI. \emph{Fusion Is Composition: Kernel Fusion as the Functor Law.} Paper 07.1.
\bibitem{tune} Hanzo AI. \emph{The Schedule Is a Value: Comptime-Specialized Autotuning.} Paper 07.2.
\bibitem{island} Hanzo AI. \emph{Intrinsic Islands: An Oracle-Anchored Escape Hatch.} Paper 07.3.
\bibitem{oracle} Hanzo AI. \emph{One Oracle, Every Backend: The CPU Bit-Exactness Law.} Paper 07.4.
\bibitem{mmq} Hanzo AI. \emph{When the Product Won't Collapse: An int8 Tensor-Core MMQ Negative Result.} Paper 07.5.
\bibitem{method} Hanzo AI. \emph{Verify, Then Delete: Collapsing a Kernel Cartesian Product.} Paper 07.6.
\end{thebibliography}

\end{document}