mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-22 17:20:17 +00:00
papers: publish "One Source, Every Backend" 08 and 09 to papers.hanzo.ai
Add the kernel-DSL campaign pair: - refutation-driven-perf (Paper 08): the refutation-log methodology and measured win ladders across gfx1151 / GB10 / M4 Max. - evolutionary-schedule-search (Paper 09): schedule search over the monomorphized DSL and the continual-improvement loop it anchors. Each ships root .tex + .pdf, a served copy under site/public/pdfs, and a site catalog entry (site/src/config/papers.ts) with a companion cross-link. INDEX.md regenerated via `make index` (218 papers, 211 compiled).
This commit is contained in:
@@ -29,8 +29,11 @@ Auto-generated catalogue of research papers. Regenerate: `scripts/gen-index.sh`
|
||||
| `hanzo-zap-protocol` | ✓ | `defense/hanzo-zap-protocol.tex` |
|
||||
| `logo-convert` | — | `defense/logo-convert.tex` |
|
||||
| `earle-genetic-marketing` | ✓ | `earle-genetic-marketing/earle-genetic-marketing.tex` |
|
||||
| `enso` | ✓ | `enso/main.tex` |
|
||||
| `evolutionary-schedule-search` | ✓ | `evolutionary-schedule-search.tex` |
|
||||
| `hanzo-4-0-launch` | ✓ | `hanzo-4-0-launch/hanzo-4-0-launch.tex` |
|
||||
| `hanzo-aci` | ✓ | `hanzo-aci/hanzo-aci.tex` |
|
||||
| `hanzo-adversarial-mean-field-feedback` | ✓ | `hanzo-adversarial-mean-field-feedback.tex` |
|
||||
| `hanzo-agent-grpo` | ✓ | `hanzo-agent-grpo/hanzo-agent-grpo.tex` |
|
||||
| `hanzo-agent-sdk` | ✓ | `hanzo-agent-sdk/hanzo-agent-sdk.tex` |
|
||||
| `hanzo-agi-platform` | ✓ | `hanzo-agi-platform/hanzo-agi-platform.tex` |
|
||||
@@ -48,6 +51,7 @@ Auto-generated catalogue of research papers. Regenerate: `scripts/gen-index.sh`
|
||||
| `hanzo-chat` | ✓ | `hanzo-chat/hanzo-chat.tex` |
|
||||
| `hanzo-checkout` | ✓ | `hanzo-checkout/hanzo-checkout.tex` |
|
||||
| `hanzo-cloud-infrastructure` | ✓ | `hanzo-cloud-infrastructure/hanzo-cloud-infrastructure.tex` |
|
||||
| `hanzo-cloud-network-bench` | — | `hanzo-cloud-network-bench.tex` |
|
||||
| `hanzo-collapse-methodology` | ✓ | `hanzo-collapse-methodology.tex` |
|
||||
| `hanzo-commerce-payments` | ✓ | `hanzo-commerce-payments.tex` |
|
||||
| `hanzo-comptime-autotune` | ✓ | `hanzo-comptime-autotune.tex` |
|
||||
@@ -68,6 +72,7 @@ Auto-generated catalogue of research papers. Regenerate: `scripts/gen-index.sh`
|
||||
| `hanzo-edge-inference` | ✓ | `hanzo-edge-inference.tex` |
|
||||
| `hanzo-encryption-security` | ✓ | `hanzo-encryption-security/hanzo-encryption-security.tex` |
|
||||
| `hanzo-engine-ml` | ✓ | `hanzo-engine-ml.tex` |
|
||||
| `hanzo-evolutionary-orchestration` | ✓ | `hanzo-evolutionary-orchestration.tex` |
|
||||
| `hanzo-exchange-architecture` | ✓ | `hanzo-exchange-architecture/hanzo-exchange-architecture.tex` |
|
||||
| `hanzo-federated-agents` | ✓ | `hanzo-federated-agents/hanzo-federated-agents.tex` |
|
||||
| `hanzo-fhe-inference` | ✓ | `hanzo-fhe-inference/hanzo-fhe-inference.tex` |
|
||||
@@ -97,6 +102,7 @@ Auto-generated catalogue of research papers. Regenerate: `scripts/gen-index.sh`
|
||||
| `hanzo-llm-gateway` | ✓ | `hanzo-llm-gateway/hanzo-llm-gateway.tex` |
|
||||
| `hanzo-longform-media` | ✓ | `hanzo-longform-media.tex` |
|
||||
| `hanzo-mcp-server` | ✓ | `hanzo-mcp-server/hanzo-mcp-server.tex` |
|
||||
| `hanzo-mean-field-alignment` | ✓ | `hanzo-mean-field-alignment.tex` |
|
||||
| `hanzo-ml-framework` | ✓ | `hanzo-ml-framework/hanzo-ml-framework.tex` |
|
||||
| `hanzo-native-stack-thesis` | ✓ | `hanzo-native-stack-thesis.tex` |
|
||||
| `hanzo-native-training` | ✓ | `hanzo-native-training.tex` |
|
||||
@@ -128,7 +134,9 @@ Auto-generated catalogue of research papers. Regenerate: `scripts/gen-index.sh`
|
||||
| `hip-0302-replicate` | ✓ | `hip-0302-replicate/hip-0302-replicate.tex` |
|
||||
| `lightspeed-dex` | ✓ | `lightspeed-dex/lightspeed-dex.tex` |
|
||||
| `ntt-transform` | ✓ | `ntt-transform/ntt-transform.tex` |
|
||||
| `refutation-driven-perf` | ✓ | `refutation-driven-perf.tex` |
|
||||
| `verus-analytics` | ✓ | `verus-analytics/verus-analytics.tex` |
|
||||
| `training-without-retention` | ✓ | `zen/training-without-retention.tex` |
|
||||
| `zen-3d` | ✓ | `zen/zen-3d.tex` |
|
||||
| `zen3-embedding_whitepaper` | ✓ | `zen/zen3-embedding_whitepaper.tex` |
|
||||
| `zen3-guard_whitepaper` | ✓ | `zen/zen3-guard_whitepaper.tex` |
|
||||
@@ -215,8 +223,4 @@ Auto-generated catalogue of research papers. Regenerate: `scripts/gen-index.sh`
|
||||
| `ZEN_WHITEPAPER_2025` | ✓ | `zen/ZEN_WHITEPAPER_2025.tex` |
|
||||
| `zen-world` | ✓ | `zen/zen-world.tex` |
|
||||
|
||||
**Total**: 210 papers, 204 with compiled PDFs
|
||||
|
||||
| `hanzo-mean-field-alignment` | ✓ | `hanzo-mean-field-alignment.tex` |
|
||||
| `hanzo-evolutionary-orchestration` | ✓ | `hanzo-evolutionary-orchestration.tex` |
|
||||
| `hanzo-adversarial-mean-field-feedback` | ✓ | `hanzo-adversarial-mean-field-feedback.tex` |
|
||||
**Total**: 218 papers, 211 with compiled PDFs
|
||||
|
||||
Binary file not shown.
@@ -0,0 +1,578 @@
|
||||
\documentclass[11pt]{article}
|
||||
\input{shared/kernel-preamble.tex}
|
||||
\usepackage{algorithm}
|
||||
\usepackage{algpseudocode}
|
||||
|
||||
\title{\textbf{Evolutionary Schedule Search in a One-Source Kernel DSL,\\
|
||||
and the Continual-Improvement Loop It Anchors}}
|
||||
\author{Hanzo AI Research}
|
||||
\date{hanzo-kernel Paper 09 --- part of the \emph{One Source, Every Backend} series}
|
||||
|
||||
\begin{document}
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
A one-source kernel DSL exposes each operation's schedule --- tile dimensions, buffering depth, vector
|
||||
width, padding stride, per-shape selection --- as compile-time knobs, so every knob assignment is a distinct
|
||||
monomorphized kernel and the space of assignments is a discrete search problem. We already own the base
|
||||
algebra for it: a per-\code{(device,\,op,\,shape)} winner cache (\code{tune.rs}) that times a small variant
|
||||
set once and reuses the winner. This paper does three things. (1)~It reports the campaign's manual search as
|
||||
the only empirical sample --- roughly 15 hand-evaluated configurations --- and extracts the constants a
|
||||
search must respect: a static-rejection filter that is \emph{free} (register spill, LDS over the occupancy
|
||||
budget), a per-op fitness signal at $\pm0.4\%$ and a full-bench signal at $\pm0.8\%$, strongly
|
||||
\emph{non-additive} lever interaction (BK=64 lost alone; single-buffering won; the cross-product was never
|
||||
explored), and hard \emph{device non-transfer} (the reference's best-occupancy $64{\times}64$ tile is the
|
||||
\emph{worst} time on gfx1151). (2)~It specifies a multi-fidelity evolutionary schedule search over that
|
||||
space --- static rejection before any GPU time, bit-exactness as viability not penalty, sustained-only
|
||||
fitness through the honest instrument, refutation-log negative priors seeding the population, and an island
|
||||
model across a heterogeneous fleet --- with an eval-budget analysis showing $\sim$300 configurations/night
|
||||
against $\sim$15 by hand. (3)~It embeds the search in a continual-improvement loop: campaign transcripts
|
||||
distil into decision-point evaluations (the shipped 22-case \code{roofline} bench is the pilot) and
|
||||
process-supervision data; a \emph{calibrated judge flock}, weighted by measured ground truth rather than
|
||||
preference, selects training signal under a Goodhart tripwire; and the whole agent fleet coordinates through
|
||||
a shared field rather than pairwise, which we frame as the small-$N$ precursor of a mean-field game. The
|
||||
search is not yet run: its arithmetic is derived from measured constants, and the manual search is the only
|
||||
empirical evidence. We mark that boundary throughout.
|
||||
\end{abstract}
|
||||
|
||||
\section{The schedule is a value, and the space is searchable}
|
||||
\label{sec:intro}
|
||||
|
||||
In a multi-backend engine the recurring correctness hazard is ``the same operation, a different hand-tuned
|
||||
kernel per device, therefore a different numeric behavior.'' The DSL removes it by writing each operation
|
||||
once and lowering that one source to every backend. Autotuning closes the remaining loop: the \emph{schedule}
|
||||
(tile size, vector width, unroll factor, buffering, cube dimension) becomes a set of \code{\#[comptime]}
|
||||
knobs on that single source. Because the compiler monomorphizes on comptime state, each knob assignment is a
|
||||
distinct compiled kernel with identical semantics --- so exploring schedules is a search over compiled
|
||||
variants of one bit-exact source, never a fork.
|
||||
|
||||
The campaign of the companion study~\cite{refutation} produced the evidence that this search is worth
|
||||
automating: the winning schedules were \emph{not} the ones intuition or the reference implementation
|
||||
predicted, they interacted non-additively, and they did not transfer across devices. Those three facts
|
||||
(\S\ref{sec:handsearch}) are precisely the conditions under which a cached, per-device, measured search
|
||||
beats hand-tuning. We first ground the machinery in what already exists (\S\ref{sec:tuners}), then specify
|
||||
the search (\S\ref{sec:algorithm}), then embed it in a learning loop (\S\S\ref{sec:loop}--\ref{sec:meanfield}).
|
||||
|
||||
\section{Grounding: the existing schedule algebra}
|
||||
\label{sec:tuners}
|
||||
|
||||
The base layer is three orthogonal values in \code{hanzo-kernel/src/tune.rs}, deliberately kept as pure
|
||||
data so the mechanism is testable without a GPU.
|
||||
|
||||
\begin{itemize}[leftmargin=1.4em,itemsep=1pt,topsep=2pt]
|
||||
\item \textbf{\code{Variant}} --- a named schedule: a closure \code{f(iters) -> (output, ms)}. The same
|
||||
closure benchmarks (many iters) and executes (one iter), so a variant is both a candidate and a runnable
|
||||
kernel.
|
||||
\item \textbf{\code{Tuned}} --- a fluent builder collecting the variants for one \code{(op, shape-key)}.
|
||||
\item \textbf{\code{Tuner}} --- the cache: an in-memory \code{(device, op, key) $\to$ winner} map backed by
|
||||
one TSV file per device under the XDG cache dir. A cache \emph{miss} times every variant \code{TUNE\_ITERS=25}
|
||||
times and persists the fastest; a \emph{hit} launches only the winner once. It is a plain value over an
|
||||
explicit root, so the select/cache/reload logic is unit-tested with pure closures --- no GPU, no globals.
|
||||
\end{itemize}
|
||||
|
||||
The result of a run is a \code{Pick\{output, winner, from\_cache, benched, timings\}}; \code{benched} is $N$
|
||||
on a miss and $0$ on a hit, making ``the second call skips timing'' an observable. A first-party design was
|
||||
chosen over \code{cubecl\_runtime::tune} for three concrete reasons, not for their own sake: its tuner is
|
||||
re-exported only from \code{cubecl\_runtime} (adopting it would force a second engine crate into a facade
|
||||
architected so only \code{Cargo.toml} names the engine); its persistent cache lives under \code{target/} via
|
||||
an MD5 checksum and \code{serde}, not the required XDG path; and its \code{TuneInputs::At} GAT plus
|
||||
\code{KeyGenerator}/\code{InputGenerator} traits are far more surface than ``pick the fastest of $N$ named
|
||||
closures'' needs.
|
||||
|
||||
This layer already proves the loop on the CPU runtime: two ops (\code{rms\_norm}, \code{matvec\_q8\_dp4a})
|
||||
each expose one comptime-knobbed source with a 4-variant set; the tests show every variant bit-exact against
|
||||
the CPU oracle, a winner selected and cached in memory and on disk, and a fresh \code{Tuner} reloading the
|
||||
winner from disk. What is \emph{missing} is the search itself: the variant set is hand-written and tiny, the
|
||||
objective is a single timing with no multi-fidelity gating, and there is no population, no cross-device
|
||||
migration, and no negative prior. This paper specifies those.
|
||||
|
||||
\section{The only empirical sample: the manual search}
|
||||
\label{sec:handsearch}
|
||||
|
||||
The campaign hand-evaluated on the order of 15 schedule configurations for the two dominant prefill GEMMs on
|
||||
gfx1151. That sample --- small, but real and hardware-verified --- fixes the constants any automated search
|
||||
must obey.
|
||||
|
||||
\subsection{The fidelity ladder and its costs}
|
||||
\label{sec:fidelity}
|
||||
|
||||
Evaluation is not one operation but a ladder of increasing cost and decreasing throughput (Table~\ref{tab:fidelity}).
|
||||
The load-bearing fact is that the first rung is \emph{free}: a compile plus a read of shader statistics
|
||||
(registers, LDS bytes, scratch) rejects a large fraction of candidates --- anything that spills scratch
|
||||
($>0$) or exceeds the LDS occupancy budget --- before a single GPU dispatch. The campaign repeatedly killed
|
||||
configurations at this rung: a 32\,KB accumulator spill (occupancy capped at 2), a 40\,KB double-buffered
|
||||
K-tile (occupancy capped at 6 where registers alone allowed $\sim$9), an f16-accumulator variant whose VGPRs
|
||||
went \emph{up} $168\to180$. None of these needed a timed run to be disqualified.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering\small
|
||||
\setlength{\tabcolsep}{6pt}
|
||||
\begin{tabular}{@{}>{\raggedright\arraybackslash}p{0.225\textwidth}>{\raggedright\arraybackslash}p{0.265\textwidth}ll@{}}
|
||||
\toprule
|
||||
\textbf{Fidelity} & \textbf{Signal} & \textbf{Cost / config} & \textbf{Noise} \\
|
||||
\midrule
|
||||
static (compile + shaderstats) & scratch, LDS, VGPR, occupancy & seconds, no GPU & exact (deterministic) \\
|
||||
bit-exact oracle & \code{scale\_rel} vs CPU reference & seconds & exact (a gate, not a score) \\
|
||||
sustained per-op (in-engine) & GPU $\mu$s at the live shape & seconds & $\pm0.4\%$ \\
|
||||
full bench (in-engine) & pp512 / tg128 vs \code{llama} & minutes & $\pm0.8\%$ \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{The evaluation fidelity ladder. Static rejection and the bit-exact gate are effectively free
|
||||
relative to timed runs; the two timed rungs carry the $\pm0.4\%$ / $\pm0.8\%$ noise floors measured after the
|
||||
harness fix of~\cite{refutation}. Per-config compile time (a single-shader \code{glslc}/\code{spirv-opt} is
|
||||
seconds; a cubecl re-monomorphization plus engine relink is longer) is treated as an order-of-magnitude
|
||||
modeling constant, not a logged datapoint (\S\ref{sec:limits}).}
|
||||
\label{tab:fidelity}
|
||||
\end{table}
|
||||
|
||||
\subsection{Non-additivity and device non-transfer}
|
||||
\label{sec:nonadditive}
|
||||
|
||||
Two properties of the sample make greedy, one-lever-at-a-time tuning provably incomplete.
|
||||
|
||||
\textbf{Non-additivity.} The schedule knobs contend for shared occupancy resources (LDS bytes and VGPRs),
|
||||
so their effects are not separable. \code{BK}=64 \emph{lost} when changed alone (it doubled LDS and left
|
||||
occupancy at 6); single-buffering \emph{won} (occupancy $6\to8$, GEMM $232\to181\,\mu$s); the recorded losers
|
||||
also include \code{sb64} (single-buffer $\times$ BK=64) and \code{sb\_bm64} (single-buffer $\times$ BM=64).
|
||||
The eventual shipping schedule was \emph{single-buffer $\wedge$ BK=32 $\wedge$ BM=64 $\wedge$ packed-f16vec2
|
||||
LDS} --- reached by a chain of single-lever A/Bs, with the full cross-product never explored. A greedy walk
|
||||
that had accepted BK=64's solo loss as final would have masked a lever that only pays in combination; a
|
||||
search must be able to evaluate joint assignments.
|
||||
|
||||
\textbf{Device non-transfer.} The reference implementation selects a tuned tile \emph{family}
|
||||
(\{64{\texttt{x}}64, 128{\texttt{x}}128, 128{\texttt{x}}256\}) per GEMM shape for its target GPUs. On gfx1151
|
||||
the smallest of these, the $64{\times}64$ RN4 tile, achieved the \emph{best} occupancy of any variant (10
|
||||
subgroups/SIMD) and the \emph{worst} sustained time ($1915\,\mu$s vs the $128{\times}128$ RN8 tile's
|
||||
$1634\,\mu$s): the larger tile's activation-reuse amortization beats occupancy on this silicon. Independently,
|
||||
the optimal thread count for the block-reduce MoE matvec \emph{differs by kernel} on the same device (Q4\_K at
|
||||
\code{nt}=64, Q6\_K at \code{nt}=32). A schedule optimal on one device or one op is not merely suboptimal
|
||||
elsewhere --- it can be the worst choice. This is the empirical justification for a winner cache keyed on
|
||||
\code{(device, op, shape)} rather than a global constant, and for running the search on the deployment
|
||||
hardware.
|
||||
|
||||
\section{Multi-fidelity evolutionary schedule search}
|
||||
\label{sec:algorithm}
|
||||
|
||||
The search treats a schedule as a genome and the fidelity ladder as a cost-ordered filter, so that GPU time
|
||||
is spent almost entirely on candidates that have already survived free static analysis and the correctness
|
||||
gate.
|
||||
|
||||
\paragraph{Genome.} A schedule is the tuple
|
||||
\[
|
||||
g \;=\; \bigl(\underbrace{\mathrm{BM},\mathrm{BN},\mathrm{BK}}_{\text{tile}},\;
|
||||
\underbrace{\beta}_{\text{buffering}\in\{1,2\}},\;
|
||||
\underbrace{v}_{\text{vec width}\in\{\mathrm{u16},\mathrm{f16{\times}2},\mathrm{uvec4}\}},\;
|
||||
\underbrace{\sigma}_{\text{pad stride}},\;
|
||||
\underbrace{\pi}_{\text{per-shape selector}}\bigr),
|
||||
\]
|
||||
each coordinate a comptime knob, so $g$ names one monomorphized, bit-exact kernel. Selection is elitist with
|
||||
tournaments; recombination is uniform crossover over coordinates; mutation perturbs one coordinate within its
|
||||
legal set. The refutation log contributes two things to the genetics: it \emph{seeds} the initial population
|
||||
away from known-dead regions, and it \emph{masks} operators that only produce dead offspring (e.g.\ the
|
||||
coopmat-with-f16-accumulator direction, or BM${\ge}128$ on this device).
|
||||
|
||||
\begin{algorithm}[h]
|
||||
\caption{\textsc{EvoSchedule} --- multi-fidelity search over comptime schedules}
|
||||
\label{alg:evo}
|
||||
\begin{algorithmic}[1]
|
||||
\Require op $o$, shape $s$, device $d$, budget $B$, log $L$, population size $\mu$, offspring $\lambda$
|
||||
\State $P \gets \textsc{SeedPopulation}(o, d, L)$ \Comment{negative priors; mask dead operators}
|
||||
\State $\mathit{best} \gets \textsc{Incumbent}(o, s, d)$
|
||||
\While{$B$ not exhausted}
|
||||
\State $Q \gets \varnothing$
|
||||
\For{$i \gets 1 \dots \lambda$}
|
||||
\State $(a,b) \gets \textsc{Tournament}(P),\ \textsc{Tournament}(P)$
|
||||
\State $c \gets \textsc{Mutate}(\textsc{UniformCrossover}(a,b))$ \Comment{or \textsc{LlmMutate}$(a,L)$, \S\ref{sec:llmop}}
|
||||
\If{$\textsc{Spill}(c,d) > 0 \ \textbf{or}\ \textsc{Lds}(c,d) > \textsc{OccBudget}(d)$}
|
||||
\State \textbf{continue} \Comment{FREE static reject; no GPU time} \label{ln:static}
|
||||
\EndIf
|
||||
\State $Q \gets Q \cup \{c\}$
|
||||
\EndFor
|
||||
\For{$c \in Q$} \Comment{only static survivors reach the GPU}
|
||||
\If{$\textsc{ScaleRel}(c, \textsc{Oracle}(s)) > \tau$}
|
||||
\State $c.\mathit{fit} \gets \infty$ \Comment{infeasible: viability, not a penalty} \label{ln:viability}
|
||||
\Else
|
||||
\State $c.\mathit{fit} \gets \textsc{SustainedPerOp}(c, s, d)$ \Comment{honest instrument, $\pm0.4\%$} \label{ln:fit}
|
||||
\EndIf
|
||||
\EndFor
|
||||
\State $P \gets \textsc{ElitistSelect}(P \cup Q,\ \mu)$
|
||||
\State $\mathit{best} \gets \arg\min_{c} c.\mathit{fit}$
|
||||
\State $\textsc{Migrate}(d, \mathit{best})$ \Comment{island model: publish winner to the fleet, \S\ref{sec:islands}}
|
||||
\EndWhile
|
||||
\State \textbf{assert} \textsc{FullBenchWin}$(\mathit{best})$ vs \code{llama}, same run, $\pm0.8\%$ \Comment{final gate}
|
||||
\State \Return $\mathit{best}$
|
||||
\end{algorithmic}
|
||||
\end{algorithm}
|
||||
|
||||
Three design choices distinguish this from a generic evolutionary tuner. Line~\ref{ln:static} makes static
|
||||
rejection \emph{free and first}: no fitness is defined for a candidate that spills or overflows LDS, so those
|
||||
never consume GPU time. Line~\ref{ln:viability} treats bit-exactness as \emph{viability, not a soft penalty}
|
||||
--- an incorrect kernel does not get a discounted fitness, it does not reproduce. Line~\ref{ln:fit} scores on
|
||||
sustained in-engine per-op time, never a cache-warm microbenchmark, because for memory-bound kernels the warm
|
||||
number systematically overstates the real regime (the campaign burned several otherwise-correct kernels on
|
||||
exactly that lie).
|
||||
|
||||
\subsection{Island model over a heterogeneous fleet}
|
||||
\label{sec:islands}
|
||||
|
||||
Because fitness is device-specific (\S\ref{sec:nonadditive}), the search is an island model: one deme per
|
||||
box (evo/gfx1151, spark/GB10, dbc/M4~Max), each evolving against its own hardware. Migration carries a
|
||||
winning \emph{genome}, not a fitness --- an immigrant is always re-evaluated locally, because a schedule that
|
||||
tops one island can be the worst on another. Migration is therefore a source of good \emph{structure} to
|
||||
recombine, not a claim of cross-device optimality. This is the fleet-level view whose coordination we
|
||||
formalize in \S\ref{sec:meanfield}.
|
||||
|
||||
\subsection{LLM-guided mutation as an optional operator}
|
||||
\label{sec:llmop}
|
||||
|
||||
The mutation operator can be replaced or augmented by a language model that proposes a new genome from the
|
||||
parent plus the refutation log --- the pattern of program-search systems such as FunSearch~\cite{funsearch}
|
||||
and AlphaEvolve~\cite{alphaevolve}, where an LLM proposes and an external evaluator selects. Here the
|
||||
selector is exactly the gate ladder of~\cite{refutation}: an LLM proposal enjoys no privilege, it must pass
|
||||
static rejection, the bit-exact oracle, and sustained timing like any other candidate. The LLM supplies
|
||||
search \emph{bias} (informed guesses about which joint assignments to try, drawn from the log's mechanisms);
|
||||
the hardware supplies \emph{truth}. This keeps the creative operator unfalsifiable-proof: it can only ever
|
||||
propose, never accept.
|
||||
|
||||
\subsection{Eval-budget arithmetic}
|
||||
\label{sec:budget}
|
||||
|
||||
The point of cost-ordering is a favorable generation budget. Take $\lambda=16$ offspring per generation. The
|
||||
campaign's static-rejection experience (spills and LDS-overflows were common among hand-tried variants)
|
||||
puts the free-reject rate near $50\%$, so $\sim$8 candidates per generation reach the GPU. Each GPU
|
||||
evaluation is a bit-exact check plus a 25-iteration sustained per-op timing --- seconds of GPU time --- plus
|
||||
a monomorphization/relink amortized across the batch. At an effective few minutes per surviving candidate, a
|
||||
single box clears on the order of a few hundred configurations in an overnight window; against the
|
||||
$\sim$15 the campaign evaluated by hand over weeks, that is a $10$--$20\times$ throughput increase in
|
||||
schedules examined, with the full-bench rung (minutes, $\pm0.8\%$) reserved for the handful of per-op
|
||||
finalists. The arithmetic is a projection from measured constants, not a measured result
|
||||
(\S\ref{sec:limits}); its purpose is to show the search is budget-feasible on existing hardware, not to
|
||||
promise a specific speedup.
|
||||
|
||||
\section{The continual-improvement loop}
|
||||
\label{sec:loop}
|
||||
|
||||
The search consumes agent-run optimization campaigns and, in turn, those campaigns produce data that
|
||||
improves the agents. The loop has four stages.
|
||||
|
||||
\begin{enumerate}[leftmargin=1.6em,itemsep=2pt,topsep=2pt]
|
||||
\item \textbf{Run} campaigns (human- or agent-driven) under the gate ladder, logging every decision point:
|
||||
the evidence on the table, the action space, and the measured outcome.
|
||||
\item \textbf{Distil} each transcript into \emph{decision-point} triples $\langle$evidence-snapshot,
|
||||
action-space, measured-outcome$\rangle$. These become (a)~new evaluation cases and (b)~process-supervision
|
||||
targets.
|
||||
\item \textbf{Select} training signal with an evaluation that rewards the right \emph{process}, not only the
|
||||
right answer (trap-resistance and an overthink penalty as auxiliary rewards).
|
||||
\item \textbf{Improve} the models, which become better mutation operators (\S\ref{sec:llmop}) and better
|
||||
campaign agents --- closing the loop.
|
||||
\end{enumerate}
|
||||
|
||||
\subsection{Decision-point distillation and its schema}
|
||||
\label{sec:distill}
|
||||
|
||||
The pipeline's pilot already ships: the 22-case \code{roofline} benchmark is decision-point distillation run
|
||||
by hand over the campaign of~\cite{refutation}. Each case is one record:
|
||||
\begin{quote}\small
|
||||
\code{\{id, category, trap, question:\{evidence, choices\}, answer, rationale\}}
|
||||
\end{quote}
|
||||
where \code{answer} is the hardware-verified outcome and the distractors are the campaign's \emph{actual}
|
||||
refuted theories. Categories span \code{diagnosis}, \code{experiment}, \code{trap}, \code{update},
|
||||
\code{calculation}, \code{discipline}, and \code{synthesis}; 6 of 22 are traps (the prompt asserts a false
|
||||
premise the evidence contradicts). Ground truth is unpublished,
|
||||
so the set is uncontaminated by pretraining. Crucially, the standard results JSONL already logs
|
||||
\code{usages.completion\_tokens} per item, so two post-hoc metrics need no new instrumentation: the
|
||||
\emph{overthink index} (mean completion tokens on \emph{correct} answers) and \emph{trap resistance}
|
||||
(accuracy on the trap subset).
|
||||
|
||||
\subsection{Reward shaping}
|
||||
\label{sec:reward}
|
||||
|
||||
Selection optimizes a shaped reward rather than raw accuracy, so that a model is rewarded for reasoning
|
||||
\emph{to} the measurement and penalized for reasoning \emph{past} it:
|
||||
\[
|
||||
R \;=\; \underbrace{\mathbb{1}[\hat a = a^\star]}_{\text{accuracy}}
|
||||
\;+\; \lambda_{\mathrm{trap}}\cdot \mathbb{1}[\text{trap} \wedge \hat a = a^\star]
|
||||
\;-\; \lambda_{\mathrm{ot}}\cdot \frac{\mathrm{tokens\_on\_correct}}{T_{\mathrm{ref}}} .
|
||||
\]
|
||||
The trap term overweights the cases where the prompt's framing is adversarial (verify-before-comply); the
|
||||
overthink term, normalized by a reference length $T_{\mathrm{ref}}$, taxes the models that talk themselves
|
||||
back into a refuted theory. The \code{rationale} field supplies process-supervision targets: the reward is
|
||||
not only on the boxed letter but on recovering the decisive experiment. This is implementable directly
|
||||
against the existing harness --- the arms are the answer letters, the results JSONL carries the token usages,
|
||||
and the golds are the hardware-verified answers.
|
||||
|
||||
\subsection{Calibrated judge flocks}
|
||||
\label{sec:judges}
|
||||
|
||||
Selecting training signal at scale needs judges, and LLM-as-judge pipelines are usually \emph{preference-only}:
|
||||
they rank outputs by what a model prefers, inheriting position, verbosity, and self-preference
|
||||
biases~\cite{judgebias}. This stack has a rarer asset --- \emph{anchored calibration}: subsets with measured
|
||||
ground truth against which a judge's reliability is a fact, not an opinion. Four anchors exist today: the 22
|
||||
hardware-verified \code{roofline} answers (6 of them traps); the router corpus of \textbf{2{,}981}
|
||||
counterfactual outcomes (373 prompts $\times$ 8 model arms); the bit-exact kernel gates; and the enso-bench
|
||||
golds. The router corpus is
|
||||
concrete evidence that calibration is separable from preference: a first learned routing head scored
|
||||
$\mathrm{AIQ}=0.6248$ against a $0.7427$ heuristic and was \emph{not} shipped; after a train/serve featurizer
|
||||
skew was fixed (it had trained without prompt text), the head reached held-out decision quality
|
||||
$0.9215\,{\pm}\,0.011$ versus the rule's $0.9162\,{\pm}\,0.007$, with an oracle ceiling of $0.9402$. The
|
||||
anchored eval, not a preference signal, is what exposed the skew.
|
||||
|
||||
\begin{algorithm}[h]
|
||||
\caption{\textsc{CalibratedJudgeFlock} --- anchored, diverse, disagreement-aware judging}
|
||||
\label{alg:flock}
|
||||
\begin{algorithmic}[1]
|
||||
\Require judges $J$ (family-diverse), anchored sets $A_{1..m}$ with golds, candidate pool $C$, student $\theta$
|
||||
\For{$j \in J$, domain $k$} \Comment{(1) reliability from measured truth, per domain}
|
||||
\State $w_{j,k} \gets \textsc{Reliability}\bigl(j \text{ on } A_k\bigr)$ \Comment{accuracy vs golds; also tokens/correct-verdict}
|
||||
\EndFor
|
||||
\For{$x \in C$}
|
||||
\State $\{s_{j}\}_{j\in J} \gets \textsc{Verdicts}(J, x)$
|
||||
\State $\bar s(x) \gets \dfrac{\sum_j w_{j,\mathrm{dom}(x)}\, s_j}{\sum_j w_{j,\mathrm{dom}(x)}}$ \Comment{(3) reliability-weighted aggregation}
|
||||
\If{$\textsc{Var}_j(s_j) > \eta$} \Comment{(4) disagreement-as-curriculum}
|
||||
\State escalate $x$ to a debate round or human; route to the high-value training pool \label{ln:escalate}
|
||||
\EndIf
|
||||
\EndFor
|
||||
\State $\theta' \gets \textsc{Finetune}_{\mathrm{DPO/RLAIF}}(\theta,\ \bar s,\ \text{anchored items overweighted})$ \Comment{(5)}
|
||||
\If{$\textsc{Anchored}(\theta') < \textsc{Anchored}(\theta) \ \wedge\ \textsc{FlockPref}(\theta') > \textsc{FlockPref}(\theta)$}
|
||||
\State \textbf{halt}; recalibrate \Comment{(6) Goodhart tripwire: judge-hacking detected} \label{ln:goodhart}
|
||||
\EndIf
|
||||
\State \textbf{re-measure} $w_{j,k}$ next cycle; version the flock \Comment{(7) a frozen flock is the student's ceiling}
|
||||
\State \Return $\theta'$
|
||||
\end{algorithmic}
|
||||
\end{algorithm}
|
||||
|
||||
Five properties are load-bearing. \textbf{Diversity is not decoration}: same-family judges share correlated
|
||||
blind spots that vote aggregation cannot remove, so the flock is drawn across model families --- consistent
|
||||
with the Panel-of-LLM-evaluators result that a small, diverse jury beats a single frontier judge on both bias
|
||||
and cost~\cite{poll}. \textbf{Reliability is per-domain}: a judge trustworthy on kernel-perf reasoning is not
|
||||
automatically trustworthy on prose, so weights are conditioned on the domain of the item. \textbf{Disagreement
|
||||
is a hardness detector} (line~\ref{ln:escalate}): high inter-judge variance marks the items worth escalating
|
||||
to a debate round~\cite{debate} or a human --- averaging them away discards the highest-value training data.
|
||||
\textbf{The Goodhart tripwire} (line~\ref{ln:goodhart}) is the anchored holdout's job: a student that improves
|
||||
on flock \emph{preference} while regressing on measured \emph{truth} is hacking the judges, and the cycle
|
||||
halts. \textbf{The flock is versioned and re-measured} each cycle, because a frozen flock is a hard ceiling on
|
||||
the student it trains. Judges are themselves scored on tokens-per-correct-verdict --- the overthink index of
|
||||
\S\ref{sec:distill} applies to judges too, keeping the panel cheap.
|
||||
|
||||
This training-time flock is the mirror of a shipped inference-time one: the \code{enso-ultra} product line's
|
||||
verify-then-select over a diverse candidate set \emph{is} a flock at inference. The contribution here is to
|
||||
run the same diverse-panel discipline at training time, and to anchor it --- because we can --- on measured
|
||||
ground truth rather than preference alone. The base techniques are external and cited plainly: learning from
|
||||
AI feedback~\cite{cai}, preference optimization~\cite{dpo}, and the judge-bias and debate literatures
|
||||
\cite{judgebias,debate}; the anchoring and the Goodhart tripwire are what the measured corpora add.
|
||||
|
||||
\section{Generalization: the oracle-substitution rule}
|
||||
\label{sec:general}
|
||||
|
||||
The architecture --- cheap-reject ladder, honest instrument, refutation-log prior, evolutionary search under
|
||||
a viability gate, distillation into decision-point evals --- is not specific to kernels. It ports wherever
|
||||
two things exist: a \emph{trustworthy fitness oracle} and a \emph{cheap rejection ladder} beneath it. Only
|
||||
the oracle changes.
|
||||
|
||||
\begin{itemize}[leftmargin=1.4em,itemsep=2pt,topsep=2pt]
|
||||
\item \textbf{Coding agents.} Fitness oracle: the test suite, the type checker, the runtime verifier. Cheap
|
||||
rejects: compile failure and static analysis stand in for shaderstats; a failing test stands in for a
|
||||
bit-exactness miss. Verified-campaign transcripts distil into decision-point evals exactly as here, and the
|
||||
process-supervision target is the sequence of decisive checks, not just the final diff.
|
||||
\item \textbf{Marketing agents.} Fitness oracle: an A/B conversion metric. The \emph{same} disciplines
|
||||
transfer unchanged --- contended-measurement refusal (a concurrent campaign contaminates the lift the way a
|
||||
foreign GPU job contaminates a kernel A/B), a noise floor below which a lift is unreadable, and a
|
||||
refutation log of dead creative directions. What changes is only the oracle: a \emph{statistical-significance
|
||||
gate} replaces bit-exactness. Viability becomes ``the lift clears the significance threshold,'' not ``the
|
||||
bits match.''
|
||||
\end{itemize}
|
||||
|
||||
The rule has a hard boundary, which we state rather than paper over: the architecture works wherever fitness
|
||||
is \emph{measurable and cheap enough to gate on}, and it fails where fitness is not. A task whose only
|
||||
signal is diffuse long-horizon human judgment, with no anchor and no cheap reject, has no honest instrument
|
||||
for the ladder to escalate through --- and there the search degenerates into preference-chasing, precisely
|
||||
the failure mode the judge flock (\S\ref{sec:judges}) is built to detect. The method is as strong as its
|
||||
weakest oracle.
|
||||
|
||||
\section{Mean-field coordination of the agent fleet}
|
||||
\label{sec:meanfield}
|
||||
|
||||
The search (\S\ref{sec:islands}), the judge flock (\S\ref{sec:judges}), and the campaigns themselves are run
|
||||
by a fleet of agents sharing scarce resources. The campaign wrote its coordination laws by hand; we formalize
|
||||
them here as the small-$N$ precursor of a mean-field game, and are explicit about where the analogy holds and
|
||||
where it does not.
|
||||
|
||||
\subsection{The observed game}
|
||||
\label{sec:mf-game}
|
||||
|
||||
$N$ agents contend for shared resources: GPUs (a hard ``one agent per box'' law), the \code{git main}
|
||||
version queue (publish sequencing), \code{cargo} locks, and the search-space basins themselves. The log
|
||||
records two failures of \emph{pairwise} coordination and one success of \emph{field} coordination.
|
||||
|
||||
\begin{itemize}[leftmargin=1.4em,itemsep=2pt,topsep=2pt]
|
||||
\item \textbf{Failure --- the patch-line race.} An agent's local \code{[patch.crates-io]} redirect raced into
|
||||
a peer's release commit; a pre-commit grep passed, then the line was re-added before the commit landed, so
|
||||
the tagged release did not build off-box until a follow-up fixed it. A pairwise ``check then act'' lost to a
|
||||
concurrent writer.
|
||||
\item \textbf{Failure --- branch-delete before merge.} An agent deleted a remote feature branch before
|
||||
verifying the merge had landed; the fast-forward had aborted because \code{main} moved under it, and the
|
||||
commits had to be rescued from the local object store and re-landed. Again: the shared state moved between
|
||||
one agent's fetch and its act.
|
||||
\item \textbf{Success --- the front map.} Publishing a \emph{population-level} summary --- which fronts had
|
||||
explorers, which trees were owned --- caused a peer agent to \emph{stand down} of its own accord, twice,
|
||||
leaving its checkout clean and detached. The peer best-responded to the field (the occupancy of fronts), not
|
||||
to any individual agent.
|
||||
\end{itemize}
|
||||
|
||||
The pattern is exactly the mean-field intuition: pairwise coordination is $O(N^2)$ interactions and races on
|
||||
shared mutable state; field coordination is each agent best-responding to an aggregate, at $O(N)$
|
||||
communication.
|
||||
|
||||
\subsection{The formalism}
|
||||
\label{sec:mf-formalism}
|
||||
|
||||
Let $\mu$ be the \emph{field}: a distribution over resource states --- per-box GPU occupancy, per-tree
|
||||
ownership density, the version queue, and per-basin explorer density in schedule space. Each agent solves a
|
||||
best-response
|
||||
\[
|
||||
\pi^\star \;=\; \arg\max_{\pi}\; \mathbb{E}\bigl[\, \underbrace{V(\pi)}_{\text{lever value}}
|
||||
\;-\; \underbrace{c\bigl(\mu(\pi)\bigr)}_{\text{congestion}}
|
||||
\;-\; \underbrace{\kappa\!\cdot\!\mathrm{switch}(\pi)}_{\text{switching cost}} \,\bigr],
|
||||
\]
|
||||
and an equilibrium is a field $\mu^\star$ induced by the agents' best responses to $\mu^\star$ --- a fixed
|
||||
point, in the sense of the mean-field game limit of Lasry and Lions~\cite{lasrylions} and the Nash-certainty-equivalence
|
||||
of Huang, Malhamé and Caines~\cite{hmc}. The solver the campaign already ran is \emph{fictitious play}:
|
||||
publish the current field, let agents best-respond, republish. Convergence is expected under monotone
|
||||
congestion costs (the Lasry--Lions monotonicity condition): a second agent on a box, a second explorer in a
|
||||
basin, is strictly worse for everyone there, which is the ``one agent per box'' and anti-crowding rules made
|
||||
quantitative.
|
||||
|
||||
\begin{algorithm}[h]
|
||||
\caption{\textsc{MeanFieldFleetCoordinator} --- interact through the field, not pairwise}
|
||||
\label{alg:mfc}
|
||||
\begin{algorithmic}[1]
|
||||
\Require agents $\{1..N\}$, resources $\mathcal{R}$ (boxes, trees, version queue, basins)
|
||||
\Loop \Comment{fictitious play: the publish/respond cycle the campaign ran}
|
||||
\State $\mu \gets \textsc{Field}(\mathcal{R})$: \{GPU busy, tree-ownership density, version queue, basin explorer density\}
|
||||
\For{agent $a$ \textbf{in parallel}} \Comment{$O(N)$ communication: each reads $\mu$, none reads peers}
|
||||
\State $a.\pi \gets \arg\max_{\pi}\ V_a(\pi) - c(\mu(\pi)) - \kappa\,\mathrm{switch}_a(\pi)$
|
||||
\State \textbf{if} $\mu(\text{box}(\pi)) > 0$ \textbf{then} best-respond elsewhere \Comment{one agent per box}
|
||||
\EndFor
|
||||
\State \textsc{Publish}$(\mu')$ \Comment{next field; version-queue writes take the next free patch, index-verified}
|
||||
\EndLoop
|
||||
\end{algorithmic}
|
||||
\end{algorithm}
|
||||
|
||||
\subsection{The router is a textbook congestion game}
|
||||
\label{sec:mf-router}
|
||||
|
||||
The same structure appears one level down, at inference routing. A population of requests routed to a set of
|
||||
model arms degrades those arms' latency and cost as load rises --- a congestion game whose equilibrium is
|
||||
Wardrop routing~\cite{wardrop}: traffic distributes so that no request can lower its own cost by switching
|
||||
arms, i.e.\ each request best-responds to the \emph{arm-load field}. This is directly implementable on the
|
||||
existing seam: the \code{hanzo-router} replica layer already tracks per-arm load and prefix affinity, so the
|
||||
learned routing head of \S\ref{sec:judges} need only take expected-load arguments in its SLO terms to route
|
||||
against the field rather than against a static cost table.
|
||||
|
||||
\subsection{Search and judges as low-density equilibria}
|
||||
\label{sec:mf-searchjudge}
|
||||
|
||||
Two earlier constructions are special cases. The island search (\S\ref{sec:islands}) becomes an
|
||||
\emph{anti-crowding} field over schedule-space basins --- explorers are pushed toward under-occupied basins ---
|
||||
of which the refutation log is the static, infinite-density limit (a killed basin has permanent congestion
|
||||
cost $\infty$, so no agent ever re-enters). The judge flock (\S\ref{sec:judges}) is a low-density equilibrium
|
||||
in \emph{error-correlation} space: the diversity premium is exactly a congestion cost that penalizes adding a
|
||||
judge whose blind spots correlate with the panel's, driving the flock toward independent error.
|
||||
|
||||
\subsection{Honest boundaries}
|
||||
\label{sec:mf-bounds}
|
||||
|
||||
Two caveats are mandatory. First, this is \emph{mean-field control} (MFC), not a true mean-field
|
||||
\emph{game}: we own every agent's policy, so the fleet is a cooperative planner minimizing a shared cost, not
|
||||
a population of selfish players. The genuine-game regime --- selfish agents, external model arms, multi-tenant
|
||||
SaaS pricing --- is where the Nash rather than the social-optimum solution matters, and we do not claim the
|
||||
equilibrium analysis there. Second, the fleet is \emph{small}: $N\approx3$--$6$. Mean-field approximation
|
||||
error scales as $O(1/\sqrt{N})$, so at this scale the equilibrium \emph{guarantees} are asymptotic and not in
|
||||
force. What \emph{is} validated at small $N$ is the \emph{architecture}: interact through a published field,
|
||||
$O(N)$ communication, no pairwise races --- which turned the two $O(N^2)$ coordination failures above into
|
||||
the one $O(N)$ success. The asymptotics are the reason to keep the architecture as the fleet grows, not a
|
||||
claim about today's fleet.
|
||||
|
||||
\section{Limitations}
|
||||
\label{sec:limits}
|
||||
|
||||
\begin{itemize}[leftmargin=1.4em,itemsep=2pt,topsep=2pt]
|
||||
\item \textbf{The search is designed, and its first hunt is only now in progress.} Algorithm~\ref{alg:evo}
|
||||
and its budget (\S\ref{sec:budget}) are derived from measured constants (the fidelity noise floors, the
|
||||
static-reject experience, the non-additivity and non-transfer of the sample); no completed evolutionary run
|
||||
exists yet, and its result against the hand-search is exactly the controlled experiment that will confirm or
|
||||
refute the loop's value over an expert (\cite{refutation}, velocity section). The $\sim$15-config manual
|
||||
search is the only empirical schedule-search data in this paper, and it is a convenience sample from one
|
||||
campaign, not a controlled sweep.
|
||||
\item \textbf{The compile-cost constant is modeled.} The per-config compile time used in the budget is an
|
||||
order-of-magnitude estimate from build observations, not a logged measurement; the $\pm0.4\%$/$\pm0.8\%$
|
||||
noise floors and the ``shaderstats is free'' rung are the sourced constants.
|
||||
\item \textbf{The LLM operator is unproven here.} We adopt the propose-and-select pattern on external
|
||||
evidence (FunSearch~\cite{funsearch}, AlphaEvolve~\cite{alphaevolve}); we have not measured an LLM mutation
|
||||
operator against random mutation on our own search.
|
||||
\item \textbf{The judge flock is specified, partially anchored.} The four anchor corpora are real and sized
|
||||
(notably the 2{,}981-event router corpus), but the full flock loop --- disagreement escalation, the Goodhart
|
||||
tripwire, multi-cycle reliability re-measurement --- is specified, not yet run end-to-end.
|
||||
\item \textbf{Mean-field is small-$N$.} As \S\ref{sec:mf-bounds} states, the equilibrium guarantees are
|
||||
asymptotic; at $N{=}3$--$6$ only the architecture is validated, by three logged anecdotes (two failures, one
|
||||
success), which is illustration, not proof.
|
||||
\end{itemize}
|
||||
|
||||
\section{Conclusion}
|
||||
\label{sec:conc}
|
||||
|
||||
The schedule is a value, the space of schedules is searchable, and the campaign proved the search worth
|
||||
automating by showing its winners were non-obvious, non-additive, and non-transferable. We specified a
|
||||
multi-fidelity evolutionary search that spends GPU time only on candidates already past a free static filter
|
||||
and a correctness gate, seeded by a log of what is already known to be dead. We embedded it in a loop that
|
||||
turns campaign transcripts into decision-point evaluations and process supervision, selects training signal
|
||||
through a judge flock anchored on measured ground truth under a Goodhart tripwire, and coordinates the whole
|
||||
agent fleet through a shared field rather than pairwise. The through-line is a single discipline: build the
|
||||
honest instrument first, let it --- not intuition, not the reference, not a preference model --- decide, and
|
||||
write down every refutation so it is never bought twice. The search and the flock are that discipline made
|
||||
mechanical; the mean-field frame is that discipline made to scale.
|
||||
|
||||
\begin{thebibliography}{99}
|
||||
\small
|
||||
\bibitem{refutation} Hanzo AI Research. \emph{Refutation-Driven Performance Engineering: An Empirical Study
|
||||
of a Multi-Week GPU Kernel Campaign.} hanzo-kernel Paper 08 (companion).
|
||||
\bibitem{tunepaper} Hanzo AI Research. \emph{The Schedule Is a Value: Comptime-Specialized Autotuning.}
|
||||
hanzo-kernel Paper 07.2. The \code{Variant}/\code{Tuned}/\code{Tuner} algebra of \S\ref{sec:tuners}.
|
||||
\bibitem{cubecl} Tracel AI. \emph{CubeCL: multi-platform GPU compute in Rust.}
|
||||
\url{https://github.com/tracel-ai/cubecl}. Comptime monomorphization; the studied \code{cubecl\_runtime::tune}.
|
||||
\bibitem{llamacpp} G.~Gerganov and the \code{ggml}/\code{llama.cpp} contributors. \emph{\code{llama.cpp}.}
|
||||
\url{https://github.com/ggml-org/llama.cpp}. The same-box baseline and tile-family reference.
|
||||
\bibitem{funsearch} B.~Romera-Paredes, M.~Barekatain, A.~Novikov, et al. \emph{Mathematical discoveries from
|
||||
program search with large language models.} Nature \textbf{625}, 468--475 (2024). LLM-proposes /
|
||||
evaluator-selects.
|
||||
\bibitem{alphaevolve} Google DeepMind. \emph{AlphaEvolve: A coding agent for scientific and algorithmic
|
||||
discovery.} Technical report (2025). Evolutionary program search with an LLM mutation operator.
|
||||
\bibitem{ansor} L.~Zheng, C.~Jia, M.~Sun, et al. \emph{Ansor: Generating High-Performance Tensor Programs for
|
||||
Deep Learning.} OSDI (2020). Schedule search over tensor programs.
|
||||
\bibitem{tvm} T.~Chen, T.~Moreau, Z.~Jiang, et al. \emph{TVM: An Automated End-to-End Optimizing Compiler for
|
||||
Deep Learning.} OSDI (2018).
|
||||
\bibitem{opentuner} J.~Ansel, S.~Kamil, K.~Veeramachaneni, et al. \emph{OpenTuner: An Extensible Framework
|
||||
for Program Autotuning.} PACT (2014). Ensemble search over program-configuration spaces.
|
||||
\bibitem{atlas} R.~C.~Whaley and J.~Dongarra. \emph{Automatically Tuned Linear Algebra Software (ATLAS).}
|
||||
Supercomputing (1998). Empirical per-machine tuning of a numeric kernel library.
|
||||
\bibitem{poll} P.~Verga, S.~Hofstatter, S.~Althammer, et al. \emph{Replacing Judges with Juries: Evaluating
|
||||
LLM Generations with a Panel of Diverse Models (PoLL).} Cohere (2024). Small diverse juries beat a single
|
||||
frontier judge on bias and cost.
|
||||
\bibitem{cai} Y.~Bai, S.~Kadavath, S.~Kundu, et al. \emph{Constitutional AI: Harmlessness from AI Feedback.}
|
||||
Anthropic (2022). Learning from model feedback (RLAIF).
|
||||
\bibitem{dpo} R.~Rafailov, A.~Sharma, E.~Mitchell, et al. \emph{Direct Preference Optimization.} NeurIPS
|
||||
(2023).
|
||||
\bibitem{judgebias} L.~Zheng, W.-L.~Chiang, Y.~Sheng, et al. \emph{Judging LLM-as-a-Judge with MT-Bench and
|
||||
Chatbot Arena.} NeurIPS (2023). Documents position, verbosity, and self-preference bias in LLM judges.
|
||||
\bibitem{debate} G.~Irving, P.~Christiano, and D.~Amodei. \emph{AI Safety via Debate.} arXiv:1805.00899
|
||||
(2018). Escalation of hard items to a debate round.
|
||||
\bibitem{lasrylions} J.-M.~Lasry and P.-L.~Lions. \emph{Mean field games.} Japanese Journal of Mathematics
|
||||
\textbf{2}, 229--260 (2007).
|
||||
\bibitem{hmc} M.~Huang, R.~P.~Malhamé, and P.~E.~Caines. \emph{Large-population stochastic dynamic games:
|
||||
closed-loop McKean--Vlasov systems and the Nash certainty equivalence principle.} Communications in
|
||||
Information and Systems \textbf{6}(3), 221--252 (2006).
|
||||
\bibitem{wardrop} J.~G.~Wardrop. \emph{Some theoretical aspects of road traffic research.} Proceedings of the
|
||||
Institution of Civil Engineers \textbf{1}(3), 325--362 (1952). Equilibrium routing under congestion.
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
Binary file not shown.
@@ -0,0 +1,539 @@
|
||||
\documentclass[11pt]{article}
|
||||
\input{shared/kernel-preamble.tex}
|
||||
\usepackage{algorithm}
|
||||
\usepackage{algpseudocode}
|
||||
|
||||
\title{\textbf{Refutation-Driven Performance Engineering:\\
|
||||
An Empirical Study of a Multi-Week GPU Kernel Campaign}}
|
||||
\author{Hanzo AI Research}
|
||||
\date{hanzo-kernel Paper 08 --- part of the \emph{One Source, Every Backend} series}
|
||||
|
||||
\begin{document}
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
We report an observational study of a single multi-week campaign to close the inference-throughput gap
|
||||
against \code{llama.cpp} on three accelerators (AMD gfx1151 RDNA3.5 APU, NVIDIA GB10 Blackwell, Apple M4
|
||||
Max), conducted inside a one-source GPU kernel DSL. The campaign's primary artifact is not a kernel but a
|
||||
\emph{refutation log}: twenty-five numbered hypotheses, most of them plausible and several of them
|
||||
our own, each killed by a cheap decisive experiment. We formalize the practice behind it. First, the
|
||||
quantified outcomes: a Vulkan-prefill \emph{win ladder} of $212\to1734$ tok/s ($8.2\x$ of the gap closed in
|
||||
a single $\sim$36-hour release cadence, from $11.6\x$ behind the reference to $1.40\x$), decomposed per
|
||||
lever; a CUDA decode gap closed $1.23\x\to1.003\x$ by a single model-eligibility list entry with
|
||||
\emph{zero} kernel work; and an instrument fix that collapsed
|
||||
prefill benchmark noise from $\pm42\%$ to $\pm0.8\%$ ($52\x$) and thereby made the second half of the win
|
||||
ladder measurable at all. Second, three algorithms that recur across the log: a multi-fidelity
|
||||
\emph{Gate Ladder} that rejects candidates cheapest-first (free static rejection $\to$ bit-exact oracle
|
||||
$\to$ sustained per-op timing $\to$ in-engine wall $\to$ coherence); a \emph{decisive-experiment}
|
||||
selection rule (the cheapest single-variable control that can falsify a live hypothesis); and
|
||||
\emph{refutation-log-as-prior}, negative constraints injected into later work, which provably prevented at
|
||||
least nine documented re-chases. Third, the measurement doctrine as testable claims (same-run ratios only;
|
||||
sustained-vs-sustained; contended-timing refusal; warm-microbench distrust). We close by describing how the
|
||||
discipline was operationalized as a 22-case language-model evaluation whose distractors are the campaign's
|
||||
own refuted theories. We do not overclaim the standing: across the measured fronts the campaign holds two
|
||||
wins (ROCm decode $1.07\x$; a $470\x$ margin over the fork's upstream on Metal), one parity (CUDA decode),
|
||||
and three open gaps (Vulkan prefill/decode, CUDA prefill) --- ``beat everywhere'' is the objective, not the
|
||||
achieved state. This is an $N{=}1$ study with no control group; we state that boundary explicitly.
|
||||
\end{abstract}
|
||||
|
||||
\section{Thesis: the refutation is the deliverable}
|
||||
\label{sec:thesis}
|
||||
|
||||
Conventional performance write-ups report wins. This campaign's ledger is dominated by \emph{kills}: of the
|
||||
levers seriously proposed and evaluated, the majority were refuted, and refuting them was the value
|
||||
produced --- because a refuted lever, recorded with its mechanism, removes an entire branch of future work.
|
||||
The campaign ran against a fixed external baseline (\code{llama.cpp}, the reference LLM inference engine)
|
||||
on the same silicon and the same model files, so every claim reduces to a measured ratio on shared
|
||||
hardware. The engineering question was never ``is our kernel fast?'' but ``\emph{which} of these competing
|
||||
stories about the bottleneck survives contact with an instrument?''
|
||||
|
||||
The stakes are concrete. On the AMD gfx1151 APU (evo), our Vulkan prefill began at $212$ tok/s against a
|
||||
same-box \code{llama.cpp} figure of $2459$ --- an $11.6\x$ deficit. Twenty-five refutations later it stood
|
||||
at $1739.6\,{\pm}\,7.4$ vs $2441.9\,{\pm}\,22.2$ same-run, $1.40\x$ behind. The distance between those endpoints is a
|
||||
sequence of decisions, and most of the decisions were \emph{not to do the attractive thing}. This paper is
|
||||
about how those decisions were made and what they cost.
|
||||
|
||||
\paragraph{Environment.} Three self-hosted boxes, each its own backend: \textbf{evo} --- AMD Ryzen AI
|
||||
Max+~395 / Radeon 8060S, gfx1151 (RDNA3.5 APU, unified memory), bare-metal Linux 6.17, ROCm 7.2.4, Vulkan
|
||||
via RADV; \textbf{spark} --- NVIDIA GB10, compute capability 12.1 (sm\_121, Blackwell), aarch64;
|
||||
\textbf{dbc} --- Apple M4 Max, 40-core GPU, macOS~26. Models: \code{zen-eco-4b} (a Qwen2-3B derivative)
|
||||
Q4\_K\_M, and Qwen3-30B-A3B-Instruct Q4\_K\_M (18.5\,GB). Reference: \code{llama.cpp} (Vulkan/CUDA/Metal/HIP
|
||||
backends) and its \code{llama-bench}, run on the identical box and GGUF file. All comparisons in this paper
|
||||
are same-box, same-model; cross-box numbers are labelled as such.
|
||||
|
||||
\section{The refutation dataset}
|
||||
\label{sec:dataset}
|
||||
|
||||
Table~\ref{tab:refutations} is the study's core observation: a census of the campaign's hypotheses,
|
||||
classified by the \emph{kind} of story each told and annotated with the decisive experiment that settled
|
||||
it, its approximate cost to kill, and what killing it saved. We use four hypothesis classes:
|
||||
|
||||
\begin{itemize}[leftmargin=1.4em,itemsep=1pt,topsep=2pt]
|
||||
\item \textbf{mechanistic-kernel} --- a claim that some change to a compute kernel (fuse these ops, cut this
|
||||
dispatch count, use tensor cores, enlarge the tile) will move the wall clock.
|
||||
\item \textbf{configuration-gap} --- the bottleneck is not the kernel but a build flag, an eligibility
|
||||
list, a harness setting, or a dispatch-orchestration pattern.
|
||||
\item \textbf{measurement-artifact} --- the ``finding'' was an instrument lie: a warm microbenchmark, a
|
||||
warmup-shape mismatch, contended timing, or mixed-phase telemetry sampling.
|
||||
\item \textbf{hardware-story} --- a narrative about the silicon: thermal throttling, a DVFS
|
||||
utilization spiral, a sustained power limit.
|
||||
\end{itemize}
|
||||
|
||||
\begin{table}[t]
|
||||
\centering\scriptsize
|
||||
\setlength{\tabcolsep}{3.5pt}
|
||||
\renewcommand{\arraystretch}{1.18}
|
||||
\begin{tabular}{@{}>{\raggedright\arraybackslash}p{0.205\textwidth}>{\raggedright\arraybackslash}p{0.115\textwidth}>{\raggedright\arraybackslash}p{0.30\textwidth}>{\raggedright\arraybackslash}p{0.075\textwidth}>{\raggedright\arraybackslash}p{0.175\textwidth}@{}}
|
||||
\toprule
|
||||
\textbf{Hypothesis} & \textbf{Class} & \textbf{Decisive experiment} & \textbf{Cost} & \textbf{What it saved} \\
|
||||
\midrule
|
||||
Cut router GPU$\to$CPU$\to$GPU id syncs ($144$/tok) speeds decode & mechanistic-kernel & Removed the syncs; \code{VK\_PROFILE} decode A/B & minutes & Retired ``sync count'' as a decode lever (8.97$\to$9.14, flat) \\
|
||||
Fuse the MoE router op-chain (softmax+argsort) & mechanistic-kernel & Fused to one kernel; dispatches $55272\to43656$ ($-21\%$); measure t/s & hours & Retired ``cheap-dispatch count'' (decode $\sim$9$\to$8.56, flat) \\
|
||||
Fused GQA flash attention is the decode lever & mechanistic-kernel & Built \code{sdpa\_blk} (roofline $284$ GB/s); fused ON/OFF A/B & days & ``Attention was never the lever'' (2.2 vs 2.3 t/s, identical) \\
|
||||
Prefill is submission-bound; a command graph is the fix & measurement-artifact & \code{fence\_wait}$=$per-op GPU sum $\Rightarrow$ GPU is computing & minutes & Graph saves $\sim$0.6\% of prefill, not the $11.6\x$; redirected to the GEMM \\
|
||||
Q6\_K prefill needs a tiled GEMM (not a column matvec) & mechanistic-kernel & Per-op profile: \code{mul\_mat\_q6k} $=76\%$ of prefill & minutes & The first real lever: $212\to704$ t/s \\
|
||||
32KB accumulator spill caps occupancy & mechanistic-kernel & shaderstats before/after (scratch $32768\to0$) & free & occupancy $2\to6$; op $1722\to1634\,\mu$s \\
|
||||
\code{llama} 64$\times$64 tile (RN4) is better (max occupancy) & mechanistic-kernel & Sustained per-op: RN4 $1915$ vs RN8 $1634\,\mu$s & hours & Occupancy-maximalism killed; kept RN8 \\
|
||||
Sustained power throttle / DVFS spiral (29\%@713 vs 77\%@2431 MHz) & hardware-story & Pin clock 2900 MHz; forward still $540$ ms & minutes & Throttle + downclock narratives both dead \\
|
||||
Per-forward state accumulates (``120ms$\to$523ms'') & measurement-artifact & \code{warmup=0}: forward \#1 $=573$ ms, all flat & minutes & Killed the ``memory leak'' hunt \\
|
||||
The ``120ms fast window'' is real GPU capability & measurement-artifact & Byte counters: $334.6$ vs $8364$ MB at equal dispatch count & minutes & 32-tok warmup phantom; killed 3 theories at once \\
|
||||
Brief: port \code{llama}'s BK=64 into our coopmat & configuration-gap & Read source: \code{:3968} overrides BK=32 for coopmat & minutes & Refuted the brief; BK=64 later tried and lost \\
|
||||
f16 accumulators raise occupancy past 8 & mechanistic-kernel & shaderstats: LDS-limited (20480B), VGPR $168\to180$ & minutes & ``\code{llama} does it'' is not a mechanism \\
|
||||
Double-buffer the K-tile to hide latency & mechanistic-kernel & Weight stream $=10.8\%$ of peak (no latency crisis) & hours & Single-buffer: occ $6\to8$; $1383\to1734$ t/s \\
|
||||
Coopmat (tensor cores) beats \code{dp4a} on gfx1151 & mechanistic-kernel & Cache-cold in-engine: coopmat $4\x$ worse e2e & hours & Killed until the access pattern, not the cores, was fixed \\
|
||||
Bigger BM tile cuts cold-weight re-reads & mechanistic-kernel & \code{VK\_Q4K\_BM} A/B: BM128/256 $2.4\x$ slower & minutes & Occupancy collapse; BM64 is the sweet spot \\
|
||||
Coalesced-cooperative layout will speed Q4\_K matvec too & mechanistic-kernel & In-engine A/B: $115$ vs $169$ GB/s ($1.46\x$ slower) & hours & Lever value $=$ distance-from-wall; Q4\_K already at 83\% \\
|
||||
CUDA prefill needs the \code{flash-attn} feature & configuration-gap & Idle-window A/B: flash-ON $3.6\%$ slower & hours & Redirected to \code{flashinfer} + glue, not v2 flash \\
|
||||
CUDA decode is $1.23\x$ because of a slow kernel & configuration-gap & \code{Model::Qwen} absent from graph-eligibility list & minutes & One arm $\Rightarrow$ $1.23\x\to1.003\x$, no kernel touched \\
|
||||
Metal is $21\x$ behind: the tiled GEMM was never written & configuration-gap & Read backend: \code{kernel\_mul\_mm\_*} exists for every quant & minutes & Refused to port an existing kernel; re-measure first \\
|
||||
Quantize-fusion (drop 96 dispatches/tok) is a perf win & mechanistic-kernel & Decode A/B: $+2.2\%$, error bars touching & hours & Kept as cleanliness, not perf; ``10 for 10'' on counts \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{The refutation census (representative subset of the log). ``Cost'' is order-of-magnitude
|
||||
engineer time to the decisive result. The modal class is \emph{mechanistic-kernel} (the attractive story
|
||||
is almost always ``write a better kernel''); the modal \emph{outcome} is refutation. Several rows kill our
|
||||
own prior conclusions --- the log is adversarial toward its author.}
|
||||
\label{tab:refutations}
|
||||
\end{table}
|
||||
|
||||
Two structural facts fall out of the census. (1)~The attractive hypothesis is nearly always
|
||||
mechanistic-kernel --- ``fuse it,'' ``use the tensor cores,'' ``cut the dispatches'' --- and it is the
|
||||
class most often refuted. (2)~The largest single wins were \emph{configuration-gap}: a Cargo feature, an
|
||||
enum arm, a warmup shape. Kernels get the glory; configuration eats the throughput. This asymmetry is the
|
||||
study's headline qualitative finding, and \S\ref{sec:findings} quantifies it.
|
||||
|
||||
\section{Five quantified findings}
|
||||
\label{sec:findings}
|
||||
|
||||
\subsection{The win ladder: $212\to1734$ tok/s, per lever}
|
||||
\label{sec:ladder}
|
||||
|
||||
Table~\ref{tab:ladder} attributes the Vulkan-prefill gain to individual shipped levers. The kernel-level
|
||||
mechanism of each is recorded so the ladder is auditable, not a single aggregate number. Every rung is
|
||||
bit-exact-gated against the CPU oracle and coherence-checked before shipping.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering\small
|
||||
\setlength{\tabcolsep}{5pt}
|
||||
\renewcommand{\arraystretch}{1.12}
|
||||
\begin{tabular}{@{}l >{\raggedright\arraybackslash}p{0.28\textwidth} r >{\raggedright\arraybackslash}p{0.34\textwidth}@{}}
|
||||
\toprule
|
||||
\textbf{Ship} & \textbf{Lever} & \textbf{pp512 (t/s)} & \textbf{Op-level result} \\
|
||||
\midrule
|
||||
--- start & column Q6\_K matvec, no tiled path & $212$ & $11.6\x$ behind \code{llama} \\
|
||||
ml 0.11.64 & Q6\_K 2D-tiled \code{dp4a} GEMM & $704$ & Q6\_K op $7.5$--$29.8\x$; it was $76\%$ of prefill \\
|
||||
eng v1.7.60 & warmup-shape bench fix $+$ Q4\_K coalesced coopmat & $946.5\,{\pm}\,7.6$ & noise $\pm42\%\!\to\!\pm0.8\%$; prefill A/B now legible \\
|
||||
ml 0.11.69 & packed f16vec2 LDS (wide \code{ds\_load}) & $1264.8\,{\pm}\,4.1$ & Q4\_K GEMM $356\!\to\!233$ ms ($1.53\x$); VGPR $204\!\to\!168$ \\
|
||||
ml 0.11.70 & Q6\_K \code{dp4a}$\to$coopmat & $1383.1\,{\pm}\,5.0$ & Q6\_K op $104\!\to\!62$ ms ($1.67\x$) \\
|
||||
ml 0.11.71 & single-buffer, BK=32 & $1734.4\,{\pm}\,4.4$ & Q4\_K $232\!\to\!181$ ms (occ $6\!\to\!8$); Q6\_K $62\!\to\!38.7$ ms \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Vulkan prefill win ladder on evo (gfx1151, \code{zen-eco-4b} Q4\_K\_M), CTO-frozen figures.
|
||||
$212\to1734.4$ is $8.2\x$ shipped in one campaign; the same-run gap to \code{llama} went $11.6\x$ to
|
||||
$1.40\x$ behind. Current standing (ml 0.11.72 / engine v1.7.65): $1739.6\,{\pm}\,7.4$ vs \code{llama}
|
||||
$2441.9\,{\pm}\,22.2$ same run ($0.71\x$ of the reference). The $704$ rung predates the warmup-shape harness
|
||||
fix (\S\ref{sec:variance}); $946.5$ onward are on the $\pm0.8\%$ honest bench. Every rung is bit-exact-gated
|
||||
and coherence-checked.}
|
||||
\label{tab:ladder}
|
||||
\end{table}
|
||||
|
||||
The ladder's shape is itself evidence for the thesis. The first rung ($3.3\x$, Q6\_K tiling) was found not
|
||||
by intuition but by a per-op profiler that named \code{mul\_mat\_q6k} as $76\%$ of prefill GPU time --- a
|
||||
quant type that had no tiled path because it appears in only $\sim$2 tensors per layer (\code{attn\_v},
|
||||
\code{ffn\_down}) and was therefore invisible to attention aimed at the dominant Q4\_K weights. Every later
|
||||
rung was a small, measured, single-variable change read out of the reference implementation's source and
|
||||
A/B'd in-engine. No rung came from a redesign.
|
||||
|
||||
\subsection{The CUDA gap decomposition: configuration vs kernel}
|
||||
\label{sec:cuda}
|
||||
|
||||
The CUDA front is the cleanest natural experiment separating configuration from kernel work, because its two
|
||||
phases resolved differently. On spark (GB10, \code{zen-eco-4b} Q4\_K\_M, same-run):
|
||||
|
||||
\begin{itemize}[leftmargin=1.4em,itemsep=1pt,topsep=2pt]
|
||||
\item \textbf{Decode} began at $63.3\,{\pm}\,0.2$ vs \code{llama} $77.6\,{\pm}\,0.9$ ($1.23\x$). The
|
||||
profiler (\code{nsys}) showed $\sim$1{,}110 kernel launches per token against \code{llama}'s $\sim$1 graph
|
||||
launch. The engine already had a decode command-graph path, fail-closed behind a bit-exact gate and a
|
||||
\emph{model-eligibility list}. \code{Model::Qwen} was simply not in the list (and decode RoPE read a
|
||||
host-side offset that freezes under graph capture --- fixed by the device-positions pattern the eligible
|
||||
models already used). Adding the one arm collapsed launches to $\sim$1 per token and took decode to the
|
||||
CTO-frozen $81.9\,{\pm}\,0.6$ vs \code{llama} $82.15\,{\pm}\,1.76$ --- $1.23\x\to1.003\x$ \textbf{parity,
|
||||
with zero kernel rewrites}.
|
||||
\item \textbf{Prefill} remains open at $5003.4\,{\pm}\,23.0$ vs \code{llama} $6079\,{\pm}\,326$ ($0.82\x$)
|
||||
and is glue-bound: naive attention \code{softmax\_f32} $\sim$13--15\%, \code{badd\_bf16} $10.5\%$, dtype
|
||||
casts $6.2\%$, \code{ucopy} $4.2\%$. Here the attractive lever was itself refuted (\#25): building with the
|
||||
\code{flash-attn} feature made prefill $3.6\%$ \emph{slower} ($4820.8\,{\pm}\,166.3$), because the paged
|
||||
bench's gather path forces naive attention regardless and Ampere-era flash-v2 kernels do not win on
|
||||
Blackwell. The residual is genuine kernel/dtype work (route \code{flashinfer} prefill, tighten
|
||||
bf16$\leftrightarrow$f32 discipline) --- in progress.
|
||||
\end{itemize}
|
||||
|
||||
The decomposition is stark: the \emph{entire} CUDA decode deficit was configuration (a graph-eligibility
|
||||
list entry plus a device-placement fix), closed to statistical parity without touching a compute kernel;
|
||||
the CUDA prefill deficit is what is actually left for kernel engineering. When a fresh front is opened, the
|
||||
prior that ``the gap is plumbing'' pays before the prior that ``the gap is a slow kernel.''
|
||||
|
||||
\subsection{Instrument variance collapse: $\pm42\% \to \pm0.8\%$}
|
||||
\label{sec:variance}
|
||||
|
||||
For much of the campaign the prefill benchmark was unusable for A/B testing: repeated runs of the same
|
||||
configuration reported $728\,{\pm}\,306$ tok/s ($\pm42\%$), and in the worst observations $\pm48$--$62\%$.
|
||||
The cause was a harness defect, not the GPU: warmup ran a fixed 32-token prefill while the measured batch
|
||||
was 512 tokens, so the first measured forward paid a one-time Vulkan pipeline compile and the warmup
|
||||
touched a different shape than the measurement. The fix (engine v1.7.60) warms the \emph{measured} shapes;
|
||||
prefill went to $946.5\,{\pm}\,7.6$ ($\pm0.8\%$) --- a $\mathbf{52\x}$ variance collapse.
|
||||
|
||||
The downstream effect is what makes this a headline rather than a footnote. A change worth $+5\%$ is
|
||||
invisible under $\pm42\%$ noise (the ratio of signal to noise is $0.12$); it is unambiguous under
|
||||
$\pm0.8\%$. Decode was always tight ($\pm1.3\%$), so decode A/Bs were resolvable throughout --- which is
|
||||
exactly why the decode wins landed earlier. Prefill A/Bs became resolvable \emph{only after} the harness
|
||||
fix, and the entire second half of the win ladder ($946\to1734$, the packed-LDS, Q6\_K-coopmat, and
|
||||
single-buffer rungs, each a $1.2$--$1.7\x$ op-level change) was A/B-decided on the honest bench. The
|
||||
instrument fix did not speed up a kernel; it made three subsequent kernel wins \emph{legible}.
|
||||
|
||||
\subsection{The 32-token warmup phantom: one artifact, three dead theories}
|
||||
\label{sec:phantom}
|
||||
|
||||
The most expensive single artifact of the campaign was the same warmup-shape bug viewed from the other
|
||||
side. A profiler labelled every prefill batch \code{dispatch=1949} --- because dispatch count is
|
||||
per-layer-ops $\times$ 36 layers, hence \emph{token-count-independent}. The 32-token warmup batches
|
||||
therefore printed an identical dispatch count to the 512-token measured batches while doing $16\x$ less
|
||||
work (byte counters: $334.6$ vs $8364$ MB). The visible symptom --- ``the first $\sim$5 forwards run
|
||||
$\sim$120 ms, then collapse to $\sim$523 ms'' --- looked like accumulating degradation, and spawned three
|
||||
competing hardware/leak stories:
|
||||
|
||||
\begin{enumerate}[leftmargin=1.6em,itemsep=1pt,topsep=2pt]
|
||||
\item a \emph{DVFS utilization death-spiral} (mixed-phase telemetry read $29\%$ busy @ 713 MHz for us vs
|
||||
$77\%$ @ 2431 MHz for \code{llama} on the same silicon);
|
||||
\item \emph{accumulating engine state} / a per-forward memory leak;
|
||||
\item a \emph{sustained power throttle}.
|
||||
\end{enumerate}
|
||||
|
||||
All three were killed by two single-variable controls plus one counter read. Pinning the clock at 2900 MHz
|
||||
left the 512-token forward at $540$ ms (throttle and downclock require the clock to move --- it did not).
|
||||
Running \code{warmup=0} made forward \#1 $573$ ms and every subsequent forward $\sim$545--561 ms flat (no
|
||||
accumulation; stable from the first forward). And the byte counters showed the ``fast'' forwards were
|
||||
simply smaller. The tell, in hindsight, was that \emph{the fast-window length always equalled the warmup
|
||||
count}. One mislabelled invariant --- treating a shape-independent dispatch count as workload identity ---
|
||||
manufactured three plausible mechanisms and consumed multi-agent effort until the counters ended it. The
|
||||
lesson entered the log as a rule: \emph{dispatch count is not workload identity; check tokens and bytes
|
||||
before comparing two batches.}
|
||||
|
||||
\subsection{Optimization velocity: the campaign clock}
|
||||
\label{sec:velocity}
|
||||
|
||||
If the tooling of \S\ref{sec:algorithms} has a reason to exist, it is velocity: the loop must close gaps
|
||||
faster than unaided hand-tuning, or it is ceremony. The campaign's release history is the direct evidence,
|
||||
and its timestamps are real (Table~\ref{tab:velocity}, from the \code{ml} and \code{engine} git logs). The
|
||||
Vulkan-prefill ladder of \S\ref{sec:ladder} --- $212\to1734.4$ tok/s, $8.2\x$ of gap closed --- shipped
|
||||
across nine tagged releases spanning $\sim$36 hours, and its four consecutive coopmat rungs
|
||||
($946.5\to1734.4$) landed inside a single window of $\sim$5.5 hours (11:28--14:58 on 2026-07-18). The CUDA
|
||||
decode close to parity (\S\ref{sec:cuda}) shipped the same evening. Twenty-five numbered refutations gate
|
||||
this cadence, each with a cost-to-kill of minutes to hours (Table~\ref{tab:refutations}).
|
||||
|
||||
\begin{table}[h]
|
||||
\centering\small
|
||||
\setlength{\tabcolsep}{5pt}
|
||||
\renewcommand{\arraystretch}{1.1}
|
||||
\begin{tabular}{@{}l l >{\raggedright\arraybackslash}p{0.23\textwidth} >{\raggedright\arraybackslash}p{0.27\textwidth}@{}}
|
||||
\toprule
|
||||
\textbf{Release} & \textbf{Landed (PDT)} & \textbf{Change} & \textbf{Measured effect} \\
|
||||
\midrule
|
||||
ml 0.11.64 & 07-17 & Q6\_K tiled GEMM & pp512 $212\to704$ \\
|
||||
ml 0.11.67 / eng 1.7.58 & 07-17 21{:}23 & coalesced coopmat Q4\_K & Q4\_K GEMM $473\to369$ ms \\
|
||||
eng v1.7.60 & 07-18 09{:}45 & warmup-shape bench fix & noise $\pm42\%\to\pm0.8\%$ \\
|
||||
ml 0.11.69 / eng 1.7.61 & 07-18 11{:}28 & packed f16vec2 LDS & pp512 $946.5\to1264.8$ \\
|
||||
ml 0.11.70 / eng 1.7.62 & 07-18 12{:}30 & Q6\_K coopmat & pp512 $1264.8\to1383.1$ \\
|
||||
ml 0.11.71 / eng 1.7.63 & 07-18 14{:}58 & single-buffer coopmat & pp512 $1383.1\to1734.4$ \\
|
||||
ml 0.11.72 / eng 1.7.64 & 07-18 17{:}07 & coalesced Q6\_K decode matvec & decode $43.0\to50.6$ \\
|
||||
eng v1.7.65 & 07-18 17{:}18 & qwen2 CUDA/ROCm decode graph & CUDA decode $\to 1.003\x$ parity \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Release cadence with real git commit timestamps (\code{ml} and \code{engine} logs, PDT). Nine
|
||||
releases in $\sim$36 hours; the four coopmat prefill rungs within a single $\sim$5.5-hour window. This is
|
||||
the ``why the tooling exists'' evidence: the loop's throughput is in the timestamps.}
|
||||
\label{tab:velocity}
|
||||
\end{table}
|
||||
|
||||
\paragraph{The honest contrast.} It is tempting to set this cadence against the maturity of the baseline,
|
||||
and the comparison must be made carefully. \code{llama.cpp}'s Vulkan matmul lead was built incrementally by
|
||||
many contributors over roughly two years: its Vulkan shader path dates to mid-2024, cooperative-matrix
|
||||
support to December 2024 (PRs \#10206/\#10713), the DP4A MMQ + Q8\_1 quantization shader to March 2025
|
||||
(PR \#12135), and a \code{mul\_mm} shader cleanup to September 2025 (PR \#15987), among others --- dates read
|
||||
directly from the local \code{llama.cpp} checkout's git history. We closed $8.2\x$ of the gap to that
|
||||
artifact in $\sim$36 hours. This is \textbf{not} a claim that our method is hundreds of times more
|
||||
productive than theirs: the comparison is method-versus-method \emph{on different starting points}. They
|
||||
pioneered the kernels with no map; we chased, using their shipped source as the map (the BK, tile, and
|
||||
buffering constants were read out of their code, then A/B'd on our silicon). \textbf{Chasing a known-good
|
||||
target is strictly easier than pioneering it}, and the velocity figure reflects that asymmetry as much as it
|
||||
reflects the loop.
|
||||
|
||||
\paragraph{The falsification path.} The velocity claim --- that the loop out-paces hand-tuning --- is
|
||||
falsifiable, and two controlled experiments are the honest way to settle it, both listed here rather than
|
||||
assumed. (a)~The in-flight automated schedule search~\cite{tune} is machine-search against our own
|
||||
hand-search on the \emph{same} kernels: if the machine cannot match or beat the hand-picked schedules per
|
||||
unit compute, the loop's marginal value over an expert is refuted. (b)~A from-zero replication on an
|
||||
untouched front (Metal, currently blocked by an offline box) measures time-to-parity with no prior map on
|
||||
that backend; a slow replication would show the $36$-hour figure was mostly borrowed from the reference,
|
||||
not generated by the loop. Until those run, the honest statement is narrow: \emph{on this campaign, the
|
||||
measured cadence out-paced this team's prior hand-tuning}, and the mechanism (cheap refutation plus a map)
|
||||
is legible in the timestamps.
|
||||
|
||||
\section{Three algorithms}
|
||||
\label{sec:algorithms}
|
||||
|
||||
The practices above are not ad hoc; they instantiate three procedures that recur throughout the log.
|
||||
|
||||
\subsection{The Gate Ladder}
|
||||
\label{sec:gateladder}
|
||||
|
||||
Verification is multi-fidelity: each candidate passes through gates ordered by increasing cost, and is
|
||||
rejected at the cheapest gate that can reject it. Free static analysis (compile + shader statistics) filters
|
||||
occupancy-killers and register spills before any GPU time is spent; the bit-exact oracle enforces
|
||||
correctness as \emph{viability} (an incorrect kernel is not a slow candidate, it is a non-candidate);
|
||||
sustained per-op timing on the honest instrument decides ``is it a win''; the in-engine wall A/B and a
|
||||
coherence check confirm the win survives to the deployed regime.
|
||||
|
||||
\begin{algorithm}[h]
|
||||
\caption{\textsc{GateLadder} --- multi-fidelity acceptance, cheapest rejection first}
|
||||
\label{alg:gate}
|
||||
\begin{algorithmic}[1]
|
||||
\Require candidate schedule $C$, shape $s$, device $d$, incumbent time $t_\star$
|
||||
\State $(\mathit{spill}, \mathit{lds}, \mathit{occ}) \gets \textsc{Compile+ShaderStats}(C, d)$ \Comment{free; no GPU}
|
||||
\If{$\mathit{spill} > 0$ \textbf{or} $\mathit{lds} > \textsc{OccBudget}(d)$} \State \Return \textsc{Reject}(``static'') \EndIf
|
||||
\State $y_C \gets \textsc{Run}(C, s, d)$; \; $y_{\mathrm{ref}} \gets \textsc{CpuOracle}(s)$
|
||||
\If{$\textsc{ScaleRel}(y_C, y_{\mathrm{ref}}) > \tau$} \State \Return \textsc{Reject}(``not bit-exact'') \Comment{viability, not penalty} \EndIf
|
||||
\State $t_C \gets \textsc{SustainedPerOp}(C, s, d)$ \Comment{median of 25, $\pm0.4\%$, in-engine} \label{ln:sustained}
|
||||
\If{$t_C \ge t_\star$} \State \Return \textsc{Reject}(``no op-level win'') \EndIf
|
||||
\State $(w_C, w_\star, w_{\mathrm{ref}}) \gets \textsc{SameRunWall}(C, \text{incumbent}, \text{llama})$ \Comment{$\pm0.8\%$}
|
||||
\If{$w_C \not> w_\star$} \State \Return \textsc{Reject}(``no wall win'') \EndIf
|
||||
\If{\textbf{not} \textsc{Coherent}(C)} \State \Return \textsc{Reject}(``incoherent output'') \EndIf
|
||||
\State \Return \textsc{Accept}(C) \Comment{ship: bit-exact $\wedge$ op-win $\wedge$ wall-win $\wedge$ coherent}
|
||||
\end{algorithmic}
|
||||
\end{algorithm}
|
||||
|
||||
The ladder's economic point is the ordering. Static rejection is free and, empirically, disqualified a
|
||||
large fraction of hand-tried variants (register spills, LDS over the occupancy budget) before they ever
|
||||
reached the GPU. Line~\ref{ln:sustained} is deliberately \emph{not} a cache-warm microbenchmark
|
||||
(\S\ref{sec:doctrine}); the honest per-op timer is the first gate that costs real GPU seconds, and it is
|
||||
placed after the two free/cheap gates for that reason.
|
||||
|
||||
\subsection{Decisive-experiment selection}
|
||||
\label{sec:decisive}
|
||||
|
||||
Given a set of live hypotheses, the campaign repeatedly chose the \emph{cheapest single-variable
|
||||
experiment that could falsify the most of them at once}. The pinned-clock control (\S\ref{sec:phantom})
|
||||
is the exemplar: one knob, minutes, and it bears on both the throttle and the downclock stories
|
||||
simultaneously. The rule prefers controls that change exactly one variable, because a confounded
|
||||
experiment cannot cleanly kill anything.
|
||||
|
||||
\begin{algorithm}[h]
|
||||
\caption{\textsc{DecisiveExperiment} --- maximize hypotheses-killed per unit cost}
|
||||
\label{alg:decisive}
|
||||
\begin{algorithmic}[1]
|
||||
\Require live hypotheses $H$, available experiments $X$
|
||||
\State $\mathit{best} \gets \varnothing$; \; $\mathit{score}_\star \gets \infty$
|
||||
\For{$x \in X$ \textbf{such that} $x$ varies exactly one factor} \Comment{single-variable controls only}
|
||||
\State $K_x \gets \{\, h \in H : x \text{ can falsify } h \,\}$ \Comment{the kill set}
|
||||
\If{$|K_x| = 0$} \State \textbf{continue} \EndIf
|
||||
\State $\mathit{score} \gets \textsc{Cost}(x) \,/\, |K_x|$ \Comment{cost per hypothesis killed}
|
||||
\If{$\mathit{score} < \mathit{score}_\star$} \State $\mathit{score}_\star \gets \mathit{score}$; \; $\mathit{best} \gets x$ \EndIf
|
||||
\EndFor
|
||||
\State \Return $\mathit{best}$ \Comment{e.g.\ pin-clock kills throttle \emph{and} downclock in one run}
|
||||
\end{algorithmic}
|
||||
\end{algorithm}
|
||||
|
||||
\subsection{Refutation-log-as-prior}
|
||||
\label{sec:logprior}
|
||||
|
||||
A refutation is only worth its cost if it is not re-purchased. Every kill was written to a durable log with
|
||||
its mechanism, and subsequent task briefs were seeded with the log's negative constraints: forbidden
|
||||
re-chases and masked dead operators. This converts a one-time experiment into a permanent search-space
|
||||
restriction.
|
||||
|
||||
\begin{algorithm}[h]
|
||||
\caption{\textsc{SeedBrief} --- inject refutations as negative priors}
|
||||
\label{alg:seed}
|
||||
\begin{algorithmic}[1]
|
||||
\Require task $T$, refutation log $L$
|
||||
\State $\mathit{forbid} \gets \varnothing$; \; $\mathit{deadOps} \gets \varnothing$
|
||||
\For{$r \in L$}
|
||||
\If{$\textsc{Class}(r) \in \{\text{mechanistic-kernel}, \text{hardware-story}\}$}
|
||||
\State $\mathit{forbid} \gets \mathit{forbid} \cup \{(\,r.\text{lever},\ r.\text{mechanism}\,)\}$
|
||||
\EndIf
|
||||
\If{$r.\text{outcome} = \text{regress}$} \State $\mathit{deadOps} \gets \mathit{deadOps} \cup \{r.\text{operator}\}$ \EndIf
|
||||
\EndFor
|
||||
\State \Return \textsc{Brief}$(T,\ \mathbf{forbid}{=}\mathit{forbid},\ \mathbf{mask}{=}\mathit{deadOps})$
|
||||
\end{algorithmic}
|
||||
\end{algorithm}
|
||||
|
||||
The log records at least nine re-chases this prior provably prevented, each carrying an explicit ``do NOT
|
||||
re-chase'' marker traceable to a prior measurement: coopmat on gfx1151 (killed, then partially un-refuted
|
||||
only when the access pattern --- not the tensor cores --- was identified as the cause); \code{BM}${\ge}128$
|
||||
tiles ($2.4\x$ slower, occupancy collapse); \code{BK}${=}64$ on the coopmat path (llama's override read
|
||||
correctly); the RN4 / BM=64 tile family (best occupancy, worst time); f16 accumulators (LDS-limited, not
|
||||
register-limited); the per-shape tile family (skinny projections are $1.9\%$ of Q4\_K time); the Q4\_K
|
||||
coalesced-cooperative rewrite (already at $83\%$ of peak); dispatch/instruction-count reduction as a
|
||||
\emph{perf} lever (``10 for 10 --- counts never predict wall-clock on this decode path''); and Ampere-era
|
||||
\code{flash-attn} v2 on Blackwell ($3.6\%$ slower). Each avoided re-chase is a multi-hour experiment not
|
||||
re-run.
|
||||
|
||||
\section{The measurement doctrine, as testable claims}
|
||||
\label{sec:doctrine}
|
||||
|
||||
The doctrine is four falsifiable rules, each earned by a specific scar.
|
||||
|
||||
\paragraph{Same-run ratios only.} The same \code{llama.cpp} on the same box and model reported prefill
|
||||
figures of $2390.1$, $2404.1$, $2417.8$, $2441.9$, $2459$, $2465.3$, and $2486$ tok/s across the campaign's
|
||||
days --- a $\pm2\%$ spread from run conditions alone. A ratio computed from our number today against a
|
||||
\code{llama} number from last week is therefore uninterpretable; only both sides measured in the same
|
||||
session count. Every ratio in this paper obeys this.
|
||||
|
||||
\paragraph{Sustained-vs-sustained.} On the gfx1151 APU, a kernel's \emph{burst} per-op time ($\sim$416 $\mu$s)
|
||||
and its \emph{sustained} per-op time ($\sim$1634 $\mu$s) differ by $\sim$$4\x$ --- an intrinsic power-limiting
|
||||
swing confirmed under a pinned clock. Comparing a burst measurement of one variant to a sustained
|
||||
measurement of another manufactures a $4\x$ phantom. All A/Bs compare sustained to sustained.
|
||||
|
||||
\paragraph{Contended-timing refusal.} When a foreign 7\,GB job held the GPU at $94\%$ utilization, the
|
||||
candidate-vs-baseline A/B returned ``candidate $\approx$ baseline.'' The correct report is not ``refuted''
|
||||
but ``blocked'': symmetric contention does not make a masked signal fair, it makes it \emph{absent}.
|
||||
Contention-independent facts (compiles, engages, produces correct output) are still reported as fact; the
|
||||
timing is deferred to an idle window.
|
||||
|
||||
\paragraph{Warm-microbench distrust.} For a memory-bound kernel the cache-warm microbenchmark systematically
|
||||
lies. The dense \code{dp4a} matvec measured $74\to255$ GB/s ($3.4\x$) with weights hot in the 32 MB
|
||||
last-level cache, but only $74\to97$ GB/s ($\sim$1.3$\x$) in-engine, where decode reads each weight once,
|
||||
cold, from unified memory. In the other direction, a warm microbenchmark rated the f16 coopmat GEMM at
|
||||
$0.07$--$0.20\x$ of \code{dp4a} and it was written off --- yet with a coalesced access pattern the same
|
||||
matrix-core path became the shipping prefill default. Benchmarks are proxies for deployment; when they
|
||||
disagree, deployment governs.
|
||||
|
||||
\section{Operationalizing the discipline as an evaluation}
|
||||
\label{sec:eval}
|
||||
|
||||
The campaign's reasoning discipline is itself a measurable property, and we operationalized it as a
|
||||
22-case multiple-choice evaluation (\code{roofline}) distilled directly from the log. Every case is a real
|
||||
decision point: the evidence shown is what was on the table at that moment; the gold answer is what
|
||||
hardware measurement subsequently proved; and the distractors are the \emph{actual} refuted theories from
|
||||
Table~\ref{tab:refutations} --- power throttling, occupancy-maximalism, ``port what the reference does,''
|
||||
warm-microbench trust, contended-timing acceptance, dispatch-count-as-identity. Six of the 22 are
|
||||
\emph{traps}: the prompt itself asserts a false premise that the provided evidence contradicts, testing
|
||||
whether a model verifies before it complies. Ground truth is hardware-verified and unpublished, so the set
|
||||
is uncontaminated by pretraining.
|
||||
|
||||
Beyond accuracy, two post-hoc metrics fall out of the standard results log, whose per-item token usages are
|
||||
already recorded:
|
||||
\begin{itemize}[leftmargin=1.4em,itemsep=1pt,topsep=2pt]
|
||||
\item \textbf{overthink index} --- mean completion tokens on \emph{correct} answers. A discipline that
|
||||
trusts a decisive measurement answers in hundreds of tokens; an overthinker burns thousands and, on the
|
||||
trap cases, sometimes reasons its way back \emph{into} the refuted theory.
|
||||
\item \textbf{trap resistance} --- accuracy on the trap subset alone, where the prompt's own framing is
|
||||
adversarial.
|
||||
\end{itemize}
|
||||
|
||||
This turns ``measure, don't theorize'' from a slogan into a benchmark: the same skill the campaign
|
||||
rewarded in engineers is scored, per model, on held-out decision points. It is the pilot instance of a
|
||||
distillation pipeline (a companion paper develops the loop in full).
|
||||
|
||||
\section{Limitations}
|
||||
\label{sec:limits}
|
||||
|
||||
This is an observational $N{=}1$ study and we do not overclaim from it.
|
||||
|
||||
\begin{itemize}[leftmargin=1.4em,itemsep=2pt,topsep=2pt]
|
||||
\item \textbf{No control group.} We did not run a parallel team practising non-refutation-driven
|
||||
engineering on the same problem. We cannot attribute the $11.6\x\to1.40\x$ compression to the methodology
|
||||
versus the specific people, tools, or the mere passage of time. The claim is that the recorded decisions
|
||||
were sound given the evidence, not that an alternative method would have failed.
|
||||
\item \textbf{Single campaign, one team.} All data come from one multi-week effort by one group against one
|
||||
reference implementation. The hypothesis-class distribution (mechanistic-kernel dominant, configuration-gap
|
||||
highest-yield) may not generalize beyond low-level GPU work against a mature baseline.
|
||||
\item \textbf{Hardware specificity.} The mechanisms are tied to particular silicon: gfx1151's unified
|
||||
memory and LDS-limited occupancy, GB10's Blackwell scheduling, the APU's utilization-slaved DVFS. A
|
||||
lever's sign can flip across devices (\S on the RN4 tile; the Q4\_K coalescing reversal), so the specific
|
||||
verdicts do not transfer even though the \emph{method} may.
|
||||
\item \textbf{Survivorship in the log.} The log records hypotheses that were seriously evaluated. Ideas
|
||||
discarded before measurement, or never conceived, leave no trace; the census is of considered hypotheses,
|
||||
not of the full space.
|
||||
\item \textbf{The win ladder mixes measurement bases.} Its early rungs predate the instrument fix of
|
||||
\S\ref{sec:variance}; we mark this in Table~\ref{tab:ladder} rather than silently splicing two noise
|
||||
regimes. The endpoints ($212$ and $1734$) are individually defensible, but the intermediate slope should be
|
||||
read with the basis change in mind.
|
||||
\end{itemize}
|
||||
|
||||
\section{Conclusion}
|
||||
\label{sec:conclusion}
|
||||
|
||||
Across a multi-week campaign the reliable path to throughput was not a better idea about kernels but a
|
||||
cheaper way to be wrong about them. The largest wins were configuration gaps that a profiler named in
|
||||
minutes; the most expensive costs were measurement artifacts that dressed as hardware physics; and the most
|
||||
durable asset produced was a log of refutations that stopped the same attractive mistakes from being bought
|
||||
twice. The three procedures --- a cost-ordered gate ladder, a decisive-experiment rule, and
|
||||
refutation-as-prior --- are simple, and their simplicity is the point: they make the discipline auditable
|
||||
and, as \S\ref{sec:eval} shows, measurable. The companion paper closes the loop by turning the same gate
|
||||
ladder into the selector of an automated search, and the same decision log into training signal.
|
||||
|
||||
Finally, the standing must be stated without inflation (Table~\ref{tab:scoreboard}). The campaign holds two
|
||||
wins (ROCm decode $1.07\x$ over \code{llama}-ROCm; a $470\x$ margin over the fork's own upstream on Metal),
|
||||
one parity (CUDA decode, $1.003\x$), and three still-open gaps (Vulkan prefill $0.71\x$, Vulkan decode
|
||||
$0.55\x$, CUDA prefill $0.82\x$), with the Metal-vs-\code{llama} front blocked by an offline box. ``Beat
|
||||
\code{llama} everywhere'' is the standing objective, not the achieved state, and this paper claims only what
|
||||
the same-run measurements support.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering\small
|
||||
\setlength{\tabcolsep}{5pt}
|
||||
\renewcommand{\arraystretch}{1.12}
|
||||
\begin{tabular}{@{}llllc@{}}
|
||||
\toprule
|
||||
\textbf{Front (backend, phase)} & \textbf{Ours} & \textbf{Baseline, same box} & \textbf{Ratio} & \textbf{Standing} \\
|
||||
\midrule
|
||||
ROCm gfx1151, decode & $70.9$ & \code{llama}-ROCm $66.5$ & $1.07\x$ & win \\
|
||||
Metal M4 Max, decode & $94.4$ & upstream \code{mistral.rs} $0.20$ & $470\x$ & win \\
|
||||
CUDA GB10, decode & $81.9\,{\pm}\,0.6$ & \code{llama} $82.15\,{\pm}\,1.76$ & $1.003\x$ & parity \\
|
||||
CUDA GB10, prefill & $5003\,{\pm}\,23$ & \code{llama} $6079\,{\pm}\,326$ & $0.82\x$ & open \\
|
||||
Vulkan gfx1151, prefill & $1739.6\,{\pm}\,7.4$ & \code{llama} $2441.9\,{\pm}\,22.2$ & $0.71\x$ & open \\
|
||||
Vulkan gfx1151, decode & $50.6\,{\pm}\,0.1$ & \code{llama} $92.6\,{\pm}\,0.2$ & $0.55\x$ & open \\
|
||||
Metal M4 Max, vs \code{llama} & \multicolumn{4}{l}{\emph{blocked} --- box offline; last (stale) reading decode $0.80\x$, prefill $0.047\x$} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Standing scoreboard, CTO-frozen same-run figures (\code{zen-eco-4b} Q4\_K\_M except the Metal rows,
|
||||
Qwen3-30B-A3B). The $470\x$ is measured against the fork's \emph{upstream} \code{mistral.rs} on Metal, not
|
||||
against \code{llama}. Two wins, one parity, three open; Metal-vs-\code{llama} blocked.}
|
||||
\label{tab:scoreboard}
|
||||
\end{table}
|
||||
|
||||
\begin{thebibliography}{9}
|
||||
\small
|
||||
\bibitem{llamacpp} G.~Gerganov and the \code{ggml}/\code{llama.cpp} contributors. \emph{\code{llama.cpp}:
|
||||
LLM inference in C/C++.} \url{https://github.com/ggml-org/llama.cpp}. Reference implementation and
|
||||
\code{llama-bench} baseline; \code{ggml-vulkan.cpp} coopmat pipeline referenced by line in \S\ref{sec:dataset}.
|
||||
\bibitem{cubecl} Tracel AI. \emph{CubeCL: multi-platform GPU compute in Rust.}
|
||||
\url{https://github.com/tracel-ai/cubecl}. The DSL substrate lowered to SPIR-V / HIP / PTX / MSL.
|
||||
\bibitem{oracle} Hanzo AI Research. \emph{One Oracle, Every Backend: The CPU Bit-Exactness Law and How It
|
||||
Is Enforced.} hanzo-kernel Paper 07.4. The bit-exact gate of Algorithm~\ref{alg:gate}.
|
||||
\bibitem{tune} Hanzo AI Research. \emph{Evolutionary Schedule Search in a One-Source Kernel DSL.}
|
||||
hanzo-kernel Paper 09 (companion). The automated successor to the manual gate ladder.
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
Binary file not shown.
Binary file not shown.
@@ -320,6 +320,38 @@ export const siteConfig: SiteConfig = {
|
||||
tags: ['Cloud', 'Multi-Tenancy', 'SQLite', 'Benchmarks'],
|
||||
},
|
||||
|
||||
// --- One Source, Every Backend (kernel-DSL campaign) ---
|
||||
{
|
||||
id: 'refutation-driven-perf',
|
||||
title: 'Refutation-Driven Performance Engineering',
|
||||
subtitle: 'An Empirical Study of a Multi-Week GPU Kernel Campaign',
|
||||
abstract: 'An N=1 observational study of a multi-week campaign to close the llama.cpp inference gap on three accelerators (AMD gfx1151 RDNA3.5, NVIDIA GB10 Blackwell, Apple M4 Max) inside a one-source GPU kernel DSL. The primary artifact is not a kernel but a refutation log: twenty-five numbered hypotheses, most plausible, each killed by a cheap decisive experiment. Reports a Vulkan-prefill win ladder (212→1734 tok/s, 8.2× of the gap closed in one ~36-hour cadence), a CUDA decode gap closed to parity by a single model-eligibility entry with zero kernel work, and an instrument fix that collapsed prefill benchmark noise from ±42% to ±0.8% (52×). Formalizes three recurring algorithms — a multi-fidelity Gate Ladder, a decisive-experiment selection rule, and refutation-log-as-prior — and a measurement doctrine of same-run, sustained-only ratios. Honest about standing: two wins, one parity, three open gaps.',
|
||||
pdfUrl: '/pdfs/refutation-driven-perf.pdf',
|
||||
latexUrl: 'https://github.com/hanzoai/papers/blob/main/refutation-driven-perf.tex',
|
||||
githubUrl: 'https://github.com/hanzoai/papers',
|
||||
date: '2026-07-18',
|
||||
authors: ['Hanzo AI Research'],
|
||||
tags: ['Engine', 'GPU Kernels', 'Benchmarks', 'Methodology'],
|
||||
relatedLinks: [
|
||||
{ label: 'Companion: Evolutionary Schedule Search', url: 'https://papers.hanzo.ai/evolutionary-schedule-search' },
|
||||
],
|
||||
},
|
||||
{
|
||||
id: 'evolutionary-schedule-search',
|
||||
title: 'Evolutionary Schedule Search in a One-Source Kernel DSL',
|
||||
subtitle: 'And the Continual-Improvement Loop It Anchors',
|
||||
abstract: 'A one-source kernel DSL exposes each operation\'s schedule — tile dimensions, buffering depth, vector width, per-shape selection — as compile-time knobs, so schedule search becomes a discrete search over monomorphized, bit-exact kernels. Building on an existing per-(device, op, shape) winner cache, this paper extracts the constants such a search must respect from the campaign\'s ~15 hand-evaluated configurations (free static rejection, a ±0.4% per-op fitness signal, strongly non-additive lever interaction, and hard device non-transfer), specifies a multi-fidelity evolutionary schedule search (~300 configurations/night against ~15 by hand), and embeds it in a continual-improvement loop where a calibrated judge flock — weighted by measured ground truth rather than preference — selects training signal under a Goodhart tripwire, framed as the small-N precursor of a mean-field game. The search is specified, not yet run; that boundary is marked throughout.',
|
||||
pdfUrl: '/pdfs/evolutionary-schedule-search.pdf',
|
||||
latexUrl: 'https://github.com/hanzoai/papers/blob/main/evolutionary-schedule-search.tex',
|
||||
githubUrl: 'https://github.com/hanzoai/papers',
|
||||
date: '2026-07-18',
|
||||
authors: ['Hanzo AI Research'],
|
||||
tags: ['Engine', 'Autotuning', 'Evolutionary Search', 'Kernels'],
|
||||
relatedLinks: [
|
||||
{ label: 'Companion: Refutation-Driven Performance Engineering', url: 'https://papers.hanzo.ai/refutation-driven-perf' },
|
||||
],
|
||||
},
|
||||
|
||||
// --- Hanzo Core Papers ---
|
||||
{
|
||||
id: 'aso',
|
||||
|
||||
Reference in New Issue
Block a user