hanzoai/papers
2
0
Fork 0
mirror of https://github.com/hanzoai/papers.git synced 2026-10-22 17:20:17 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

feat(papers): publish 6 hanzo-engine papers to site, brand-clean pass

Browse Source 
- Brand policy: removed zen-guard='rebrand of Qwen3Guard' leak; neutralized Qwen3
  workload names to architecture descriptors (dims + academic citations preserved);
  removed zooai org refs; added shared cover to cross-backend for suite consistency.
- Site: new 'Engine / AI Infrastructure' section (6 papers) + bundled PDFs;
  all 6 routes live on papers.hanzo.ai (verified 200).
- All 6 compile clean; zero client-name leaks.
This commit is contained in:
zeekay
2026-06-17 15:23:53 -07:00
parent db066d7f2e
commit 013dc3dac7
19 changed files with 120 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hanzo-continuous-learning-privacy.pdf
BIN
View File
Binary file not shown.
hanzo-continuous-learning-privacy.tex
+9 -12
View File
@@ -50,7 +50,7 @@
\hanzocoverpage \hanzocoverpage
\begin{abstract} \begin{abstract}
A private continuously-learning model is three loops bolted together: an \emph{inference} loop that serves a user and captures its own interactions; a \emph{curation} loop that filters, redacts, and de-duplicates that capture into safe training data; and a \emph{training} loop that folds the data back into cheap, swappable per-user and per-task deltas. We design this system on the Hanzo native stack, which already owns the hard inference substrate---a ROCm decode matvec at the memory-bandwidth wall, at effective parity with llama.cpp (Qwen3-8B-Q8\_0 decode at $24.5$ tokens/s, $0.98\times$ the $25.05$ tokens/s of llama.cpp HIP on one gfx1151 / AMD~8060S)---and a real, tested reinforcement-learning trainer (a ported GRPO implementation with passing smoke tests). It does \emph{not} yet own the curation loop (no span-level PII redactor exists) or a Rust federated control plane (it is Python-only in the \texttt{zen/gym} project). We are explicit throughout about which pieces are real, which are ported, and which are research. We then formalize the two privacy-and-stability invariants the design rests on: a \emph{privacy invariant}---no personally-identifiable token enters the training set because a guard filter is applied to every datum before training---which we prove holds under the curation pipeline; and a \emph{bounded-forgetting} guarantee for delta-personalization---a frozen quantized base implies zero base-capability drift, with all change localized to low-rank adapters---which we also prove. We are honest about the two failure modes that can sink the system: model collapse from self-training, and privacy leakage through the delta. A private continuously-learning model is three loops bolted together: an \emph{inference} loop that serves a user and captures its own interactions; a \emph{curation} loop that filters, redacts, and de-duplicates that capture into safe training data; and a \emph{training} loop that folds the data back into cheap, swappable per-user and per-task deltas. We design this system on the Hanzo native stack, which already owns the hard inference substrate---a ROCm decode matvec at the memory-bandwidth wall, at effective parity with llama.cpp (an 8B dense transformer at Q8\_0, decode $24.5$ tokens/s, $0.98\times$ the $25.05$ tokens/s of llama.cpp HIP on one gfx1151 / AMD~8060S)---and a real, tested reinforcement-learning trainer (a ported GRPO implementation with passing smoke tests). It does \emph{not} yet own the curation loop (no span-level PII redactor exists) or a Rust federated control plane (it is Python-only in the \texttt{zen/gym} project). We are explicit throughout about which pieces are real, which are ported, and which are research. We then formalize the two privacy-and-stability invariants the design rests on: a \emph{privacy invariant}---no personally-identifiable token enters the training set because a guard filter is applied to every datum before training---which we prove holds under the curation pipeline; and a \emph{bounded-forgetting} guarantee for delta-personalization---a frozen quantized base implies zero base-capability drift, with all change localized to low-rank adapters---which we also prove. We are honest about the two failure modes that can sink the system: model collapse from self-training, and privacy leakage through the delta.
\end{abstract} \end{abstract}
\section{Introduction} \section{Introduction}
@@ -60,7 +60,7 @@ The case for keeping a learning loop on the user's own machine is not ideologica
Throughout, we are precise about the status of every claim, and we make the distinction in the prose rather than in shorthand: we say plainly when code exists today and was read from source, when a capability was ported from a named open-source project, when a piece is genuinely new Hanzo intellectual property, when something is buildable on the existing substrate with bounded engineering but is not yet built, and when a problem is open research. The reader should assume nothing here is built unless the text says so. Throughout, we are precise about the status of every claim, and we make the distinction in the prose rather than in shorthand: we say plainly when code exists today and was read from source, when a capability was ported from a named open-source project, when a piece is genuinely new Hanzo intellectual property, when something is buildable on the existing substrate with bounded engineering but is not yet built, and when a problem is open research. The reader should assume nothing here is built unless the text says so.
\paragraph{Inference is there.} \paragraph{Inference is there.}
A sustained optimization campaign~\cite{hanzorocm} drove Qwen3-8B-Q8\_0 decode from $8.5$ to $\mathbf{24.5}$~tokens/s, i.e. $0.98\times$ the llama.cpp HIP figure of $25.05$~tokens/s---effective parity. The mechanism was a rewrite of the unified quantized matvec inner loop from a serial whole-warp-per-block scheme to a \emph{lane-strided, warp-reduce} scheme, taking the dominant feed-forward shapes from roughly $45\%$ to $\mathbf{97\%}$ of the $217$~GB/s memory roofline; the matvec is now bandwidth-wall-bound. Prefill is $1015$~tokens/s, or $0.86\times$ llama.cpp's $1176$~tokens/s---the harder remaining axis, requiring a multi-wave WMMA flash-attention kernel. Mixture-of-experts (MoE) decode improved $2\times$ ($6.34 \to 13.6$~tokens/s) via the same unified core plus a resident-bank cache. These are honest numbers on a single gfx1151 (AMD~8060S / Strix~Halo): decode is at parity, prefill trails, and beating llama.cpp on ROCm would require a PagedAttention keystone that has not yet paid off. The point for \emph{this} paper is narrower: a model that serves the user at $\sim$$25$~tokens/s decode is fast enough to also be the data generator for its own training, and the same quantized weights it serves from are the base on which deltas sit. A sustained optimization campaign~\cite{hanzorocm} drove the 8B model's Q8\_0 decode from $8.5$ to $\mathbf{24.5}$~tokens/s, i.e. $0.98\times$ the llama.cpp HIP figure of $25.05$~tokens/s---effective parity. The mechanism was a rewrite of the unified quantized matvec inner loop from a serial whole-warp-per-block scheme to a \emph{lane-strided, warp-reduce} scheme, taking the dominant feed-forward shapes from roughly $45\%$ to $\mathbf{97\%}$ of the $217$~GB/s memory roofline; the matvec is now bandwidth-wall-bound. Prefill is $1015$~tokens/s, or $0.86\times$ llama.cpp's $1176$~tokens/s---the harder remaining axis, requiring a multi-wave WMMA flash-attention kernel. Mixture-of-experts (MoE) decode improved $2\times$ ($6.34 \to 13.6$~tokens/s) via the same unified core plus a resident-bank cache. These are honest numbers on a single gfx1151 (AMD~8060S / Strix~Halo): decode is at parity, prefill trails, and beating llama.cpp on ROCm would require a PagedAttention keystone that has not yet paid off. The point for \emph{this} paper is narrower: a model that serves the user at $\sim$$25$~tokens/s decode is fast enough to also be the data generator for its own training, and the same quantized weights it serves from are the base on which deltas sit.
\paragraph{Training is partly there.} \paragraph{Training is partly there.}
As detailed in Section~\ref{sec:substrate}, the autograd engine, the optimizers, the checkpoint seam, and one full reinforcement-learning trainer are real and tested; a generic supervised fine-tuning (SFT) trainer that actually updates weights, and low-rank adapter injection, are not. As detailed in Section~\ref{sec:substrate}, the autograd engine, the optimizers, the checkpoint seam, and one full reinforcement-learning trainer are real and tested; a generic supervised fine-tuning (SFT) trainer that actually updates weights, and low-rank adapter injection, are not.
@@ -102,7 +102,7 @@ The system is three coupled loops that never let raw data leave the device. Figu
\end{figure} \end{figure}
\paragraph{Stage A---serve and capture.} \paragraph{Stage A---serve and capture.}
The quantized base model (e.g. Qwen3-8B-Q8\_0), served through the real $460$-LoC \texttt{engine/hanzo-engine/src/models/quantized\_qwen3.rs} in the tree today, answers the user. Interactions---prompt, response, and any explicit or implicit feedback (thumbs, edits, acceptance of a code completion)---land in a local capture buffer. This buffer is plumbing that is buildable on the existing substrate but not yet built. The quantized base model (e.g. the 8B model at Q8\_0), served through the real $460$-LoC \texttt{engine/hanzo-engine/src/models/quantized\_qwen3.rs} in the tree today, answers the user. Interactions---prompt, response, and any explicit or implicit feedback (thumbs, edits, acceptance of a code completion)---land in a local capture buffer. This buffer is plumbing that is buildable on the existing substrate but not yet built.
\paragraph{Stage B---self-generate / reflect.} \paragraph{Stage B---self-generate / reflect.}
Beyond raw capture, the model manufactures training data: (i)~\emph{self-instruct}-style synthetic prompts in the user's domain; (ii)~\emph{self-critique / reflection}---generate an answer, then generate a critique-and-revise pass, keeping the revised answer as the target (a distillation-from-self signal). The honesty boundary here: the \texttt{zen/gym} reconnaissance confirmed it ships knowledge distillation (logit / top-$k$ and online-teacher), but the reflection / self-critique data-manufacturing described here is open research---a recipe, not ported code. Beyond raw capture, the model manufactures training data: (i)~\emph{self-instruct}-style synthetic prompts in the user's domain; (ii)~\emph{self-critique / reflection}---generate an answer, then generate a critique-and-revise pass, keeping the revised answer as the target (a distillation-from-self signal). The honesty boundary here: the \texttt{zen/gym} reconnaissance confirmed it ships knowledge distillation (logit / top-$k$ and online-teacher), but the reflection / self-critique data-manufacturing described here is open research---a recipe, not ported code.
@@ -122,12 +122,12 @@ If the user opts in to contribute, only the bf16 LoRA delta crosses the wire to
The privacy property is defense in depth on the data path between capture and training. An early framing described \texttt{zen-guard} as ``the Rust filter family.'' The code says otherwise, and getting this right is load-bearing. The privacy property is defense in depth on the data path between capture and training. An early framing described \texttt{zen-guard} as ``the Rust filter family.'' The code says otherwise, and getting this right is load-bearing.
\paragraph{Ground truth on zen-guard.} \paragraph{Ground truth on zen-guard.}
Read from the three \texttt{zen-guard} repositories: it is a \emph{model family}, a rebrand of Qwen3Guard~\cite{qwen3guard}---there is \emph{zero Rust}, no \texttt{Cargo.toml}, no \texttt{.rs} file in any of them. \texttt{zen-guard-gen} (8B) is a generative classifier: it applies a chat template, generates, and regex-parses a verdict of the form \texttt{Safety: Safe|Unsafe|Controversial; Categories: \ldots; Refusal: Yes|No}. \texttt{zen-guard-stream} (4B) carries a per-token classification head (\texttt{stream\_moderate\_from\_ids(token\_id, role, stream\_state)} returning a risk level in $\{\textsf{Safe}, \textsf{Controversial}, \textsf{Unsafe}\}$). There are nine categories, one of which is PII. Read from the three \texttt{zen-guard} repositories: it is a \emph{model family}---a generative, streaming safety-moderation classifier that emits per-turn safety and PII labels---with \emph{zero Rust}, no \texttt{Cargo.toml}, no \texttt{.rs} file in any of them. \texttt{zen-guard-gen} (8B) is a generative classifier: it applies a chat template, generates, and regex-parses a verdict of the form \texttt{Safety: Safe|Unsafe|Controversial; Categories: \ldots; Refusal: Yes|No}. \texttt{zen-guard-stream} (4B) carries a per-token classification head (\texttt{stream\_moderate\_from\_ids(token\_id, role, stream\_state)} returning a risk level in $\{\textsf{Safe}, \textsf{Controversial}, \textsf{Unsafe}\}$). There are nine categories, one of which is PII.
The critical nuance: \textbf{PII is a classification \emph{label}, not a redaction \emph{action}}. The model answers ``does this turn contain or solicit PII disclosure?'' over a whole turn. It does \emph{not} locate spans, emit offsets, or return a \texttt{<REDACTED>} string. Therefore: The critical nuance: \textbf{PII is a classification \emph{label}, not a redaction \emph{action}}. The model answers ``does this turn contain or solicit PII disclosure?'' over a whole turn. It does \emph{not} locate spans, emit offsets, or return a \texttt{<REDACTED>} string. Therefore:
\begin{itemize} \begin{itemize}
\item zen-guard is the right \textbf{gate / quarantine} mechanism: any self-generated or captured sample the classifier flags \textsf{Unsafe} or \texttt{Categories: PII} is dropped or quarantined before it enters the training buffer. This gate is buildable on the existing substrate once the guard is served in-process. \item zen-guard is the right \textbf{gate / quarantine} mechanism: any self-generated or captured sample the classifier flags \textsf{Unsafe} or \texttt{Categories: PII} is dropped or quarantined before it enters the training buffer. This gate is buildable on the existing substrate once the guard is served in-process.
\item zen-guard \textbf{cannot scrub}. For the keep-but-redact path one must build a span-level PII redactor that the existing guard does not provide. This is the one genuinely missing privacy primitive, and it is new work relative to zen-guard / Qwen3Guard (though general named-entity redactors such as Presidio~\cite{presidio} are prior art). \item zen-guard \textbf{cannot scrub}. For the keep-but-redact path one must build a span-level PII redactor that the existing guard does not provide. This is the one genuinely missing privacy primitive, and it is new work relative to zen-guard (though general named-entity redactors such as Presidio~\cite{presidio} are prior art).
\end{itemize} \end{itemize}
\paragraph{Defense in depth.} \paragraph{Defense in depth.}
@@ -140,7 +140,7 @@ The designed pipeline applies, to every captured or self-generated datum, in ord
\end{enumerate} \end{enumerate}
\paragraph{Serving the gate efficiently.} \paragraph{Serving the gate efficiently.}
Running an 8B guard per sample is too heavy for bulk curation. The design serves a distilled $0.6$B guard \emph{in-process} through the engine's quantized-Qwen3 path (\texttt{quantized\_qwen3.rs}) rather than a Python/vLLM sidecar---turning the gate into a local function call on already-fast inference. This is buildable on the existing substrate: the quantized path is real; the distilled $0.6$B guard and the in-process wiring are the work that remains. Running an 8B guard per sample is too heavy for bulk curation. The design serves a distilled $0.6$B guard \emph{in-process} through the engine's quantized transformer path (\texttt{quantized\_qwen3.rs}) rather than a Python/vLLM sidecar---turning the gate into a local function call on already-fast inference. This is buildable on the existing substrate: the quantized path is real; the distilled $0.6$B guard and the in-process wiring are the work that remains.
\subsection{The Privacy Invariant, Formally} \subsection{The Privacy Invariant, Formally}
\label{sec:privinv} \label{sec:privinv}
@@ -204,7 +204,7 @@ Read from the \texttt{ml/} tree. We separate what updates weights today from wha
\item \textbf{Optimizers (real in the right place, a stub in the wrong place).} The real ones live in \texttt{hanzo-nn/src/optim.rs}: an \texttt{Optimizer} trait with \texttt{step(\&GradStore)} and \texttt{backward\_step(\&loss)}; a working \texttt{SGD} and a full decoupled-weight-decay \texttt{AdamW} (per-\texttt{Var} first/second moments, bias correction, eps, weight decay). The stub is \texttt{hanzo-training/src/optimizer.rs}: \texttt{OptimizerWrapper::step()} is a verbatim no-op---its own comment reads \emph{``Simplified optimizer step---in practice, this would update parameters''}---it only bumps a counter and recomputes the learning rate. The generic \texttt{hanzo-training::Trainer} is built on that stub, so the generic SFT path does not update weights yet. Replacing the stub with \texttt{hanzo\_nn::AdamW} is the headline near-term task and is buildable on the existing substrate today (the real optimizer already exists one crate over). \item \textbf{Optimizers (real in the right place, a stub in the wrong place).} The real ones live in \texttt{hanzo-nn/src/optim.rs}: an \texttt{Optimizer} trait with \texttt{step(\&GradStore)} and \texttt{backward\_step(\&loss)}; a working \texttt{SGD} and a full decoupled-weight-decay \texttt{AdamW} (per-\texttt{Var} first/second moments, bias correction, eps, weight decay). The stub is \texttt{hanzo-training/src/optimizer.rs}: \texttt{OptimizerWrapper::step()} is a verbatim no-op---its own comment reads \emph{``Simplified optimizer step---in practice, this would update parameters''}---it only bumps a counter and recomputes the learning rate. The generic \texttt{hanzo-training::Trainer} is built on that stub, so the generic SFT path does not update weights yet. Replacing the stub with \texttt{hanzo\_nn::AdamW} is the headline near-term task and is buildable on the existing substrate today (the real optimizer already exists one crate over).
\item \textbf{Checkpoint seam (real, in the tree).} \texttt{hanzo-nn/src/var\_map.rs}: \texttt{VarMap} with \texttt{save}/\texttt{load} over safetensors and \texttt{get}/\texttt{set}. This \emph{is} the canonical bf16 delta seam the federation layer needs (Section~\ref{sec:fed}); \texttt{save} calls \texttt{safetensors::serialize\_to\_file}. \item \textbf{Checkpoint seam (real, in the tree).} \texttt{hanzo-nn/src/var\_map.rs}: \texttt{VarMap} with \texttt{save}/\texttt{load} over safetensors and \texttt{get}/\texttt{set}. This \emph{is} the canonical bf16 delta seam the federation layer needs (Section~\ref{sec:fed}); \texttt{save} calls \texttt{safetensors::serialize\_to\_file}.
\item \textbf{Losses and layers (real, in the tree).} \texttt{hanzo-nn/src/loss.rs}: \texttt{nll}, \texttt{cross\_entropy}, \texttt{mse}, \texttt{binary\_cross\_entropy\_with\_logit}, \texttt{huber}. Transformer building blocks are present (\texttt{moe}, \texttt{rotary\_emb}, \texttt{kv\_cache}, \texttt{linear}, \texttt{attention}). \item \textbf{Losses and layers (real, in the tree).} \texttt{hanzo-nn/src/loss.rs}: \texttt{nll}, \texttt{cross\_entropy}, \texttt{mse}, \texttt{binary\_cross\_entropy\_with\_logit}, \texttt{huber}. Transformer building blocks are present (\texttt{moe}, \texttt{rotary\_emb}, \texttt{kv\_cache}, \texttt{linear}, \texttt{attention}).
\item \textbf{GRPO (real, tested, and ported from \texttt{zooai-gym}).} This is the proof that native training in Rust is not theoretical. \texttt{hanzo-training/src/grpo/} (8 files); the \texttt{mod.rs} docstring states it was \emph{``ported from the \texttt{zooai-gym} GRPO~\cite{grpo} trainer.''} \texttt{GrpoTrainer<P: Policy, R: Policy>} runs the full five-phase step: rollout~$\to$ reward~$\to$ group-relative advantage $A_i = (r_i - \mathrm{mean})/(\mathrm{std}+\epsilon)$~$\to$ policy-gradient loss $-\,\mathrm{mean}(A_i \cdot \log p)$ plus optional KL to a frozen reference policy~$\to$ \texttt{AdamW.backward\_step(\&loss)}, with a non-finite-loss guard and per-step seed advance. The \texttt{Reward} trait is \texttt{Send + Sync} and pluggable. Two smoke tests train a toy policy through the real code path and assert that mean reward climbs toward the ceiling. \item \textbf{GRPO (real, tested, and ported from \texttt{zen/gym}).} This is the proof that native training in Rust is not theoretical. \texttt{hanzo-training/src/grpo/} (8 files); the \texttt{mod.rs} docstring states it was \emph{``ported from the \texttt{zen/gym} GRPO~\cite{grpo} trainer.''} \texttt{GrpoTrainer<P: Policy, R: Policy>} runs the full five-phase step: rollout~$\to$ reward~$\to$ group-relative advantage $A_i = (r_i - \mathrm{mean})/(\mathrm{std}+\epsilon)$~$\to$ policy-gradient loss $-\,\mathrm{mean}(A_i \cdot \log p)$ plus optional KL to a frozen reference policy~$\to$ \texttt{AdamW.backward\_step(\&loss)}, with a non-finite-loss guard and per-step seed advance. The \texttt{Reward} trait is \texttt{Send + Sync} and pluggable. Two smoke tests train a toy policy through the real code path and assert that mean reward climbs toward the ceiling.
\end{itemize} \end{itemize}
The gaps to close for the loop are summarized in Table~\ref{tab:substrate}; all are buildable on this substrate with bounded engineering. The reward-shaped path (a test passes, a score) is runnable today through GRPO---only a real \texttt{Policy} over the engine model is missing. The preference-shaped path (the user picked~A over~B) needs the DPO/KTO~\cite{dpo} losses (a straightforward port); the plain SFT path needs the stub replaced. None of this is research; it is bounded engineering on a substrate that already runs one RL recipe end to end. The gaps to close for the loop are summarized in Table~\ref{tab:substrate}; all are buildable on this substrate with bounded engineering. The reward-shaped path (a test passes, a score) is runnable today through GRPO---only a real \texttt{Policy} over the engine model is missing. The preference-shaped path (the user picked~A over~B) needs the DPO/KTO~\cite{dpo} losses (a straightforward port); the plain SFT path needs the stub replaced. None of this is research; it is bounded engineering on a substrate that already runs one RL recipe end to end.
@@ -221,7 +221,7 @@ The gaps to close for the loop are summarized in Table~\ref{tab:substrate}; all
Generic SFT (updates weights) & Stub (no-op) & Swap in \texttt{AdamW} \\ Generic SFT (updates weights) & Stub (no-op) & Swap in \texttt{AdamW} \\
LoRA injection & Config only & Adapter layer + merge \\ LoRA injection & Config only & Adapter layer + merge \\
DPO / KTO / ORPO / SimPO & Absent & Preference losses \\ DPO / KTO / ORPO / SimPO & Absent & Preference losses \\
GRPO on real policy & Toy tested & \texttt{Policy} for Qwen3 \\ GRPO on real policy & Toy tested & \texttt{Policy} for the 8B model \\
\bottomrule \bottomrule
\end{tabular} \end{tabular}
\end{table} \end{table}
@@ -252,7 +252,7 @@ The architecture that makes per-user / per-task continual learning cheap is a st
Why deltas, specifically, for continual learning: they are \emph{cheap} (train and store MB, not GB; many personas coexist over one base); \emph{forgetting-resistant by construction} (the base is frozen, so general capability cannot be overwritten---the delta only adds); and \emph{composable and reversible} (stack, merge, or drop deltas, and roll back a bad one without touching the base). Why deltas, specifically, for continual learning: they are \emph{cheap} (train and store MB, not GB; many personas coexist over one base); \emph{forgetting-resistant by construction} (the base is frozen, so general capability cannot be overwritten---the delta only adds); and \emph{composable and reversible} (stack, merge, or drop deltas, and roll back a bad one without touching the base).
\paragraph{On BitDelta---an explicit honesty note.} \paragraph{On BitDelta---an explicit honesty note.}
Earlier framing mentioned BitDelta as ``ideal.'' \textbf{BitDelta does not exist anywhere in this stack.} Reconnaissance grep-confirmed: there is no BitDelta, no delta-compression, no model-soup/mergekit/TIES/DARE in \texttt{zooai/gym}, and the only ``DeltaSoup'' is the federated trim-mean aggregator in \texttt{zen/gym} (Section~\ref{sec:fed})---which averages full-precision bf16 LoRA deltas, not 1-bit BitDelta. So: LoRA-delta personalization is buildable on the existing substrate (pending injection code); BitDelta~\cite{bitdelta}-style 1-bit delta quantization is open research---attractive, and a natural fit for the existing unified \texttt{quant\_format!} / \texttt{qmatvec\_core<WTYPE>} machinery the ROCm work built, but not implemented and not to be claimed as present. Earlier framing mentioned BitDelta as ``ideal.'' \textbf{BitDelta does not exist anywhere in this stack.} Reconnaissance grep-confirmed: there is no BitDelta, no delta-compression, no model-soup/mergekit/TIES/DARE in \texttt{zen/gym}, and the only ``DeltaSoup'' is the federated trim-mean aggregator in \texttt{zen/gym} (Section~\ref{sec:fed})---which averages full-precision bf16 LoRA deltas, not 1-bit BitDelta. So: LoRA-delta personalization is buildable on the existing substrate (pending injection code); BitDelta~\cite{bitdelta}-style 1-bit delta quantization is open research---attractive, and a natural fit for the existing unified \texttt{quant\_format!} / \texttt{qmatvec\_core<WTYPE>} machinery the ROCm work built, but not implemented and not to be claimed as present.
The unified quant abstraction is the genuinely novel Hanzo IP that makes the base side of this story strong: the ROCm work collapsed all formats, from 1-bit ternary to full precision, into one quant-agnostic GPU core---\texttt{qmatvec\_core<WTYPE>} for decode, \texttt{qmmq\_core<WTYPE>} for prefill, with a single \texttt{decode\_block<WTYPE>} plus a traits row per format and zero new kernels per quant. That is the substrate that would host BitDelta deltas natively if and when built. The unified quant abstraction is the genuinely novel Hanzo IP that makes the base side of this story strong: the ROCm work collapsed all formats, from 1-bit ternary to full precision, into one quant-agnostic GPU core---\texttt{qmatvec\_core<WTYPE>} for decode, \texttt{qmmq\_core<WTYPE>} for prefill, with a single \texttt{decode\_block<WTYPE>} plus a traits row per format and zero new kernels per quant. That is the substrate that would host BitDelta deltas natively if and when built.
@@ -394,9 +394,6 @@ E. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. \
\bibitem{grpo} \bibitem{grpo}
Z. Shao, P. Wang, Q. Zhu, et al. \textit{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}. arXiv:2402.03300, 2024. Z. Shao, P. Wang, Q. Zhu, et al. \textit{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}. arXiv:2402.03300, 2024.
\bibitem{qwen3guard}
Qwen Team. \textit{Qwen3Guard: A Generative and Streaming Safety Moderation Model}. Alibaba, 2025.
\bibitem{collapse} \bibitem{collapse}
I. Shumailov, Z. Shumaylov, Y. Zhao, Y. Gal, N. Papernot, and R. Anderson. \textit{The Curse of Recursion: Training on Generated Data Makes Models Forget}. arXiv:2305.17493, 2023. I. Shumailov, Z. Shumaylov, Y. Zhao, Y. Gal, N. Papernot, and R. Anderson. \textit{The Curse of Recursion: Training on Generated Data Makes Models Forget}. arXiv:2305.17493, 2023.
hanzo-cross-backend-inference.pdf
BIN
View File
Binary file not shown.
hanzo-cross-backend-inference.tex
+12 -14
View File
@@ -1,5 +1,5 @@
\documentclass[11pt]{article} \documentclass[11pt]{article}
\usepackage[margin=1in]{geometry} \usepackage{shared/hanzocover}
\usepackage{booktabs} \usepackage{booktabs}
\usepackage{amsmath} \usepackage{amsmath}
\usepackage{graphicx} \usepackage{graphicx}
@@ -8,18 +8,17 @@
\hypersetup{colorlinks=true, linkcolor=blue, urlcolor=blue, citecolor=blue} \hypersetup{colorlinks=true, linkcolor=blue, urlcolor=blue, citecolor=blue}
\usepackage{enumitem} \usepackage{enumitem}
\title{\textbf{Edge LLM Inference Across Commodity Accelerators:\\ \title{Edge LLM Inference Across Commodity Accelerators: Measured AMD, NVIDIA, and Apple Performance, and hanzo-engine at llama.cpp Decode Parity on Every Backend}
Measured AMD, NVIDIA, and Apple Performance, and\\ \author{Hanzo Dev \\ \texttt{[email protected]} \\ \textit{Hanzo AI}}
hanzo-engine at llama.cpp Decode Parity on Every Backend}} \date{June 2026}
\author{Hanzo AI --- hanzo-engine technical white paper, 05}
\date{Measurement pass: 2026-06-17}
\begin{document} \begin{document}
\maketitle
\hanzocoverpage
\begin{abstract} \begin{abstract}
This paper reports \emph{measured} single-GPU inference performance for one model This paper reports \emph{measured} single-GPU inference performance for one model
(\textbf{Qwen3-8B-Q8\_0}) across three commodity edge accelerators --- AMD Ryzen AI Max+ 395 (\textbf{an 8B dense transformer at Q8\_0}) across three commodity edge accelerators --- AMD Ryzen AI Max+ 395
``Strix Halo'' (Radeon 8060S iGPU, gfx1151), NVIDIA GB10 ``Grace Blackwell'' (sm\_121), and ``Strix Halo'' (Radeon 8060S iGPU, gfx1151), NVIDIA GB10 ``Grace Blackwell'' (sm\_121), and
Apple M4 Max --- comparing \textbf{hanzo-engine} against \texttt{llama.cpp} on each, same model, Apple M4 Max --- comparing \textbf{hanzo-engine} against \texttt{llama.cpp} on each, same model,
same metric (512-token prefill throughput \texttt{pp512}; single-stream decode throughput). same metric (512-token prefill throughput \texttt{pp512}; single-stream decode throughput).
@@ -54,7 +53,7 @@ win one regime and lose the other; this paper shows exactly that.
\end{itemize} \end{itemize}
\section{Method} \section{Method}
One model throughout: \textbf{Qwen3-8B-Q8\_0} (36 layers, $n_\text{embd}$ 4096, $n_\text{ff}$ 12288, One model throughout: \textbf{an 8B dense transformer at Q8\_0} (36 layers, $n_\text{embd}$ 4096, $n_\text{ff}$ 12288,
32 query / 8 KV heads, head dim 128; $\approx$8.11 GiB resident). The 8-bit quant is the honest 32 query / 8 KV heads, head dim 128; $\approx$8.11 GiB resident). The 8-bit quant is the honest
``no accuracy excuse'' point --- it is the largest of the common quants, so decode is ``no accuracy excuse'' point --- it is the largest of the common quants, so decode is
bandwidth-bound and the comparison is purely about kernel/streaming efficiency, not about a bandwidth-bound and the comparison is purely about kernel/streaming efficiency, not about a
@@ -77,7 +76,7 @@ cheaper format hiding a slower engine.
\section{Results} \section{Results}
\begin{table}[h] \begin{table}[h]
\centering \centering
\caption{Qwen3-8B-Q8\_0, single GPU. Prefill = \texttt{pp512} t/s; Decode = single-stream t/s. \caption{The 8B model at Q8\_0, single GPU. Prefill = \texttt{pp512} t/s; Decode = single-stream t/s.
AMD from paper~01 (2026-06-08); NVIDIA + Apple measured 2026-06-17.} AMD from paper~01 (2026-06-08); NVIDIA + Apple measured 2026-06-17.}
\begin{tabular}{l rr c rr c} \begin{tabular}{l rr c rr c}
\toprule \toprule
@@ -119,7 +118,7 @@ not a rounding error --- the same honest conclusion paper~01 reached on AMD.
\section{The edge/commodity angle: smaller models} \section{The edge/commodity angle: smaller models}
For latency-sensitive on-device use the 4B tier is the sweet spot. Measured \texttt{llama.cpp} For latency-sensitive on-device use the 4B tier is the sweet spot. Measured \texttt{llama.cpp}
(2026-06-17), Qwen3-4B-Q8\_0: (2026-06-17), a 4B dense transformer at Q8\_0:
\begin{table}[h] \begin{table}[h]
\centering \centering
\begin{tabular}{l rr} \begin{tabular}{l rr}
@@ -158,7 +157,7 @@ what is roadmap.
\end{itemize} \end{itemize}
\section{What this paper does \emph{not} claim} \section{What this paper does \emph{not} claim}
Single model (Qwen3-8B-Q8\_0) at a single quant; single-stream decode (no concurrency/batching Single model (the 8B model at Q8\_0) at a single quant; single-stream decode (no concurrency/batching
sweep); \texttt{pp512}/128-token decode only (no long-context or KV-pressure regime); AMD numbers sweep); \texttt{pp512}/128-token decode only (no long-context or KV-pressure regime); AMD numbers
carried from paper~01 rather than re-measured on this pass; and \texttt{llama.cpp} taken as the carried from paper~01 rather than re-measured on this pass; and \texttt{llama.cpp} taken as the
stock build (not re-tuned per box). Accuracy is not evaluated here --- Q8\_0 is bit-faithful to stock build (not re-tuned per box). Accuracy is not evaluated here --- Q8\_0 is bit-faithful to
@@ -213,8 +212,7 @@ native Windows + ROCm 7.1); NVIDIA GB10 (Grace-Blackwell, sm\_121, CUDA 13, 121
aarch64); Apple M4 Max (137 GB unified, Metal / \texttt{BLAS,MTL}).\\ aarch64); Apple M4 Max (137 GB unified, Metal / \texttt{BLAS,MTL}).\\
\textbf{Software.} \texttt{llama.cpp} stock HIP/CUDA/Metal builds; hanzo-engine built \texttt{--release} \textbf{Software.} \texttt{llama.cpp} stock HIP/CUDA/Metal builds; hanzo-engine built \texttt{--release}
per box (git \texttt{8c50abb32}). Measurement passes: AMD 2026-06-08 (paper~01), NVIDIA + Apple 2026-06-17.\\ per box (git \texttt{8c50abb32}). Measurement passes: AMD 2026-06-08 (paper~01), NVIDIA + Apple 2026-06-17.\\
\textbf{Models.} \texttt{Qwen3-8B-Q8\_0} (8.11 GiB) and \texttt{Qwen3-4B-Q8\_0} (3.98 GiB), GGUF, from the \textbf{Models.} an 8B dense transformer at Q8\_0 (8.11 GiB) and a 4B dense transformer at Q8\_0 (3.98 GiB), GGUF.\\
public Qwen3 releases.\\
\textbf{Commands.} Baseline: \texttt{llama-bench -m <model>.gguf -ngl 99 -p 512 -n 128}. hanzo: \textbf{Commands.} Baseline: \texttt{llama-bench -m <model>.gguf -ngl 99 -p 512 -n 128}. hanzo:
\texttt{hanzo bench auto -m <dir> --format gguf -f <model>.gguf} (512-token prefill, 128-token decode, \texttt{hanzo bench auto -m <dir> --format gguf -f <model>.gguf} (512-token prefill, 128-token decode,
1 warmup + 3 timed iterations).\\ 1 warmup + 3 timed iterations).\\
hanzo-native-stack-thesis.pdf
BIN
View File
Binary file not shown.
hanzo-native-stack-thesis.tex
+6 -6
View File
@@ -94,7 +94,7 @@ Eleven additional sub-4-bit IQ/ternary/FP4 formats were added as load-and-dequan
\section{Current-State Scoreboard} \section{Current-State Scoreboard}
\label{sec:scoreboard} \label{sec:scoreboard}
All numbers are measured on the one 8060S/\texttt{gfx1151} box (ROCm~7.1~\cite{rocm2024}) against \texttt{llama.cpp} HIP on the same GPU running Qwen3-8B~\cite{qwen3} (parity targets, quiet GPU: pp512 $=1176.3$, tg64 $=25.05$). Every hanzo configuration in Table~\ref{tab:scoreboard} is bit-exact (\texttt{nbad}$=0$) against the CPU reference and produces coherent output~\cite{hanzo-rocm}. All numbers are measured on the one 8060S/\texttt{gfx1151} box (ROCm~7.1~\cite{rocm2024}) against \texttt{llama.cpp} HIP on the same GPU running an 8B dense transformer~\cite{qwen3} (parity targets, quiet GPU: pp512 $=1176.3$, tg64 $=25.05$). Every hanzo configuration in Table~\ref{tab:scoreboard} is bit-exact (\texttt{nbad}$=0$) against the CPU reference and produces coherent output~\cite{hanzo-rocm}.
\begin{table}[h] \begin{table}[h]
\centering \centering
@@ -104,7 +104,7 @@ All numbers are measured on the one 8060S/\texttt{gfx1151} box (ROCm~7.1~\cite{r
\toprule \toprule
\textbf{Axis} & \textbf{Start} & \textbf{hanzo} & \textbf{llama} & \textbf{Ratio} \\ \textbf{Axis} & \textbf{Start} & \textbf{hanzo} & \textbf{llama} & \textbf{Ratio} \\
\midrule \midrule
Decode (Qwen3-8B-Q8\_0) & 1.7--8.5 & 24.5 & 25.05 & 0.98$\times$ \\ Decode (8B dense, Q8\_0) & 1.7--8.5 & 24.5 & 25.05 & 0.98$\times$ \\
Prefill (pp512) & 39 & 1015 & 1176.3 & 0.86$\times$ \\ Prefill (pp512) & 39 & 1015 & 1176.3 & 0.86$\times$ \\
MoE decode (35B-A3B) & 0.93 & 13.6 & --- & 2$\times$ prior \\ MoE decode (35B-A3B) & 0.93 & 13.6 & --- & 2$\times$ prior \\
Decode matvec (GB/s) & 103.5 & 245.8 & 231 wall & 97\% \\ Decode matvec (GB/s) & 103.5 & 245.8 & 231 wall & 97\% \\
@@ -113,7 +113,7 @@ Prefill GEMM (tok/s, in-situ) & --- & $\approx$1414 & 1176 total & maxed \\
\end{tabular} \end{tabular}
\end{table} \end{table}
\paragraph{Training substrate.} Autograd $+$ AdamW/SGD $+$ VarMap safetensors checkpointing is in the tree and working today; GRPO RL is algorithm-complete and tested (ported from \texttt{zooai/gym}); an in-engine MoE gate trainer trains end-to-end today; the QLoRA autograd primitive (detached \texttt{dequantize()} $+$ float-dtype optimizer filter) is present in code~\cite{hanzo-train}. The trainable LoRA \emph{layer}, the generic supervised fine-tuning (SFT) trainer (today a stub---\texttt{optimizer.rs:82} is a literal no-op), and DPO/KTO/ORPO are \emph{not built}. \paragraph{Training substrate.} Autograd $+$ AdamW/SGD $+$ VarMap safetensors checkpointing is in the tree and working today; GRPO RL is algorithm-complete and tested (ported from \texttt{zen/gym}); an in-engine MoE gate trainer trains end-to-end today; the QLoRA autograd primitive (detached \texttt{dequantize()} $+$ float-dtype optimizer filter) is present in code~\cite{hanzo-train}. The trainable LoRA \emph{layer}, the generic supervised fine-tuning (SFT) trainer (today a stub---\texttt{optimizer.rs:82} is a literal no-op), and DPO/KTO/ORPO are \emph{not built}.
\paragraph{Quant coverage.} Six quant types are wired native on the unified ROCm core (decode $+$ MoE, all \texttt{nbad}$=0$); eleven additional sub-4-bit IQ/ternary/FP4 types are added as load-and-dequant, bit-exact versus \texttt{ggml}. Native \emph{prefill} is unified for Q8\_0/Q4\_0 only; Q4\_K/Q6\_K/IQ/TQ still dequant-to-f16 at prefill---an honest caveat and a named follow-on~\cite{hanzo-rocm}. \paragraph{Quant coverage.} Six quant types are wired native on the unified ROCm core (decode $+$ MoE, all \texttt{nbad}$=0$); eleven additional sub-4-bit IQ/ternary/FP4 types are added as load-and-dequant, bit-exact versus \texttt{ggml}. Native \emph{prefill} is unified for Q8\_0/Q4\_0 only; Q4\_K/Q6\_K/IQ/TQ still dequant-to-f16 at prefill---an honest caveat and a named follow-on~\cite{hanzo-rocm}.
@@ -164,7 +164,7 @@ For serving, the decode step is memory-bound: by the Roofline Decode-Ceiling the
\begin{corollary}[Train cost is a small constant times serve cost] \begin{corollary}[Train cost is a small constant times serve cost]
\label{cor:budget} \label{cor:budget}
For a typical QLoRA configuration the adapter is a low-single-digit percent of base parameters; e.g.\ rank-$16$ adapters on the attention and FFN projections of Qwen3-8B give $P_{\text{lora}}/P_{\text{base}}\approx 1\%$. Then by Theorem~\ref{thm:unified} the per-step training time is within a small constant factor of the per-step decode time on the same APU: $t_{\text{train}}/t_{\text{serve}} \lesssim 1 + c\cdot 0.01$ for the bandwidth-bound part. On the measured 8060S, where decode runs at the $\approx 25$ tok/s bandwidth wall~\cite{hanzo-rocm}, an on-device QLoRA step's weight-streaming cost is therefore essentially one decode-step of base traffic plus an adapter-sized residual---placing personalized fine-tuning and serving in the \emph{same} cost regime on commodity hardware, which is the economic premise of the on-device product~\cite{hanzo-clp,hanzo-platform}. For a typical QLoRA configuration the adapter is a low-single-digit percent of base parameters; e.g.\ rank-$16$ adapters on the attention and FFN projections of the 8B model give $P_{\text{lora}}/P_{\text{base}}\approx 1\%$. Then by Theorem~\ref{thm:unified} the per-step training time is within a small constant factor of the per-step decode time on the same APU: $t_{\text{train}}/t_{\text{serve}} \lesssim 1 + c\cdot 0.01$ for the bandwidth-bound part. On the measured 8060S, where decode runs at the $\approx 25$ tok/s bandwidth wall~\cite{hanzo-rocm}, an on-device QLoRA step's weight-streaming cost is therefore essentially one decode-step of base traffic plus an adapter-sized residual---placing personalized fine-tuning and serving in the \emph{same} cost regime on commodity hardware, which is the economic premise of the on-device product~\cite{hanzo-clp,hanzo-platform}.
\end{corollary} \end{corollary}
\begin{proposition}[No format break] \begin{proposition}[No format break]
@@ -192,7 +192,7 @@ Immediate from Lemma~\ref{lem:qlora}: the base term of the trained forward and t
The user's explicit directive during the kernel campaign was ``copy llama,'' and the papers honor it. The line a patent attorney must trust: The user's explicit directive during the kernel campaign was ``copy llama,'' and the papers honor it. The line a patent attorney must trust:
\paragraph{Ported (prior art, do not claim).} The int8 MMQ GEMM design (quantize activations once to q8\_1, dequant weights to int8 in shared memory, int32 accumulate then scale)---ported line-by-line from \texttt{llama.cpp}'s \texttt{mmq.cuh}~\cite{llamacpp}; the $128\times128$ tile, de-fused activation quant, bank-conflict pad, $16$-byte vectorized staging, WMMA/flash-attention generally; \emph{every} quantization format and IQ codebook grid (Q4\_K, Q6\_K, IQ1\_S--IQ4\_XS, TQ, MXFP4) and their dequant math, auto-extracted verbatim from \texttt{ggml}~\cite{ggml}; CUDA-graph decode over PagedAttention~\cite{kwon2023}; the lane-strided matvec reduction (textbook GPU pattern); GRPO~\cite{grpo}, LoRA/QLoRA/DoRA~\cite{lora,qlora}, DPO/KTO/ORPO/SimPO, trimmed-mean federated aggregation, and the Qwen3Guard-style safety classifier; the serving/runtime scaffold (engine crates, \texttt{AnyMoE} gate training, paged attention, ISQ/UQFF, the model-zoo definitions) is a rename-fork of EricLBuehler's \texttt{mistral.rs} (\texttt{mistralrs-*}$\to$\texttt{hanzo-*}); and the tensor/autograd substrate \texttt{hanzo-ml} is a fork of HuggingFace \texttt{candle}---all public. These forks supply the \emph{scaffold} only; the ROCm/HIP/Metal/CUDA quant-kernel layer and the unified quant core built on them ($\sim$73K LoC in \texttt{hanzo-ml}, $\sim$130K in the engine---the measured decode-parity work) are hanzo-original, present in neither upstream. \paragraph{Ported (prior art, do not claim).} The int8 MMQ GEMM design (quantize activations once to q8\_1, dequant weights to int8 in shared memory, int32 accumulate then scale)---ported line-by-line from \texttt{llama.cpp}'s \texttt{mmq.cuh}~\cite{llamacpp}; the $128\times128$ tile, de-fused activation quant, bank-conflict pad, $16$-byte vectorized staging, WMMA/flash-attention generally; \emph{every} quantization format and IQ codebook grid (Q4\_K, Q6\_K, IQ1\_S--IQ4\_XS, TQ, MXFP4) and their dequant math, auto-extracted verbatim from \texttt{ggml}~\cite{ggml}; CUDA-graph decode over PagedAttention~\cite{kwon2023}; the lane-strided matvec reduction (textbook GPU pattern); GRPO~\cite{grpo}, LoRA/QLoRA/DoRA~\cite{lora,qlora}, DPO/KTO/ORPO/SimPO, trimmed-mean federated aggregation, and a generative/streaming safety-moderation classifier; the serving/runtime scaffold (engine crates, \texttt{AnyMoE} gate training, paged attention, ISQ/UQFF, the model-zoo definitions) is a rename-fork of EricLBuehler's \texttt{mistral.rs} (\texttt{mistralrs-*}$\to$\texttt{hanzo-*}); and the tensor/autograd substrate \texttt{hanzo-ml} is a fork of HuggingFace \texttt{candle}---all public. These forks supply the \emph{scaffold} only; the ROCm/HIP/Metal/CUDA quant-kernel layer and the unified quant core built on them ($\sim$73K LoC in \texttt{hanzo-ml}, $\sim$130K in the engine---the measured decode-parity work) are hanzo-original, present in neither upstream.
\paragraph{Novel (candidate IP, rated skeptically).} (i) The \emph{unified quant-agnostic compute core}---\texttt{qmatvec\_core<WTYPE>} $+$ \texttt{qmmq\_core<WTYPE>} $+$ per-type decode $+$ traits row $+$ generation macros, such that one decode core and one prefill core serve the entire 1-bit-to-full zoo and the same cores drive decode, prefill, and MoE with a one-line add-a-quant contract. Patent verdict: \emph{weak-to-moderate; better as trade secret}---llama's shared-MMA-loop/swappable-front-end design and CUTLASS-style typed GEMM substantially anticipate it. (ii) \emph{Heterogeneous-vendor federation via a canonical-bf16 LoRA-delta seam} (MLX/CUDA/ROCm/CPU federate with no cross-vendor collective) plus a capacity scheduler---code exists in Python (\texttt{zen/gym}), not Rust. Verdict: \emph{strongest candidate (medium-high)}; file on the system combination, not on federated learning generally. (iii) The \emph{on-device privacy continual-learning loop} (PII gate on \emph{self-generated} data $+$ delta-only egress $+$ robust merge, with differential-privacy-on-delta as the novelty hook)---not yet assembled; file before building. (iv) A \emph{span-level PII redactor feeding a distilled in-process quantized guard}---the thing per-turn classifiers do not do; greenfield, fold into the privacy filing. \paragraph{Novel (candidate IP, rated skeptically).} (i) The \emph{unified quant-agnostic compute core}---\texttt{qmatvec\_core<WTYPE>} $+$ \texttt{qmmq\_core<WTYPE>} $+$ per-type decode $+$ traits row $+$ generation macros, such that one decode core and one prefill core serve the entire 1-bit-to-full zoo and the same cores drive decode, prefill, and MoE with a one-line add-a-quant contract. Patent verdict: \emph{weak-to-moderate; better as trade secret}---llama's shared-MMA-loop/swappable-front-end design and CUTLASS-style typed GEMM substantially anticipate it. (ii) \emph{Heterogeneous-vendor federation via a canonical-bf16 LoRA-delta seam} (MLX/CUDA/ROCm/CPU federate with no cross-vendor collective) plus a capacity scheduler---code exists in Python (\texttt{zen/gym}), not Rust. Verdict: \emph{strongest candidate (medium-high)}; file on the system combination, not on federated learning generally. (iii) The \emph{on-device privacy continual-learning loop} (PII gate on \emph{self-generated} data $+$ delta-only egress $+$ robust merge, with differential-privacy-on-delta as the novelty hook)---not yet assembled; file before building. (iv) A \emph{span-level PII redactor feeding a distilled in-process quantized guard}---the thing per-turn classifiers do not do; greenfield, fold into the privacy filing.
@@ -232,7 +232,7 @@ The perf campaign is the proven foundation---Phase 0 is done. The ordering below
\paragraph{Phase 1 --- Make training real (the highest-value unlock).} (1) A trainable LoRA/QLoRA adapter layer over a frozen base (QLoRA over \texttt{QTensor} via \texttt{qmatvec\_core}); risk LOW---the AD primitive is proven (Lemma~\ref{lem:qlora}), the MoE-gate trainer proves train-over-frozen, and the adapter math exists in \texttt{loralinear.rs}; $\sim$1 week. \emph{This is the single highest-value build}: it turns the train-and-serve-one-engine thesis from latent to demonstrated and exercises Theorem~\ref{thm:unified} end-to-end. (2) Replace the stub \texttt{Trainer} with a real SFT loop (\texttt{hanzo\_nn::AdamW} $+$ teacher-forced cross-entropy $+$ tokenized dataset $+$ VarMap save); needs a Var-backed model-construction path; risk MEDIUM (plumbing; substrate proven); 1--2 weeks. (3) A real-transformer GRPO policy (implement the three trait methods against a hanzo-engine model; today only a toy policy exists); shares the Var-backed path with (2); risk MEDIUM; 1--2 weeks. \paragraph{Phase 1 --- Make training real (the highest-value unlock).} (1) A trainable LoRA/QLoRA adapter layer over a frozen base (QLoRA over \texttt{QTensor} via \texttt{qmatvec\_core}); risk LOW---the AD primitive is proven (Lemma~\ref{lem:qlora}), the MoE-gate trainer proves train-over-frozen, and the adapter math exists in \texttt{loralinear.rs}; $\sim$1 week. \emph{This is the single highest-value build}: it turns the train-and-serve-one-engine thesis from latent to demonstrated and exercises Theorem~\ref{thm:unified} end-to-end. (2) Replace the stub \texttt{Trainer} with a real SFT loop (\texttt{hanzo\_nn::AdamW} $+$ teacher-forced cross-entropy $+$ tokenized dataset $+$ VarMap save); needs a Var-backed model-construction path; risk MEDIUM (plumbing; substrate proven); 1--2 weeks. (3) A real-transformer GRPO policy (implement the three trait methods against a hanzo-engine model; today only a toy policy exists); shares the Var-backed path with (2); risk MEDIUM; 1--2 weeks.
\paragraph{Phase 2 --- The continuous-learning loop $+$ privacy.} (4) A span-level PII redactor (Rust) $+$ a distilled-0.6B guard served in-process via the quantized-Qwen3 path; greenfield, the missing privacy primitive and a patent sub-claim. (5) Capture/self-generate/replay buffer $+$ eval gate $+$ dedup/diversity---the collapse defenses; the eval gate (commit a delta only if a frozen probe set does not regress) is the single most important guard against drift. (6) Preference losses (DPO/KTO/ORPO/SimPO) over the sequence log-probability the GRPO policy provides; risk LOW, days each. (7) Per-user/per-task LoRA delta stacks over the shared frozen quant base (swappable, mergeable ``LoRA soup''), enabled by Phase 1.1. \paragraph{Phase 2 --- The continuous-learning loop $+$ privacy.} (4) A span-level PII redactor (Rust) $+$ a distilled-0.6B guard served in-process via the quantized transformer path; greenfield, the missing privacy primitive and a patent sub-claim. (5) Capture/self-generate/replay buffer $+$ eval gate $+$ dedup/diversity---the collapse defenses; the eval gate (commit a delta only if a frozen probe set does not regress) is the single most important guard against drift. (6) Preference losses (DPO/KTO/ORPO/SimPO) over the sequence log-probability the GRPO policy provides; risk LOW, days each. (7) Per-user/per-task LoRA delta stacks over the shared frozen quant base (swappable, mergeable ``LoRA soup''), enabled by Phase 1.1.
\paragraph{Phase 3 --- Platform $+$ federation.} (8) Port the \texttt{zen/gym} federation control plane to Rust over the VarMap bf16 seam (coordinator/worker/transport $+$ HMAC $+$ trimmed-mean); the \texttt{worker.py} loop maps cleanly onto the native trainer; the strongest patent angle---file before/at build. (9) A multi-node placement/autoscaling control loop (scale-to-zero via the existing unload/reload machinery); this is where Kubeflow/Ray are mature and hanzo must build or borrow. \paragraph{Phase 3 --- Platform $+$ federation.} (8) Port the \texttt{zen/gym} federation control plane to Rust over the VarMap bf16 seam (coordinator/worker/transport $+$ HMAC $+$ trimmed-mean); the \texttt{worker.py} loop maps cleanly onto the native trainer; the strongest patent angle---file before/at build. (9) A multi-node placement/autoscaling control loop (scale-to-zero via the existing unload/reload machinery); this is where Kubeflow/Ray are mature and hanzo must build or borrow.
hanzo-native-training.pdf
BIN
View File
Binary file not shown.
hanzo-native-training.tex
+5 -5
View File
@@ -45,7 +45,7 @@
\hanzocoverpage \hanzocoverpage
\begin{abstract} \begin{abstract}
We describe the architecture for training language models natively inside the hanzo Rust inference stack (\texttt{hanzo-ml} / \texttt{hanzo-engine}), eliminating the conversion boundary between a Python training stack and a separate serving engine. The existing engine already ships a reverse-mode autograd (\texttt{hanzo-ml}, a fork of HuggingFace \texttt{candle}; $809$ lines, full matmul/conv/gather rules), bias-corrected AdamW with decoupled weight decay, safetensors checkpoint I/O, and a unified quantization kernel that serves \texttt{Q8\_0} decode at $24.5$ tokens/s, equal to $0.98\times$ llama.cpp ($25.05$ tokens/s) on a gfx1151 / Radeon~8060S (Strix~Halo) device. Two trainers prove the substrate end-to-end: an algorithm-complete, tested GRPO reinforcement-learning loop ported from \texttt{zooai/gym}, and an in-engine AnyMoE (the MoE-distillation trainer inherited from \texttt{mistral.rs}) gate trainer that performs forward, cross-entropy, backward, optimizer step, and checkpoint save over frozen quantized expert weights. We show that the QLoRA primitive---a detached \texttt{dequantize()} of the quantized base plus a trainable \texttt{bf16} adapter pair---is already present in the code, so a model can be quantized once, served by the unified kernel, and trained by attaching adapters over the exact bytes that serve it, with zero dequant-to-train and zero re-quant-to-serve. We formalize the parameter and memory accounting of rank-$r$ QLoRA over a frozen quantized base, and prove that linear LoRA adapters over a shared base compose additively (the ``LoRA-soup'' property). We give an honest accounting of what is real, what needs building, and what is deliberately out of scope. We describe the architecture for training language models natively inside the hanzo Rust inference stack (\texttt{hanzo-ml} / \texttt{hanzo-engine}), eliminating the conversion boundary between a Python training stack and a separate serving engine. The existing engine already ships a reverse-mode autograd (\texttt{hanzo-ml}, a fork of HuggingFace \texttt{candle}; $809$ lines, full matmul/conv/gather rules), bias-corrected AdamW with decoupled weight decay, safetensors checkpoint I/O, and a unified quantization kernel that serves \texttt{Q8\_0} decode at $24.5$ tokens/s, equal to $0.98\times$ llama.cpp ($25.05$ tokens/s) on a gfx1151 / Radeon~8060S (Strix~Halo) device. Two trainers prove the substrate end-to-end: an algorithm-complete, tested GRPO reinforcement-learning loop ported from \texttt{zen/gym}, and an in-engine AnyMoE (the MoE-distillation trainer inherited from \texttt{mistral.rs}) gate trainer that performs forward, cross-entropy, backward, optimizer step, and checkpoint save over frozen quantized expert weights. We show that the QLoRA primitive---a detached \texttt{dequantize()} of the quantized base plus a trainable \texttt{bf16} adapter pair---is already present in the code, so a model can be quantized once, served by the unified kernel, and trained by attaching adapters over the exact bytes that serve it, with zero dequant-to-train and zero re-quant-to-serve. We formalize the parameter and memory accounting of rank-$r$ QLoRA over a frozen quantized base, and prove that linear LoRA adapters over a shared base compose additively (the ``LoRA-soup'' property). We give an honest accounting of what is real, what needs building, and what is deliberately out of scope.
\end{abstract} \end{abstract}
\section{Introduction} \section{Introduction}
@@ -91,7 +91,7 @@ The following components are not stubs; each is cited and several are covered by
\subsection{GRPO reinforcement learning (\textsc{Ported}, tested)} \subsection{GRPO reinforcement learning (\textsc{Ported}, tested)}
\texttt{ml/hanzo-training/src/grpo/} contains \texttt{trainer}, \texttt{math}, \texttt{policy}, \texttt{sampler}, \texttt{config}, \texttt{reward}, and \texttt{metrics}. The module is ported from the \texttt{zooai/gym} GRPO~\cite{shao2024grpo} trainer. \texttt{GrpoTrainer::step} (\texttt{trainer.rs:129}) runs the full five-phase update: rollout (\texttt{Policy::sample\_group}); reward (weighted multi-reward sum, \texttt{score()} at \texttt{trainer.rs:119}); group-relative advantages (\texttt{math::batch\_advantages}, unbiased standard deviation with $\mathrm{ddof}=1$, \texttt{math.rs:45--52}); a differentiable, sign-correct, length-normalized policy-gradient loss (\texttt{math::policy\_gradient\_loss}, \texttt{math.rs:87--120}); an optional low-variance \texttt{k3} KL penalty to a frozen reference ($k3 = \exp(\Delta) - \Delta - 1$, \texttt{math.rs:134--166}); and \texttt{AdamW.backward\_step(\&loss)} (\texttt{trainer.rs:228}). It is tested: a smoke test learns a toy reward to ceiling (asserts mean reward rises past $0.6 \cdot \texttt{max\_len}$), a KL-branch test stays finite, and math unit tests verify advantage zero-mean, unit-scale scaling, and policy-gradient sign. \texttt{ml/hanzo-training/src/grpo/} contains \texttt{trainer}, \texttt{math}, \texttt{policy}, \texttt{sampler}, \texttt{config}, \texttt{reward}, and \texttt{metrics}. The module is ported from the \texttt{zen/gym} GRPO~\cite{shao2024grpo} trainer. \texttt{GrpoTrainer::step} (\texttt{trainer.rs:129}) runs the full five-phase update: rollout (\texttt{Policy::sample\_group}); reward (weighted multi-reward sum, \texttt{score()} at \texttt{trainer.rs:119}); group-relative advantages (\texttt{math::batch\_advantages}, unbiased standard deviation with $\mathrm{ddof}=1$, \texttt{math.rs:45--52}); a differentiable, sign-correct, length-normalized policy-gradient loss (\texttt{math::policy\_gradient\_loss}, \texttt{math.rs:87--120}); an optional low-variance \texttt{k3} KL penalty to a frozen reference ($k3 = \exp(\Delta) - \Delta - 1$, \texttt{math.rs:134--166}); and \texttt{AdamW.backward\_step(\&loss)} (\texttt{trainer.rs:228}). It is tested: a smoke test learns a toy reward to ceiling (asserts mean reward rises past $0.6 \cdot \texttt{max\_len}$), a KL-branch test stays finite, and math unit tests verify advantage zero-mean, unit-scale scaling, and policy-gradient sign.
\paragraph{The gap, stated honestly.} The \texttt{Policy} trait (\texttt{policy.rs:33}) requires exactly three methods: \texttt{trainable\_vars()}, \texttt{sample\_group()}, and \texttt{sequence\_logprob()}. The only implementor is \texttt{ToyPolicy}, a \texttt{[max\_len, vocab]} logits bandit in tests and \texttt{examples/grpo\_toy.rs}. No hanzo-engine transformer implements \texttt{Policy} yet; wiring a real model to those three methods is \textsc{Needs-Building} (Section~\ref{sec:migration}). \paragraph{The gap, stated honestly.} The \texttt{Policy} trait (\texttt{policy.rs:33}) requires exactly three methods: \texttt{trainable\_vars()}, \texttt{sample\_group()}, and \texttt{sequence\_logprob()}. The only implementor is \texttt{ToyPolicy}, a \texttt{[max\_len, vocab]} logits bandit in tests and \texttt{examples/grpo\_toy.rs}. No hanzo-engine transformer implements \texttt{Policy} yet; wiring a real model to those three methods is \textsc{Needs-Building} (Section~\ref{sec:migration}).
@@ -244,7 +244,7 @@ Soup / delta-soup & Real & Elementwise mean of checkpoints; \\
\paragraph{DoRA.} Weight-decomposed LoRA~\cite{liu2024dora}: $W' = m \cdot (W_0 + \Delta W)/\lVert W_0 + \Delta W \rVert_{\text{col}}$, where $m$ is a trainable per-column magnitude vector and $\Delta W = BA$. On the engine: add one magnitude \texttt{Var} of shape $[d_{\text{out}}]$, compute the column norm with existing ops, and scale. All differentiable; no new backward. The only cost is recomputing the directional norm each step, a known DoRA expense, not a hanzo-specific one. \paragraph{DoRA.} Weight-decomposed LoRA~\cite{liu2024dora}: $W' = m \cdot (W_0 + \Delta W)/\lVert W_0 + \Delta W \rVert_{\text{col}}$, where $m$ is a trainable per-column magnitude vector and $\Delta W = BA$. On the engine: add one magnitude \texttt{Var} of shape $[d_{\text{out}}]$, compute the column norm with existing ops, and scale. All differentiable; no new backward. The only cost is recomputing the directional norm each step, a known DoRA expense, not a hanzo-specific one.
\paragraph{BitDelta and soups (\textsc{Real}).} \texttt{model\_delta/bitdelta.rs} implements the BitDelta~\cite{liu2024bitdelta} $\mathrm{encode}$/$\mathrm{decode\_apply}$: encode is sign-pack$(W_{\!f}-W_0)$ plus a per-tensor scale $\alpha = \mathrm{mean}|\Delta|$; reconstruct is $W_0 + \alpha \cdot \mathrm{sign}(\Delta)$. \texttt{model\_delta/soup.rs} implements $\mathrm{soup}$ (elementwise mean of checkpoints) and $\mathrm{delta\_soup}$ ($W_0 + \mathrm{mean}_i(W_{\!f,i}-W_0)$), both tested. They operate on saved checkpoints, so they compose with any trainer above: train, save \texttt{Var} weights, then \texttt{bitdelta::encode} for 1-bit delta storage or \texttt{delta\_soup} to average several fine-tunes. Lemma~\ref{lem:soup} is the formal justification for the LoRA case of delta-soup. For the patent record: neither BitDelta nor any model-soup/TIES/DARE routine exists anywhere in \texttt{zooai/gym}; these are hanzo-native with no Axolotl analog. The DeltaSoup \emph{federation} variant (trim-mean of LoRA deltas) lives only in the Python \texttt{zen/gym} and is not yet ported to Rust. \paragraph{BitDelta and soups (\textsc{Real}).} \texttt{model\_delta/bitdelta.rs} implements the BitDelta~\cite{liu2024bitdelta} $\mathrm{encode}$/$\mathrm{decode\_apply}$: encode is sign-pack$(W_{\!f}-W_0)$ plus a per-tensor scale $\alpha = \mathrm{mean}|\Delta|$; reconstruct is $W_0 + \alpha \cdot \mathrm{sign}(\Delta)$. \texttt{model\_delta/soup.rs} implements $\mathrm{soup}$ (elementwise mean of checkpoints) and $\mathrm{delta\_soup}$ ($W_0 + \mathrm{mean}_i(W_{\!f,i}-W_0)$), both tested. They operate on saved checkpoints, so they compose with any trainer above: train, save \texttt{Var} weights, then \texttt{bitdelta::encode} for 1-bit delta storage or \texttt{delta\_soup} to average several fine-tunes. Lemma~\ref{lem:soup} is the formal justification for the LoRA case of delta-soup. For the patent record: neither BitDelta nor any model-soup/TIES/DARE routine exists anywhere in \texttt{zen/gym}; these are hanzo-native with no Axolotl analog. The DeltaSoup \emph{federation} variant (trim-mean of LoRA deltas) lives only in the Python \texttt{zen/gym} and is not yet ported to Rust.
\section{Quant-Aware Training and Inherited Performance} \section{Quant-Aware Training and Inherited Performance}
\label{sec:perf} \label{sec:perf}
@@ -287,7 +287,7 @@ Prefill end-to-end (t/s) & 39 & 1015 & 0.86 \\
\paragraph{Build (\textsc{Needs-Building}), ordered by value/effort.} \paragraph{Build (\textsc{Needs-Building}), ordered by value/effort.}
\begin{enumerate} \begin{enumerate}
\item \textbf{Trainable LoRA/QLoRA adapter layer} (highest value, low-medium effort). A \texttt{TrainableLoraLinear} with $(A,B)$ \texttt{Var}s over a frozen base; QLoRA variant over \texttt{QTensor}. Risk: \textsc{Low}---the AD primitive is proven (Section~\ref{sec:quant}), AnyMoE proves the train-over-frozen pattern (Section~\ref{sec:trainers}), and the adapter math already exists. Estimate: about one week to a tested LoRA $+$ QLoRA on one architecture (Qwen3), reusing \texttt{qmatvec\_core} for the base. \item \textbf{Trainable LoRA/QLoRA adapter layer} (highest value, low-medium effort). A \texttt{TrainableLoraLinear} with $(A,B)$ \texttt{Var}s over a frozen base; QLoRA variant over \texttt{QTensor}. Risk: \textsc{Low}---the AD primitive is proven (Section~\ref{sec:quant}), AnyMoE proves the train-over-frozen pattern (Section~\ref{sec:trainers}), and the adapter math already exists. Estimate: about one week to a tested LoRA $+$ QLoRA on one architecture (the 8B dense transformer), reusing \texttt{qmatvec\_core} for the base.
\item \textbf{Real transformer \texttt{Policy} for GRPO} (high value, medium effort). Implement the three trait methods (\texttt{policy.rs:33}) against a hanzo-engine model: \texttt{sample\_group} is the existing generate loop under no-grad; \texttt{sequence\_logprob} is a teacher-forced forward returning $\sum \log p$. Risk: \textsc{Medium}---needs the \texttt{Var}-backed model path (shared with SFT) and careful no-grad/grad separation. Estimate: one to two weeks; pairs naturally with item 1. \item \textbf{Real transformer \texttt{Policy} for GRPO} (high value, medium effort). Implement the three trait methods (\texttt{policy.rs:33}) against a hanzo-engine model: \texttt{sample\_group} is the existing generate loop under no-grad; \texttt{sequence\_logprob} is a teacher-forced forward returning $\sum \log p$. Risk: \textsc{Medium}---needs the \texttt{Var}-backed model path (shared with SFT) and careful no-grad/grad separation. Estimate: one to two weeks; pairs naturally with item 1.
\item \textbf{Generic SFT trainer $+$ real tokenized dataset} (high value, medium effort). Replace the stub \texttt{Trainer}/\texttt{ModelWrapper}/\texttt{OptimizerWrapper} (Section~\ref{sec:stubs}) with a real epoch/step loop calling \texttt{hanzo\_nn::AdamW} directly, a teacher-forced LM \texttt{cross\_entropy}, and a real \texttt{tokenizers}-backed dataset. Risk: \textsc{Medium}, mostly plumbing. Estimate: one to two weeks. \item \textbf{Generic SFT trainer $+$ real tokenized dataset} (high value, medium effort). Replace the stub \texttt{Trainer}/\texttt{ModelWrapper}/\texttt{OptimizerWrapper} (Section~\ref{sec:stubs}) with a real epoch/step loop calling \texttt{hanzo\_nn::AdamW} directly, a teacher-forced LM \texttt{cross\_entropy}, and a real \texttt{tokenizers}-backed dataset. Risk: \textsc{Medium}, mostly plumbing. Estimate: one to two weeks.
\item \textbf{Preference losses (DPO~\cite{rafailov2023dpo}/KTO/ORPO/SimPO)} (medium value, low effort once item 2 exists). Closed-form differentiable losses over (chosen, rejected) log-probs, using the same \texttt{sequence\_logprob}. Risk: \textsc{Low}. Estimate: a few days each. \item \textbf{Preference losses (DPO~\cite{rafailov2023dpo}/KTO/ORPO/SimPO)} (medium value, low effort once item 2 exists). Closed-form differentiable losses over (chosen, rejected) log-probs, using the same \texttt{sequence\_logprob}. Risk: \textsc{Low}. Estimate: a few days each.
@@ -328,7 +328,7 @@ We state the boundaries of the present architecture plainly.
\section{Conclusion} \section{Conclusion}
The hanzo training substrate is real: a \texttt{candle}-derived autograd, AdamW/SGD, VarMap checkpointing, a differentiable loss family, and a unified quant core that serves \texttt{Q8\_0} decode at parity ($24.5$ tokens/s, $0.98\times$ llama.cpp) and prefill at $0.86\times$. Two trainers prove it trains end-to-end: the GRPO RL loop (algorithm-complete, tested, ported from \texttt{zooai/gym}) and the AnyMoE in-engine gate trainer. The QLoRA primitive---a detached \texttt{dequantize()} base plus trainable \texttt{bf16} adapter \texttt{Var}s---is present in the code; only the adapter \emph{layer} is missing. We formalized the consequences: the trainable footprint is the adapter, not the base (Proposition~\ref{prop:qlora}), and linear adapters over a shared base compose additively (Lemma~\ref{lem:soup}). The build order is clear: \texttt{TrainableLoraLinear} $+$ QLoRA over \texttt{qmatvec\_core}; a real-transformer \texttt{Policy} to unlock GRPO; a real SFT loop with a tokenized dataset; then preference losses and ergonomics. The end state is to train and serve a model in one Rust engine, one quant format, one device backend---with QLoRA adapters learned over the exact kernel that serves them. The hanzo training substrate is real: a \texttt{candle}-derived autograd, AdamW/SGD, VarMap checkpointing, a differentiable loss family, and a unified quant core that serves \texttt{Q8\_0} decode at parity ($24.5$ tokens/s, $0.98\times$ llama.cpp) and prefill at $0.86\times$. Two trainers prove it trains end-to-end: the GRPO RL loop (algorithm-complete, tested, ported from \texttt{zen/gym}) and the AnyMoE in-engine gate trainer. The QLoRA primitive---a detached \texttt{dequantize()} base plus trainable \texttt{bf16} adapter \texttt{Var}s---is present in the code; only the adapter \emph{layer} is missing. We formalized the consequences: the trainable footprint is the adapter, not the base (Proposition~\ref{prop:qlora}), and linear adapters over a shared base compose additively (Lemma~\ref{lem:soup}). The build order is clear: \texttt{TrainableLoraLinear} $+$ QLoRA over \texttt{qmatvec\_core}; a real-transformer \texttt{Policy} to unlock GRPO; a real SFT loop with a tokenized dataset; then preference losses and ergonomics. The end state is to train and serve a model in one Rust engine, one quant format, one device backend---with QLoRA adapters learned over the exact kernel that serves them.
\begin{thebibliography}{10} \begin{thebibliography}{10}
hanzo-platform-infra-desktop.pdf
BIN
View File
Binary file not shown.
hanzo-platform-infra-desktop.tex
+8 -8
View File
@@ -45,7 +45,7 @@
\hanzocoverpage \hanzocoverpage
\begin{abstract} \begin{abstract}
We present the serving economics, platform architecture, and on-device thesis that follow from a native Rust train-and-inference engine whose decode kernel runs at the memory-bandwidth wall. On a single AMD Ryzen AI Max 8060S (Strix Halo, gfx1151, RDNA3.5 iGPU) with an achievable read peak of $\sim$253~GB/s, the unified quant-matvec core reaches 245.8~GB/s after a lane-strided rewrite---$97\%$ of that peak, above the conservative 231~GB/s stream-copy figure. From this roofline we derive a cost law: for a memory-bound autoregressive decoder, tokens per second per dollar scales as achieved bandwidth divided by the product of per-parameter quantization width and active parameter count, making the quantization choice---not the purchase of larger accelerators---the primary cost lever. We prove this proposition and a companion multi-tenant throughput result under paged attention. The engine attains decode parity (0.98$\times$ llama.cpp) on Qwen3-8B-Q8\_0 and 2$\times$ throughput on a 35B-A3B mixture-of-experts model, both on consumer silicon, supports eleven sub-4-bit quantization formats bit-exactly against \texttt{ggml}, and shares one tensor type and one quantization core across training and serving. We describe the platform control plane and an on-device continuous-learning product direction, and we are explicit throughout about what exists today versus what remains roadmap. The honest claims are: AMD parity makes AMD a cost lever; a genuinely unified native train-and-serve runtime exists; and a structurally private on-device path is feasible---each with its gaps named. We present the serving economics, platform architecture, and on-device thesis that follow from a native Rust train-and-inference engine whose decode kernel runs at the memory-bandwidth wall. On a single AMD Ryzen AI Max 8060S (Strix Halo, gfx1151, RDNA3.5 iGPU) with an achievable read peak of $\sim$253~GB/s, the unified quant-matvec core reaches 245.8~GB/s after a lane-strided rewrite---$97\%$ of that peak, above the conservative 231~GB/s stream-copy figure. From this roofline we derive a cost law: for a memory-bound autoregressive decoder, tokens per second per dollar scales as achieved bandwidth divided by the product of per-parameter quantization width and active parameter count, making the quantization choice---not the purchase of larger accelerators---the primary cost lever. We prove this proposition and a companion multi-tenant throughput result under paged attention. The engine attains decode parity (0.98$\times$ llama.cpp) on an 8B dense transformer at Q8\_0 and 2$\times$ throughput on a 35B-A3B mixture-of-experts model, both on consumer silicon, supports eleven sub-4-bit quantization formats bit-exactly against \texttt{ggml}, and shares one tensor type and one quantization core across training and serving. We describe the platform control plane and an on-device continuous-learning product direction, and we are explicit throughout about what exists today versus what remains roadmap. The honest claims are: AMD parity makes AMD a cost lever; a genuinely unified native train-and-serve runtime exists; and a structurally private on-device path is feasible---each with its gaps named.
\end{abstract} \end{abstract}
\section{Introduction} \section{Introduction}
@@ -110,7 +110,7 @@ Two consequences for serving economics follow immediately from Theorem~\ref{prop
\paragraph{Right-size by quant, not by buying bigger accelerators.} Halving bytes per token (for example, Q8\_0 $\to$ Q4\_K) roughly doubles the decode ceiling on the \emph{same} silicon. Since the matrix-vector kernel is already at the bandwidth wall, the lever a serving operator actually controls is the on-disk quantization, not the kernel. \paragraph{Right-size by quant, not by buying bigger accelerators.} Halving bytes per token (for example, Q8\_0 $\to$ Q4\_K) roughly doubles the decode ceiling on the \emph{same} silicon. Since the matrix-vector kernel is already at the bandwidth wall, the lever a serving operator actually controls is the on-disk quantization, not the kernel.
\paragraph{MoE is the cheapest-per-quality tier.} By Definition~\ref{def:footprint}, a sparsely-gated mixture-of-experts model~\cite{shazeer2017} reads only its active parameters per token. A 35B-A3B model reads only the $\sim$3B activated parameters, so a 35B-capacity model decodes like a $\sim$3B dense model. The campaign measured 13.6 tokens per second for Qwen3.6-35B-A3B at Q4\_K on the consumer APU (measured), a $2\times$ improvement over the prior MoE path delivered by routing each expert through the same banked decode core. \paragraph{MoE is the cheapest-per-quality tier.} By Definition~\ref{def:footprint}, a sparsely-gated mixture-of-experts model~\cite{shazeer2017} reads only its active parameters per token. A 35B-A3B model reads only the $\sim$3B activated parameters, so a 35B-capacity model decodes like a $\sim$3B dense model. The campaign measured 13.6 tokens per second for a 35B-A3B MoE at Q4\_K on the consumer APU (measured), a $2\times$ improvement over the prior MoE path delivered by routing each expert through the same banked decode core.
Table~\ref{tab:roofline} reports the measured decode figures and the bandwidth-implied ceilings on the single 8060S. Table~\ref{tab:roofline} reports the measured decode figures and the bandwidth-implied ceilings on the single 8060S.
@@ -122,14 +122,14 @@ Table~\ref{tab:roofline} reports the measured decode figures and the bandwidth-i
\toprule \toprule
\textbf{Model} & \textbf{Quant} & \textbf{Resident} & \textbf{Decode (t/s)} \\ \textbf{Model} & \textbf{Quant} & \textbf{Resident} & \textbf{Decode (t/s)} \\
\midrule \midrule
Qwen3-8B & Q8\_0 & $\sim$8.7~GB & 24.5 (0.98$\times$) \\ 8B dense & Q8\_0 & $\sim$8.7~GB & 24.5 (0.98$\times$) \\
Qwen3-8B & Q4\_K & $\sim$4.9~GB & kernel 181~GB/s \\ 8B dense & Q4\_K & $\sim$4.9~GB & kernel 181~GB/s \\
Qwen3.6-35B-A3B & Q4\_K & $\sim$3~GB active & 13.6 (2$\times$) \\ 35B-A3B MoE & Q4\_K & $\sim$3~GB active & 13.6 (2$\times$) \\
\bottomrule \bottomrule
\end{tabular} \end{tabular}
\end{table} \end{table}
The Qwen3-8B-Q8\_0 figure of 24.5 tokens per second is 0.98$\times$ the reference \texttt{llama.cpp} decode rate of 25.05 on the same chip---effective parity, and the honest decode ceiling for this kernel-plus-fusion approach. The Q4\_K kernel itself runs at 181~GB/s (79\% of peak) at decode shapes and is already bandwidth-efficient; the 6--8 tokens per second observed on a 14B-Q4\_K model is engine-side overhead amplified by depth, not a kernel deficiency---an honest gap discussed in Section~\ref{sec:limitations}. The 8B-model Q8\_0 figure of 24.5 tokens per second is 0.98$\times$ the reference \texttt{llama.cpp} decode rate of 25.05 on the same chip---effective parity, and the honest decode ceiling for this kernel-plus-fusion approach. The Q4\_K kernel itself runs at 181~GB/s (79\% of peak) at decode shapes and is already bandwidth-efficient; the 6--8 tokens per second observed on a 14B-Q4\_K model is engine-side overhead amplified by depth, not a kernel deficiency---an honest gap discussed in Section~\ref{sec:limitations}.
\subsection{One Quant-Agnostic Core, Not a Kernel-per-Quant Zoo} \subsection{One Quant-Agnostic Core, Not a Kernel-per-Quant Zoo}
@@ -231,7 +231,7 @@ The honest framing is that the moat is the unified native engine, not the orches
The product thesis (aspirational) is a personal AI that runs, learns, and improves entirely on the user's machine: inference is local, personalization is local, and a continuous-learning loop adapts the model to the user, with no cloud required and raw data never leaving the device. The same native stack that serves the cloud tier also fits a laptop-class APU, so personal AI is the full train-and-infer engine running privately, not a stripped cloud client. There is no dedicated desktop application crate in the tree today; Section~\ref{sec:desktop} is product direction grounded in what the runtime can already do locally, not a shipped app. The product thesis (aspirational) is a personal AI that runs, learns, and improves entirely on the user's machine: inference is local, personalization is local, and a continuous-learning loop adapts the model to the user, with no cloud required and raw data never leaving the device. The same native stack that serves the cloud tier also fits a laptop-class APU, so personal AI is the full train-and-infer engine running privately, not a stripped cloud client. There is no dedicated desktop application crate in the tree today; Section~\ref{sec:desktop} is product direction grounded in what the runtime can already do locally, not a shipped app.
The technical credibility rests on three present-day facts. First, the entire performance campaign ran on a single 8060S, reaching decode parity on Qwen3-8B-Q8\_0 and 2$\times$ on a 35B-A3B MoE---desktop-class silicon, not a datacenter accelerator (measured). Second, the training primitives are native and local: autograd, AdamW and SGD, VarMap checkpointing, and a working, tested trainer for group relative policy optimization (GRPO)~\cite{shahul2024} all run in-process, so on-device reinforcement-learning personalization is not theoretical (available). Third, ISQ lets the desktop load one checkpoint and pick the quantization that fits local memory---a quantized low-rank-adaptation base in the manner of QLoRA~\cite{dettmers2023}---trading quality for the device's bandwidth budget per the roofline of Section~\ref{sec:serving} (available). The technical credibility rests on three present-day facts. First, the entire performance campaign ran on a single 8060S, reaching decode parity on the 8B model at Q8\_0 and 2$\times$ on a 35B-A3B MoE---desktop-class silicon, not a datacenter accelerator (measured). Second, the training primitives are native and local: autograd, AdamW and SGD, VarMap checkpointing, and a working, tested trainer for group relative policy optimization (GRPO)~\cite{shahul2024} all run in-process, so on-device reinforcement-learning personalization is not theoretical (available). Third, ISQ lets the desktop load one checkpoint and pick the quantization that fits local memory---a quantized low-rank-adaptation base in the manner of QLoRA~\cite{dettmers2023}---trading quality for the device's bandwidth budget per the roofline of Section~\ref{sec:serving} (available).
\subsection{Continuous Learning: Design and Honest Gaps} \subsection{Continuous Learning: Design and Honest Gaps}
@@ -260,7 +260,7 @@ This section's arithmetic is illustrative and derived only from the measured roo
\end{equation} \end{equation}
where $C_{\mathrm{hr}}$ is hardware dollars per hour and $R$ is decode tokens per second, itself set by quantization and bandwidth through Equation~\eqref{eq:tps}. Two levers, both owned by the operator, appear: choose cheaper hardware (an AMD APU versus an NVIDIA datacenter accelerator) and choose a smaller quantization (Q4\_K versus Q8\_0, roughly $2\times$ throughput on the same chip). where $C_{\mathrm{hr}}$ is hardware dollars per hour and $R$ is decode tokens per second, itself set by quantization and bandwidth through Equation~\eqref{eq:tps}. Two levers, both owned by the operator, appear: choose cheaper hardware (an AMD APU versus an NVIDIA datacenter accelerator) and choose a smaller quantization (Q4\_K versus Q8\_0, roughly $2\times$ throughput on the same chip).
Worked from measured throughput on the 8060S for Qwen3-8B: at Q8\_0 and 24.5 tokens per second, one million tokens is about 11.3 GPU-hours of decode by Equation~\eqref{eq:dollarmtok}; at Q4\_K with a kernel-bound ceiling near 44 tokens per second, about 6.3 GPU-hours---provided the engine tail is closed to realize it, since a 14B-Q4\_K model is engine-limited to 6--8 tokens per second today (an honest gap, not the kernel). The shape of the model dominates: a 35B-A3B MoE at Q4\_K decodes about $2\times$ faster than a dense 8B at Q8\_0 while carrying far more capacity, so MoE-at-low-quant is the cheapest quality tier. Worked from measured throughput on the 8060S for the 8B model: at Q8\_0 and 24.5 tokens per second, one million tokens is about 11.3 GPU-hours of decode by Equation~\eqref{eq:dollarmtok}; at Q4\_K with a kernel-bound ceiling near 44 tokens per second, about 6.3 GPU-hours---provided the engine tail is closed to realize it, since a 14B-Q4\_K model is engine-limited to 6--8 tokens per second today (an honest gap, not the kernel). The shape of the model dominates: a 35B-A3B MoE at Q4\_K decodes about $2\times$ faster than a dense 8B at Q8\_0 while carrying far more capacity, so MoE-at-low-quant is the cheapest quality tier.
\begin{table}[h] \begin{table}[h]
\centering \centering
hanzo-rocm-inference.pdf
BIN
View File
Binary file not shown.
hanzo-rocm-inference.tex
+6 -6
View File
@@ -54,7 +54,7 @@ A consumer APU is a brutally honest target for inference engineering. The Radeon
The thesis of this paper is narrow and measured. Starting from a native-ROCm backend that was the slowest of the project's five---an order of magnitude or worse behind \texttt{llama.cpp} (f32 prefill $39$ tok/s $=0.033\times$; decode $1.7$ tok/s $=0.068\times$; even the project's best pre-existing path, the Vulkan backend, reached only $\approx 0.34\times$ decode and $\approx 0.09\times$ prefill)---a kernel-plus-engine campaign reached effective decode parity ($0.98\times$) and $\approx 0.86\times$ prefill on this APU, while keeping every quantization format bit-exact against the CPU reference. We separate, throughout, techniques \emph{ported} from \texttt{llama.cpp}/\texttt{ggml}~\cite{ggml,llamacpp} (which are excellent and which we copied deliberately) from genuinely \emph{novel} hanzo IP, and we mark each roadmap item \textbf{FEASIBLE-now} versus \textbf{ASPIRATIONAL}. The thesis of this paper is narrow and measured. Starting from a native-ROCm backend that was the slowest of the project's five---an order of magnitude or worse behind \texttt{llama.cpp} (f32 prefill $39$ tok/s $=0.033\times$; decode $1.7$ tok/s $=0.068\times$; even the project's best pre-existing path, the Vulkan backend, reached only $\approx 0.34\times$ decode and $\approx 0.09\times$ prefill)---a kernel-plus-engine campaign reached effective decode parity ($0.98\times$) and $\approx 0.86\times$ prefill on this APU, while keeping every quantization format bit-exact against the CPU reference. We separate, throughout, techniques \emph{ported} from \texttt{llama.cpp}/\texttt{ggml}~\cite{ggml,llamacpp} (which are excellent and which we copied deliberately) from genuinely \emph{novel} hanzo IP, and we mark each roadmap item \textbf{FEASIBLE-now} versus \textbf{ASPIRATIONAL}.
All performance numbers in this paper are measured on that one machine against \texttt{llama.cpp} (HIP backend) on the \emph{same} GPU, same model, same quiet-GPU conditions. Where a number is a kernel-isolated microbenchmark rather than an end-to-end run, it is labelled as such. The model used as the primary benchmark is \textbf{Qwen3-8B-Q8\_0}~\cite{qwen3} ($36$ layers, $n_{\text{embd}}=4096$, $n_{\text{ff}}=12288$, $32$ query / $8$ KV heads, head dim $128$); its resident weights are $\approx 8.04$ GB in Q8\_0 ($\approx 1.06$ bytes/element). All performance numbers in this paper are measured on that one machine against \texttt{llama.cpp} (HIP backend) on the \emph{same} GPU, same model, same quiet-GPU conditions. Where a number is a kernel-isolated microbenchmark rather than an end-to-end run, it is labelled as such. The model used as the primary benchmark is \textbf{an 8B dense transformer}~\cite{qwen3} ($36$ layers, $n_{\text{embd}}=4096$, $n_{\text{ff}}=12288$, $32$ query / $8$ KV heads, head dim $128$); its resident weights are $\approx 8.04$ GB in Q8\_0 ($\approx 1.06$ bytes/element).
\paragraph{Contributions.} \paragraph{Contributions.}
\begin{itemize} \begin{itemize}
@@ -66,7 +66,7 @@ All performance numbers in this paper are measured on that one machine against \
\section{Starting Point and the Shape of the Problem} \section{Starting Point and the Shape of the Problem}
The matmul shapes that dominate Qwen3-8B are the FFN gate/up ($N=12288$, $K=4096$), FFN down ($N=4096$, $K=12288$), the four attention projections ($N\in\{1024,4096\}$, $K=4096$), and the LM head ($N=151936$, $K=4096$). Two regimes matter and they have opposite bottlenecks. The matmul shapes that dominate the 8B model are the FFN gate/up ($N=12288$, $K=4096$), FFN down ($N=4096$, $K=12288$), the four attention projections ($N\in\{1024,4096\}$, $K=4096$), and the LM head ($N=151936$, $K=4096$). Two regimes matter and they have opposite bottlenecks.
\paragraph{Decode (batch $=1$, one new token).} Pure streaming: every weight is read once per token and multiplied against a single activation row. This is \textbf{memory-bandwidth bound}. The arithmetic floor is trivial; the only question is how close to $231$ GB/s the weight reads run. As Section~\ref{sec:roofline} proves, $8.04$ GB at the conservative $217$ GB/s is $\approx 37$ ms, and the $\approx 8.7$ GB total resident footprint gives $\approx 40$ ms $\approx 25$ tok/s---\emph{exactly} the figure \texttt{llama.cpp} achieves (tg64 $=25.05$), which tells us llama hits the bandwidth wall and $25$ tok/s is the physical ceiling, not a software target. \paragraph{Decode (batch $=1$, one new token).} Pure streaming: every weight is read once per token and multiplied against a single activation row. This is \textbf{memory-bandwidth bound}. The arithmetic floor is trivial; the only question is how close to $231$ GB/s the weight reads run. As Section~\ref{sec:roofline} proves, $8.04$ GB at the conservative $217$ GB/s is $\approx 37$ ms, and the $\approx 8.7$ GB total resident footprint gives $\approx 40$ ms $\approx 25$ tok/s---\emph{exactly} the figure \texttt{llama.cpp} achieves (tg64 $=25.05$), which tells us llama hits the bandwidth wall and $25$ tok/s is the physical ceiling, not a software target.
@@ -84,7 +84,7 @@ The decisive move was adopting \texttt{llama.cpp}'s \textbf{\texttt{mul\_mat\_q}
MMQ then progressed $170 \to 224$ (fuse activation-quant into staging) $\to 340$ (llama's actual $128{\times}128$ tile) $\to 473$ (a correctness-and-speed insight: our fused in-kernel quant was re-quantizing each activation row $\approx 96\times$, once per $N$-column tile of the $12288$-wide FFN; llama quantizes activations \emph{once} into a q8\_1 buffer, so we de-fused it) $\to 636$ (llama's $+4$-byte bank-conflict pad, $16$-byte vectorized staging, shared-staged scales) $\to 651$ stable. \textbf{Lesson: don't blind-tune---copy llama's actual config.} Two ISA-backed counter-examples to naive porting: llama's literal $8$-warp / $8$-fragment layout \emph{regresses} here ($\approx 425$ versus $474$ tok/s)---with our hand-rolled \texttt{v4i} fragments it spills ($192$ VGPR $+\approx 541$ spills, $7$ waves) versus the $16$-warp / $4$-fragment layout ($132$ VGPR, $10$ waves, $0$ spills); and stream-K is not llama's RDNA3 path (disabled for non-CDNA AMD), so we did not build it. MMQ then progressed $170 \to 224$ (fuse activation-quant into staging) $\to 340$ (llama's actual $128{\times}128$ tile) $\to 473$ (a correctness-and-speed insight: our fused in-kernel quant was re-quantizing each activation row $\approx 96\times$, once per $N$-column tile of the $12288$-wide FFN; llama quantizes activations \emph{once} into a q8\_1 buffer, so we de-fused it) $\to 636$ (llama's $+4$-byte bank-conflict pad, $16$-byte vectorized staging, shared-staged scales) $\to 651$ stable. \textbf{Lesson: don't blind-tune---copy llama's actual config.} Two ISA-backed counter-examples to naive porting: llama's literal $8$-warp / $8$-fragment layout \emph{regresses} here ($\approx 425$ versus $474$ tok/s)---with our hand-rolled \texttt{v4i} fragments it spills ($192$ VGPR $+\approx 541$ spills, $7$ waves) versus the $16$-warp / $4$-fragment layout ($132$ VGPR, $10$ waves, $0$ spills); and stream-K is not llama's RDNA3 path (disabled for non-CDNA AMD), so we did not build it.
The disassembly's honest conclusion: \emph{we are more occupancy-efficient than llama here} ($124$ VGPR / $10$ waves versus llama's measured $168$--$240$). The residual prefill-GEMM gap is therefore not registers or occupancy---it is \textbf{weight-load coalescing}. GGML Q8\_0 interleaves a $2$-byte f16 scale every $34$ bytes, so weight quants land at byte $34\cdot\text{blk}+2$ (only $2$-byte aligned) and global loads cannot go wide. A weight \textbf{repack} to contiguous int8 plus separate f16 scales (llama's \texttt{block\_q8\_1\_mmq}) fixes it---\emph{proven} at $739$ tok/s on Qwen3-4B-Q8\_0---but on 8B it doubles resident weight VRAM (raw Q8\_0 for decode \emph{and} the repacked arrays for prefill $\approx 17$ GB), silently spills on the 8060S budget, and \emph{collapses} prefill to $58$ tok/s. Honest partial result: a win \emph{where memory fits}; landing it on 8B means serving \emph{both} paths from the flat repacked arrays ($\approx 8.7$ GB)---a FEASIBLE-now follow-on, not done. The disassembly's honest conclusion: \emph{we are more occupancy-efficient than llama here} ($124$ VGPR / $10$ waves versus llama's measured $168$--$240$). The residual prefill-GEMM gap is therefore not registers or occupancy---it is \textbf{weight-load coalescing}. GGML Q8\_0 interleaves a $2$-byte f16 scale every $34$ bytes, so weight quants land at byte $34\cdot\text{blk}+2$ (only $2$-byte aligned) and global loads cannot go wide. A weight \textbf{repack} to contiguous int8 plus separate f16 scales (llama's \texttt{block\_q8\_1\_mmq}) fixes it---\emph{proven} at $739$ tok/s on a 4B dense transformer at Q8\_0---but on 8B it doubles resident weight VRAM (raw Q8\_0 for decode \emph{and} the repacked arrays for prefill $\approx 17$ GB), silently spills on the 8060S budget, and \emph{collapses} prefill to $58$ tok/s. Honest partial result: a win \emph{where memory fits}; landing it on 8B means serving \emph{both} paths from the flat repacked arrays ($\approx 8.7$ GB)---a FEASIBLE-now follow-on, not done.
\subsection{Decode: the lane-strided unified core---the parity lever} \subsection{Decode: the lane-strided unified core---the parity lever}
\label{sec:lanestride} \label{sec:lanestride}
@@ -132,7 +132,7 @@ The fusion batch that closed it (each red-verified against a numeric oracle, out
\subsection{The hipGraph Detour---Correct, Exhaustively Debugged, and \emph{Not} the Lever} \subsection{The hipGraph Detour---Correct, Exhaustively Debugged, and \emph{Not} the Lever}
The most instructive negative result in the campaign is the HIP-graph effort. The premise was textbook: decode was launch-bound ($\approx 2312$ launches/token), so capture the per-token decode once and replay it as one graph. The project was large---it required first building a ROCm PagedAttention~\cite{kwon2023} backend (the engine's existing decode-graph machinery is CUDA-only, gated on PagedAttention holding dynamic KV state in stable device buffers), then making three synchronous default-stream ops async so capture would not trip \texttt{hipErrorStreamCaptureImplicit}, then chasing a deep coherence bug. Capture succeeded but replay produced a degenerate loop; a 508-step probe found the compute graph \emph{bit-exact} (all $423$ forward intermediates, including the $151936$-wide LM-head logits, matched eager to \texttt{max\_abs\_diff}$=0.0$). The bug was in no kernel: Qwen3 has no sliding window, so paged-attention takes the \texttt{use\_full} path, and the ROCm graph metadata refreshed only the windowed buffers, cloning the full ones verbatim from warmup, so the captured kernel attended a \emph{frozen span} every token. The fix (mirroring \texttt{cuda\_graph.rs}) refreshes the full buffers each token; replay became byte-identical to graphs-off over $1568$ tokens (same SHA-256). The most instructive negative result in the campaign is the HIP-graph effort. The premise was textbook: decode was launch-bound ($\approx 2312$ launches/token), so capture the per-token decode once and replay it as one graph. The project was large---it required first building a ROCm PagedAttention~\cite{kwon2023} backend (the engine's existing decode-graph machinery is CUDA-only, gated on PagedAttention holding dynamic KV state in stable device buffers), then making three synchronous default-stream ops async so capture would not trip \texttt{hipErrorStreamCaptureImplicit}, then chasing a deep coherence bug. Capture succeeded but replay produced a degenerate loop; a 508-step probe found the compute graph \emph{bit-exact} (all $423$ forward intermediates, including the $151936$-wide LM-head logits, matched eager to \texttt{max\_abs\_diff}$=0.0$). The bug was in no kernel: the benchmark model has no sliding window, so paged-attention takes the \texttt{use\_full} path, and the ROCm graph metadata refreshed only the windowed buffers, cloning the full ones verbatim from warmup, so the captured kernel attended a \emph{frozen span} every token. The fix (mirroring \texttt{cuda\_graph.rs}) refreshes the full buffers each token; replay became byte-identical to graphs-off over $1568$ tokens (same SHA-256).
The punchline: \textbf{zero speedup} (graphs-on $12.7$ tok/s $=$ graphs-off $12.7$). Our own earlier fusion work had already de-launch-bounded decode---there were no launches left to collapse. The graph project ran on a stale premise the fusion had invalidated. \textbf{Lesson: re-measure the bottleneck after every major change.} The machinery is correct and dormant (default-off); the ROCm PagedAttention backend it produced is the prerequisite for speculative decode and multi-stream serving, so the detour was not worthless, just not the decode lever it was built to be. The punchline: \textbf{zero speedup} (graphs-on $12.7$ tok/s $=$ graphs-off $12.7$). Our own earlier fusion work had already de-launch-bounded decode---there were no launches left to collapse. The graph project ran on a stale premise the fusion had invalidated. \textbf{Lesson: re-measure the bottleneck after every major change.} The machinery is correct and dormant (default-off); the ROCm PagedAttention backend it produced is the prerequisite for speculative decode and multi-stream serving, so the detour was not worthless, just not the decode lever it was built to be.
@@ -189,7 +189,7 @@ One decode step must, by definition of the weight-streaming model, transfer at l
\begin{corollary}[Q8\_0 ceiling on \texttt{gfx1151}] \begin{corollary}[Q8\_0 ceiling on \texttt{gfx1151}]
\label{cor:q80} \label{cor:q80}
For Qwen3-8B-Q8\_0 with total resident footprint $W \approx 8.7$ GB streamed at the conservative achieved bandwidth $B \approx 217$ GB/s, the decode ceiling is For the 8B model at Q8\_0 with total resident footprint $W \approx 8.7$ GB streamed at the conservative achieved bandwidth $B \approx 217$ GB/s, the decode ceiling is
\[ \[
T \leq \frac{217\ \text{GB/s}}{8.7\ \text{GB}} \approx 24.9\ \text{tok/s} \approx 25\ \text{tok/s}. T \leq \frac{217\ \text{GB/s}}{8.7\ \text{GB}} \approx 24.9\ \text{tok/s} \approx 25\ \text{tok/s}.
\] \]
@@ -225,7 +225,7 @@ A complementary hardware finding killed a class of bad ideas: \textbf{RDNA3.5 \t
\section{Results} \section{Results}
End-to-end, eager path, quiet GPU, Qwen3-8B-Q8\_0 on \texttt{gfx1151}, versus \texttt{llama.cpp} HIP on the \emph{same} GPU (llama parity targets: pp512 $=1176.3$, tg64 $=25.05$). All hanzo configs are bit-exact (\texttt{nbad}$=0$) and produce coherent output. End-to-end, eager path, quiet GPU, the 8B model at Q8\_0 on \texttt{gfx1151}, versus \texttt{llama.cpp} HIP on the \emph{same} GPU (llama parity targets: pp512 $=1176.3$, tg64 $=25.05$). All hanzo configs are bit-exact (\texttt{nbad}$=0$) and produce coherent output.
\begin{table}[h] \begin{table}[h]
\centering \centering
site/public/pdfs/hanzo-continuous-learning-privacy.pdf
BIN
View File
Binary file not shown.
site/public/pdfs/hanzo-cross-backend-inference.pdf
BIN
View File
Binary file not shown.
site/public/pdfs/hanzo-native-stack-thesis.pdf
BIN
View File
Binary file not shown.
site/public/pdfs/hanzo-native-training.pdf
BIN
View File
Binary file not shown.
site/public/pdfs/hanzo-platform-infra-desktop.pdf
BIN
View File
Binary file not shown.
site/public/pdfs/hanzo-rocm-inference.pdf
BIN
View File
Binary file not shown.
site/src/config/papers.ts
+74
View File
@@ -233,6 +233,80 @@ export const siteConfig: SiteConfig = {
tags: ['Defense', 'Test & Evaluation', 'Digital Twin', 'Observability'], tags: ['Defense', 'Test & Evaluation', 'Digital Twin', 'Observability'],
}, },
// --- Engine / AI Infrastructure (measured, on-device) ---
{
id: 'hanzo-native-stack-thesis',
title: 'One Native Stack for Private, Continuously-Learning AI',
subtitle: 'Inference at the Bandwidth Wall, On-Device QLoRA, One Engine',
abstract: 'A synthesis paper: train and serve collapse into a single Rust engine with one tensor type, one quantization core, and one device backend — eliminating the format-break tax of training in PyTorch, exporting to GGUF/ONNX, and serving on a third engine. Formalizes a unified bandwidth-budget theorem and QLoRA bit-exactness, with every claim sourced to a measured ROCm campaign on consumer hardware. Scrupulously scoped: claims no speed win over NVIDIA and no zero-leakage guarantee.',
pdfUrl: '/pdfs/hanzo-native-stack-thesis.pdf',
latexUrl: 'https://github.com/hanzoai/papers/blob/main/hanzo-native-stack-thesis.tex',
githubUrl: 'https://github.com/hanzoai/papers',
date: '2026-01-01',
authors: ['Zach Kelling'],
tags: ['Engine', 'Inference', 'On-Device', 'Privacy'],
},
{
id: 'hanzo-rocm-inference',
title: 'Native ROCm Inference on a Consumer RDNA3.5 APU',
subtitle: 'Reaching llama.cpp Decode Parity via a Unified 1-bit-to-Full Quant Core',
abstract: 'A measurement-driven engineering campaign that took native-ROCm LLM inference on a consumer AMD Ryzen AI Max+ 395 "Strix Halo" (Radeon 8060S, gfx1151) from the slowest of five backends to decode parity (0.98×) and 0.86× prefill versus llama.cpp on the same GPU. A lane-strided decode kernel reaches 245.8 GB/s — 97% of the memory-bandwidth wall. Documents negative results in detail and derives the physical ~25 tok/s ceiling that bounds both engines.',
pdfUrl: '/pdfs/hanzo-rocm-inference.pdf',
latexUrl: 'https://github.com/hanzoai/papers/blob/main/hanzo-rocm-inference.tex',
githubUrl: 'https://github.com/hanzoai/papers',
date: '2026-01-01',
authors: ['Zach Kelling'],
tags: ['Engine', 'ROCm', 'Inference', 'Benchmarks'],
},
{
id: 'hanzo-cross-backend-inference',
title: 'Edge LLM Inference Across Commodity Accelerators',
subtitle: 'Measured AMD, NVIDIA, and Apple Performance',
abstract: 'Measured single-GPU inference for one 8B dense transformer at Q8_0 across three commodity edge accelerators — AMD "Strix Halo", NVIDIA GB10 "Grace Blackwell", and Apple M4 Max — comparing the Hanzo engine to llama.cpp on identical model and metric. Two findings: decode parity on all three backends, and the fastest box flips by workload (NVIDIA wins prefill, Apple wins decode ~2.5×). Includes a full reproducibility appendix with exact hardware, commands, and git SHA.',
pdfUrl: '/pdfs/hanzo-cross-backend-inference.pdf',
latexUrl: 'https://github.com/hanzoai/papers/blob/main/hanzo-cross-backend-inference.tex',
githubUrl: 'https://github.com/hanzoai/papers',
date: '2026-01-01',
authors: ['Zach Kelling'],
tags: ['Engine', 'Inference', 'Edge', 'Benchmarks'],
},
{
id: 'hanzo-native-training',
title: 'Native Training in the Hanzo Engine',
subtitle: 'One Engine, One Quant Format, One Backend',
abstract: 'The architecture for training language models natively inside the Hanzo Rust inference stack, eliminating the conversion boundary between a Python training stack and a separate serving engine. Shows the QLoRA primitive already exists in the engine\'s reverse-mode autograd, and formalizes QLoRA parameter/memory accounting plus additive LoRA-soup composition. Unusually candid: an entire section names the stack\'s own non-functional stubs to replace.',
pdfUrl: '/pdfs/hanzo-native-training.pdf',
latexUrl: 'https://github.com/hanzoai/papers/blob/main/hanzo-native-training.tex',
githubUrl: 'https://github.com/hanzoai/papers',
date: '2026-01-01',
authors: ['Zach Kelling'],
tags: ['Engine', 'Training', 'QLoRA', 'On-Device'],
},
{
id: 'hanzo-continuous-learning-privacy',
title: 'Continuously-Learning Private AI',
subtitle: 'Self-Improvement and Privacy in the Hanzo Native Stack',
abstract: 'A private continuously-learning model as three loops: an inference loop that serves and captures its own interactions, a curation loop that filters/redacts/de-duplicates into safe training data, and a training loop that folds it back into cheap swappable adapters. Formalizes a privacy invariant (no PII token enters the training set under a sound gate) and bounded-forgetting (frozen base → zero base drift), and is honest about model-collapse and delta-leakage failure modes and which components are still research.',
pdfUrl: '/pdfs/hanzo-continuous-learning-privacy.pdf',
latexUrl: 'https://github.com/hanzoai/papers/blob/main/hanzo-continuous-learning-privacy.tex',
githubUrl: 'https://github.com/hanzoai/papers',
date: '2026-01-01',
authors: ['Zach Kelling'],
tags: ['Engine', 'Privacy', 'Continual Learning', 'On-Device'],
},
{
id: 'hanzo-platform-infra-desktop',
title: 'Economical Serving and the hanzo.ai Platform',
subtitle: 'A Bandwidth-Optimal Unified Train-and-Serve Runtime',
abstract: 'The serving economics, platform architecture, and on-device thesis that follow from a native Rust train-and-inference engine whose decode kernel runs at the memory-bandwidth wall. Derives a cost law (tokens/$/s ∝ B / (q·N_active)) showing quantization — not bigger accelerators — is the primary cost lever, and proves paged-attention tenant concurrency and a structural egress bound (raw data never crosses the wire). Honestly marks train/personalize/federate as stub/roadmap.',
pdfUrl: '/pdfs/hanzo-platform-infra-desktop.pdf',
latexUrl: 'https://github.com/hanzoai/papers/blob/main/hanzo-platform-infra-desktop.tex',
githubUrl: 'https://github.com/hanzoai/papers',
date: '2026-01-01',
authors: ['Zach Kelling'],
tags: ['Engine', 'Economics', 'Platform', 'On-Device'],
},
// --- Hanzo Core Papers --- // --- Hanzo Core Papers ---
{ {
id: 'aso', id: 'aso',