papers/hanzo-ai-cloud/sections/06-hanzo-candle.tex

\section{Hanzo Candle}

\label{sec:candle}
% ============================================================================

Hanzo Candle is a Rust ML framework purpose-built for inference
workloads~\cite{hanzocandle}. It provides the GPU acceleration layer for
Hanzo AI Cloud.

\subsection{Design Principles}

\begin{itemize}[leftmargin=1.1em]
    \item \textbf{Inference-first}: Candle optimizes for inference, not
    training. The tensor graph is static (no autograd by default), enabling
    aggressive kernel fusion and memory planning.
    \item \textbf{Rust-native}: No Python runtime, no GIL, no garbage
    collection pauses. Candle runs as a compiled binary with deterministic
    memory management.
    \item \textbf{Multi-backend}: CUDA (NVIDIA), Metal (Apple), Vulkan
    (cross-platform), and CPU (SIMD-optimized) backends share the same API.
    \item \textbf{Model format agnostic}: Loads ONNX, GGUF, SafeTensors, and
    native Candle format. Automatic quantization (INT8/INT4) at load time.
\end{itemize}

\subsection{Performance}

\begin{table}[H]
\centering
\small
\begin{tabular}{lrrr}
\toprule
\textbf{Model} & \textbf{Candle} & \textbf{PyTorch} & \textbf{llama.cpp} \\
\midrule
Qwen3-8B (FP16, A100) & 142 tok/s & 128 tok/s & --- \\
Qwen3-72B (INT4, A100) & 38 tok/s & 31 tok/s & 35 tok/s \\
Qwen3-8B (INT4, M4 CPU) & 28 tok/s & 11 tok/s & 26 tok/s \\
MobileNet-V3 (FP16, A100) & 0.4~ms & 0.6~ms & --- \\
Stable Diffusion (FP16, A100) & 2.8~s & 3.4~s & --- \\
\bottomrule
\end{tabular}
\caption{Inference throughput comparison. Candle's advantage comes from static
graph optimization, kernel fusion, and zero-copy tensor management.}
\label{tab:candle}
\end{table}

\subsection{Base Functions Integration}

Candle models can be invoked from Base Functions via the \texttt{candle}
module in the Goja VM. This enables on-device inference inside a Base
application without an external model serving infrastructure:

\begin{lstlisting}[language=JavaScript,caption={Candle inference from a Base Function.}]
// Base Function: classify an image
onRecordCreate((e) => {
  const img = e.record.get("image");
  const result = candle.infer("mobilenet-v3", img);
  e.record.set("label", result.top_class);
  e.record.set("confidence", result.score);
});
\end{lstlisting}

% ============================================================================