papers/hanzo-stack-comparison/sections/16-ai-inference-hanzo-engine.tex

\section{AI Inference: Hanzo Engine}

\label{sec:inference}
%% ============================================================================

\subsection{Overview}

Hanzo Engine is a Rust-based AI inference server built on the Candle framework. It supports transformer models (LLMs, vision-language models, embedding models) with continuous batching, PagedAttention, speculative decoding, and quantized inference (GPTQ, AWQ, GGUF). The engine exposes an OpenAI-compatible API and integrates with Hanzo IAM for per-model access control.

\subsection{Feature Matrix}

\begin{table}[H]
\centering
\caption{AI inference server comparison.}
\footnotesize
\begin{tabular}{@{}lcccc@{}}
\toprule
Feature & \rotatebox{60}{Engine} & \rotatebox{60}{vLLM} & \rotatebox{60}{TGI} & \rotatebox{60}{Triton} \\
\midrule
Continuous batch & \yes & \yes & \yes & \yes \\
PagedAttention & \yes & \yes & \yes & \pmark \\
Speculative dec. & \yes & \yes & \yes & \no \\
GPTQ quant & \yes & \yes & \yes & \pmark \\
AWQ quant & \yes & \yes & \yes & \no \\
GGUF quant & \yes & \no & \yes & \no \\
Metal support & \yes & \no & \no & \no \\
Multi-GPU & \yes & \yes & \yes & \yes \\
OpenAI compat. & \yes & \yes & \yes & \pmark \\
Tool calling & \yes & \yes & \yes & \no \\
Vision models & \yes & \yes & \yes & \yes \\
IAM-native & \yes & \no & \no & \no \\
Rust native & \yes & \no & \no & \no \\
WASM target & \yes & \no & \no & \no \\
Model marketplace & \yes & \no & \no & \no \\
Open source & \yes & \yes & \yes & \yes \\
K8s CRD & \yes & \pmark & \pmark & \yes \\
\bottomrule
\end{tabular}
\end{table}

\subsection{Performance}

Benchmarked on a single NVIDIA A100 (80~GB), Qwen3-72B-AWQ (4-bit), 2048 input / 512 output tokens, 64 concurrent requests:

\begin{table}[H]
\centering
\caption{LLM inference throughput (tokens/second, output).}
\footnotesize
\begin{tabular}{@{}lrrr@{}}
\toprule
System & Throughput & TTFT (ms) & p99 ITL (ms) \\
\midrule
Hanzo Engine & 3,420 & 185 & 42 \\
vLLM 0.7.x & 3,180 & 210 & 48 \\
TGI 3.x & 2,950 & 240 & 55 \\
Triton + TRT-LLM & 3,600 & 165 & 38 \\
\bottomrule
\end{tabular}
\end{table}

Triton + TensorRT-LLM achieves the highest raw throughput due to NVIDIA's proprietary kernel optimizations. Hanzo Engine is 7.5\% faster than vLLM and 16\% faster than TGI. The Rust implementation eliminates Python GIL contention in the scheduling loop, which is the primary bottleneck in vLLM's Python-based scheduler at high concurrency.

\subsection{Apple Silicon Performance}

Hanzo Engine is the only inference server with first-class Metal support:

\begin{table}[H]
\centering
\caption{LLM inference on Apple Silicon (Qwen3-32B-Q4).}
\footnotesize
\begin{tabular}{@{}lrr@{}}
\toprule
Hardware & Tokens/s & TTFT (ms) \\
\midrule
M3 Max (64 GB) & 28.5 & 380 \\
M4 Ultra (192 GB) & 82.1 & 145 \\
\bottomrule
\end{tabular}
\end{table}

vLLM and TGI require NVIDIA GPUs. Triton supports only NVIDIA hardware. Hanzo Engine runs on any Metal-capable Mac, enabling local development and edge inference on Apple devices.

\subsection{Key Differentiator}

Rust-native implementation with Metal support. No other production inference server runs on Apple Silicon natively. The IAM integration enables per-model, per-organization access control---critical for multi-tenant AI platforms where different organizations have different model access rights. The WASM target enables model inference in the browser, which no competing server supports.

%% ============================================================================