papers/hanzo-gpu-inference/sections/07-production-results.tex

\section{Production Results}

\label{sec:results}
% ============================================================================

We report results from 90 days of production operation (January--March 2026)
across a fleet of 128 H100 GPUs and 64 A100 GPUs serving Zen models.

\begin{table}[H]
\centering
\small
\begin{tabular}{lrrr}
\toprule
\textbf{Metric} & \textbf{Before} & \textbf{After} & \textbf{Improvement} \\
\midrule
Throughput (req/s/GPU) & 42 & 134 & 3.2$\times$ \\
p50 TTFT (ms) & 85 & 31 & 2.7$\times$ \\
p99 TTFT (ms) & 520 & 178 & 2.9$\times$ \\
GPU utilization (\%) & 61 & 94 & +33pp \\
Models per GPU & 1.0 & 3.2 & 3.2$\times$ \\
Cost per 1M tokens (\$) & 0.42 & 0.13 & 3.2$\times$ \\
\bottomrule
\end{tabular}
\caption{Production metrics before and after the optimizations described in this
paper. ``Before'' is the baseline Candle serving with static batching.}
\label{tab:results}
\end{table}

\subsection{Batch Size Distribution}

Adaptive batching produces a bimodal batch size distribution: realtime requests
form small batches (4--16 sequences), while batch workloads fill to the maximum
(64--128 sequences). The scheduler correctly prioritizes realtime requests,
achieving p99 TTFT $<$ 200ms even at 90\% GPU utilization.

\subsection{Multi-Model Sharing Efficiency}

On a typical H100 node, we co-locate Qwen3-7B (14GB), Qwen3-1.5B (3GB), and
an embedding model (1.2GB), leaving 61.8GB for KV-cache and workspace. This
configuration serves 95\% of production traffic without model swapping.

\subsection{Speculative Decoding Impact}

Speculative decoding with Qwen3-0.6B as draft model reduces median inter-token
latency from 18ms to 7.5ms for Qwen3-32B, a 2.4$\times$ improvement.
Acceptance rates vary by task: code generation achieves 0.85 (highly
predictable), creative writing achieves 0.62 (less predictable).

% ============================================================================