papers/hanzo-gpu-inference/sections/01-introduction.tex

\section{Introduction}

\label{sec:intro}
% ============================================================================

GPU inference at scale presents a fundamental tension: maximizing throughput
(requests per second per GPU) requires batching, but batching increases latency
for individual requests. Naive approaches---static batch sizes, one-model-per-GPU
allocation, FIFO scheduling---waste GPU cycles and fail to meet the diverse
latency requirements of production workloads~\cite{orca2022,vllm2023}.

Hanzo Cloud serves a heterogeneous workload: real-time chat completions (p99
$<$ 200ms TTFT), batch document processing (throughput-optimized), agent tool
calls (latency-sensitive), and embedding generation (compute-light). Each
workload has different optimal serving parameters. A single static configuration
cannot serve all efficiently.

This paper describes three core contributions:
\begin{enumerate}[leftmargin=1.4em]
    \item \textbf{Adaptive continuous batching} (\S\ref{sec:batching}): A
    scheduler that dynamically adjusts batch composition based on request
    priority, sequence length distribution, and GPU memory pressure.
    \item \textbf{Multi-model spatial sharing} (\S\ref{sec:sharing}): A memory
    manager that co-locates multiple models on a single GPU, switching between
    them with sub-millisecond overhead via pre-allocated memory partitions.
    \item \textbf{GPU-aware cluster scheduling} (\S\ref{sec:scheduling}): A
    cluster-level scheduler that routes requests to optimal GPU nodes based on
    model affinity, current load, thermal state, and network topology.
\end{enumerate}

The system is deployed on Hanzo ACI~\cite{hanzoaci} infrastructure, where GPU
node operators participate in a staked compute market. Integration with the
Candle ML framework~\cite{hanzocandle} provides Rust-native inference kernels,
and the Jin architecture~\cite{hanzojin} handles multimodal routing.

% ============================================================================