papers/hanzo-gpu-inference/sections/06-latency-optimization.tex

\section{Latency Optimization}

\label{sec:latency}
% ============================================================================

\subsection{Speculative Decoding}

Speculative decoding uses a small draft model to propose $k$ tokens, which the
target model verifies in a single forward pass. If all $k$ tokens are accepted,
we achieve $k\times$ speedup for those tokens.

\begin{definition}[Acceptance Rate]
Let $p(x_t | x_{<t})$ be the target model distribution and $q(x_t | x_{<t})$
be the draft model distribution. The acceptance probability for token $x_t$ is:
\begin{equation}
\alpha(x_t) = \min\left(1, \frac{p(x_t | x_{<t})}{q(x_t | x_{<t})}\right).
\end{equation}
\end{definition}

With a well-aligned draft model (Qwen3-0.6B drafting for Qwen3-32B), we
achieve an average acceptance rate of 0.78 with $k=5$ speculative tokens,
yielding 2.4$\times$ decode speedup.

\subsection{KV-Cache Compression}

Long sequences consume proportionally large KV-cache. We apply two compression
techniques:
\begin{enumerate}[leftmargin=1.4em]
    \item \textbf{Grouped-query attention (GQA)}: Models use 8 KV heads instead
    of 32, reducing cache size by 4$\times$.
    \item \textbf{Quantized cache}: KV-cache entries are stored in FP8 (E4M3)
    instead of FP16, halving cache memory with negligible quality loss
    ($<$0.1\% perplexity increase on validation sets).
\end{enumerate}

Combined, these reduce per-sequence cache from 2.1MB to 262KB for a 7B model
at 4096 context length, enabling 8$\times$ more concurrent sequences.

\subsection{Prefix Caching}

Many requests share common prefixes (system prompts, few-shot examples). We
hash prefix token sequences and cache their KV states:
\begin{equation}
\text{cache\_key} = \texttt{SHA256}(m \| \mathbf{x}_{1:j}),
\end{equation}
where $j$ is the prefix boundary. Cache hits skip prefill for the shared prefix,
reducing TTFT by 60--80\% for repeated system prompts.

% ============================================================================