papers/hanzo-platform/sections/05-cost-optimized-autoscaler.tex

\section{Cost-Optimized Autoscaler}

\label{sec:autoscaler}

\subsection{Problem Formulation}

Traditional autoscalers use CPU/memory utilization as scaling signals, which poorly correlate with AI workload performance. We formulate autoscaling as a constrained optimization problem:

\begin{align}
\min_{k(t)} \quad & \int_0^T c(k(t)) \, dt \label{eq:autoscale} \\
\text{s.t.} \quad & L_{P99}(k(t), \lambda(t)) \le L_{\max}, \nonumber \\
& \text{TPS}(k(t), \lambda(t)) \ge \text{TPS}_{\min}, \nonumber \\
& k_{\min} \le k(t) \le k_{\max}, \nonumber
\end{align}

where $k(t)$ is the number of replicas at time $t$, $c(k)$ is the cost function, $\lambda(t)$ is the request arrival rate, $L_{P99}$ is the P99 latency, $\text{TPS}$ is tokens per second throughput, and $L_{\max}$, $\text{TPS}_{\min}$ are SLA thresholds.

\subsection{AI-Specific Scaling Signals}

The autoscaler uses four AI-specific signals:

\begin{enumerate}[leftmargin=1.4em]
    \item \textbf{Token throughput}: Tokens generated per second across all replicas.
    \item \textbf{Queue depth}: Number of requests waiting for inference.
    \item \textbf{GPU utilization}: Percentage of GPU compute capacity in use.
    \item \textbf{KV cache pressure}: Fraction of KV cache memory occupied (for transformer inference).
\end{enumerate}

\begin{equation}
\text{Scale signal} = \alpha_1 \cdot \frac{\text{TPS}}{\text{TPS}_{\text{target}}} + \alpha_2 \cdot \frac{\text{Queue}}{\text{Queue}_{\max}} + \alpha_3 \cdot \text{GPU\%} + \alpha_4 \cdot \text{KV\%},
\end{equation}

where $\alpha_i$ are learned weights optimized to minimize cost subject to SLA constraints.

\subsection{Predictive Scaling}

We augment reactive scaling with a time-series forecasting model that predicts load 15 minutes ahead:

\begin{algorithm}[H]
\caption{Predictive Autoscaler}
\label{alg:autoscaler}
\begin{algorithmic}[1]
\Require Current replicas $k$, load history $\lambda_{t-W:t}$, SLA thresholds
\State \Comment{Reactive component}
\State $s_{\text{react}} \gets \text{ScaleSignal}(\text{current metrics})$
\State \Comment{Predictive component}
\State $\hat{\lambda}_{t+\Delta} \gets \text{LSTM.Predict}(\lambda_{t-W:t})$
\State $\hat{k}_{\text{pred}} \gets \text{CapacityModel}(\hat{\lambda}_{t+\Delta}, \text{SLA})$
\State \Comment{Combine with safety margin}
\State $k' \gets \max(k \cdot s_{\text{react}}, \hat{k}_{\text{pred}})$
\State $k' \gets \text{clamp}(k', k_{\min}, k_{\max})$
\State \Comment{Cost gate: prevent over-scaling}
\If{$\text{ProjectedCost}(k') > 1.5 \times \text{Budget}$}
    \State $k' \gets \text{CostConstrained}(k', \text{Budget})$
\EndIf
\State \Return $k'$
\end{algorithmic}
\end{algorithm}

\subsection{Spot Instance Integration}

For GPU workloads, the autoscaler integrates with spot/preemptible instance markets:

\begin{itemize}[leftmargin=1.1em]
    \item \textbf{Price monitoring}: Continuously monitor spot prices across regions and instance types.
    \item \textbf{Hybrid scaling}: Maintain a base of on-demand instances for SLA guarantees; scale burst capacity with spot instances.
    \item \textbf{Graceful preemption}: When spot instances are reclaimed, drain requests and redirect to on-demand capacity.
\end{itemize}

\begin{table}[H]
\centering
\small
\begin{tabular}{lccc}
\toprule
\textbf{Strategy} & \textbf{Avg. Cost} & \textbf{P99 Lat.} & \textbf{Avail.} \\
\midrule
On-demand only & \$1.00 & 850ms & 99.99\% \\
Spot only & \$0.35 & 920ms & 97.2\% \\
Hybrid (Hanzo) & \$0.62 & 870ms & 99.95\% \\
\bottomrule
\end{tabular}
\caption{Cost-availability trade-off with spot instance integration.}
\label{tab:spot}
\end{table}

\subsection{Scaling Performance}

Comparison of autoscaler responsiveness:

\begin{table}[H]
\centering
\small
\begin{tabular}{lccc}
\toprule
\textbf{Platform} & \textbf{Scale-up} & \textbf{Scale-down} & \textbf{Overprov.} \\
\midrule
Vercel & 12s & 300s & 41\% \\
Railway & 45s & 180s & 28\% \\
Render & 120s & 600s & 52\% \\
Fly.io & 8s & 120s & 23\% \\
\textbf{Hanzo} & \textbf{6s} (CPU) & \textbf{90s} & \textbf{14\%} \\
 & \textbf{45s} (GPU) & & \\
\bottomrule
\end{tabular}
\caption{Autoscaler responsiveness. Overprov. = average over-provisioning.}
\label{tab:scaling}
\end{table}