papers/hanzo-chat/sections/03-multi-model-routing.tex

\section{Multi-Model Routing}

\label{sec:routing}

\subsection{Problem Formulation}

Given a query context $C_t$ and a set of available models $\mathcal{M} = \{M_1, \ldots, M_K\}$, select model $\hat{M}$ that maximizes expected quality subject to cost and latency constraints:

\begin{align}
\hat{M} &= \arg\max_{M \in \mathcal{M}} \; \mathbb{E}[Q(M, C_t)] \label{eq:routing} \\
\text{s.t.} \quad & \text{Cost}(M, |C_t|) \le B_t, \nonumber \\
& \text{Latency}(M, |C_t|) \le L_{\max}. \nonumber
\end{align}

\subsection{Query Classification}

We classify queries into categories that predict model performance using a lightweight classifier (distilled BERT, 67M parameters) trained on 50K labeled examples:

\begin{table}[H]
\centering
\small
\begin{tabular}{lcc}
\toprule
\textbf{Category} & \textbf{Best Model Family} & \textbf{Accuracy} \\
\midrule
Code generation & Claude, GPT-4 & 94.2\% \\
Mathematical reasoning & Claude, o1 & 91.7\% \\
Creative writing & GPT-4, Claude & 89.3\% \\
Factual Q\&A & Gemini, GPT-4 & 92.8\% \\
Multimodal & Gemini, GPT-4V & 88.6\% \\
Summarization & Claude, Gemini & 93.1\% \\
Translation & GPT-4, Gemini & 90.4\% \\
Tool use & Claude, GPT-4 & 91.9\% \\
\bottomrule
\end{tabular}
\caption{Query category classification accuracy and best-performing model families.}
\label{tab:categories}
\end{table}

\subsection{Multi-Armed Bandit Formulation}

We model the routing problem as a contextual multi-armed bandit~\cite{li2010contextual} where:

\begin{itemize}[leftmargin=1.1em]
    \item \textbf{Arms}: Available models $\mathcal{M}$.
    \item \textbf{Context}: Query embedding $\bm{e}_t$, category $c_t$, conversation length $|H_{<t}|$, user preferences.
    \item \textbf{Reward}: User satisfaction signal (explicit rating, implicit signals such as follow-up corrections, regeneration requests, copy actions).
\end{itemize}

We use Thompson Sampling with a neural reward model:

\begin{algorithm}[H]
\caption{Neural Thompson Sampling for Model Routing}
\label{alg:thompson}
\begin{algorithmic}[1]
\Require Context $\bm{x}_t = (\bm{e}_t, c_t, |H_{<t}|, U)$, models $\mathcal{M}$
\For{each model $M_k \in \mathcal{M}$}
    \State Sample $\hat{\theta}_k \sim \mathcal{N}(\mu_k(\bm{x}_t), \sigma_k^2(\bm{x}_t))$
    \State $\hat{Q}_k \gets f_\theta(\bm{x}_t, k)$ \Comment{Neural reward estimate}
    \State $\hat{R}_k \gets \hat{Q}_k / \text{Cost}(M_k, |\bm{x}_t|)$ \Comment{Quality per dollar}
\EndFor
\State $\hat{M} \gets \arg\max_k \hat{R}_k$ subject to constraints
\State \Return $\hat{M}$
\end{algorithmic}
\end{algorithm}

The neural reward model $f_\theta$ is a 3-layer MLP with ReLU activations, trained online with replay buffer of size 100K. The posterior variance $\sigma_k^2$ is estimated via MC Dropout~\cite{gal2016dropout}.

\subsection{Cascading Strategy}

For complex queries, we employ a cascading strategy that starts with a cheaper model and escalates if quality is insufficient:

\begin{algorithm}[H]
\caption{Model Cascade}
\label{alg:cascade}
\begin{algorithmic}[1]
\Require Context $C_t$, model tiers $[M_{\text{fast}}, M_{\text{mid}}, M_{\text{best}}]$
\State $r_t \gets \text{Generate}(M_{\text{fast}}, C_t)$
\State $q \gets \text{QualityEstimate}(r_t, C_t)$
\If{$q < \tau_1$}
    \State $r_t \gets \text{Generate}(M_{\text{mid}}, C_t)$
    \State $q \gets \text{QualityEstimate}(r_t, C_t)$
    \If{$q < \tau_2$}
        \State $r_t \gets \text{Generate}(M_{\text{best}}, C_t)$
    \EndIf
\EndIf
\State \Return $r_t$
\end{algorithmic}
\end{algorithm}

The quality estimator is a small classifier trained to predict human preference rankings from response features (length, perplexity, tool call correctness, factual consistency).

\subsection{Cost Optimization}

With the cascade and routing combined, Hanzo Chat achieves significant cost savings:

\begin{table}[H]
\centering
\small
\begin{tabular}{lccc}
\toprule
\textbf{Strategy} & \textbf{Quality} & \textbf{Cost/query} & \textbf{Latency} \\
\midrule
Always best model & 8.92/10 & \$0.042 & 3.2s \\
Always cheapest & 6.14/10 & \$0.003 & 0.8s \\
Random selection & 7.31/10 & \$0.019 & 1.9s \\
Hanzo routing & 8.73/10 & \$0.025 & 2.1s \\
Hanzo cascade & \textbf{8.81/10} & \textbf{\$0.018} & 2.4s \\
\bottomrule
\end{tabular}
\caption{Quality-cost trade-offs across routing strategies. Hanzo cascade achieves 98.8\% of best-model quality at 43\% of the cost.}
\label{tab:routing_cost}
\end{table}