papers/hanzo-chat/sections/06-evaluation.tex

\section{Evaluation}

\label{sec:evaluation}

\subsection{Benchmarks}

We evaluate Hanzo Chat on four benchmarks:

\begin{enumerate}[leftmargin=1.4em]
    \item \textbf{MT-Bench}~\cite{zheng2023judging}: 80 multi-turn questions across 8 categories, judged by GPT-4.
    \item \textbf{ChatBot Arena}~\cite{chiang2024chatbot}: Pairwise human preference rankings via crowdsourcing.
    \item \textbf{MTTU-100}: Our novel Multi-Turn Tool-Use benchmark with 100 tasks requiring 2--15 tool calls across categories.
    \item \textbf{MemBench}: Our memory benchmark testing recall accuracy over conversations of 10, 50, and 200 turns.
\end{enumerate}

\subsection{MT-Bench Results}

\begin{table}[H]
\centering
\small
\begin{tabular}{lcccc}
\toprule
\textbf{System} & \textbf{Turn 1} & \textbf{Turn 2} & \textbf{Avg.} \\
\midrule
GPT-4 (single) & 8.96 & 9.03 & 8.99 \\
Claude 3.5 (single) & 8.81 & 8.97 & 8.89 \\
Gemini Ultra (single) & 8.72 & 8.45 & 8.58 \\
Hanzo Chat (routed) & \textbf{9.12} & \textbf{9.18} & \textbf{9.15} \\
Hanzo Chat (w/ memory) & 9.08 & \textbf{9.24} & 9.16 \\
\bottomrule
\end{tabular}
\caption{MT-Bench scores. Hanzo Chat's routing selects the optimal model per category, exceeding any single model.}
\label{tab:mtbench}
\end{table}

The improvement on Turn 2 with memory (+0.06) demonstrates that retrieved context from Turn 1 improves second-turn responses.

\subsection{MTTU-100 Benchmark}

We designed MTTU-100 to evaluate multi-turn tool use in realistic scenarios. Each task specifies a goal, available tools, and expected outputs. Scoring is automated via deterministic verification of outputs.

\begin{table}[H]
\centering
\small
\begin{tabular}{lccc}
\toprule
\textbf{System} & \textbf{Success} & \textbf{Tool Acc.} & \textbf{Turns} \\
\midrule
GPT-4 + plugins & 62\% & 78\% & 4.2 \\
Claude + MCP & 71\% & 84\% & 3.8 \\
Gemini + extensions & 58\% & 73\% & 5.1 \\
Hanzo Chat & \textbf{83\%} & \textbf{91\%} & \textbf{3.4} \\
\bottomrule
\end{tabular}
\caption{MTTU-100 results. Hanzo Chat achieves higher success rates with fewer turns due to dynamic tool selection and model routing.}
\label{tab:mttu}
\end{table}

\subsection{MemBench Results}

MemBench tests the ability to recall information introduced earlier in a conversation. We measure precision and recall of factual retrieval at different conversation depths.

\begin{table}[H]
\centering
\small
\begin{tabular}{lcccc}
\toprule
\textbf{System} & \textbf{10 turns} & \textbf{50 turns} & \textbf{200 turns} \\
\midrule
GPT-4 (128K) & 98.2\% & 87.4\% & 42.1\% \\
Claude (200K) & 98.7\% & 91.2\% & 58.3\% \\
Hanzo (no mem.) & 97.9\% & 85.1\% & 39.8\% \\
Hanzo (w/ mem.) & \textbf{99.1\%} & \textbf{96.8\%} & \textbf{93.2\%} \\
\bottomrule
\end{tabular}
\caption{MemBench factual recall accuracy by conversation length. Memory-augmented Hanzo Chat maintains high recall even at 200 turns.}
\label{tab:membench}
\end{table}

The dramatic improvement at 200 turns (93.2\% vs. 58.3\% for the best single-model baseline) demonstrates the value of hierarchical memory with semantic retrieval.

\subsection{Ablation Studies}

\begin{table}[H]
\centering
\small
\begin{tabular}{lccc}
\toprule
\textbf{Configuration} & \textbf{MT-Bench} & \textbf{MTTU} & \textbf{Cost} \\
\midrule
Full system & 9.15 & 83\% & \$0.018 \\
$-$ routing (fixed model) & 8.89 & 71\% & \$0.042 \\
$-$ memory & 9.08 & 79\% & \$0.017 \\
$-$ tool selection & 9.12 & 74\% & \$0.022 \\
$-$ cascading & 9.15 & 83\% & \$0.025 \\
$-$ all (single model, no mem) & 8.81 & 68\% & \$0.042 \\
\bottomrule
\end{tabular}
\caption{Ablation study showing contribution of each component.}
\label{tab:ablation}
\end{table}