papers/hanzo-search/sections/07-evaluation.tex

\section{Evaluation}

\label{sec:evaluation}

\subsection{Answer Quality (GenSearch-500)}

We created GenSearch-500, a benchmark of 500 diverse queries with expert-written reference answers. Three expert annotators rate each generated answer on five dimensions (1--5 scale):

\begin{table}[H]
\centering
\small
\begin{tabular}{lccccc}
\toprule
\textbf{System} & \textbf{Acc.} & \textbf{Comp.} & \textbf{Clar.} & \textbf{Cite.} & \textbf{Avg.} \\
\midrule
Perplexity & 4.1 & 3.8 & 4.2 & 3.6 & 3.93 \\
Bing Copilot & 3.9 & 3.9 & 4.0 & 3.8 & 3.90 \\
Google SGE & 4.0 & 3.7 & 4.1 & 3.5 & 3.83 \\
You.com & 3.8 & 3.6 & 3.9 & 3.4 & 3.68 \\
\textbf{Hanzo Search} & \textbf{4.3} & \textbf{4.2} & \textbf{4.3} & \textbf{4.4} & \textbf{4.30} \\
\bottomrule
\end{tabular}
\caption{GenSearch-500 results. Acc = Accuracy, Comp = Completeness, Clar = Clarity, Cite = Citation quality.}
\label{tab:gensearch}
\end{table}

\subsection{Natural Questions}

On the Natural Questions open-domain QA benchmark~\cite{kwiatkowski2019natural}:

\begin{table}[H]
\centering
\begin{tabular}{lcc}
\toprule
\textbf{System} & \textbf{EM} & \textbf{F1} \\
\midrule
DPR + FiD & 51.4 & 56.7 \\
Atlas & 64.0 & --- \\
REPLUG & 55.8 & 62.1 \\
Self-RAG & 54.9 & 60.3 \\
\textbf{Hanzo Search} & \textbf{65.2} & \textbf{71.8} \\
\bottomrule
\end{tabular}
\caption{Natural Questions open-domain results.}
\label{tab:nq}
\end{table}

\subsection{Latency Analysis}

End-to-end latency breakdown for a typical query:

\begin{table}[H]
\centering
\begin{tabular}{lc}
\toprule
\textbf{Stage} & \textbf{Latency (ms)} \\
\midrule
Query understanding & 35 \\
BM25 retrieval & 28 \\
Dense retrieval & 42 \\
Fusion + top-100 & 8 \\
Cross-encoder reranking & 45 \\
Passage extraction & 12 \\
LLM generation (TTFT) & 380 \\
Citation verification & 55 \\
UI rendering (client) & 120 \\
\midrule
\textbf{Total (to first token)} & \textbf{550} \\
\textbf{Total (complete answer)} & \textbf{2,100} \\
\bottomrule
\end{tabular}
\caption{Latency breakdown. TTFT = time to first token.}
\label{tab:latency}
\end{table}