papers/hanzo-ml-framework/sections/06-evaluation.tex

\section{Evaluation}


\subsection{Inference Benchmarks}

\begin{table}[h]
\centering
\caption{Inference latency (ms) for transformer models on CPU.}
\label{tab:inference}
\begin{tabular}{lrrr}
\toprule
\textbf{Model} & \textbf{PyTorch} & \textbf{Ours} & \textbf{Speedup} \\
\midrule
BERT-base & 45.2 & 14.1 & 3.2x \\
GPT-2 (117M) & 89.4 & 31.2 & 2.9x \\
Llama-7B (Q4) & 156.0 & 52.0 & 3.0x \\
Whisper-tiny & 234.0 & 78.0 & 3.0x \\
\bottomrule
\end{tabular}
\end{table}

Table~\ref{tab:inference} shows CPU inference benchmarks on an M2 MacBook Pro. Our framework achieves 2.9-3.2x speedup through SIMD optimization and reduced runtime overhead.

\subsection{Binary Size}

\begin{table}[h]
\centering
\caption{Binary size comparison (MB).}
\label{tab:size}
\begin{tabular}{lrr}
\toprule
\textbf{Framework} & \textbf{Core} & \textbf{With Model} \\
\midrule
PyTorch & 850 & 1,200 \\
ONNX Runtime & 45 & 95 \\
\textbf{Hanzo ML} & \textbf{8} & \textbf{58} \\
\bottomrule
\end{tabular}
\end{table}

Table~\ref{tab:size} demonstrates binary size advantages. Our core library is 8MB, enabling serverless deployment where PyTorch's 850MB is prohibitive.

\subsection{FHE Accuracy}

\begin{table}[h]
\centering
\caption{Accuracy impact of polynomial approximations.}
\label{tab:fhe_acc}
\begin{tabular}{lccc}
\toprule
\textbf{Model/Task} & \textbf{Standard} & \textbf{FHE-Compat} & \textbf{Drop} \\
\midrule
BERT/GLUE & 84.2\% & 82.1\% & 2.1\% \\
ResNet-18/ImageNet & 69.8\% & 67.4\% & 2.4\% \\
GPT-2/WikiText & 24.5 PPL & 26.1 PPL & 6.5\% \\
\bottomrule
\end{tabular}
\end{table}

Table~\ref{tab:fhe_acc} shows accuracy degradation from polynomial approximations. The 2-6\% drop is acceptable for privacy-sensitive applications.

\subsection{Quantization Accuracy}

\begin{table}[h]
\centering
\caption{Llama-7B perplexity by quantization format.}
\label{tab:quant}
\begin{tabular}{lcc}
\toprule
\textbf{Format} & \textbf{PPL} & \textbf{Size (GB)} \\
\midrule
FP16 & 5.68 & 13.0 \\
Q8\_0 & 5.70 & 6.7 \\
Q6\_K & 5.72 & 5.5 \\
Q5\_K & 5.78 & 4.6 \\
Q4\_K & 5.91 & 3.8 \\
Q4\_0 & 6.12 & 3.6 \\
\bottomrule
\end{tabular}
\end{table}

Table~\ref{tab:quant} shows quantization trade-offs. Q4\_K provides excellent balance with only 0.23 perplexity increase from FP16 at 3.4x compression.