papers/hanzo-llm-gateway/sections/10-evaluation.tex

\section{Evaluation}

\label{sec:evaluation}

\subsection{Experimental Setup}

We evaluate the gateway using production traffic from the Hanzo AI platform over a 30-day period (January 2026). The workload comprises:

\begin{itemize}[leftmargin=*]
    \item 2.8 billion tokens processed (1.1B input, 1.7B output)
    \item 4.2 million requests across 23 distinct models
    \item 127 provider endpoints across 14 providers
    \item Peak throughput: 847 requests/second
\end{itemize}

\subsection{Latency}

\begin{table}[t]
\centering
\caption{End-to-end latency overhead (milliseconds)}
\label{tab:latency}
\begin{tabular}{@{}lrrrr@{}}
\toprule
\textbf{Component} & \textbf{p50} & \textbf{p95} & \textbf{p99} & \textbf{Max} \\
\midrule
Auth + Rate Limit & 0.3 & 0.8 & 1.2 & 4.1 \\
Routing Decision & 0.1 & 0.4 & 0.7 & 2.3 \\
Cache Lookup & 0.4 & 1.1 & 2.8 & 8.7 \\
Transform (req) & 0.2 & 0.6 & 1.1 & 3.4 \\
Transform (resp) & 0.3 & 0.7 & 1.3 & 4.2 \\
\midrule
\textbf{Total Overhead} & \textbf{1.3} & \textbf{3.6} & \textbf{7.1} & \textbf{22.7} \\
\bottomrule
\end{tabular}
\end{table}

The gateway adds a median of 1.3ms of overhead per request, which is negligible compared to typical LLM inference latencies of 500ms--30s. The p99 overhead of 7.1ms is dominated by cache lookup in the Redis tier.

Intelligent routing reduces median end-to-end latency by 23\% compared to fixed-provider routing by directing requests to the fastest available provider. The improvement is most pronounced during provider degradation events, where failover avoids multi-second timeouts.

\subsection{Cache Performance}

Over the evaluation period, the semantic cache achieved:

\begin{itemize}[leftmargin=*]
    \item \textbf{Hit rate}: 34.2\% of all requests served from cache.
    \item \textbf{Precision}: 97.2\% of cache hits returned responses equivalent to fresh provider responses (validated on a 10,000-request sample).
    \item \textbf{Latency reduction}: Cache hits served in median 1.8ms vs. 1,240ms for cache misses (690$\times$ speedup).
    \item \textbf{Cost savings}: Cache hits eliminated \$47,200 in provider costs over the evaluation period.
\end{itemize}

Cache hit rates vary significantly by use case: embedding requests achieve 71\% hit rate (high repetition), while creative generation requests achieve only 8\% (high temperature, unique prompts).

\subsection{Cost Optimization}

\begin{table}[t]
\centering
\caption{Monthly cost comparison (USD)}
\label{tab:cost}
\begin{tabular}{@{}lrrr@{}}
\toprule
\textbf{Strategy} & \textbf{Cost} & \textbf{Savings} & \textbf{Qual.} \\
\midrule
Single provider (OpenAI) & 183,400 & --- & 1.00 \\
Manual multi-provider & 152,100 & 17\% & 0.98 \\
Gateway: cost-optimized & 126,500 & 31\% & 0.96 \\
Gateway: quality-first & 171,200 & 7\% & 1.02 \\
Gateway: balanced & 141,300 & 23\% & 0.99 \\
\bottomrule
\end{tabular}
\end{table}

The cost-optimized strategy reduces costs by 31\% with a 4\% quality degradation as measured by a composite benchmark of MMLU, HumanEval, and MT-Bench scores. The balanced strategy achieves 23\% cost reduction with negligible quality impact.

\subsection{Availability}

During the 30-day evaluation, the gateway achieved 99.97\% availability (8 minutes of cumulative downtime across two rolling restart events). Provider-level availability varied from 99.2\% (worst single provider) to 99.99\% (best). The gateway's multi-provider failover elevated effective availability from the worst single-provider level to the aggregate level.

\begin{table}[t]
\centering
\caption{Provider failover events during evaluation}
\label{tab:failover}
\begin{tabular}{@{}lrrr@{}}
\toprule
\textbf{Event Type} & \textbf{Count} & \textbf{Avg Duration} & \textbf{Requests Saved} \\
\midrule
Rate limit & 1,247 & 12s & 34,200 \\
5xx error & 89 & 4.2min & 12,100 \\
Timeout & 312 & 8s & 8,900 \\
Full outage & 3 & 47min & 28,400 \\
\bottomrule
\end{tabular}
\end{table}

\subsection{Throughput}

Under load testing with synthetic traffic, the gateway sustains:

\begin{itemize}[leftmargin=*]
    \item 2,400 requests/second with non-streaming responses (single node, 8 Uvicorn workers).
    \item 1,800 concurrent streaming connections with sub-5ms chunk relay latency.
    \item Linear horizontal scaling up to 12 nodes (tested), with shared Redis state for cache and rate limits.
\end{itemize}