mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-29 07:34:31 +00:00
112 lines
2.7 KiB
TeX
112 lines
2.7 KiB
TeX
\section{Evaluation}
|
|
|
|
|
|
\subsection{Experimental Setup}
|
|
|
|
We evaluated Hanzo across three production deployments:
|
|
|
|
\begin{itemize}
|
|
\item \textbf{Fashion retailer}: 2M monthly active users, 50K SKUs.
|
|
\item \textbf{Electronics store}: 500K MAU, 10K SKUs.
|
|
\item \textbf{Marketplace}: 1M MAU, 200K SKUs.
|
|
\end{itemize}
|
|
|
|
Each deployment ran for 6 months. We measured against pre-Hanzo baselines.
|
|
|
|
\subsection{Recommendation Performance}
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\caption{Recommendation metrics improvement}
|
|
\label{tab:reco}
|
|
\begin{tabular}{lrrr}
|
|
\hline
|
|
Metric & Fashion & Electronics & Marketplace \\
|
|
\hline
|
|
CTR & +45\% & +38\% & +52\% \\
|
|
Conversion & +22\% & +18\% & +31\% \\
|
|
AOV & +28\% & +24\% & +32\% \\
|
|
\hline
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
Recommendations drive significant revenue improvement. Average order value increases as recommendations surface complementary products.
|
|
|
|
\subsection{Search Quality}
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\caption{Search metrics improvement}
|
|
\label{tab:search}
|
|
\begin{tabular}{lrrr}
|
|
\hline
|
|
Metric & Fashion & Electronics & Marketplace \\
|
|
\hline
|
|
NDCG@10 & +35\% & +28\% & +42\% \\
|
|
Zero-result rate & -62\% & -55\% & -71\% \\
|
|
Search-to-purchase & +24\% & +19\% & +28\% \\
|
|
\hline
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
Intelligent ranking improves search relevance. Query understanding reduces zero-result searches through spelling correction and synonym expansion.
|
|
|
|
\subsection{Fraud Detection}
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\caption{Fraud metrics}
|
|
\label{tab:fraud}
|
|
\begin{tabular}{lrrr}
|
|
\hline
|
|
Metric & Fashion & Electronics & Marketplace \\
|
|
\hline
|
|
Fraud loss reduction & -42\% & -51\% & -45\% \\
|
|
False positive rate & -35\% & -28\% & -38\% \\
|
|
Manual review rate & -55\% & -48\% & -52\% \\
|
|
\hline
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
Improved fraud detection reduces both losses and operational burden. Lower false positive rates mean fewer legitimate customers face friction.
|
|
|
|
\subsection{Inventory Optimization}
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\caption{Inventory metrics improvement}
|
|
\label{tab:inventory}
|
|
\begin{tabular}{lrrr}
|
|
\hline
|
|
Metric & Fashion & Electronics & Marketplace \\
|
|
\hline
|
|
Stockout rate & -38\% & -42\% & -35\% \\
|
|
Inventory turnover & +18\% & +22\% & +15\% \\
|
|
Markdown rate & -25\% & -18\% & -28\% \\
|
|
\hline
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
Better demand forecasting reduces both stockouts (lost sales) and overstock (markdowns).
|
|
|
|
\subsection{System Performance}
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\caption{Production latency (ms)}
|
|
\label{tab:prod_latency}
|
|
\begin{tabular}{lrr}
|
|
\hline
|
|
Component & p50 & p99 \\
|
|
\hline
|
|
Feature retrieval & 2 & 8 \\
|
|
Recommendation inference & 12 & 35 \\
|
|
Search re-ranking & 18 & 45 \\
|
|
Fraud scoring & 45 & 180 \\
|
|
\hline
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
All components meet latency requirements in production.
|
|
|