papers/crowdstart-commerce/sections/05-evaluation.tex

\section{Evaluation}


\subsection{Experimental Setup}

We evaluate Crowdstart under production-representative workloads:

\begin{itemize}
\item 70\% product catalog reads
\item 20\% cart operations
\item 8\% search queries
\item 2\% checkout/payment flows
\end{itemize}

Load generation uses Locust with geographically distributed workers simulating realistic user behavior.

\subsection{Latency Results}

Table~\ref{tab:latency} summarizes latency measurements across percentiles.

\begin{table}[h]
\centering
\caption{Request latency by operation type (ms)}
\label{tab:latency}
\begin{tabular}{lrrr}
\hline
Operation & p50 & p95 & p99 \\
\hline
Product Read & 12 & 28 & 45 \\
Cart Update & 18 & 42 & 78 \\
Search & 35 & 85 & 142 \\
Checkout & 245 & 520 & 890 \\
\hline
\end{tabular}
\end{table}

Checkout latency is dominated by external payment processing (Stripe API calls average 180ms).

\subsection{Scaling Behavior}

Figure~\ref{fig:scaling} illustrates automatic scaling under a synthetic load ramp from 100 to 10,000 requests per second.

\begin{figure}[h]
\centering
\fbox{\parbox{0.9\linewidth}{
\centering
\textit{[Instance count and latency vs. request rate]}\\[1em]
Instances scale from 2 to 47 as load increases.\\
p99 latency remains below 100ms until saturation.\\
Saturation occurs at approximately 8,500 RPS.
}}
\caption{Automatic scaling behavior}
\label{fig:scaling}
\end{figure}

App Engine's automatic scaling responds to load increases within 15 seconds, adding instances as needed. Cold start latency (first request to a new instance) averages 850ms but affects fewer than 0.1\% of requests under stable load.

\subsection{Cost Analysis}

Cloud-native architecture enables pay-per-use pricing. For a merchant processing 100,000 orders per month:

\begin{table}[h]
\centering
\caption{Monthly infrastructure cost breakdown}
\label{tab:cost}
\begin{tabular}{lr}
\hline
Component & Cost (USD) \\
\hline
App Engine instances & \$145 \\
Cloud Datastore & \$78 \\
Cloud Storage & \$12 \\
Network egress & \$23 \\
\hline
Total & \$258 \\
\hline
\end{tabular}
\end{table}

Comparable self-hosted infrastructure would require minimum \$800/month for equivalent reliability (multi-AZ deployment, managed database, load balancing).