papers/earle-genetic-marketing/sections/06-evaluation.tex

\section{Evaluation}


\subsection{Experimental Setup}

We evaluated Earle across three verticals:

\begin{itemize}
\item \textbf{E-commerce}: Fashion retailer, conversion = purchase.
\item \textbf{SaaS}: B2B software, conversion = trial signup.
\item \textbf{Lead generation}: Financial services, conversion = form submission.
\end{itemize}

Each experiment ran for 30 days with \$50,000 budget. We compared:

\begin{enumerate}
\item \textbf{Human}: Campaigns designed by professional marketers.
\item \textbf{MAB}: Thompson Sampling over human-designed variants.
\item \textbf{Earle}: Genetic algorithm optimization.
\end{enumerate}

\subsection{Conversion Rate Results}

\begin{table}[h]
\centering
\caption{Conversion rate by method (\%)}
\label{tab:cvr}
\begin{tabular}{lrrr}
\hline
Vertical & Human & MAB & Earle \\
\hline
E-commerce & 2.1 & 2.4 & 2.9 \\
SaaS & 4.8 & 5.2 & 6.3 \\
Lead gen & 8.3 & 9.1 & 10.8 \\
\hline
Average & 5.1 & 5.6 & 6.7 \\
\hline
\end{tabular}
\end{table}

Earle achieves 31\% improvement over human baselines and 20\% over MAB.

\subsection{Convergence Speed}

We measured generations to reach 90\% of final fitness:

\begin{table}[h]
\centering
\caption{Generations to 90\% fitness}
\label{tab:convergence}
\begin{tabular}{lrr}
\hline
Vertical & MAB & Earle \\
\hline
E-commerce & 42 & 15 \\
SaaS & 38 & 14 \\
Lead gen & 35 & 12 \\
\hline
Average & 38.3 & 13.7 \\
\hline
\end{tabular}
\end{table}

Earle converges 2.8$\times$ faster than MAB by generating new variants rather than only selecting among fixed options.

\subsection{Discovered Insights}

Earle discovered non-obvious optimizations:

\begin{itemize}
\item \textbf{E-commerce}: Shorter headlines (< 25 characters) outperformed longer variants by 18\%.
\item \textbf{SaaS}: Targeting ``small business owners'' + ``recently changed jobs'' increased conversion 23\%.
\item \textbf{Lead gen}: Images with faces looking toward CTA button improved click-through 15\%.
\end{itemize}

These insights, discovered automatically, would require significant human experimentation to uncover.

\subsection{Cost Efficiency}

\begin{table}[h]
\centering
\caption{Cost per acquisition (\$)}
\label{tab:cpa}
\begin{tabular}{lrrr}
\hline
Vertical & Human & MAB & Earle \\
\hline
E-commerce & 42.50 & 38.20 & 31.80 \\
SaaS & 185.00 & 162.00 & 128.00 \\
Lead gen & 28.50 & 24.80 & 19.20 \\
\hline
\end{tabular}
\end{table}

Earle reduces CPA by 25--30\% across verticals.