papers/hanzo-aci/sections/08-evaluation.tex

\section{Evaluation}

\label{sec:evaluation}

\subsection{Testnet Configuration}

We deployed ACI on a testnet with:

\begin{itemize}[leftmargin=1.1em]
    \item \textbf{Validators}: 100 nodes across 10 geographic regions.
    \item \textbf{Workers}: 500 GPU nodes (mixed A100, H100, RTX 4090).
    \item \textbf{Clients}: 1,000 simulated agents submitting Poisson-distributed workloads.
    \item \textbf{Duration}: 30 days of continuous operation.
\end{itemize}

\subsection{Throughput and Latency}

\begin{table}[H]
\centering
\small
\begin{tabular}{lcc}
\toprule
\textbf{Metric} & \textbf{ACI} & \textbf{Ethereum L2} \\
\midrule
Job matches/second & 12,000 & 350 \\
Settlement TPS & 2,000 & 150 \\
Match latency (P50) & 87ms & 2,400ms \\
Match latency (P99) & 198ms & 8,700ms \\
Settlement finality & 2.0s & 12s \\
\bottomrule
\end{tabular}
\caption{Throughput and latency comparison.}
\label{tab:throughput}
\end{table}

\subsection{PoAI Verification Results}

\begin{table}[H]
\centering
\small
\begin{tabular}{lcc}
\toprule
\textbf{Metric} & \textbf{Value} \\
\midrule
Total jobs processed & 8,412,893 \\
Jobs sampled for verification & 25,238 (0.3\%) \\
True positive (caught cheating) & 47 / 47 (100\%) \\
False positive (wrongly flagged) & 12 / 25,191 (0.05\%) \\
Verification overhead & 0.31\% of total compute \\
Detection rate (injected faults) & 99.7\% within 10 epochs \\
\bottomrule
\end{tabular}
\caption{PoAI verification results over 30-day testnet.}
\label{tab:poai_results}
\end{table}

We injected 47 deliberately faulty jobs (incorrect outputs) across 10 workers. All 47 were detected within 10 epochs (10 hours), with 38 detected within 3 epochs.

\subsection{Gas Cost Comparison}

\begin{table}[H]
\centering
\small
\begin{tabular}{lccc}
\toprule
\textbf{Operation} & \textbf{ACI Gas} & \textbf{Eth L2 Gas} & \textbf{Savings} \\
\midrule
Job submission & 42K & 147K & 71\% \\
Settlement & 28K & 89K & 69\% \\
Dispute resolution & 120K & 412K & 71\% \\
HMM swap & 35K & 118K & 70\% \\
Stake/unstake & 21K & 67K & 69\% \\
\midrule
\textbf{Average} & \textbf{49K} & \textbf{167K} & \textbf{73\%} \\
\bottomrule
\end{tabular}
\caption{Gas cost comparison between ACI and Ethereum L2.}
\label{tab:gas}
\end{table}

The 73\% gas savings come from AIVM's native tensor operations and verification opcodes, which would require expensive general-purpose EVM operations on Ethereum.

\subsection{Stress Testing}

\begin{enumerate}[leftmargin=1.4em]
    \item \textbf{10x load surge}: Network maintained 99.8\% job success rate under 10x normal load for 1 hour. Fast-path latency increased to 420ms (from 87ms), but remained within SLA.
    \item \textbf{33\% validator failure}: With 33 of 100 validators offline, the network continued operating with degraded settlement throughput (1,200 TPS, down from 2,000).
    \item \textbf{Coordinated attack}: 10 colluding workers attempting to submit false attestations were detected within 4 epochs, with all 10 slashed.
\end{enumerate}