papers/hanzo-operative/sections/08-evaluation.tex

\section{Evaluation}

\label{sec:evaluation}

\subsection{Benchmarks}

We evaluate Operative on three established benchmarks:

\begin{enumerate}[leftmargin=*]
    \item \textbf{OSWorld}~\cite{xie2024osworld}: 369 real-world desktop tasks spanning 9 applications (Ubuntu environment).
    \item \textbf{WebArena}~\cite{zhou2024webarena}: 812 web-based tasks across 4 self-hosted web applications.
    \item \textbf{VisualWebBench}~\cite{liu2024visualwebbench}: 1,500 web understanding and interaction tasks with fine-grained annotations.
\end{enumerate}

\subsection{Task Completion Results}

\begin{table}[t]
\centering
\caption{Task completion rates (\%) across benchmarks}
\label{tab:results}
\begin{tabular}{@{}lccc@{}}
\toprule
\textbf{System} & \textbf{OSWorld} & \textbf{WebArena} & \textbf{VWB} \\
\midrule
GPT-4V (direct) & 12.2 & 14.4 & 18.7 \\
Claude 3.5 CU & 22.0 & 35.1 & 41.2 \\
SeeAct & 31.6 & --- & 38.4 \\
CogAgent & --- & 29.8 & 35.9 \\
WebVoyager & --- & 42.7 & --- \\
\midrule
Operative (Claude) & \textbf{38.1} & \textbf{54.3} & \textbf{52.8} \\
Operative (GPT-4o) & 34.7 & 48.9 & 49.1 \\
Operative (Gemini) & 31.2 & 44.2 & 46.7 \\
\bottomrule
\end{tabular}
\end{table}

Operative with Claude achieves state-of-the-art results across all three benchmarks. The improvement over raw Claude computer use (22.0\% $\rightarrow$ 38.1\% on OSWorld) demonstrates the value of the structured perception pipeline and hierarchical planning.

\subsection{Ablation Study}

\begin{table}[t]
\centering
\caption{Ablation study on OSWorld (Claude backend)}
\label{tab:ablation}
\begin{tabular}{@{}lr@{}}
\toprule
\textbf{Configuration} & \textbf{Completion (\%)} \\
\midrule
Full system & 38.1 \\
$-$ Accessibility tree fusion & 31.4 \\
$-$ Hierarchical planning & 29.7 \\
$-$ Error recovery & 33.2 \\
$-$ Context summarization & 35.6 \\
$-$ Pre-condition validation & 36.8 \\
Screenshot-only (no fusion) & 24.3 \\
\bottomrule
\end{tabular}
\end{table}

The hierarchical planning architecture contributes the largest improvement ($+$8.4 points), followed by accessibility tree fusion ($+$6.7 points). Error recovery accounts for $+$4.9 points, confirming the importance of graceful failure handling.

\subsection{Perception Accuracy}

\begin{table}[t]
\centering
\caption{Element detection accuracy by source}
\label{tab:perception}
\begin{tabular}{@{}lccc@{}}
\toprule
\textbf{Source} & \textbf{Precision} & \textbf{Recall} & \textbf{F1} \\
\midrule
Visual only & 89.1 & 72.3 & 79.8 \\
OCR only & 91.4 & 58.2 & 71.1 \\
A11y only & 96.2 & 81.7 & 88.3 \\
Fused (all) & 94.7 & 91.2 & 92.9 \\
\bottomrule
\end{tabular}
\end{table}

The fused pipeline achieves 92.9\% F1, significantly outperforming any individual source. The accessibility tree provides the highest precision but misses visual-only elements (icons without labels, decorative elements that serve as click targets). Visual detection catches these at the cost of some false positives, while OCR fills in text content missing from accessibility labels.

\subsection{Latency Analysis}

\begin{table}[t]
\centering
\caption{Per-step latency breakdown (ms)}
\label{tab:latency}
\begin{tabular}{@{}lrrr@{}}
\toprule
\textbf{Component} & \textbf{p50} & \textbf{p95} & \textbf{p99} \\
\midrule
Screen capture & 8 & 14 & 23 \\
Visual detection & 23 & 31 & 42 \\
OCR & 67 & 112 & 189 \\
A11y tree query & 12 & 28 & 54 \\
Element fusion & 3 & 7 & 12 \\
LLM planning & 1,840 & 4,200 & 8,100 \\
Action execution & 45 & 180 & 520 \\
Post-verification & 31 & 67 & 124 \\
\midrule
\textbf{Total per step} & \textbf{2,029} & \textbf{4,639} & \textbf{9,064} \\
\bottomrule
\end{tabular}
\end{table}

LLM planning dominates the per-step latency at 91\% of total time. The perception pipeline (capture + detection + OCR + A11y + fusion) completes in a median of 113ms, enabling real-time perception. With local models (e.g., CogVLM running on a 4090 GPU), planning latency drops to a median of 340ms.

\subsection{Safety Boundary Effectiveness}

We evaluate safety boundaries by running Operative on 100 adversarial tasks designed to trick agents into harmful actions (deleting files, exfiltrating data, making purchases):

\begin{itemize}[leftmargin=*]
    \item \textbf{Blocked by policy}: 94 of 100 harmful actions were blocked by the safety boundary system.
    \item \textbf{Caught by confirmation}: 4 additional harmful actions were caught by the confirmation gate.
    \item \textbf{Missed}: 2 actions circumvented boundaries through indirect means (e.g., using a text editor to write a shell script, then executing it through a different application). These cases informed additional heuristic rules added to the default policy.
\end{itemize}