papers/hanzo-flow/sections/06-evaluation.tex

\section{Evaluation}

\label{sec:evaluation}

\subsection{AgentBench Results}

We evaluate Hanzo Flow on AgentBench~\cite{liu2023agentbench}, a benchmark suite for AI agents:

\begin{table}[H]
\centering
\small
\begin{tabular}{lcccc}
\toprule
\textbf{Framework} & \textbf{OS} & \textbf{DB} & \textbf{Web} & \textbf{Avg.} \\
\midrule
LangChain & 42.3 & 38.7 & 51.2 & 44.1 \\
CrewAI & 39.8 & 41.2 & 48.9 & 43.3 \\
AutoGen & 45.1 & 43.8 & 53.4 & 47.4 \\
\textbf{Hanzo Flow} & \textbf{51.7} & \textbf{48.2} & \textbf{58.3} & \textbf{52.7} \\
\bottomrule
\end{tabular}
\caption{AgentBench success rates (\%). Hanzo Flow benefits from adaptive execution and model fallback.}
\label{tab:agentbench}
\end{table}

\subsection{FlowBench-200}

We created FlowBench-200, a benchmark of 200 workflow construction tasks spanning seven pattern categories. Participants are given a task description and must build a working workflow. Scoring is based on correctness (automated test cases) and efficiency (number of nodes, execution cost).

\begin{table}[H]
\centering
\small
\begin{tabular}{lccc}
\toprule
\textbf{Approach} & \textbf{Correct} & \textbf{Time (min)} & \textbf{Nodes} \\
\midrule
LangChain (code) & 72\% & 34.2 & --- \\
LangGraph (code) & 78\% & 28.7 & --- \\
Flowise (visual) & 61\% & 19.4 & 12.3 \\
\textbf{Hanzo Flow} & \textbf{89\%} & \textbf{10.6} & \textbf{8.7} \\
\bottomrule
\end{tabular}
\caption{FlowBench-200 results. Visual workflow construction with Hanzo Flow is 3.2x faster than code-based approaches.}
\label{tab:flowbench}
\end{table}

\subsection{User Study}

We conducted a controlled user study with 48 developers (24 experienced with agent frameworks, 24 novices) building three workflows of increasing complexity:

\begin{table}[H]
\centering
\small
\begin{tabular}{lcccc}
\toprule
\textbf{Group} & \textbf{Tool} & \textbf{Task 1} & \textbf{Task 2} & \textbf{Task 3} \\
\midrule
Expert & Code & 8.2 min & 22.4 min & 41.7 min \\
Expert & Flow & 3.1 min & 8.3 min & 15.2 min \\
Novice & Code & 24.1 min & DNF & DNF \\
Novice & Flow & 5.4 min & 14.7 min & 28.3 min \\
\bottomrule
\end{tabular}
\caption{User study completion times. DNF = did not finish within 60 min.}
\label{tab:userstudy}
\end{table}

Key findings:
\begin{enumerate}[leftmargin=1.4em]
    \item Experts were 2.6x faster with Hanzo Flow than with code.
    \item Novices who could not complete tasks in code successfully completed them in Hanzo Flow.
    \item Visual debugging was cited by 92\% of participants as the most valuable feature.
    \item Static type checking caught 78\% of errors before execution.
\end{enumerate}

\subsection{Satisfaction Survey}

Post-study satisfaction (Likert scale 1--5):

\begin{table}[H]
\centering
\small
\begin{tabular}{lcc}
\toprule
\textbf{Dimension} & \textbf{Code} & \textbf{Hanzo Flow} \\
\midrule
Ease of use & 2.8 & 4.4 \\
Debugging experience & 2.3 & 4.6 \\
Confidence in correctness & 3.1 & 4.2 \\
Would recommend & 3.4 & 4.7 \\
\bottomrule
\end{tabular}
\caption{User satisfaction survey results.}
\label{tab:satisfaction}
\end{table}