mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-29 07:34:31 +00:00
82 lines
2.3 KiB
TeX
82 lines
2.3 KiB
TeX
\section{Evaluation}
|
|
|
|
|
|
\subsection{Latency Benchmarks}
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\caption{Tool invocation latency (milliseconds).}
|
|
\label{tab:latency}
|
|
\begin{tabular}{lrrr}
|
|
\toprule
|
|
\textbf{Tool} & \textbf{P50} & \textbf{P95} & \textbf{P99} \\
|
|
\midrule
|
|
read\_file (1KB) & 8 & 15 & 23 \\
|
|
read\_file (1MB) & 45 & 89 & 142 \\
|
|
write\_file & 12 & 28 & 45 \\
|
|
shell (echo) & 89 & 156 & 234 \\
|
|
shell (npm test) & 2,340 & 5,670 & 12,400 \\
|
|
text\_search & 156 & 890 & 2,340 \\
|
|
ast\_search & 234 & 1,200 & 3,400 \\
|
|
vector\_search & 89 & 234 & 456 \\
|
|
ui\_navigate & 1,200 & 3,400 & 8,900 \\
|
|
ui\_click & 234 & 890 & 1,500 \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
Table~\ref{tab:latency} shows latency distributions across tool categories. File operations are fastest (median 8-45ms), while UI automation has highest latency due to browser communication overhead.
|
|
|
|
\subsection{Search Accuracy}
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\caption{Search accuracy on CodeSearchNet benchmark.}
|
|
\label{tab:search}
|
|
\begin{tabular}{lccc}
|
|
\toprule
|
|
\textbf{Strategy} & \textbf{Precision} & \textbf{Recall} & \textbf{F1} \\
|
|
\midrule
|
|
Text only & 0.78 & 0.92 & 0.84 \\
|
|
AST only & 0.91 & 0.67 & 0.77 \\
|
|
Vector only & 0.82 & 0.79 & 0.80 \\
|
|
\midrule
|
|
\textbf{Unified} & \textbf{0.94} & \textbf{0.89} & \textbf{0.91} \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
Table~\ref{tab:search} compares search strategies on CodeSearchNet~\cite{codesearchnet2019}. The unified approach achieves 91\% F1, outperforming any single strategy.
|
|
|
|
\subsection{Throughput}
|
|
|
|
Under load testing with 100 concurrent clients:
|
|
\begin{itemize}
|
|
\item File operations: 8,900 req/s
|
|
\item Shell commands: 450 req/s (process spawn limited)
|
|
\item Search: 1,200 req/s
|
|
\item UI automation: 45 req/s (browser limited)
|
|
\end{itemize}
|
|
|
|
\subsection{Comparison with Alternatives}
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\caption{Feature comparison with MCP implementations.}
|
|
\label{tab:compare}
|
|
\begin{tabular}{lccc}
|
|
\toprule
|
|
\textbf{Feature} & \textbf{Ours} & \textbf{Ref SDK} & \textbf{Claude} \\
|
|
\midrule
|
|
Built-in tools & 25+ & 0 & 8 \\
|
|
Custom tools & Yes & Yes & No \\
|
|
UI automation & Yes & No & No \\
|
|
Search engine & Yes & No & No \\
|
|
Multi-transport & Yes & Stdio & Stdio \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
Table~\ref{tab:compare} shows our implementation provides the most comprehensive feature set while maintaining extensibility.
|
|
|