papers/hanzo-unified-harness/sections/12-swe-bench-evaluation-setup.tex

\section{SWE-bench Evaluation Setup}

\label{app:swebench-setup}

Evaluation followed the SWE-bench Verified~\cite{jimenez2024swebench}
protocol:
\begin{itemize}[leftmargin=1.5em]
    \item \textbf{Dataset:} SWE-bench Verified, 500 GitHub issues from 12
          Python repositories.
    \item \textbf{Evaluation server:} Isolated Docker containers with the
          exact repository versions specified in each issue.
    \item \textbf{Success criterion:} All tests in the test suite pass after
          applying the agent's patch.
    \item \textbf{Compute:} Tier 1 (Terminal Pod) for patch generation;
          Tier 0 for issue analysis.
    \item \textbf{Model:} Zen-Code-480B (internal) via the UAH LLM gateway,
          with GPT-4o as a fallback for tool-use steps.
    \item \textbf{Maximum steps:} 50 agent steps per issue.
    \item \textbf{Time limit:} 20 minutes per issue.
    \item \textbf{Runs:} 3 independent runs; results are mean $\pm$ std.
          Standard deviation: $\pm 1.2\%$.
\end{itemize}