mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-22 17:20:17 +00:00
zeekay
ca478225e8
- Agent GRPO: training-free continuous learning for tool-using agents - Federated Agents: federated agent intelligence - Harness Hacking: agents that safely self-modify their harness - Self-Improving Agents: telemetry-driven self-improvement - Unified Harness: vertically integrated agent harness - Zen creative AI: 3D, director, foley, medical, musician, reasoning, technical, translator, video, voice-clone, voyager, world models - papers.hanzo.ai Next.js site
51 lines
2.5 KiB
TeX
51 lines
2.5 KiB
TeX
% SWE-bench Evaluation section
|
|
\section{SWE-bench Evaluation Protocol}
|
|
\subsection{Benchmark Overview}
|
|
SWE-bench~\cite{jimenez2024swebench} provides 2,294 real-world GitHub issues from 12 popular Python repositories. SWE-bench Verified~\cite{openai2024swebench-verified} is a high-quality subset of 500 issues with human-verified patches. Each task requires:
|
|
\begin{itemize}[leftmargin=1.1em]
|
|
\item Understanding the issue description
|
|
\item Locating relevant code files
|
|
\item Implementing a correct fix
|
|
\item Passing all unit tests
|
|
\end{itemize}
|
|
|
|
\subsection{Hanzo Dev Workflow}
|
|
Our evaluation protocol follows this reproducible workflow:
|
|
\begin{enumerate}
|
|
\item \textbf{Setup:} Clone repository, checkout commit hash, create isolated environment
|
|
\item \textbf{Planning:} Agent analyzes issue and generates implementation plan
|
|
\item \textbf{TF-GRPO Rollouts:} Generate \(G=4\) candidate solutions with PoE decoding
|
|
\item \textbf{Test Execution:} Run unit tests, collect feedback as reward signal \(r^{(i)}\)
|
|
\item \textbf{Advantage Extraction:} Compute group-relative advantages, extract semantic priors
|
|
\item \textbf{Refinement:} Use updated priors for next iteration (max 3 iterations)
|
|
\item \textbf{Verification:} Final patch must pass all tests without manual intervention
|
|
\end{enumerate}
|
|
|
|
\subsection{Evaluation Metrics}
|
|
\begin{itemize}[leftmargin=1.1em]
|
|
\item \textbf{Resolved Rate:} Percentage of issues fully resolved (all tests pass)
|
|
\item \textbf{Patch Similarity:} Edit distance to ground-truth patch
|
|
\item \textbf{Iteration Count:} Average number of TF-GRPO iterations needed
|
|
\item \textbf{Test Coverage:} Percentage of test files modified/added
|
|
\end{itemize}
|
|
|
|
\subsection{Reproducibility Requirements}
|
|
All runs are:
|
|
\begin{itemize}[leftmargin=1.1em]
|
|
\item \textbf{Verifier-replayable:} Complete logs + environment snapshots
|
|
\item \textbf{Deterministic:} Fixed seeds for sampling (where applicable)
|
|
\item \textbf{Containerized:} Docker images with exact dependency versions
|
|
\item \textbf{Auditable:} All LLM API calls logged with timestamps
|
|
\end{itemize}
|
|
|
|
\subsection{Baseline Comparisons}
|
|
We compare against:
|
|
\begin{itemize}[leftmargin=1.1em]
|
|
\item GPT-4 with standard prompting
|
|
\item Claude 3.5 Sonnet with agentic workflow
|
|
\item Open-source agents (AutoCodeRover, SWE-agent)
|
|
\item Fine-tuned code models (CodeLlama-Instruct, Phind-CodeLlama)
|
|
\end{itemize}
|
|
|
|
Target performance: \textbf{15--25\% resolved rate} on SWE-bench Verified (competitive with current SOTA).
|