mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-30 14:58:42 +00:00
zeekayandHanzo Dev
1d1e7be5ab
Both report benchmarks that already live in hanzoai/cloud under bench/, so the numbers have a script behind them rather than a spreadsheet. hanzo-dormant-agents takes apart the estimate that costs an agent as a process. State, execution and isolation are separable and differ by five orders of magnitude: a dormant agent is 477 bytes on disk and resumes in 0.034 ms, a parked loop is 601 bytes of heap, and the isolation boundary — the expensive one — is only owed by an agent actually running. The paper argues the container is worth its price where it is needed and says so; the claim is about which agents are charged for it. hanzo-multi-hop-retrieval is a negative result. On LoCoMo a dense retriever finds every required turn for 78.2% of single-hop questions and 22.7% of multi-hop ones, but finds at least one required turn about 80% of the time in both — so it understands the questions and cannot reach all of the answer. Four retrieval policies over one fixed index move that by at most 0.7 points and two make it worse. A diagnostic says why it was never going to work: 95.4% of multi-hop evidence spans sessions and the median worst-ranked gold turn sits at rank 67, so the missing signal is linkage rather than similarity. Publishing the dead end is the useful part. Each paper carries results-data.tex, generated from the runs, and prints XXX in place of any constant that file does not supply — a number cannot drift from its measurement without the drift showing on the page. Both compile under the CI gate's flags, -Werror included. INDEX.md is regenerated by make index, which also picks up the papers added since it was last written and the header its current generator emits. Co-authored-by: Hanzo Dev <[email protected]>
49 lines
2.3 KiB
TeX
49 lines
2.3 KiB
TeX
\section{The baseline, and the shape of the failure}
|
|
\label{sec:baseline}
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\begin{tabular}{lrr}
|
|
\toprule
|
|
& \textbf{single-hop (\rQsingle{}\,Q)} & \textbf{multi-hop (\rQmulti{}\,Q)} \\
|
|
\midrule
|
|
recall@5\ \ \textbf{all} & \rSallFive{}\% & \rMallFive{}\% \\
|
|
recall@10\ \textbf{all} & \rSallTen{}\% & \rMallTen{}\% \\
|
|
recall@20\ \textbf{all} & \textbf{\rSallTwenty{}\%} & \textbf{\rMallTwenty{}\%} \\
|
|
\midrule
|
|
recall@5\ \ \textbf{any} & \rSanyFive{}\% & \rManyFive{}\% \\
|
|
recall@10\ \textbf{any} & \rSanyTen{}\% & \rManyTen{}\% \\
|
|
recall@20\ \textbf{any} & \rSanyTwenty{}\% & \rManyTwenty{}\% \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\caption{Dense retrieval on LoCoMo, \texttt{zen-embedding-0.6b}, brute-force
|
|
cosine \meas{}. The bottom row is the one that reframes the problem.}
|
|
\label{tab:baseline}
|
|
\end{table}
|
|
|
|
Read the table by column and it says the retriever is bad at multi-hop
|
|
questions: \rSallTwenty{}\% against \rMallTwenty{}\% at $k{=}20$ is a collapse of
|
|
more than three to one.
|
|
|
|
Read it by row and it says something different and more useful. On the
|
|
\textbf{any} rows the two columns converge: \rSanyTwenty{}\% and
|
|
\rManyTwenty{}\% at $k{=}20$ are within a point of each other. Whatever the
|
|
retriever is doing, it does it equally well on both categories --- it puts a
|
|
relevant turn in the top 20 about four times in five, regardless of how many
|
|
turns the question actually needs.
|
|
|
|
So the failure is not comprehension and not embedding quality. Those would
|
|
depress \textbf{any} as well, and \textbf{any} is not depressed. The failure is
|
|
specifically in \emph{completeness}: the gap between \rManyTwenty{}\% and
|
|
\rMallTwenty{}\% --- some 57 percentage points --- is questions where the
|
|
retriever found part of the evidence and missed the remainder.
|
|
|
|
\paragraph{Why more $k$ is not obviously the answer.}
|
|
The single-hop column improves steadily with $k$ (\rSallFive{} $\to$
|
|
\rSallTen{} $\to$ \rSallTwenty{}), which is what a retriever that is
|
|
merely under-budgeted looks like. The multi-hop \textbf{all} column improves too
|
|
(\rMallFive{} $\to$ \rMallTen{} $\to$ \rMallTwenty{}) but from so low a base that
|
|
extrapolating it to a useful number requires a $k$ that defeats the purpose of
|
|
retrieving. \S\ref{sec:diagnostic} measures how large that $k$ would have to be,
|
|
and the answer rules the approach out.
|