mirror of
https://github.com/hanzoai/papers.git
synced 2026-11-03 02:27:30 +00:00
zeekayandHanzo Dev
1d1e7be5ab
Both report benchmarks that already live in hanzoai/cloud under bench/, so the numbers have a script behind them rather than a spreadsheet. hanzo-dormant-agents takes apart the estimate that costs an agent as a process. State, execution and isolation are separable and differ by five orders of magnitude: a dormant agent is 477 bytes on disk and resumes in 0.034 ms, a parked loop is 601 bytes of heap, and the isolation boundary — the expensive one — is only owed by an agent actually running. The paper argues the container is worth its price where it is needed and says so; the claim is about which agents are charged for it. hanzo-multi-hop-retrieval is a negative result. On LoCoMo a dense retriever finds every required turn for 78.2% of single-hop questions and 22.7% of multi-hop ones, but finds at least one required turn about 80% of the time in both — so it understands the questions and cannot reach all of the answer. Four retrieval policies over one fixed index move that by at most 0.7 points and two make it worse. A diagnostic says why it was never going to work: 95.4% of multi-hop evidence spans sessions and the median worst-ranked gold turn sits at rank 67, so the missing signal is linkage rather than similarity. Publishing the dead end is the useful part. Each paper carries results-data.tex, generated from the runs, and prints XXX in place of any constant that file does not supply — a number cannot drift from its measurement without the drift showing on the page. Both compile under the CI gate's flags, -Werror included. INDEX.md is regenerated by make index, which also picks up the papers added since it was last written and the header its current generator emits. Co-authored-by: Hanzo Dev <[email protected]>
62 lines
3.1 KiB
TeX
62 lines
3.1 KiB
TeX
\section{Why no number of hops helps}
|
|
\label{sec:diagnostic}
|
|
|
|
Rather than try a fifth policy we measured the evidence itself.
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\begin{tabular}{lr}
|
|
\toprule
|
|
\textbf{Structure of multi-hop evidence (\rQmulti{} questions)} & \\
|
|
\midrule
|
|
Gold turns per question, mean & \rGoldAvg{} \meas{} \\
|
|
Gold turns per question, max & \rGoldMax{} \meas{} \\
|
|
Evidence spanning more than one session & \textbf{\rSpanSessions{}\%} \meas{} \\
|
|
\midrule
|
|
Median rank of a gold turn & \rMedRank{} \meas{} \\
|
|
Median rank of the \emph{worst} gold turn & \textbf{\rMedWorst{}} \meas{} \\
|
|
Worst gold turn beyond $k{=}20$ & \rBeyondTwenty{}\% \meas{} \\
|
|
Worst gold turn beyond $k{=}100$ & \rBeyondHundred{}\% \meas{} \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\caption{The evidence a multi-hop question needs, and where a single similarity
|
|
search puts it.}
|
|
\label{tab:diagnostic}
|
|
\end{table}
|
|
|
|
Three facts in that table between them close the case.
|
|
|
|
\paragraph{The evidence is not one neighbourhood; it is several.}
|
|
\rSpanSessions{}\% of multi-hop evidence spans more than one conversation
|
|
session. These are turns separated by simulated weeks, about different immediate
|
|
topics, phrased differently, related only because they concern the same entity.
|
|
Nothing about that relation is expressed in the wording, so nothing about it is
|
|
expressed in a similarity metric over wordings. A second search in the same space
|
|
is a second look at the same neighbourhood.
|
|
|
|
\paragraph{The hard turn is not near the easy one.}
|
|
The median gold turn is at rank \rMedRank{}, comfortably retrievable. The median
|
|
\emph{worst} gold turn is at rank \rMedWorst{}. That distance is the entire
|
|
failure: a question is answered only when the last of its evidence arrives, so
|
|
recall@$k$ \textbf{all} is governed by the worst-ranked gold turn and by nothing
|
|
else. Improving the rank of turns that already rank 24 --- which is what an extra
|
|
hop from a top hit does --- moves a number the metric does not depend on.
|
|
|
|
\paragraph{No affordable $k$ rescues it.}
|
|
The worst gold turn is beyond $k{=}20$ for \rBeyondTwenty{}\% of questions and
|
|
still beyond $k{=}100$ for \rBeyondHundred{}\%. Retrieving 100 turns to answer
|
|
one question is most of a short conversation; it defeats the purpose of
|
|
retrieving and it degrades the answer by burying the evidence in context. So the
|
|
brute-force escape is closed too, and it is closed by measurement rather than by
|
|
taste.
|
|
|
|
\paragraph{Putting it together.}
|
|
The retriever finds \emph{a} relevant turn nearly always (\S\ref{sec:baseline}),
|
|
because for any multi-hop question at least one piece of evidence does resemble
|
|
the question. It misses the others because they resemble the question far less
|
|
--- they are linked to it through an entity and a timeline, not through
|
|
vocabulary. Every policy in \S\ref{sec:policies} is a way of spending more
|
|
similarity, and similarity is the axis along which the missing evidence is
|
|
\emph{far}. This is why the negative result is not a tuning failure: no setting
|
|
of a similarity-based policy has access to the signal that would fix it.
|