mirror of
https://github.com/hanzoai/papers.git
synced 2026-11-03 02:27:30 +00:00
zeekayandHanzo Dev
1d1e7be5ab
Both report benchmarks that already live in hanzoai/cloud under bench/, so the numbers have a script behind them rather than a spreadsheet. hanzo-dormant-agents takes apart the estimate that costs an agent as a process. State, execution and isolation are separable and differ by five orders of magnitude: a dormant agent is 477 bytes on disk and resumes in 0.034 ms, a parked loop is 601 bytes of heap, and the isolation boundary — the expensive one — is only owed by an agent actually running. The paper argues the container is worth its price where it is needed and says so; the claim is about which agents are charged for it. hanzo-multi-hop-retrieval is a negative result. On LoCoMo a dense retriever finds every required turn for 78.2% of single-hop questions and 22.7% of multi-hop ones, but finds at least one required turn about 80% of the time in both — so it understands the questions and cannot reach all of the answer. Four retrieval policies over one fixed index move that by at most 0.7 points and two make it worse. A diagnostic says why it was never going to work: 95.4% of multi-hop evidence spans sessions and the median worst-ranked gold turn sits at rank 67, so the missing signal is linkage rather than similarity. Publishing the dead end is the useful part. Each paper carries results-data.tex, generated from the runs, and prints XXX in place of any constant that file does not supply — a number cannot drift from its measurement without the drift showing on the page. Both compile under the CI gate's flags, -Werror included. INDEX.md is regenerated by make index, which also picks up the papers added since it was last written and the header its current generator emits. Co-authored-by: Hanzo Dev <[email protected]>
62 lines
3.1 KiB
TeX
62 lines
3.1 KiB
TeX
\section{Four policies, one index}
|
|
\label{sec:policies}
|
|
|
|
If one search finds part of the evidence, the standard move is to search again
|
|
from what the first search found. We implemented four policies over the cached
|
|
index. Every policy sees identical vectors and identical scoring; the retrieval
|
|
policy is the only variable.
|
|
|
|
\begin{description}[leftmargin=1.5cm,style=nextline]
|
|
\item[single] One search on the question. The baseline.
|
|
\item[prf] Pseudo-relevance feedback~\cite{rocchio}: blend the question vector
|
|
toward the centroid of its own top 3 hits ($w{=}0.4$) and search again,
|
|
keeping the first hop's best and filling from the second.
|
|
\item[multi] Search again from each of the top 3 hits; union the results,
|
|
scoring each neighbour by the product of the seed's confidence and its own.
|
|
\item[chain] \textbf{multi}, then one further hop from what that found --- the
|
|
retrieve--reason--retrieve shape~\cite{ircot} with similarity standing in for
|
|
the reasoning step.
|
|
\end{description}
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\begin{tabular}{lrrrrr}
|
|
\toprule
|
|
& \multicolumn{2}{c}{\textbf{single-hop}} & \multicolumn{2}{c}{\textbf{multi-hop}} & \\
|
|
\cmidrule(lr){2-3}\cmidrule(lr){4-5}
|
|
\textbf{Policy} & \textbf{@20 all} & $\Delta$ & \textbf{@20 all} & $\Delta$ & \textbf{ms} \\
|
|
\midrule
|
|
single & \rPolSingleS{}\% & --- & \rPolSingleM{}\% & --- & \rPolSingleMs{} \\
|
|
prf & \rPolPrfS{}\% & $-$\rWorstDelta{} & \rPolPrfM{}\% & $+$\rBestDelta{} & \rPolPrfMs{} \\
|
|
multi & \rPolMultiS{}\% & $-0.3$ & \rPolMultiM{}\% & $-0.7$ & \rPolMultiMs{} \\
|
|
chain & \rPolChainS{}\% & $-0.1$ & \rPolChainM{}\% & $-0.4$ & \rPolChainMs{} \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\caption{Retrieval policy against recall@20 \textbf{all}, $\Delta$ in percentage
|
|
points against a single search \meas{}. Latency is per search, averaged at
|
|
$k{=}20$.}
|
|
\label{tab:policies}
|
|
\end{table}
|
|
|
|
\paragraph{The result is negative, and it is negative in three separate ways.}
|
|
|
|
\begin{enumerate}[leftmargin=*]
|
|
\item \textbf{The best case is noise.} \textbf{prf} gains \rBestDelta{}
|
|
percentage points on multi-hop. On \rQmulti{} questions that is two questions.
|
|
It is not a result, and reporting it as one would be indefensible.
|
|
\item \textbf{Two policies made it worse.} \textbf{multi} and \textbf{chain} ---
|
|
the two that most directly implement ``follow the first hit'' --- both lost
|
|
ground on multi-hop, at $5\times$ and $7\times$ the latency. Searching from a
|
|
neighbour pulls in that neighbour's neighbourhood, which is denser in
|
|
near-duplicates of what was already found than in the missing evidence.
|
|
\item \textbf{The one gain is paid for elsewhere.} \textbf{prf} costs
|
|
\rWorstDelta{} points on single-hop, where the baseline was working. Blending
|
|
the query toward its own results is a bet that the first hits are on topic;
|
|
when a single turn is the whole answer, that bet dilutes a query that was
|
|
already correct.
|
|
\end{enumerate}
|
|
|
|
There is no row in this table anyone should ship. The honest summary is that
|
|
policy is the wrong knob, and \S\ref{sec:diagnostic} shows it was never going to
|
|
be the right one.
|