mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-30 14:58:42 +00:00
zeekayandHanzo Dev
1d1e7be5ab
Both report benchmarks that already live in hanzoai/cloud under bench/, so the numbers have a script behind them rather than a spreadsheet. hanzo-dormant-agents takes apart the estimate that costs an agent as a process. State, execution and isolation are separable and differ by five orders of magnitude: a dormant agent is 477 bytes on disk and resumes in 0.034 ms, a parked loop is 601 bytes of heap, and the isolation boundary — the expensive one — is only owed by an agent actually running. The paper argues the container is worth its price where it is needed and says so; the claim is about which agents are charged for it. hanzo-multi-hop-retrieval is a negative result. On LoCoMo a dense retriever finds every required turn for 78.2% of single-hop questions and 22.7% of multi-hop ones, but finds at least one required turn about 80% of the time in both — so it understands the questions and cannot reach all of the answer. Four retrieval policies over one fixed index move that by at most 0.7 points and two make it worse. A diagnostic says why it was never going to work: 95.4% of multi-hop evidence spans sessions and the median worst-ranked gold turn sits at rank 67, so the missing signal is linkage rather than similarity. Publishing the dead end is the useful part. Each paper carries results-data.tex, generated from the runs, and prints XXX in place of any constant that file does not supply — a number cannot drift from its measurement without the drift showing on the page. Both compile under the CI gate's flags, -Werror included. INDEX.md is regenerated by make index, which also picks up the papers added since it was last written and the header its current generator emits. Co-authored-by: Hanzo Dev <[email protected]>
38 lines
2.1 KiB
TeX
38 lines
2.1 KiB
TeX
\section{What this licenses, and what it does not}
|
|
\label{sec:implications}
|
|
|
|
\paragraph{Ruled out, on this corpus.}
|
|
Retrieval policy over a dense index --- feedback, neighbour expansion, chained
|
|
hops --- as a fix for multi-hop conversational recall. It gained at most
|
|
\rBestDelta{} points, cost up to $7\times$ the latency, and lost ground where the
|
|
baseline was working. Also ruled out, by \S\ref{sec:diagnostic} rather than by
|
|
trial: enlarging $k$, since \rBeyondHundred{}\% of questions have evidence beyond
|
|
rank 100.
|
|
|
|
\paragraph{Not ruled out, and not tested here.}
|
|
A better embedding would raise both \textbf{any} and \textbf{all} and we have not
|
|
measured its ceiling; the diagnostic predicts the gain would be modest, because
|
|
cross-session evidence is far in \emph{any} similarity space over wordings, but
|
|
that is a prediction and not a measurement. A cross-encoder reranker over a
|
|
larger candidate pool addresses the same axis and inherits the same doubt.
|
|
|
|
\paragraph{What the diagnostic actually points at.}
|
|
The signal the retriever lacks is which turns concern the same entity across
|
|
sessions. That is a relation to be \emph{constructed} --- extract entities as
|
|
turns arrive, link mentions across sessions, and let retrieval traverse those
|
|
links rather than only the similarity graph. This is the family that
|
|
entity-and-graph-structured memory systems occupy~\cite{hipporag}, and it is the
|
|
one the measurement endorses.
|
|
|
|
We are naming a direction, not claiming a result. Building it is a larger project
|
|
than testing four policies over a cached index, and the honest position at the
|
|
end of this paper is that we have established what does not work and why, and
|
|
have not yet built what does. That is the whole content of the claim.
|
|
|
|
\paragraph{One consequence for anyone shipping today.}
|
|
Report \textbf{all}, not \textbf{any}, and report the two categories separately.
|
|
A system quoting \rManyTwenty{}\% on multi-hop questions is quoting a true number
|
|
that describes a system answering \rMallTwenty{}\% of them. The gap between those
|
|
figures is not a rounding difference; it is the difference between a question
|
|
answered and a confident wrong answer built from two-thirds of the evidence.
|