mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-30 14:58:42 +00:00
zeekayandHanzo Dev
1d1e7be5ab
Both report benchmarks that already live in hanzoai/cloud under bench/, so the numbers have a script behind them rather than a spreadsheet. hanzo-dormant-agents takes apart the estimate that costs an agent as a process. State, execution and isolation are separable and differ by five orders of magnitude: a dormant agent is 477 bytes on disk and resumes in 0.034 ms, a parked loop is 601 bytes of heap, and the isolation boundary — the expensive one — is only owed by an agent actually running. The paper argues the container is worth its price where it is needed and says so; the claim is about which agents are charged for it. hanzo-multi-hop-retrieval is a negative result. On LoCoMo a dense retriever finds every required turn for 78.2% of single-hop questions and 22.7% of multi-hop ones, but finds at least one required turn about 80% of the time in both — so it understands the questions and cannot reach all of the answer. Four retrieval policies over one fixed index move that by at most 0.7 points and two make it worse. A diagnostic says why it was never going to work: 95.4% of multi-hop evidence spans sessions and the median worst-ranked gold turn sits at rank 67, so the missing signal is linkage rather than similarity. Publishing the dead end is the useful part. Each paper carries results-data.tex, generated from the runs, and prints XXX in place of any constant that file does not supply — a number cannot drift from its measurement without the drift showing on the page. Both compile under the CI gate's flags, -Werror included. INDEX.md is regenerated by make index, which also picks up the papers added since it was last written and the header its current generator emits. Co-authored-by: Hanzo Dev <[email protected]>
51 lines
2.8 KiB
TeX
51 lines
2.8 KiB
TeX
\section{Introduction}
|
|
\label{sec:intro}
|
|
|
|
An agent that holds a relationship with a user over months accumulates a
|
|
transcript far larger than any context window, so every such system retrieves.
|
|
The dominant design is a dense retriever over conversation turns: embed each
|
|
turn, embed the question, return the nearest $k$. It works well enough that most
|
|
platforms ship it and report a single recall number.
|
|
|
|
A single recall number hides the failure. Questions that require \emph{one} fact
|
|
and questions that require \emph{several} fail in different ways, and averaging
|
|
them produces a figure that describes neither. We separate them, and the
|
|
separation is the paper.
|
|
|
|
On LoCoMo~\cite{locomo} --- ten multi-session conversations of 369--689 turns
|
|
with human-annotated evidence --- a dense retriever at $k{=}20$ retrieves every
|
|
required turn for \rSallTwenty{}\% of single-hop questions and \rMallTwenty{}\%
|
|
of multi-hop ones \meas{}. That gap looks like a retrieval-quality problem and
|
|
invites a retrieval-quality answer: a better embedding, a reranker, a larger $k$.
|
|
|
|
It is not a retrieval-quality problem. The same run reports a second metric that
|
|
settles it. Scoring \emph{any} rather than \emph{all} --- did the retriever find
|
|
at least one of the required turns --- gives \rSanyTwenty{}\% on single-hop and
|
|
\rManyTwenty{}\% on multi-hop, which are the same number. The retriever
|
|
understands multi-hop questions exactly as well as it understands single-hop
|
|
ones. It finds a relevant turn essentially always, and the rest of them
|
|
essentially never.
|
|
|
|
That is a precise diagnosis and it points at a known family of fixes. If one
|
|
search finds part of the answer, search again from what it found: pseudo-relevance
|
|
feedback~\cite{rocchio} in vector space, or the retrieve--reason--retrieve loop
|
|
of IRCoT~\cite{ircot}. This paper implements four such policies over a single
|
|
fixed index, measures them, and reports that \textbf{they do not work} --- and
|
|
then measures the corpus itself to explain why they cannot.
|
|
|
|
\subsection*{Contributions}
|
|
|
|
\begin{enumerate}[leftmargin=*]
|
|
\item A decomposition of dense-retrieval recall on long conversations into
|
|
\emph{all}-evidence and \emph{any}-evidence, showing the two categories differ
|
|
only in the former (\S\ref{sec:baseline}).
|
|
\item Four retrieval policies over one index, holding embedding, corpus and
|
|
scoring fixed so that the policy is the only variable, with the negative result
|
|
in full including the cases that got worse (\S\ref{sec:policies}).
|
|
\item A structural diagnostic of the evidence that explains the negative result
|
|
and predicts it would survive a larger $k$, a better embedding, or a reranker
|
|
(\S\ref{sec:diagnostic}).
|
|
\item A statement of what the measurements do and do not license anyone to build
|
|
next (\S\ref{sec:implications}--\S\ref{sec:limits}).
|
|
\end{enumerate}
|