mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-30 14:58:42 +00:00
zeekayandHanzo Dev
1d1e7be5ab
Both report benchmarks that already live in hanzoai/cloud under bench/, so the numbers have a script behind them rather than a spreadsheet. hanzo-dormant-agents takes apart the estimate that costs an agent as a process. State, execution and isolation are separable and differ by five orders of magnitude: a dormant agent is 477 bytes on disk and resumes in 0.034 ms, a parked loop is 601 bytes of heap, and the isolation boundary — the expensive one — is only owed by an agent actually running. The paper argues the container is worth its price where it is needed and says so; the claim is about which agents are charged for it. hanzo-multi-hop-retrieval is a negative result. On LoCoMo a dense retriever finds every required turn for 78.2% of single-hop questions and 22.7% of multi-hop ones, but finds at least one required turn about 80% of the time in both — so it understands the questions and cannot reach all of the answer. Four retrieval policies over one fixed index move that by at most 0.7 points and two make it worse. A diagnostic says why it was never going to work: 95.4% of multi-hop evidence spans sessions and the median worst-ranked gold turn sits at rank 67, so the missing signal is linkage rather than similarity. Publishing the dead end is the useful part. Each paper carries results-data.tex, generated from the runs, and prints XXX in place of any constant that file does not supply — a number cannot drift from its measurement without the drift showing on the page. Both compile under the CI gate's flags, -Werror included. INDEX.md is regenerated by make index, which also picks up the papers added since it was last written and the header its current generator emits. Co-authored-by: Hanzo Dev <[email protected]>
51 lines
2.3 KiB
TeX
51 lines
2.3 KiB
TeX
\section{Setup}
|
|
\label{sec:setup}
|
|
|
|
\subsection{Corpus and categories}
|
|
|
|
LoCoMo~\cite{locomo} is ten conversations, each spanning multiple sessions over
|
|
a simulated months-long relationship, 369--689 turns apiece, with questions
|
|
annotated by category and by the turns that constitute their evidence. We use
|
|
two categories:
|
|
|
|
\begin{description}[leftmargin=1.2cm,style=nextline]
|
|
\item[Category 4, single-hop] \rQsingle{} questions answerable from one turn.
|
|
\item[Category 1, multi-hop] \rQmulti{} questions requiring evidence from several
|
|
turns, frequently in different sessions.
|
|
\end{description}
|
|
|
|
Only questions carrying annotated evidence are scored. Retrieval is within a
|
|
conversation --- the correct scope, since a question about one relationship is
|
|
never answered by another's transcript.
|
|
|
|
\subsection{Retriever}
|
|
|
|
Deliberately plain, because the paper is about the retrieval \emph{policy} and a
|
|
sophisticated retriever would confound it: turns embedded with
|
|
\texttt{zenlm/zen-embedding-0.6b}, cosine similarity, brute force over every turn
|
|
in the conversation, no approximate index, no reranking, no query rewriting.
|
|
Brute force costs \rSearchMs{}\,ms per search over a whole conversation \meas{},
|
|
which is small enough that an index would buy nothing at this scale and would add
|
|
a recall variable we do not want.
|
|
|
|
Vectors are computed once and cached, so every policy in \S\ref{sec:policies}
|
|
scores against a byte-identical index. This matters: it means a difference
|
|
between policies cannot be an embedding difference.
|
|
|
|
\subsection{Two metrics, and why both are needed}
|
|
|
|
For a question with gold evidence set $E$ and retrieved set $R_k$ of size $k$:
|
|
|
|
\begin{align*}
|
|
\text{recall@}k\ \textbf{all} &= \Pr\left[E \subseteq R_k\right] \\
|
|
\text{recall@}k\ \textbf{any} &= \Pr\left[E \cap R_k \neq \emptyset\right]
|
|
\end{align*}
|
|
|
|
\textbf{all} is the one that matters for answering: a question needing three
|
|
facts is not answered by two of them, and a model handed two of three will
|
|
typically produce a confident wrong answer rather than an admission. \textbf{any}
|
|
is the diagnostic: it separates ``the retriever does not understand the
|
|
question'' from ``the retriever understands the question and cannot reach all of
|
|
the answer''. Reporting only one of the two is how the failure in this paper
|
|
stays hidden.
|