mirror of
https://github.com/hanzoai/papers.git
synced 2026-11-03 02:27:30 +00:00
zeekayandHanzo Dev
1d1e7be5ab
Both report benchmarks that already live in hanzoai/cloud under bench/, so the numbers have a script behind them rather than a spreadsheet. hanzo-dormant-agents takes apart the estimate that costs an agent as a process. State, execution and isolation are separable and differ by five orders of magnitude: a dormant agent is 477 bytes on disk and resumes in 0.034 ms, a parked loop is 601 bytes of heap, and the isolation boundary — the expensive one — is only owed by an agent actually running. The paper argues the container is worth its price where it is needed and says so; the claim is about which agents are charged for it. hanzo-multi-hop-retrieval is a negative result. On LoCoMo a dense retriever finds every required turn for 78.2% of single-hop questions and 22.7% of multi-hop ones, but finds at least one required turn about 80% of the time in both — so it understands the questions and cannot reach all of the answer. Four retrieval policies over one fixed index move that by at most 0.7 points and two make it worse. A diagnostic says why it was never going to work: 95.4% of multi-hop evidence spans sessions and the median worst-ranked gold turn sits at rank 67, so the missing signal is linkage rather than similarity. Publishing the dead end is the useful part. Each paper carries results-data.tex, generated from the runs, and prints XXX in place of any constant that file does not supply — a number cannot drift from its measurement without the drift showing on the page. Both compile under the CI gate's flags, -Werror included. INDEX.md is regenerated by make index, which also picks up the papers added since it was last written and the header its current generator emits. Co-authored-by: Hanzo Dev <[email protected]>
91 lines
4.0 KiB
TeX
91 lines
4.0 KiB
TeX
\section{Related work}
|
|
\label{sec:related}
|
|
|
|
LoCoMo~\cite{locomo} established the long-horizon conversational memory task and
|
|
supplies the per-question evidence annotations that make an \textbf{all}-evidence
|
|
metric computable; without them only \textbf{any} is measurable, and
|
|
\S\ref{sec:baseline} is the argument that \textbf{any} alone is misleading.
|
|
|
|
Dense passage retrieval~\cite{dpr} is the substrate every policy here runs on.
|
|
Pseudo-relevance feedback long predates it --- Rocchio's method~\cite{rocchio}
|
|
reformulates a query toward its own top results, and our \textbf{prf} policy is
|
|
that idea in vector space. IRCoT~\cite{ircot} interleaves retrieval with a
|
|
model's chain of thought, which is the shape our \textbf{chain} policy imitates
|
|
with similarity substituted for the reasoning step; \S\ref{sec:limits} is
|
|
explicit that this makes our result a bound on cheap hops rather than a
|
|
refutation of IRCoT.
|
|
|
|
Entity-and-graph-structured memory~\cite{hipporag} is the family
|
|
\S\ref{sec:implications} points at: it constructs the cross-session linkage that
|
|
our diagnostic identifies as the missing signal, instead of trying to recover it
|
|
from similarity. This paper's contribution to that line is negative and
|
|
supporting --- evidence, on a standard benchmark, that the cheaper alternative
|
|
does not work and cannot be made to.
|
|
|
|
\section{Conclusion}
|
|
\label{sec:conclusion}
|
|
|
|
Dense retrieval over a long conversation does not misunderstand multi-hop
|
|
questions. It answers them partially: at $k{=}20$ it finds at least one required
|
|
turn \rManyTwenty{}\% of the time and all of them \rMallTwenty{}\% of the time.
|
|
|
|
Four retrieval policies over one fixed index --- feedback, neighbour expansion,
|
|
and a two-deep chain --- move that \rMallTwenty{}\% by at most \rBestDelta{}
|
|
percentage points, and two of the four move it backwards. The reason is
|
|
structural: \rSpanSessions{}\% of multi-hop evidence spans sessions, the median
|
|
worst-ranked gold turn sits at rank \rMedWorst{}, and \rBeyondHundred{}\% of
|
|
questions have evidence past rank 100. Every policy tested spends more
|
|
similarity, and similarity is precisely the axis along which the missing turns
|
|
are distant.
|
|
|
|
The fix is not a better search over the same space. It is a different relation
|
|
--- entity linkage across sessions --- built as the conversation arrives and
|
|
traversed at retrieval time. We have not built it. We have established, on a
|
|
standard benchmark with a reproducible script, that the cheap alternative does
|
|
not work, and we would rather publish that than a tuning result of two questions
|
|
out of \rQmulti{}.
|
|
|
|
\section*{Reproducing}
|
|
|
|
\begin{lstlisting}[language=bash]
|
|
curl -sL https://raw.githubusercontent.com/snap-research/locomo/main/\
|
|
data/locomo10.json -o locomo10.json
|
|
node brain.mjs # baseline; caches vectors to brain-vectors.json
|
|
node hops.mjs # the four policies, from the cache
|
|
node evidence.mjs # the structural diagnostic
|
|
\end{lstlisting}
|
|
|
|
\noindent The scripts are in \texttt{hanzoai/cloud} under \texttt{bench/brain/}.
|
|
|
|
\begin{thebibliography}{9}
|
|
|
|
\bibitem{locomo}
|
|
A.~Maharana, D.-H.~Lee, S.~Tulyakov, M.~Bansal, F.~Barbieri, and Y.~Fang.
|
|
\newblock Evaluating Very Long-Term Conversational Memory of LLM Agents.
|
|
\newblock In \emph{ACL}, 2024.
|
|
|
|
\bibitem{dpr}
|
|
V.~Karpukhin, B.~O\u{g}uz, S.~Min, P.~Lewis, L.~Wu, S.~Edunov, D.~Chen, and
|
|
W.-t.~Yih.
|
|
\newblock Dense Passage Retrieval for Open-Domain Question Answering.
|
|
\newblock In \emph{EMNLP}, 2020.
|
|
|
|
\bibitem{rocchio}
|
|
J.~J.~Rocchio.
|
|
\newblock Relevance Feedback in Information Retrieval.
|
|
\newblock In \emph{The SMART Retrieval System}, Prentice-Hall, 1971.
|
|
|
|
\bibitem{ircot}
|
|
H.~Trivedi, N.~Balasubramanian, T.~Khot, and A.~Sabharwal.
|
|
\newblock Interleaving Retrieval with Chain-of-Thought Reasoning for
|
|
Knowledge-Intensive Multi-Step Questions.
|
|
\newblock In \emph{ACL}, 2023.
|
|
|
|
\bibitem{hipporag}
|
|
B.~J.~Guti\'errez, Y.~Shu, Y.~Gu, M.~Yasunaga, and Y.~Su.
|
|
\newblock HippoRAG: Neurobiologically Inspired Long-Term Memory for Large
|
|
Language Models.
|
|
\newblock In \emph{NeurIPS}, 2024.
|
|
|
|
\end{thebibliography}
|