mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-30 14:58:42 +00:00
zeekayandHanzo Dev
1d1e7be5ab
Both report benchmarks that already live in hanzoai/cloud under bench/, so the numbers have a script behind them rather than a spreadsheet. hanzo-dormant-agents takes apart the estimate that costs an agent as a process. State, execution and isolation are separable and differ by five orders of magnitude: a dormant agent is 477 bytes on disk and resumes in 0.034 ms, a parked loop is 601 bytes of heap, and the isolation boundary — the expensive one — is only owed by an agent actually running. The paper argues the container is worth its price where it is needed and says so; the claim is about which agents are charged for it. hanzo-multi-hop-retrieval is a negative result. On LoCoMo a dense retriever finds every required turn for 78.2% of single-hop questions and 22.7% of multi-hop ones, but finds at least one required turn about 80% of the time in both — so it understands the questions and cannot reach all of the answer. Four retrieval policies over one fixed index move that by at most 0.7 points and two make it worse. A diagnostic says why it was never going to work: 95.4% of multi-hop evidence spans sessions and the median worst-ranked gold turn sits at rank 67, so the missing signal is linkage rather than similarity. Publishing the dead end is the useful part. Each paper carries results-data.tex, generated from the runs, and prints XXX in place of any constant that file does not supply — a number cannot drift from its measurement without the drift showing on the page. Both compile under the CI gate's flags, -Werror included. INDEX.md is regenerated by make index, which also picks up the papers added since it was last written and the header its current generator emits. Co-authored-by: Hanzo Dev <[email protected]>
37 lines
2.0 KiB
TeX
37 lines
2.0 KiB
TeX
\section{Limits}
|
|
\label{sec:limits}
|
|
|
|
\paragraph{One corpus.}
|
|
LoCoMo is ten conversations. It is the standard benchmark for this
|
|
task~\cite{locomo} and its evidence annotations are what make the \textbf{all}
|
|
metric computable at all, but ten conversations is ten conversations, and the
|
|
structural facts in \S\ref{sec:diagnostic} --- particularly the
|
|
\rSpanSessions{}\% cross-session figure --- are properties of \emph{this} corpus.
|
|
A corpus whose multi-hop evidence clustered within sessions would very likely
|
|
behave differently, and the policies might help there.
|
|
|
|
\paragraph{One embedding.}
|
|
Every row uses \texttt{zen-embedding-0.6b}. The comparison between policies is
|
|
sound because they share an index, but the absolute recall numbers are that
|
|
model's, and a stronger embedding would shift the whole table. What would
|
|
\emph{not} shift is the relationship the paper is about: \textbf{any} exceeding
|
|
\textbf{all} by a wide margin is a statement about evidence distribution, not
|
|
about embedding quality.
|
|
|
|
\paragraph{Similarity standing in for reasoning.}
|
|
Our \textbf{chain} policy follows IRCoT's shape~\cite{ircot} with one important
|
|
substitution: IRCoT interleaves a language model's reasoning between retrieval
|
|
steps, and we substitute vector similarity for that step. This makes the policy
|
|
thousands of times cheaper and it is a fair test of ``does searching again from
|
|
what you found help'', but it is \emph{not} a test of IRCoT. A model that reads
|
|
the first hop and writes a genuinely new query has access to exactly the entity
|
|
linkage \S\ref{sec:diagnostic} identifies as missing. We expect it to do better;
|
|
we did not measure it; and it costs a model call per hop, which is a different
|
|
economic proposition from \rPolChainMs{}\,ms.
|
|
|
|
\paragraph{Retrieval only.}
|
|
We measure whether the evidence was retrieved, never whether a model answered
|
|
correctly given it. End-to-end accuracy depends on the reader as much as the
|
|
retriever, and conflating the two is how retrieval failures get attributed to
|
|
models. The scope here is deliberate and narrow.
|