mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-30 14:58:42 +00:00
zeekayandHanzo Dev
1d1e7be5ab
Both report benchmarks that already live in hanzoai/cloud under bench/, so the numbers have a script behind them rather than a spreadsheet. hanzo-dormant-agents takes apart the estimate that costs an agent as a process. State, execution and isolation are separable and differ by five orders of magnitude: a dormant agent is 477 bytes on disk and resumes in 0.034 ms, a parked loop is 601 bytes of heap, and the isolation boundary — the expensive one — is only owed by an agent actually running. The paper argues the container is worth its price where it is needed and says so; the claim is about which agents are charged for it. hanzo-multi-hop-retrieval is a negative result. On LoCoMo a dense retriever finds every required turn for 78.2% of single-hop questions and 22.7% of multi-hop ones, but finds at least one required turn about 80% of the time in both — so it understands the questions and cannot reach all of the answer. Four retrieval policies over one fixed index move that by at most 0.7 points and two make it worse. A diagnostic says why it was never going to work: 95.4% of multi-hop evidence spans sessions and the median worst-ranked gold turn sits at rank 67, so the missing signal is linkage rather than similarity. Publishing the dead end is the useful part. Each paper carries results-data.tex, generated from the runs, and prints XXX in place of any constant that file does not supply — a number cannot drift from its measurement without the drift showing on the page. Both compile under the CI gate's flags, -Werror included. INDEX.md is regenerated by make index, which also picks up the papers added since it was last written and the header its current generator emits. Co-authored-by: Hanzo Dev <[email protected]>
55 lines
1.9 KiB
TeX
55 lines
1.9 KiB
TeX
% GENERATED from the runs named below -- do not hand-edit.
|
|
%
|
|
% Corpus: LoCoMo (Maharana et al.), 10 conversations, locomo10.json.
|
|
% Embedding: zenlm/zen-embedding-0.6b. Brute-force cosine, no index, no rerank.
|
|
% Host: Apple M-series laptop, darwin/arm64. Date: 2026-09-07.
|
|
%
|
|
% hanzoai/cloud bench/brain/brain.mjs baseline recall, both categories
|
|
% hanzoai/cloud bench/brain/hops.mjs four retrieval policies
|
|
% hanzoai/cloud bench/brain/evidence.mjs why the policies do not help
|
|
%
|
|
% Vectors are cached in brain-vectors.json so scoring re-runs without
|
|
% re-embedding; every row below reproduces from that cache.
|
|
|
|
% -- corpus
|
|
\renewcommand{\rQsingle}{841}
|
|
\renewcommand{\rQmulti}{282}
|
|
\renewcommand{\rSearchMs}{1.27}
|
|
|
|
% -- baseline, single-hop (category 4)
|
|
\renewcommand{\rSallFive}{59.1}
|
|
\renewcommand{\rSallTen}{69.3}
|
|
\renewcommand{\rSallTwenty}{78.2}
|
|
\renewcommand{\rSanyFive}{62.2}
|
|
\renewcommand{\rSanyTen}{71.7}
|
|
\renewcommand{\rSanyTwenty}{80.7}
|
|
|
|
% -- baseline, multi-hop (category 1)
|
|
\renewcommand{\rMallFive}{10.3}
|
|
\renewcommand{\rMallTen}{14.9}
|
|
\renewcommand{\rMallTwenty}{22.7}
|
|
\renewcommand{\rManyFive}{54.3}
|
|
\renewcommand{\rManyTen}{68.4}
|
|
\renewcommand{\rManyTwenty}{80.1}
|
|
|
|
% -- four retrieval policies, recall@20 "all"
|
|
\renewcommand{\rPolSingleS}{78.2} \renewcommand{\rPolSingleM}{22.7}
|
|
\renewcommand{\rPolSingleMs}{1.25}
|
|
\renewcommand{\rPolPrfS}{75.1} \renewcommand{\rPolPrfM}{23.4}
|
|
\renewcommand{\rPolPrfMs}{2.49}
|
|
\renewcommand{\rPolMultiS}{77.9} \renewcommand{\rPolMultiM}{22.0}
|
|
\renewcommand{\rPolMultiMs}{5.88}
|
|
\renewcommand{\rPolChainS}{78.1} \renewcommand{\rPolChainM}{22.3}
|
|
\renewcommand{\rPolChainMs}{8.69}
|
|
\renewcommand{\rBestDelta}{0.7}
|
|
\renewcommand{\rWorstDelta}{3.1}
|
|
|
|
% -- diagnostic: the structure of the evidence
|
|
\renewcommand{\rGoldAvg}{3.1}
|
|
\renewcommand{\rGoldMax}{19}
|
|
\renewcommand{\rSpanSessions}{95.4}
|
|
\renewcommand{\rMedRank}{24}
|
|
\renewcommand{\rMedWorst}{67}
|
|
\renewcommand{\rBeyondTwenty}{77.2}
|
|
\renewcommand{\rBeyondHundred}{43.1}
|