mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-22 17:20:17 +00:00
zooqueenandHanzo Dev
c9d9ad5d3f
The MemoryAgentBench section now reports the exhaustive typed search over the lexicon's and Qwen3.8's plan sets (test multi-hop 89 / 89 / 67, single-hop 98 / 98 / 97, pooled 98.3 / 84.5, no reader), a table of plan recall@N against execution success@N, and the ceiling analysis: 33 questions at 262K need an older version of a key a later, unasked-for edit overwrites, six store-readable signals separate nothing, latest-wins caps at 67 and the search is on that line. Pith is compared lane by lane with no best-published claim. Co-authored-by: Hanzo Dev <[email protected]>
13 lines
717 B
TeX
13 lines
717 B
TeX
% generated by hanzoai/cloud bench/brain/tables.mjs — FC-MH per haystack, out of 100; a gold path is the shortest path from the plan's base entity to an object matching the gold; a contested key is one two questions need in different versions
|
|
\begin{tabular}{lrrrr}
|
|
\toprule
|
|
& 6k & 32k & 64k & 262k \\
|
|
\midrule
|
|
multi-hop questions with a gold path in the parsed store & 96 & 98 & 94 & 98 \\
|
|
whose gold needs an older version of an updated key & 3 & 13 & 8 & 33 \\
|
|
keys contested between questions & 4 & 6 & 2 & 5 \\
|
|
ceiling, one version per key chosen by an oracle & 97 & 94 & 98 & 94 \\
|
|
ceiling, latest version wins & 97 & 87 & 92 & 67 \\
|
|
the search, selected-of-N & 93 & 89 & 89 & 67 \\
|
|
\bottomrule
|
|
\end{tabular} |