mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-30 14:58:42 +00:00
zeekay
219e7535aa
The benchmark paper was built and never published: papers.hanzo.ai served 32 of the 50 PDFs in this repo and not this one, so the results every table in it carries were not downloadable by anyone. It is in site/public/pdfs and in the site manifest now, and the tables it inputs are regenerated from the current runs. Two of those runs finished today after four days unable to authenticate, so locomo-answers gains its oracle rows and loses two partials that were flattering — enso-flash read 54.6 F1 at n=57 and reads 45.1 at n=282. The substantive change is to what the paper may claim. Naive's 91 / 57 was described here as circulating without a published method, not locatable, and not cited. It is published, at usenaive.ai/lab/memory, and although that page names no dataset it is identifiable from its own comparison column: its baselines are CAR's to the decimal, 78.0 / 30.2 and 54, 28, 18, 7, with the multi-hop scores collapsed into one "<=7%". A table is identified by its baselines. So the benchmark is MemoryAgentBench FactConsolidation, which this paper already reports, and the comparison is ours: 98.3 / 84.5 with no reader against their 91 / 57 on a matched gpt-4o-mini backbone, +7.3 single-hop and +27.5 multi-hop. Their per-lane split and protocol are unpublished, so it is pooled figures being compared and the paper says so.
18 lines
1.4 KiB
TeX
18 lines
1.4 KiB
TeX
% generated by hanzoai/cloud bench/brain/tables.mjs — runs/mab-*; substring exact match; 6k is the dev split, the rest test; pooled = mean of the four sizes as CAR reports
|
|
\begin{tabular}{lrrrrrrrrrr}
|
|
\toprule
|
|
& \multicolumn{5}{c}{FC-SH} & \multicolumn{5}{c}{FC-MH} \\
|
|
\cmidrule(lr){2-6}\cmidrule(lr){7-11}
|
|
Configuration (reader) & 6k & 32k & 64k & 262k & pooled & 6k & 32k & 64k & 262k & pooled \\
|
|
\midrule
|
|
resolver, latest wins (no reader) & 96.0 & 97.0 & 95.0 & 94.0 & 95.5 & 76.0 & 59.0 & 59.0 & 33.0 & 56.8 \\
|
|
typed search over the plans (no reader) & 98.0 & 97.0 & 97.0 & 94.0 & 96.5 & 88.0 & 87.0 & 84.0 & 64.0 & 80.8 \\
|
|
typed search, exhaustive, lexical tie-break (no reader) & 100.0 & 98.0 & 98.0 & 97.0 & 98.3 & 93.0 & 89.0 & 89.0 & 67.0 & 84.5 \\
|
|
typed search with one repair call (no reader) & 98.0 & \ldots & \ldots & \ldots & \ldots & 89.0 & \ldots & \ldots & \ldots & \ldots \\
|
|
\midrule
|
|
\multicolumn{11}{l}{CAR (arXiv 2606.01435), pooled 6k--262k: 78.0 / 30.2 with gpt-4o-mini, 94.8 / 51.5 with gpt-4o; at 262k: 82 / 27 and 93 / 41} \\
|
|
\multicolumn{11}{l}{Na\"ive Brain, \texttt{usenaive.ai/lab/memory} (read 2026-09-15, dated July 2026), pooled: 91 / 57 on a matched gpt-4o-mini backbone} \\
|
|
\multicolumn{11}{l}{\quad that page names no dataset; it is identified as this one by its baselines, which are CAR's to the decimal --- 78.0/30.2, 54, 28, 18, 7} \\
|
|
\bottomrule
|
|
\end{tabular}
|