mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-22 17:20:17 +00:00
zeekay
219e7535aa
The benchmark paper was built and never published: papers.hanzo.ai served 32 of the 50 PDFs in this repo and not this one, so the results every table in it carries were not downloadable by anyone. It is in site/public/pdfs and in the site manifest now, and the tables it inputs are regenerated from the current runs. Two of those runs finished today after four days unable to authenticate, so locomo-answers gains its oracle rows and loses two partials that were flattering — enso-flash read 54.6 F1 at n=57 and reads 45.1 at n=282. The substantive change is to what the paper may claim. Naive's 91 / 57 was described here as circulating without a published method, not locatable, and not cited. It is published, at usenaive.ai/lab/memory, and although that page names no dataset it is identifiable from its own comparison column: its baselines are CAR's to the decimal, 78.0 / 30.2 and 54, 28, 18, 7, with the multi-hop scores collapsed into one "<=7%". A table is identified by its baselines. So the benchmark is MemoryAgentBench FactConsolidation, which this paper already reports, and the comparison is ours: 98.3 / 84.5 with no reader against their 91 / 57 on a matched gpt-4o-mini backbone, +7.3 single-hop and +27.5 multi-hop. Their per-lane split and protocol are unpublished, so it is pooled figures being compared and the paper says so.