mirror of
https://github.com/hanzoai/papers.git
synced 2026-11-03 02:27:30 +00:00
zeekayandHanzo Dev
7513388e56
The measured cells are injected from a generated file, which guarantees the table matches a file but not that the file is the only one. It was not. results-data.tex was built up across eight commits between 2026-07-15 and 2026-07-17, each naming its seed, sample size and system count. The three files the paper actually prints arrived together in54f5713, "wip: preserve in-flight work", which named no run at all and rewrote main.tex around them. The two sets disagree on GPQA Diamond (enso 92.9 vs 87.9, ultra 92.9 vs 90.4) and LiveCodeBench (93.7 vs 91.4, 89.1 vs 85.1), and agree on HLE and CharXiv exactly -- which is how we know the later file is the earlier one with two benchmarks re-run and two carried forward, and not a re-roll. enso-flash is populated for exactly the two that moved. So: rename results-data.tex to results-prior.tex, say in its header that nothing inputs it and why it is kept, and report the disagreement in Limitations with the gaps in items (ten questions of 198 on GPQA, four of 175 on LiveCodeBench) against the binomial error, and with the cost gap -- $50.6 to $11.1 on GPQA -- marked as the configuration change it is rather than sampling. enso-bench and emit_paper.py are on no machine we have, so neither run can be re-executed and neither can be adjudicated; the paper prints the later one and the spread is the honest uncertainty. Drop "edging the best single arm" from the evaluation. One accuracy point at n=198 is two questions, inside the standard error of either cell. It is a match, and the result on that row is the cost. The router repository's held-out replay agrees: over the same items the strongest single arm scores exactly what the router scores, on all four benchmarks. Also fix the build.54f5713renamed the vendor out of the prose but left the bibliography key behind, so six citations resolved to nothing and latexmk -Werror failed on a paper nobody had rebuilt. Co-authored-by: Hanzo Dev <[email protected]>
39 lines
1.5 KiB
TeX
39 lines
1.5 KiB
TeX
% GENERATED by enso-bench/scripts/emit_paper.py -- do not hand-edit.
|
|
%
|
|
% The EARLIER of two runs. Nothing inputs this file; main.tex prints
|
|
% results-measured-rows.tex. Kept because it is the only record of the run
|
|
% whose per-benchmark commits (f45579f, 2a652e8, b336a79, 66ab231, 2d26d1e,
|
|
% b679712, 1e4b204, fdd3dec) name their seed, sample size and system count,
|
|
% and because it disagrees with the printed run on two benchmarks. The
|
|
% disagreement, and what can and cannot be concluded from it, is stated in
|
|
% main.tex under Limitations -- not hidden by deleting this file.
|
|
%
|
|
% Two SKUs only: enso and enso-ultra. The enso-flash column of the printed
|
|
% table has no counterpart here, which is one reason the printed run is
|
|
% later rather than merely different.
|
|
|
|
% gpqa_diamond: enso n=198 acc=87.9 ultra n=198 acc=90.4
|
|
\renewcommand{\rEnsGPQA}{87.9}
|
|
\renewcommand{\rEnsCGPQA}{50.6}
|
|
\renewcommand{\rUltGPQA}{90.4}
|
|
\renewcommand{\rUltCGPQA}{80.2}
|
|
|
|
% hle: enso n=500 acc=29.4 ultra n=500 acc=34.6
|
|
\renewcommand{\rEnsHLE}{29.4}
|
|
\renewcommand{\rEnsCHLE}{153.1}
|
|
\renewcommand{\rUltHLE}{34.6}
|
|
\renewcommand{\rUltCHLE}{303.6}
|
|
|
|
% livecodebench: enso n=175 acc=91.4 ultra n=175 acc=85.1
|
|
\renewcommand{\rEnsLCB}{91.4}
|
|
\renewcommand{\rEnsCLCB}{29.9}
|
|
\renewcommand{\rUltLCB}{85.1}
|
|
\renewcommand{\rUltCLCB}{409.9}
|
|
|
|
% charxiv: enso n=200 acc=78.0 ultra n=200 acc=85.5
|
|
\renewcommand{\rEnsCHARXIV}{78.0}
|
|
\renewcommand{\rEnsCCHARXIV}{17.3}
|
|
\renewcommand{\rUltCHARXIV}{85.5}
|
|
\renewcommand{\rUltCCHARXIV}{31.4}
|
|
|