mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-22 17:20:17 +00:00
Enso: three of four best-arm cells come from the replay, and one does not
The router repository's stored replay names a strongest single arm per benchmark. We print fable-5 at 38.2 on HLE and gpt-5.5 at 85.5 on CharXiv, both of which agree with it to the digit, and we leave LiveCodeBench blank. On GPQA we print gpt-5.6-sol at 91.9 where the replay names grok-4.5 at 92.9, over a wider pool than this gateway serves. Three cells sharing a source and the fourth not is worth saying out loud next to the row whose margin the fourth cell decides. Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
Binary file not shown.
+6
-3
@@ -704,9 +704,12 @@ and replays the router against it with the fingerprint index refit five-fold so
|
||||
router is scored on questions absent from its own route table. Over the same item
|
||||
counts that replay reports 92.9 (GPQA, $n{=}198$), 91.4 (LiveCodeBench, $n{=}175$),
|
||||
38.2 (HLE, $n{=}500$) and 85.5 (CharXiv, $n{=}200$)---and on all four the strongest
|
||||
single arm scores exactly the same as the router. It also names \texttt{grok-4.5},
|
||||
not \texttt{\hBestGPQAname}, as the strongest arm on GPQA, over a wider eleven-arm
|
||||
pool than the one this paper serves. A stored-table replay is a different
|
||||
single arm scores exactly the same as the router. Three of the four best-arm cells we print agree with
|
||||
that replay to the digit---\texttt{fable-5} at 38.2 on HLE, \texttt{gpt-5.5} at
|
||||
85.5 on CharXiv, and the LiveCodeBench cell we leave blank. The fourth does not:
|
||||
the replay names \texttt{grok-4.5} at 92.9 as the strongest arm on GPQA, over a
|
||||
wider eleven-arm pool than the one this paper serves, where we print
|
||||
\texttt{\hBestGPQAname} at \hBestGPQAacc. A stored-table replay is a different
|
||||
measurement from the live-gateway run reported here and we do not merge the two.
|
||||
It does support reading the GPQA row as a match rather than a lead, which is how
|
||||
\S\ref{sec:eval} now reads it.
|
||||
|
||||
Reference in New Issue
Block a user