mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-22 17:20:17 +00:00
Say what the repair row cost and what it bought
mab-fc.tex carries a row — typed search with one repair call — whose test cells are ellipses and which the paper never mentioned. A reader meets an unexplained, mostly empty row and has no way to tell a missing run from a rejected one. It is a rejected one, and the traces say so precisely: across 196 dev questions the repair fired ONCE, that single plan was chosen, and it is the entire difference between 88 and 89 multi-hop. One question, for a 27B model held resident beside the search. The row beneath it — exhaustive enumeration with a lexical tie-break — buys five points with no model in the loop at all. So it was not promoted to test, and the empty cells are the method working rather than a gap in it: a configuration that loses on dev does not get a second chance on held-out data. Reporting the cells empty and saying why is cheaper than dropping the row and leaving no trace that the idea was tried. The paper already reports its losing LoCoMo variants this way. This one had the numbers and not the sentence.
This commit is contained in:
Binary file not shown.
@@ -599,9 +599,17 @@ haystack is read; it never sees a fact. Table~\ref{tab:mab-nbest} separates
|
||||
the two: plan recall@N (some plan among the first N sets reaches the gold)
|
||||
against execution success@N (the selected one does). Recall climbs with
|
||||
every set and selection tracks it within a point or two --- 68 against 67
|
||||
at 262K --- so the sets, not the selector, set the number. The fourth row
|
||||
at 262K --- so the sets, not the selector, set the number. A third row asks a language model to repair a
|
||||
plan the selector finds suspicious, and it is in the table at its dev numbers
|
||||
and nowhere else: across 196 dev questions the repair fired \emph{once}, that
|
||||
one plan was chosen, and it is the whole of the row's 88 to 89. One question,
|
||||
for a 27B model held resident beside the search. It was not promoted to test,
|
||||
which is why its test cells are empty --- a configuration that loses on dev does
|
||||
not get a second chance on held-out data, and reporting the empty cells is
|
||||
cheaper than pretending the row was never run. The fourth row
|
||||
adds an exhaustive enumeration of the relaxed plan and a lexical tie-break
|
||||
from the compiler's own templates. On dev it moved multi-hop from 88 to 93
|
||||
from the compiler's own templates, and buys five points where the repair bought
|
||||
one, with no model in the loop at all. On dev it moved multi-hop from 88 to 93
|
||||
and single-hop from 98 to 100; on test, run once, multi-hop 89 / 89 / 67 and
|
||||
single-hop 98 / 98 / 97. Pooled, 98.3 / 84.5, counting every unanswered
|
||||
question as wrong.
|
||||
|
||||
Binary file not shown.
Reference in New Issue
Block a user