Say what the repair row cost and what it bought

mab-fc.tex carries a row — typed search with one repair call — whose test cells
are ellipses and which the paper never mentioned. A reader meets an unexplained,
mostly empty row and has no way to tell a missing run from a rejected one.

It is a rejected one, and the traces say so precisely: across 196 dev questions
the repair fired ONCE, that single plan was chosen, and it is the entire
difference between 88 and 89 multi-hop. One question, for a 27B model held
resident beside the search. The row beneath it — exhaustive enumeration with a
lexical tie-break — buys five points with no model in the loop at all.

So it was not promoted to test, and the empty cells are the method working
rather than a gap in it: a configuration that loses on dev does not get a second
chance on held-out data. Reporting the cells empty and saying why is cheaper
than dropping the row and leaving no trace that the idea was tried.

The paper already reports its losing LoCoMo variants this way. This one had the
numbers and not the sentence.
This commit is contained in:
2026-09-15 15:45:43 -07:00
parent ba960bd885
commit 5a0f823482
3 changed files with 10 additions and 2 deletions
Binary file not shown.
+10 -2
View File
@@ -599,9 +599,17 @@ haystack is read; it never sees a fact. Table~\ref{tab:mab-nbest} separates
the two: plan recall@N (some plan among the first N sets reaches the gold)
against execution success@N (the selected one does). Recall climbs with
every set and selection tracks it within a point or two --- 68 against 67
at 262K --- so the sets, not the selector, set the number. The fourth row
at 262K --- so the sets, not the selector, set the number. A third row asks a language model to repair a
plan the selector finds suspicious, and it is in the table at its dev numbers
and nowhere else: across 196 dev questions the repair fired \emph{once}, that
one plan was chosen, and it is the whole of the row's 88 to 89. One question,
for a 27B model held resident beside the search. It was not promoted to test,
which is why its test cells are empty --- a configuration that loses on dev does
not get a second chance on held-out data, and reporting the empty cells is
cheaper than pretending the row was never run. The fourth row
adds an exhaustive enumeration of the relaxed plan and a lexical tie-break
from the compiler's own templates. On dev it moved multi-hop from 88 to 93
from the compiler's own templates, and buys five points where the repair bought
one, with no model in the loop at all. On dev it moved multi-hop from 88 to 93
and single-hop from 98 to 100; on test, run once, multi-hop 89 / 89 / 67 and
single-hop 98 / 98 / 97. Pooled, 98.3 / 84.5, counting every unanswered
question as wrong.
Binary file not shown.