Paper 1: the search ran over lexicon plans; no model at any point in those rows

This commit is contained in:
zooqueen
2026-09-09 11:49:21 -07:00
parent 59fa784e76
commit 21853c27cd
+10 -5
View File
@@ -558,8 +558,11 @@ phrasings. Parse coverage, entity lookup and version selection were not the
problem. The second run keeps the exact single-hop templates as rules, has a
model compile every other question into the same plan shape, validated
against the relation set, and uses the lexicon only where the model's plan
fails validation. That run scored 59 / 59 / 33 on multi-hop at 32K / 64K
/ 262K.
fails validation. On the shared GPU every model call of that pass timed
out and the lexicon filled every plan, so its plans were the lexicon's; it
scored 59 / 59 / 33 on multi-hop at 32K / 64K / 262K, the gain over the
first run being the fixed plan policy for single-hop and a corrected
lexicon, not a model.
The third resolver treats the plan as a sketch. An oracle search over the
parsed graph (\texttt{mab/oracle.mjs}) found a path from the plan's base
@@ -580,15 +583,17 @@ question names — a city, a country, a person, a language — as the final
constraint, deciding between stopping a hop early and going one further.
The best complete path wins by a fixed score; ties go to the more literal
plan and the later version. It is deterministic, it reads no gold, and it
calls no model: the compiled plan is the only thing a model produced, once,
at temperature 0. On dev it moved multi-hop from 76 to 83 and single-hop
calls no model. In the runs of Table~\ref{tab:mab} the plans it searched
over were the lexicon's (\texttt{meta.json} records the planner of every
run), so no language model took part at any point; model-compiled plan sets
are the next rows, labelled as such. On dev it moved multi-hop from 76 to 83 and single-hop
from 96 to 98; on test, run once, multi-hop 76 / 76 / 51 and single-hop
97 / 97 / 94. Pooled, 96.5 / 71.5, counting every unanswered question as
wrong.
Against the reference points: CAR's gpt-4o pipeline reports 94.8 / 51.5
pooled and 93 / 41 at 262K; this resolver reports 96.5 / 71.5 pooled and
94 / 51 at 262K with no reader at all. The unsourced ``91 / 57'' is exceeded
94 / 51 at 262K with no reader and, in these runs, no compiler model either. The unsourced ``91 / 57'' is exceeded
on both columns pooled, and we make no claim against it beyond that. Pith
reports multi-hop 95 / 83 / 84 / 68 at 6K / 32K / 64K / 262K under its own
protocol \cite{pith}, a four-lane mean near 82.5; this resolver is below it