mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-22 17:20:17 +00:00
Paper 1: the search ran over lexicon plans; no model at any point in those rows
This commit is contained in:
@@ -558,8 +558,11 @@ phrasings. Parse coverage, entity lookup and version selection were not the
|
||||
problem. The second run keeps the exact single-hop templates as rules, has a
|
||||
model compile every other question into the same plan shape, validated
|
||||
against the relation set, and uses the lexicon only where the model's plan
|
||||
fails validation. That run scored 59 / 59 / 33 on multi-hop at 32K / 64K
|
||||
/ 262K.
|
||||
fails validation. On the shared GPU every model call of that pass timed
|
||||
out and the lexicon filled every plan, so its plans were the lexicon's; it
|
||||
scored 59 / 59 / 33 on multi-hop at 32K / 64K / 262K, the gain over the
|
||||
first run being the fixed plan policy for single-hop and a corrected
|
||||
lexicon, not a model.
|
||||
|
||||
The third resolver treats the plan as a sketch. An oracle search over the
|
||||
parsed graph (\texttt{mab/oracle.mjs}) found a path from the plan's base
|
||||
@@ -580,15 +583,17 @@ question names — a city, a country, a person, a language — as the final
|
||||
constraint, deciding between stopping a hop early and going one further.
|
||||
The best complete path wins by a fixed score; ties go to the more literal
|
||||
plan and the later version. It is deterministic, it reads no gold, and it
|
||||
calls no model: the compiled plan is the only thing a model produced, once,
|
||||
at temperature 0. On dev it moved multi-hop from 76 to 83 and single-hop
|
||||
calls no model. In the runs of Table~\ref{tab:mab} the plans it searched
|
||||
over were the lexicon's (\texttt{meta.json} records the planner of every
|
||||
run), so no language model took part at any point; model-compiled plan sets
|
||||
are the next rows, labelled as such. On dev it moved multi-hop from 76 to 83 and single-hop
|
||||
from 96 to 98; on test, run once, multi-hop 76 / 76 / 51 and single-hop
|
||||
97 / 97 / 94. Pooled, 96.5 / 71.5, counting every unanswered question as
|
||||
wrong.
|
||||
|
||||
Against the reference points: CAR's gpt-4o pipeline reports 94.8 / 51.5
|
||||
pooled and 93 / 41 at 262K; this resolver reports 96.5 / 71.5 pooled and
|
||||
94 / 51 at 262K with no reader at all. The unsourced ``91 / 57'' is exceeded
|
||||
94 / 51 at 262K with no reader and, in these runs, no compiler model either. The unsourced ``91 / 57'' is exceeded
|
||||
on both columns pooled, and we make no claim against it beyond that. Pith
|
||||
reports multi-hop 95 / 83 / 84 / 68 at 6K / 32K / 64K / 262K under its own
|
||||
protocol \cite{pith}, a four-lane mean near 82.5; this resolver is below it
|
||||
|
||||
Reference in New Issue
Block a user