mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-29 07:34:31 +00:00
zeekayandHanzo Dev
2b594ab2c9
A rerun of the measurement at hanzoai/semantic v0.2.4 found the result sound and some of the prose around it not. Three statements were false. The subject resolver matches any name the conversation uses and not only the two speakers, so 11 questions fall back to searching everything, not 22. Deleting the derived facts moves six more cells than the sentence admitted, and moves them both ways: single-hop F1 up a thousandth, reach down two. And constant abstention at 0.228 is above every arm that has to retrieve, but not above the oracle's 0.252, which the paper's own Table 1 prints two pages later. Two claims outran their evidence. "At every matched silence rate" generalized from four wide-gap points past an 8% crossing where the gap is 0.002, and below that rate the flat index cannot be matched at all; the sentence now says which rates were measured and what happens at the quiet end. And the adversarial questions carry no reference answer — the field the dataset gives them holds the distractor, "yoga" for the worked example — so the paper no longer describes a reference saying the conversation does not say. "Scoped" named both a specific arm and a general property. Arms are now set in typewriter and mean one thing each, subject scope is the factor, and the abstract says which effect +0.152 is. Differences are taken from the unrounded scores and rounded once, which §2.2 states and which moves the claim-span effect to -0.029 and the interaction to +0.179. Section 13 claimed seven commands produced every number, and at least six sets came from nowhere in the repository. The harness now computes them: -stats reads a predictions file back for the constant-abstention score, the AUCs and the bootstrap intervals, and the run prints sessions, build totals and the subject counts. The commands missing from the table are in it, and the four figures that no command produces say where they come from. The measurement, the tables and the conclusion are unchanged. Five interval endpoints move by a thousandth, because the bootstrap is now enumerated over all 92,378 resamples instead of sampling 10,000 of them. Co-authored-by: Hanzo Dev <[email protected]>
62 lines
2.6 KiB
TeX
62 lines
2.6 KiB
TeX
\begin{figure}[t]
|
|
\centering
|
|
\begin{tikzpicture}
|
|
\begin{axis}[
|
|
width=0.98\linewidth, height=7.2cm,
|
|
xlabel={F1 on the 446 adversarial questions},
|
|
ylabel={F1 on the 1{,}540 answerable questions},
|
|
xmin=0, xmax=1.0, ymin=0.025, ymax=0.175,
|
|
xtick={0,0.2,0.4,0.6,0.8,1.0},
|
|
ytick={0.04,0.06,0.08,0.10,0.12,0.14,0.16},
|
|
grid=both, grid style={hzslate!12, line width=0.2pt},
|
|
scaled y ticks=false,
|
|
y tick label style={/pgf/number format/fixed,
|
|
/pgf/number format/fixed zerofill,
|
|
/pgf/number format/precision=2},
|
|
tick label style={font=\scriptsize},
|
|
label style={font=\small},
|
|
legend style={font=\scriptsize, at={(0.03,0.06)}, anchor=south west,
|
|
draw=hzslate!30, fill=white, row sep=0.5pt},
|
|
legend cell align=left,
|
|
every axis plot/.append style={line width=0.8pt, mark size=1.6pt},
|
|
]
|
|
|
|
% floor = 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.8
|
|
\addplot[hzblue, mark=*] coordinates {
|
|
(0.047,0.152) (0.070,0.152) (0.217,0.151) (0.502,0.142)
|
|
(0.693,0.126) (0.845,0.103) (0.948,0.080) (0.975,0.038)};
|
|
\addlegendentry{\texttt{graph} \; (span, scoped)}
|
|
|
|
\addplot[hzgreen, mark=square*] coordinates {
|
|
(0.043,0.154) (0.078,0.154) (0.173,0.152) (0.352,0.144)
|
|
(0.552,0.130) (0.738,0.110) (0.861,0.086) (0.951,0.040)};
|
|
\addlegendentry{\texttt{scoped} \; (turn, scoped)}
|
|
|
|
\addplot[hzslate, mark=triangle*, densely dashed] coordinates {
|
|
(0.092,0.149) (0.101,0.149) (0.132,0.148) (0.200,0.144)
|
|
(0.323,0.131) (0.491,0.115) (0.668,0.092) (0.879,0.043)};
|
|
\addlegendentry{\texttt{vector} \; (turn, flat)}
|
|
|
|
\addplot[hzred, mark=diamond*, densely dotted] coordinates {
|
|
(0.036,0.158) (0.038,0.158) (0.063,0.158) (0.170,0.151)
|
|
(0.339,0.136) (0.536,0.114) (0.700,0.090) (0.886,0.042)};
|
|
\addlegendentry{\texttt{span} \; (span, flat)}
|
|
|
|
\draw[hzslate!55, line width=0.3pt, densely dashed]
|
|
(axis cs:0.0,0.143) -- (axis cs:0.56,0.143);
|
|
\node[font=\scriptsize, color=hzslate!90, anchor=west]
|
|
at (axis cs:0.575,0.143) {at equal answerable F1};
|
|
\end{axis}
|
|
\end{tikzpicture}
|
|
\caption{The trade each index can make, swept over the abstention threshold
|
|
(\texttt{floor} $=0,\,0.1,\,0.2,\,0.3,\,0.4,\,0.5,\,0.6,\,0.8$; the marked
|
|
point fourth from the left on each curve is the operating point of
|
|
Table~\ref{tab:main}). These are four curves, not four points on one curve. At
|
|
that operating point the four arms lie within $0.009$ of each other on the
|
|
answerable half --- $0.142$, $0.144$, $0.144$ and \texttt{span}'s $0.151$,
|
|
against the dashed line at $0.143$ --- while the two subject-scoped indexes
|
|
reach $0.502$ and $0.352$ on the adversarial questions where the flat ones reach
|
|
$0.200$ and $0.170$.}
|
|
\label{fig:frontier}
|
|
\end{figure}
|