mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-30 14:58:42 +00:00
zeekayandHanzo Dev
2b594ab2c9
A rerun of the measurement at hanzoai/semantic v0.2.4 found the result sound and some of the prose around it not. Three statements were false. The subject resolver matches any name the conversation uses and not only the two speakers, so 11 questions fall back to searching everything, not 22. Deleting the derived facts moves six more cells than the sentence admitted, and moves them both ways: single-hop F1 up a thousandth, reach down two. And constant abstention at 0.228 is above every arm that has to retrieve, but not above the oracle's 0.252, which the paper's own Table 1 prints two pages later. Two claims outran their evidence. "At every matched silence rate" generalized from four wide-gap points past an 8% crossing where the gap is 0.002, and below that rate the flat index cannot be matched at all; the sentence now says which rates were measured and what happens at the quiet end. And the adversarial questions carry no reference answer — the field the dataset gives them holds the distractor, "yoga" for the worked example — so the paper no longer describes a reference saying the conversation does not say. "Scoped" named both a specific arm and a general property. Arms are now set in typewriter and mean one thing each, subject scope is the factor, and the abstract says which effect +0.152 is. Differences are taken from the unrounded scores and rounded once, which §2.2 states and which moves the claim-span effect to -0.029 and the interaction to +0.179. Section 13 claimed seven commands produced every number, and at least six sets came from nowhere in the repository. The harness now computes them: -stats reads a predictions file back for the constant-abstention score, the AUCs and the bootstrap intervals, and the run prints sessions, build totals and the subject counts. The commands missing from the table are in it, and the four figures that no command produces say where they come from. The measurement, the tables and the conclusion are unchanged. Five interval endpoints move by a thousandth, because the bootstrap is now enumerated over all 92,378 resamples instead of sampling 10,000 of them. Co-authored-by: Hanzo Dev <[email protected]>
43 lines
1.9 KiB
TeX
43 lines
1.9 KiB
TeX
\begin{figure}[t]
|
|
\centering
|
|
\begin{tikzpicture}
|
|
\begin{axis}[
|
|
width=0.98\linewidth, height=6.4cm,
|
|
ybar=2pt, bar width=10pt,
|
|
ymin=0, ymax=1.18,
|
|
ylabel={share of annotated evidence reachable},
|
|
symbolic x coords={vector, span, scoped, graph, oracle},
|
|
xtick=data,
|
|
ytick={0,0.2,0.4,0.6,0.8,1.0},
|
|
tick label style={font=\scriptsize},
|
|
xticklabel style={font=\scriptsize\ttfamily},
|
|
label style={font=\small},
|
|
grid=major, grid style={hzslate!12, line width=0.2pt},
|
|
legend style={font=\scriptsize, at={(0.5,1.06)}, anchor=south,
|
|
legend columns=2, draw=hzslate!30, fill=white},
|
|
legend cell align=left,
|
|
nodes near coords,
|
|
every node near coord/.append style={font=\fontsize{6}{7}\selectfont,
|
|
color=hzslate!90, rotate=90, anchor=west, inner sep=1.5pt,
|
|
/pgf/number format/fixed, /pgf/number format/fixed zerofill,
|
|
/pgf/number format/precision=3},
|
|
]
|
|
\addplot[draw=hzblue!70, fill=hzblue!25] coordinates {
|
|
(vector,0.992) (span,0.863) (scoped,0.885) (graph,0.756) (oracle,0.996)};
|
|
\addlegendentry{1{,}540 answerable questions}
|
|
\addplot[draw=hzred!70, fill=hzred!22] coordinates {
|
|
(vector,0.964) (span,0.916) (scoped,0.350) (graph,0.269) (oracle,1.000)};
|
|
\addlegendentry{446 adversarial questions}
|
|
\end{axis}
|
|
\end{tikzpicture}
|
|
\caption{What each index can return at all, with the result limit lifted past
|
|
the number of turns in the conversation ($k=2000$, no abstention). This is
|
|
reach, not ranking, and it owes nothing to the reader. The two flat indexes
|
|
reach almost every annotated turn of both kinds. Against \texttt{vector}, the
|
|
two subject-scoped indexes give up 11 and 24 points of the answerable evidence
|
|
and 61 and 70 points of the adversarial evidence: they are selectively blind,
|
|
and the blindness falls where the trap turns are. \texttt{oracle} is handed the
|
|
annotated evidence and is shown as the frame of the measurement.}
|
|
\label{fig:reach}
|
|
\end{figure}
|