mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-29 07:34:31 +00:00
zeekay
3c1a949bd1
A graph-structured memory beats a flat turn index on LoCoMo's adversarial questions, 0.502 against 0.200, and the paper is mostly about what that is not. The aggregate usually quoted for the benchmark is degenerate: constant abstention scores 0.228 and beats every system, because a quarter of the questions award a point for declining. Emptying the knowledge graph leaves every figure byte-identical. Removing the inference layer, which derived 44 facts from 23,942 claims, moves nothing at three decimals. What is left is a 2x2 over what is indexed against what a query may reach. Subject scope alone is worth +0.152, the claim-span unit alone -0.030, their interaction +0.180. The mechanism is reader-independent: the scoped index can reach 27% of adversarial evidence against the flat index's 96%, and only the scoped index's retrieval confidence carries any signal about whether a question is answerable at all (AUC 0.698 against 0.514, chance). Make now rebuilds a paper when its figures, sections, bibliography, local style file or the shared preamble change. It did not, so editing a figure left `make` reporting a stale PDF up to date. The touch after a successful run records what latexmk established, so make and latexmk agree on what is current.
36 lines
1.6 KiB
TeX
36 lines
1.6 KiB
TeX
\begin{figure}[t]
|
|
\centering
|
|
\begin{tikzpicture}[
|
|
font=\small,
|
|
cell/.style={draw=hzslate!45, rounded corners=2pt, minimum width=35mm,
|
|
minimum height=14mm, align=center, inner sep=2pt},
|
|
base/.style={cell, fill=hzslate!10},
|
|
new/.style={cell, fill=hzblue!9, draw=hzblue!50},
|
|
hd/.style={hzslate!85, font=\small\bfseries, align=center}
|
|
]
|
|
\newcommand{\gloss}[1]{{\scriptsize\color{hzslate!85}#1}}
|
|
|
|
\node[hd] at (2.05, 1.35) {unrestricted};
|
|
\node[hd] at (6.15, 1.35) {scoped to\\the subject};
|
|
|
|
\node[hd, anchor=east] at (0.05, 0.0) {whole\\turn};
|
|
\node[hd, anchor=east] at (0.05,-1.75) {claim\\span};
|
|
|
|
\node[base] at (2.05, 0.0) {\texttt{vector}\\[2pt]\gloss{the usual baseline}};
|
|
\node[new] at (6.15, 0.0) {\texttt{scoped}\\[2pt]\gloss{control}};
|
|
\node[new] at (2.05,-1.75) {\texttt{span}\\[2pt]\gloss{control}};
|
|
\node[base] at (6.15,-1.75) {\texttt{graph}\\[2pt]\gloss{the system under test}};
|
|
|
|
\draw[hzslate!25, line width=0.3pt] (0.25,-0.875) -- (7.95,-0.875);
|
|
\draw[hzslate!25, line width=0.3pt] (4.10, 0.85) -- (4.10,-2.60);
|
|
\end{tikzpicture}
|
|
\caption{The four indexes as a $2\times2$: the unit that is indexed down the
|
|
side, what a query may reach across the top. \arm{vector} and \arm{graph} are
|
|
the comparison usually drawn, and they differ in both factors at once. The two
|
|
middle cells are the controls that say which factor a difference belongs to.
|
|
Every cell ranks the same weighted terms by cosine through the same store, is
|
|
read by the same reader, and is marked by the same scorer, so a difference
|
|
between cells is a difference of index and of nothing else.}
|
|
\label{fig:design}
|
|
\end{figure}
|