hanzoai/papers
2
0
Fork 0
mirror of https://github.com/hanzoai/papers.git synced 2026-10-22 17:20:17 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

LoCoMo: every claim held to what the harness prints

Browse Source 
A rerun of the measurement at hanzoai/semantic v0.2.4 found the result sound
and some of the prose around it not. Three statements were false. The subject
resolver matches any name the conversation uses and not only the two speakers,
so 11 questions fall back to searching everything, not 22. Deleting the derived
facts moves six more cells than the sentence admitted, and moves them both
ways: single-hop F1 up a thousandth, reach down two. And constant abstention at
0.228 is above every arm that has to retrieve, but not above the oracle's
0.252, which the paper's own Table 1 prints two pages later.

Two claims outran their evidence. "At every matched silence rate" generalized
from four wide-gap points past an 8% crossing where the gap is 0.002, and below
that rate the flat index cannot be matched at all; the sentence now says which
rates were measured and what happens at the quiet end. And the adversarial
questions carry no reference answer — the field the dataset gives them holds
the distractor, "yoga" for the worked example — so the paper no longer
describes a reference saying the conversation does not say.

"Scoped" named both a specific arm and a general property. Arms are now set in
typewriter and mean one thing each, subject scope is the factor, and the
abstract says which effect +0.152 is. Differences are taken from the unrounded
scores and rounded once, which §2.2 states and which moves the claim-span
effect to -0.029 and the interaction to +0.179.

Section 13 claimed seven commands produced every number, and at least six sets
came from nowhere in the repository. The harness now computes them: -stats
reads a predictions file back for the constant-abstention score, the AUCs and
the bootstrap intervals, and the run prints sessions, build totals and the
subject counts. The commands missing from the table are in it, and the four
figures that no command produces say where they come from.

The measurement, the tables and the conclusion are unchanged. Five interval
endpoints move by a thousandth, because the bootstrap is now enumerated over
all 92,378 resamples instead of sampling 10,000 of them.

Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
zeekay
2026-09-09 14:00:10 -07:00
co-authored by Hanzo Dev
parent 3c1a949bd1
commit 2b594ab2c9
4 changed files with 218 additions and 140 deletions
Download Patch File Download Diff File Expand all files Collapse all files
locomo-subject-scope/figures/frontier.tex
+5 -3
View File
@@ -52,8 +52,10 @@
(\texttt{floor} $=0,\,0.1,\,0.2,\,0.3,\,0.4,\,0.5,\,0.6,\,0.8$; the marked
point fourth from the left on each curve is the operating point of
Table~\ref{tab:main}). These are four curves, not four points on one curve. At
the answerable F1 the arms all sit near ($y \approx 0.143$, dashed), the
subject-scoped indexes reach $0.502$ and $0.352$ on the adversarial questions
where the flat ones reach $0.200$ and $0.170$.}
that operating point the four arms lie within $0.009$ of each other on the
answerable half --- $0.142$, $0.144$, $0.144$ and \texttt{span}'s $0.151$,
against the dashed line at $0.143$ --- while the two subject-scoped indexes
reach $0.502$ and $0.352$ on the adversarial questions where the flat ones reach
$0.200$ and $0.170$.}
\label{fig:frontier}
\end{figure}
locomo-subject-scope/figures/reach.tex
+4 -4
View File
@@ -33,10 +33,10 @@
\caption{What each index can return at all, with the result limit lifted past
the number of turns in the conversation ($k=2000$, no abstention). This is
reach, not ranking, and it owes nothing to the reader. The two flat indexes
reach almost every annotated turn of both kinds. The two subject-scoped indexes
give up a fifth to a quarter of the answerable evidence and two-thirds to
three-quarters of the adversarial evidence: they are selectively blind, and the
blindness falls where the trap turns are. \texttt{oracle} is handed the
reach almost every annotated turn of both kinds. Against \texttt{vector}, the
two subject-scoped indexes give up 11 and 24 points of the answerable evidence
and 61 and 70 points of the adversarial evidence: they are selectively blind,
and the blindness falls where the trap turns are. \texttt{oracle} is handed the
annotated evidence and is shown as the frame of the measurement.}
\label{fig:reach}
\end{figure}
locomo-subject-scope/main.tex
+208 -132
View File
@@ -31,7 +31,7 @@ and the negative controls that locate the effect}}
Los Angeles, California \\
\texttt{https://hanzo.ai}
}
\date{Measurement pass 2026-09-09 \quad \texttt{hanzoai/semantic} v0.2.3}
\date{Measurement pass 2026-09-09 \quad \texttt{hanzoai/semantic} v0.2.4}
\begin{document}
@@ -44,38 +44,41 @@ We measure a graph-structured conversational memory against a flat
nearest-neighbour index on LoCoMo~\cite{locomo} --- 10 conversations, 272
sessions, 5{,}882 turns, 1{,}986 questions --- using the benchmark's own
scorer, ported and verified against the original Python on all 1{,}986
questions and four systems. Three results follow, and the negative controls are the
substance of the paper.
questions and every system reported here. Three results follow, and the negative
controls are the substance of the paper.
First, the aggregate over all five question categories is degenerate. A system
that replies ``No information available'' to every question scores $0.228$,
which beats every system we ran. A quarter of LoCoMo's questions are
adversarial and the official metric awards a point for declining them, so any
mean over the full set measures the abstention rate before it measures the
memory. The answerable and adversarial halves have to be reported apart.
above every system here that has to find its own evidence; only an oracle
handed the dataset's own annotated evidence scores higher, at $0.252$. A
quarter of LoCoMo's questions are adversarial and the official metric awards a
point for declining them, so any mean over the full set measures the abstention
rate before it measures the memory. The answerable and adversarial halves have
to be reported apart.
Second, the effect that survives is subject scope. Restricting retrieval to
the person a question names scores $0.502$ against $0.200$ on the 446
adversarial questions (95\% CI over conversations $[+0.247, +0.346]$) at no
adversarial questions (95\% CI over conversations $[+0.247, +0.347]$) at no
measurable cost on the 1{,}540 answerable ones ($0.142$ against $0.144$, CI
$[-0.011, +0.006]$). The mechanism is reader-independent: with the result limit
lifted past the size of the conversation, the scoped index can reach $76\%$ of
$[-0.010, +0.006]$). The mechanism is reader-independent: with the result limit
lifted past the size of the conversation, the graph memory can reach $76\%$ of
the annotated evidence for answerable questions and $27\%$ for adversarial ones,
where the flat index reaches $99\%$ and $96\%$. It is selectively blind, and
the blindness falls where the trap turns are. Abstention is the variable this
runs through: the flat index's retrieval confidence separates answerable from
adversarial questions at AUC $0.514$, which is chance; the scoped index's
adversarial questions at AUC $0.514$, which is chance; the graph memory's
separates them at $0.698$.
Third, the controls remove most of the explanation the system's name suggests.
Emptying the knowledge graph entirely leaves every reported figure
byte-identical --- retrieval never consults it. Deleting the inference step,
which derived 44 facts from 23{,}942 extracted claims, moves nothing at three
decimals. A $2\times2$ over what is indexed against what a query may reach
attributes $+0.152$ of the adversarial gain to subject scope alone, $-0.030$ to
the claim-span unit alone, and $+0.180$ to their interaction. An oracle handed
which derived 44 facts from 23{,}942 extracted claims, moves no score on either
half and no figure anywhere by more than two thousandths. A $2\times2$ over what is indexed against what a query may reach
attributes $+0.152$ of the adversarial gain to subject scope applied to the
unchanged turn index, $-0.029$ to the claim-span unit applied on its own, and
$+0.179$ to their interaction. An oracle handed
the dataset's own annotated evidence scores \emph{worse} on adversarial
questions than the scoped index does ($0.426$ against $0.502$), because on those
questions than the graph memory does ($0.426$ against $0.502$), because on those
questions the trap turn is the annotated evidence.
No language model was called, in either index or the reader, so absolute F1 is
@@ -124,7 +127,7 @@ smaller and more specific than the construction:
(\S\ref{sec:degenerate}).
\item The graph itself is inert. Emptying it changes nothing
(\S\ref{sec:control-graph}).
\item The inference step is inert. Removing it changes nothing
\item The inference step is inert. Removing it moves no score on either half
(\S\ref{sec:control-reason}).
\item What remains is a partition of the index by the subject of a sentence and
a filter on the query, and a $2\times2$ locates the effect there
@@ -170,8 +173,12 @@ answerable & the first four, together & 1{,}540 \\
The adversarial category is the one that structures this paper. Its questions
are not nonsense and not out-of-domain. They are questions about the wrong
person: an activity that speaker~A described is attributed to speaker~B, who
was present, asked about it, and never claimed it. The reference answer is that
the conversation does not say. LoCoMo's construction here follows the
was present, asked about it, and never claimed it. Only two of the 446 carry a
reference answer at all. The rest carry a field \texttt{adversarial\_answer}
holding the plausible wrong one --- for the worked example of
\S\ref{sec:result} it is literally \emph{yoga} --- and the scorer reads neither
field (\S\ref{sec:metric}). What makes a reply right on this category is that it
declines. LoCoMo's construction here follows the
unanswerable-question design of SQuAD~2.0~\cite{squad2}, where a plausible
distractor is present in the passage and the correct response is abstention.
@@ -201,15 +208,23 @@ original three ways --- the stem of all 6{,}597 words in the corpus, the
normalisation of all 10{,}382 distinct questions, answers and turns, and the
score of 2{,}400 replies spanning every category and every branch --- and then
end to end: the original Python is run over the same predictions file the Go
writes. Every figure in this paper agrees between the two to three decimals,
across 1{,}986 questions and four systems. The original scorer is fetched at
test time rather than vendored, because LoCoMo is CC BY-NC 4.0 and our
repository is MIT.
writes. Every figure in Tables~\ref{tab:main} and \ref{tab:factorial} agrees
between the two to three decimals, across 1{,}986 questions and all six systems.
That end-to-end run needs one adaptation, which lives in our script and not in
the vendored file: the official recall reads the first retrieved identifier to
tell a session id from a dialogue id, and a memory that returned nothing has no
first identifier, so a placeholder matching no evidence is substituted for it.
Without the substitution the official code awards recall $1.0$ to a system that
returned nothing. The original scorer is fetched at test time rather than
vendored, because LoCoMo is CC BY-NC 4.0 and our repository is MIT.
We report three quantities throughout. \textbf{F1} is the official metric above.
\textbf{Reach} is the share of a question's annotated evidence turns the index
returned; it is a property of retrieval alone and owes nothing to the reader.
\textbf{Silence} is how often a system declined to answer.
\textbf{Silence} is how often a system declined to answer. A difference between
two systems is computed from the unrounded per-question scores and rounded
once, so it need not equal the difference of the two rounded figures printed in
a table.
\subsection{The reader, and the ceiling it imposes}
\label{sec:reader}
@@ -243,7 +258,7 @@ we hold to both:
\end{enumerate}
The reader's other settings do not matter much. Swept over all ten
conversations, the span of words it may take moves the scoped index's overall
conversations, the span of words it may take moves \arm{graph}'s overall
score from $0.215$ to $0.224$, the band within which a second piece of evidence
counts moves it from $0.220$ to $0.223$, and the number of pieces from $0.220$
to $0.223$. The two settings that do matter, the result limit $k$ and the
@@ -304,9 +319,12 @@ stops a topic leaking from the person who raised it to the person who answered.
And a negation is kept on the predicate, so ``I do not eat dairy'' is findable
and is not mistaken for its opposite.
Resolving the subject of a \emph{question} is easier: 1{,}964 of the 1{,}986
LoCoMo questions ($98.9\%$) name one of the two speakers, and the longest known name the question
contains is taken as its subject. The remaining $1.1\%$ name nobody and fall
Resolving the subject of a \emph{question} is easier. The subject is the
longest name the memory knows that the question contains, and the names it
knows are the two speakers together with everyone the conversation mentions by
name. 1{,}964 of the 1{,}986 LoCoMo questions ($98.9\%$) name one of the two
speakers; 1{,}975 ($99.45\%$) name someone the memory knows and so resolve to a
subject. The remaining 11 questions ($0.55\%$) name nobody it knows and fall
back to searching everything, which is the honest behaviour for a memory asked
about a stranger.
@@ -314,8 +332,8 @@ about a stranger.
23{,}942 claims, folded into 14{,}810 nodes and 21{,}357 edges. The inference
engine, running one rule to closure depth one, derives 44 further facts.
Building both memories over all ten conversations and answering all 1{,}986
questions four ways costs 5.6\,s wall, 5.0\,s user and 254\,MB peak resident on
an Apple-silicon laptop.
questions four ways costs 5.2\,s of user time and 254\,MB peak resident on an
Apple-silicon laptop; wall time depends on what else the machine is doing.
\section{The aggregate over all five categories is degenerate}
\label{sec:degenerate}
@@ -327,9 +345,10 @@ for that phrase and does not consult the reference --- and $0.004$ on the
1{,}540 answerable ones, where the phrase occasionally shares a stem with the
answer.
$0.228$ beats every system in this paper on the same aggregate. It beats the
best of them, which scores $0.223$, and the oracle handed perfect evidence,
which scores $0.252$, is only $0.024$ above a system with no memory at all.
On the same aggregate $0.228$ is above every system in this paper that has to
find its own evidence: the best of those scores $0.223$. Only \arm{oracle},
handed the annotated evidence instead of retrieving it, is higher, at $0.252$
--- which puts perfect retrieval $0.024$ above a system with no memory at all.
The arithmetic is not subtle. Adversarial questions are $22.5\%$ of the set and
their score is a monotone function of the silence rate, reaching $1.000$ at
@@ -384,21 +403,24 @@ answerable & 1{,}540 & \textbf{0.142} & 0.444 & 25\% & 0.107 & 0.444 & 23\% & 0
\end{table}
Table~\ref{tab:main} is the whole measurement at one operating point. Read the
two rows that matter. On the 1{,}540 answerable questions the scoped and flat
indexes are level, $0.142$ against $0.144$. On the 446 adversarial ones they
separate, $0.502$ against $0.200$.
two rows that matter. On the 1{,}540 answerable questions \arm{graph} and
\arm{vector} are level, $0.142$ against $0.144$. On the 446 adversarial ones
they separate, $0.502$ against $0.200$.
Table~\ref{tab:ci} attaches intervals. Because the 1{,}986 questions are not
independent --- every question within a conversation is answered out of the
same memory --- the resampling unit is the conversation, not the question. We
report the percentile bootstrap~\cite{bootstrap} over 10{,}000 resamples of the
ten conversations with replacement.
report the percentile bootstrap~\cite{bootstrap} over the ten conversations
drawn with replacement, enumerated rather than sampled: there are
$\binom{19}{10} = 92{,}378$ such draws, each is weighted by how often it comes
up, and the interval is read off that distribution directly. It therefore
carries no Monte Carlo error and does not depend on a seed.
\begin{table}[t]
\centering
\small
\caption{Paired differences against \arm{vector}, with 95\% percentile bootstrap
intervals over the ten conversations (10{,}000 resamples). The adversarial gain
\caption{Paired differences, with 95\% percentile bootstrap intervals over the
ten conversations, enumerated exactly rather than sampled. The adversarial gain
is large and its interval excludes zero; the answerable difference is
indistinguishable from zero, subject to the reader ceiling of
\S\ref{sec:reader}.}
@@ -409,22 +431,22 @@ indistinguishable from zero, subject to the reader ceiling of
\cmidrule(lr){2-3}\cmidrule(lr){4-5}
difference & $\Delta$ & 95\% CI & $\Delta$ & 95\% CI \\
\midrule
\arm{graph} $-$ \arm{vector} & $+0.303$ & $[+0.247, +0.346]$ & $-0.002$ & $[-0.011, +0.006]$ \\
\arm{scoped} $-$ \arm{vector} & $+0.152$ & $[+0.107, +0.189]$ & $-0.000$ & $[-0.007, +0.007]$ \\
\arm{graph} $-$ \arm{vector} & $+0.303$ & $[+0.247, +0.347]$ & $-0.002$ & $[-0.010, +0.006]$ \\
\arm{scoped} $-$ \arm{vector} & $+0.152$ & $[+0.106, +0.189]$ & $-0.000$ & $[-0.007, +0.007]$ \\
\arm{span} $-$ \arm{vector} & $-0.029$ & $[-0.058, -0.004]$ & $+0.007$ & $[-0.000, +0.017]$ \\
\arm{graph} $-$ \arm{scoped} & $+0.150$ & $[+0.107, +0.190]$ & $-0.002$ & $[-0.008, +0.004]$ \\
\arm{oracle} $-$ \arm{vector} & $+0.226$ & $[+0.160, +0.276]$ & $+0.058$ & $[+0.048, +0.069]$ \\
\arm{oracle} $-$ \arm{vector} & $+0.226$ & $[+0.160, +0.275]$ & $+0.058$ & $[+0.048, +0.068]$ \\
\bottomrule
\end{tabular}
\end{table}
\paragraph{One question, four answers.} The separation is legible in a single
case, from \texttt{conv-43}. Turn \texttt{D20:2} is John saying \emph{``Congrats
on your success! I'm also trying out yoga to get a little extra strength and
flexibility.''} The question is \emph{``What is Tim trying out to improve his
strength and flexibility after recovery from ankle injury?''} John does yoga;
Tim asked him about it; the reference answer is that the conversation does not
say.
case, from \texttt{conv-43}. Turn \texttt{D20:2} is John saying \emph{``Hi Tim!
Congrats on your success! \ldots\ I'm also trying out yoga to get a little
extra strength and flexibility.''} The question is \emph{``What is Tim trying
out to improve his strength and flexibility after recovery from ankle
injury?''} John does yoga; Tim asked him about it; the question is adversarial,
so the reply that scores is one that declines.
\begin{center}
\small
@@ -434,17 +456,19 @@ system & reply \\
\midrule
\arm{vector} & \texttt{yoga get little extra} \quad (retrieved \texttt{D20:2} first) \\
\arm{graph} & \texttt{No information available} \quad (never retrieved \texttt{D20:2}) \\
\arm{graph+edge} & \texttt{No information available} \quad (the same evidence as \arm{graph}) \\
\arm{oracle} & \texttt{yoga get little extra} \quad (handed \texttt{D20:2}) \\
\bottomrule
\end{tabular}
\end{center}
\texttt{D20:2} is the nearest turn in the conversation to the question's words,
and the flat index is right about that and wrong about the question. The scoped
index never sees \texttt{D20:2}, because the sentence carrying \emph{yoga} has
John as its subject and the span was filed under John. Asked about Tim, it finds
nothing above the floor and says so. The oracle answers wrongly for the same
reason as the flat index, from better evidence --- see \S\ref{sec:oracle}.
and \arm{vector} is right about that and wrong about the question.
\arm{graph} never sees \texttt{D20:2}, because the sentence carrying
\emph{yoga} has John as its subject and the span was filed under John. Asked
about Tim, it finds nothing above the floor and says so. \arm{oracle} answers
wrongly for the same reason \arm{vector} does, from better evidence --- see
\S\ref{sec:oracle}.
\section{Mechanism: what each index can reach}
\label{sec:mechanism}
@@ -456,22 +480,24 @@ column stops measuring ranking and starts measuring \emph{reach}: what an index
can return at all, however hard it is asked. Figure~\ref{fig:reach} is that
measurement.
\emph{Measured.} At $k=2000$, the flat turn index reaches $0.992$ of the
annotated evidence for answerable questions and $0.964$ for adversarial ones.
The claim-span index with the filter removed reaches $0.863$ and $0.916$. Adding
the subject restriction changes the second number and not much else: whole turns
scoped by person reach $0.885$ and $0.350$; claim spans scoped by person reach
$0.756$ and $0.269$.
\emph{Measured.} At $k=2000$, the flat turn index (\arm{vector}) reaches
$0.992$ of the annotated evidence for answerable questions and $0.964$ for
adversarial ones. The claim-span index with the filter removed (\arm{span})
reaches $0.863$ and $0.916$. Adding the subject restriction changes the second
number and not much else: whole turns scoped by person (\arm{scoped}) reach
$0.885$ and $0.350$; claim spans scoped by person (\arm{graph}) reach $0.756$
and $0.269$.
\input{figures/reach}
\emph{Interpretation.} The subject restriction makes the index selectively
blind, and the blindness is concentrated where the trap turns are. It costs
$11$--$24$ points of answerable evidence, which is the price of a rule-based
extractor: a sentence whose subject it fails to resolve produces no span for
that person, and the turn becomes unreachable through them. It removes $61$--$70$
points of adversarial evidence, which is the point, because an adversarial
question's annotated evidence is a turn about somebody else.
blind, and the blindness is concentrated where the trap turns are. Measured
against \arm{vector}, it costs $11$ points of answerable evidence on whole turns
and $24$ on claim spans, which is the price of a rule-based extractor: a
sentence whose subject it fails to resolve produces no span for that person, and
the turn becomes unreachable through them. It removes $61$ and $70$ points of
adversarial evidence, which is the point, because an adversarial question's
annotated evidence is a turn about somebody else.
Stated precisely: a flat index has no representation of what a question is
\emph{about}, only of what it says, and an adversarial question says almost
@@ -486,8 +512,8 @@ that the partition be derivable from the text.
Declining is worth a full point on an adversarial question and costs whatever
the answer was worth everywhere else, so a comparison at one abstention
threshold is arguable by construction. In Table~\ref{tab:main} the scoped index
is silent on $31\%$ of questions and the flat one on $20\%$. Two measurements
threshold is arguable by construction. In Table~\ref{tab:main} \arm{graph} is
silent on $31\%$ of questions and \arm{vector} on $20\%$. Two measurements
settle whether that difference is the gain.
\subsection{The abstention decision carries information, or it does not}
@@ -519,10 +545,10 @@ system & indexed unit / reachable & AUC \\
Both flat indexes are at chance. Their confidence is high on an adversarial
question for the same reason it is high on a single-hop question --- a turn
close to the question's wording was found --- and that is the entire content of
the signal. Both scoped indexes are above chance, and the AUC tracks the scope
factor, not the unit factor: changing the unit alone moves it by $0.000$,
adding the scope moves it by $0.121$ on turns and $0.184$ on spans. The oracle is
also at chance. Perfect retrieval confers no ability to tell an answerable
the signal. Both subject-scoped indexes are above chance, and the AUC tracks the
scope factor, not the unit factor: changing the unit alone leaves it at chance,
$0.514$ either way, while adding the scope moves it by $0.120$ on turns and
$0.184$ on spans. The oracle is also at chance. Perfect retrieval confers no ability to tell an answerable
question from an unanswerable one, because an adversarial question does have
retrievable evidence and it is exactly the wrong evidence.
@@ -539,7 +565,8 @@ them apart.
Sweeping the abstention floor traces what each index can achieve, and
Figure~\ref{fig:frontier} plots the answerable score against the adversarial
score along that sweep. The four systems lie on four curves, not on one. At any
answerable score the scoped indexes are far to the right of the flat ones.
answerable score the two subject-scoped indexes are far to the right of the two
flat ones.
Matching the systems on silence rather than on threshold gives the same answer
in the units the objection is usually raised in. Reading each system at whichever
@@ -560,9 +587,14 @@ $\approx$62\% & 0.103 & 0.845 & 0.092 & 0.668 \\
\end{tabular}
\end{center}
At every matched silence rate the scoped index leads on both halves. Either
system can be made as quiet as one likes; only one of them falls silent on the
right questions.
At each of the four matched rates \arm{graph} leads on both halves. The lead
belongs to the middle of the range rather than to all of it. Matched at $8\%$
silence --- \arm{vector} declines no less often than that, even with the floor
at zero --- the adversarial scores are $0.094$ against $0.092$, a gap inside the
between-conversation variation of Table~\ref{tab:ci}, and below that rate
\arm{vector} cannot be matched at all. Either system can be made as quiet
as one likes; over the range where declining is worth anything, only one of them
falls silent on the right questions.
\section{Perfect retrieval is a liability on unanswerable questions}
\label{sec:oracle}
@@ -573,8 +605,8 @@ produces an inversion worth reporting on its own.
On answerable questions the oracle is the best system, as it must be: $0.202$
against $0.142$ and $0.144$, and $0.306$ with abstention off. On adversarial
questions it scores $0.426$, \emph{below} the scoped index's $0.502$, and with
abstention off it scores $0.045$, below even the flat index's $0.092$.
questions it scores $0.426$, \emph{below} \arm{graph}'s $0.502$, and with
abstention off it scores $0.045$, below even \arm{vector}'s $0.092$.
The reason is structural rather than incidental. An adversarial question in
LoCoMo has annotated evidence, and that evidence is the trap turn: the turn
@@ -623,19 +655,26 @@ and this control is what falsified it.
\textbf{Intervention:} delete the derived facts entirely --- run no inference,
index nothing the reasoner produced.
\textbf{Result:} single-hop F1 moves from $0.121$ to $0.122$ and its reach from
$0.507$ to $0.505$; answerable F1 is $0.142$ either way; the aggregate is
$0.223$ either way. Every other cell is unchanged at three decimals. The one
digit that moves, moves in favour of removing the inference.
\textbf{Result:} nothing moves by more than two thousandths, and what moves goes
both ways. Single-hop F1 rises from $0.121$ to $0.122$, while reach falls
wherever the graph's index is read --- single hop $0.507$ to $0.505$, answerable
$0.444$ to $0.443$, overall $0.375$ to $0.374$, in the \arm{graph} and
\arm{graph+edge} columns alike --- and \arm{graph+edge} declines on $22\%$ of
single-hop questions rather than $21\%$. No F1 on either half moves at three
decimals: answerable is $0.142$ either way, adversarial $0.502$, the aggregate
$0.223$. The 44 derived facts are worth two thousandths of single-hop reach, and
that reach does not become an answer.
The engine runs one rule --- what a thing is called is also what its owner has,
so ``I have a snake named Susie'' answers a question about the snake's name ---
to closure depth one, and derives 44 facts from 23{,}942 claims. That ratio is
the finding rather than the rule's weakness. Conversational memory is dominated
by aggregation over identity: the same person mentioned in two sessions is one
node, and that requires no inference at all. Once identity is resolved there is
very little for a reasoner to add. This says where the value in a memory graph
sits, and it is not in the deductive layer.
key --- here the folded name each indexed piece is filed under, which
\S\ref{sec:control-graph} showed is what retrieval actually reads --- and that
requires no inference at all. Once identity is resolved there is very little for
a reasoner to add. This says where the value in a memory graph sits, and it is
not in the deductive layer.
\subsection{The claim-span unit, and where the effect actually is}
\label{sec:factorial}
@@ -648,8 +687,8 @@ them by running all four cells of \S\ref{sec:indexes}.
\small
\caption{The $2\times2$ over indexed unit and reachable set, on the 446
adversarial questions and the 1{,}540 answerable ones. Effects are differences
in F1; the interaction is
$(\arm{graph}-\arm{scoped})-(\arm{span}-\arm{vector})$.}
in F1 computed from the unrounded scores (\S\ref{sec:metric}); the interaction
is $(\arm{graph}-\arm{scoped})-(\arm{span}-\arm{vector})$.}
\label{tab:factorial}
\begin{tabular}{@{}l cc c cc@{}}
\toprule
@@ -663,24 +702,27 @@ claim span & 0.170 & \textbf{0.502} & & 0.151 & 0.142 \\
\multicolumn{6}{@{}l}{\emph{simple effects, adversarial}} \\
\multicolumn{6}{@{}l}{\quad subject scope, on whole turns \hfill $+0.152$} \\
\multicolumn{6}{@{}l}{\quad subject scope, on claim spans \hfill $+0.332$} \\
\multicolumn{6}{@{}l}{\quad claim span, with everything reachable \hfill $-0.030$} \\
\multicolumn{6}{@{}l}{\quad claim span, with everything reachable \hfill $-0.029$} \\
\multicolumn{6}{@{}l}{\quad claim span, with scope in place \hfill $+0.150$} \\
\multicolumn{6}{@{}l}{\emph{main effects and interaction, adversarial}} \\
\multicolumn{6}{@{}l}{\quad subject scope \hfill $+0.242$} \\
\multicolumn{6}{@{}l}{\quad claim span \hfill $+0.060$} \\
\multicolumn{6}{@{}l}{\quad interaction \hfill $+0.180$} \\
\multicolumn{6}{@{}l}{\quad claim span \hfill $+0.061$} \\
\multicolumn{6}{@{}l}{\quad interaction \hfill $+0.179$} \\
\bottomrule
\end{tabular}
\end{table}
Three statements follow, all measured, with intervals in Table~\ref{tab:ci}.
\textbf{Subject scope alone accounts for half the effect.} Adding nothing but
the subject partition to the unchanged turn index takes the adversarial score
from $0.200$ to $0.352$, a gain of $+0.152$ against a total of $+0.302$, with
no movement on the answerable half ($0.144$ to $0.144$). No claim extraction is
needed to index by subject --- the extractor is used here only to decide which
people a turn is about --- and no graph is needed at all.
\textbf{Subject scope on the unchanged turn index accounts for half the
effect.} Adding nothing but the subject partition to the turn index takes the
adversarial score from $0.200$ to $0.352$: a simple effect of $+0.152$ against
a total of $+0.303$, with no movement on the answerable half ($0.144$ to
$0.144$). Averaged over both units the main effect of the same partition is
larger, $+0.242$, because it is worth more on claim spans than on turns;
Table~\ref{tab:factorial} reports both and they are not interchangeable. No
claim extraction is needed to index by subject --- the extractor is used here
only to decide which people a turn is about --- and no graph is needed at all.
\textbf{The claim-span unit alone is harmful.} With everything still reachable,
narrowing the indexed unit takes the adversarial score \emph{down}, from
@@ -692,11 +734,15 @@ index's $0.511$. A short span consisting only of the sentence about yoga
matches a question about yoga better than a whole turn does. Sharpening the match without restricting
what may be matched just makes the trap easier to find.
\textbf{The largest single term is the interaction.} $+0.180$ of the $+0.302$
\textbf{The largest single term is the interaction.} $+0.179$ of the $+0.303$
belongs to neither factor alone. Narrowing the unit is what makes the partition
sharp: a whole turn belongs to every person spoken of within it, so scoping
whole turns still admits the trap turn whenever both speakers appear in it,
while a claim span belongs to exactly one person. The two changes are not
while a claim span belongs to exactly one person. The worked example of
\S\ref{sec:result} is one case of it: \texttt{D20:2} congratulates Tim and then
reports John's own yoga, so the whole turn is filed under both, and \arm{scoped}
retrieves it for the question about Tim and answers \emph{yoga} exactly as
\arm{vector} does. Only the sentence-level unit keeps it out of Tim's pieces. The two changes are not
independent contributions to be added; the second is what makes the first work.
So the honest description of the system is not ``a graph memory'' and not ``an
@@ -709,9 +755,10 @@ enough that the partition is unambiguous.}
\paragraph{Multi-hop, which was the expectation.} A graph memory is supposed to
help most where one person's several answers were given months apart and are
one node's several edges. Retrieval bears that out slightly: multi-hop reach is
$0.228$ against the flat index's $0.194$ at $k=5$, and $0.419$ against $0.388$
under settings that let an answer be a list. F1 does not follow: $0.020$ against
one node's several edges. Retrieval bears that out slightly: \arm{graph}'s
multi-hop reach is $0.228$ against \arm{vector}'s $0.194$ at $k=5$, and $0.419$
against $0.388$ under settings that let an answer be a list, which
\S\ref{sec:repro} names. F1 does not follow: $0.020$ against
$0.023$, and $0.053$ against $0.057$. Gathering the turns is not the hard part
of a multi-hop answer; assembling four phrases from four sessions into one list
is, and an extractive reader cannot. The oracle scores $0.068$ with perfect
@@ -722,8 +769,8 @@ it.
There is a second caveat, and it qualifies our own earlier
work~\cite{multihop}, which proposed cross-session entity linkage as the
missing signal for multi-hop retrieval. We built that linkage, and at unbounded
$k$ the scoped index reaches $0.688$ of multi-hop evidence where the flat index
reaches $0.985$. Subject scope improves multi-hop \emph{ranking} at small $k$
$k$ \arm{graph} reaches $0.688$ of multi-hop evidence where \arm{vector} reaches
$0.985$. Subject scope improves multi-hop \emph{ranking} at small $k$
and lowers the \emph{ceiling} on what can be recovered at any $k$, because a
multi-hop question's evidence is not all filed under one person. Whether a
linkage that keeps everything reachable would help is untested here.
@@ -753,8 +800,12 @@ including it is the right design, and averaging over it is not.
Retrieval-augmented generation~\cite{rag} over dense
passage retrieval~\cite{dpr} is the construction the flat baseline stands in
for, with lexical weighting~\cite{bm25} substituted for a learned embedding for
the reasons given in \S\ref{sec:indexes}. Graph-structured and
for, with a lexical weighting substituted for a learned embedding for the
reasons given in \S\ref{sec:indexes}. That weighting is term frequency times
inverse document frequency at unit length, compared by cosine, and not
BM25~\cite{bm25}; BM25's saturation and length terms would make the baseline
stronger still, so \S\ref{sec:indexes}'s argument that the lexical arm is the
harder one does not rest on the strongest form of it. Graph-structured and
entity-structured memory --- GraphRAG~\cite{graphrag}, HippoRAG~\cite{hipporag},
MemGPT~\cite{memgpt} --- is the family the system under test belongs to. Our
contribution to that line is a decomposition rather than a system: of the
@@ -809,10 +860,10 @@ an argument, not a measurement. The subject-scoping result is a claim about what
is indexed and should transfer to a dense index; that it does transfer is
untested.
\textbf{A rule-based extractor.} The $11$--$24$ points of answerable reach the
scoped index gives up are the extractor's failures, not a property of subject
scoping. A model-based extractor would move that cost and we do not know by how
much.
\textbf{A rule-based extractor.} The $11$ and $24$ points of answerable reach
the two subject-scoped indexes give up are the extractor's failures, not a
property of subject scoping. A model-based extractor would move that cost and we
do not know by how much.
\section{What would settle it}
\label{sec:settle}
@@ -855,7 +906,7 @@ Four experiments, in the order we would run them.
\label{sec:repro}
The harness is \texttt{bench/locomo/} in \texttt{hanzoai/semantic}~\cite{semantic},
MIT, tag \texttt{v0.2.3}. It is Go with no dependencies outside the standard
MIT, tag \texttt{v0.2.4}. It is Go with no dependencies outside the standard
library and that repository, and it fetches the dataset itself on first run
(2{,}805{,}274 bytes, SHA-256
\texttt{79fa87e9\ldots ea698ff4}, from the LoCoMo GitHub repository rather than
@@ -863,31 +914,54 @@ the Hugging Face mirror, which holds no data). Neither the dataset nor the
original \texttt{evaluation.py} is redistributed: LoCoMo is CC BY-NC 4.0 and the
harness is MIT, so both are fetched at run time.
Every number in this paper comes from one of:
Every figure in this paper is produced by one of the following, run from the
repository root. The first group is \texttt{go run ./bench/locomo} with the
flags shown:
\begin{center}
\small
\footnotesize
\begin{tabular}{@{}ll@{}}
\toprule
command & what it produces \\
flags & what it produces \\
\midrule
\texttt{go run ./bench/locomo} & Table~\ref{tab:main} \\
\texttt{go run ./bench/locomo -factor} & Table~\ref{tab:factorial} \\
\texttt{go run ./bench/locomo -sweep floor} & Figure~\ref{fig:frontier}, \S\ref{sec:abstention} \\
\texttt{go run ./bench/locomo -k 2000 -floor 0} & Figure~\ref{fig:reach} \\
\texttt{go run ./bench/locomo -out <file>} & the per-question record \\
\texttt{go test ./bench/locomo/} & the metric port, three ways \\
\texttt{make check} \emph{(in \texttt{bench/locomo})} & the same predictions, scored by the original Python \\
(none) & Table~\ref{tab:main}; the corpus, build and subject counts of \S\ref{sec:setup} \\
\texttt{-factor} & Table~\ref{tab:factorial} \\
\texttt{-sweep floor} & Figure~\ref{fig:frontier}, the matched-silence table of \S\ref{sec:abstention} \\
\texttt{-sweep span} (also \texttt{band}, \texttt{parts}) & the reader settings of \S\ref{sec:reader} \\
\texttt{-k 2000 -floor 0} & Figure~\ref{fig:reach}, \S\ref{sec:mechanism}, the unbounded-$k$ reach of \S\ref{sec:where-not} \\
\texttt{-k 20 -parts 4 -band 0.5} & the multi-hop list figures of \S\ref{sec:where-not} \\
\texttt{-floor 0.13} & the low-silence crossing of \S\ref{sec:abstention} \\
\texttt{-out <file>} & the per-question record \\
\texttt{-stats <file>} & \S\ref{sec:degenerate}, the AUCs of \S\ref{sec:abstention}, Table~\ref{tab:ci} \\
\midrule
\multicolumn{2}{@{}l}{\emph{and, as whole commands:}} \\
\texttt{make golden} \emph{(in \texttt{bench/locomo})} & the two golden files the metric tests read \\
\texttt{go test -v ./bench/locomo/} & the metric port, three ways, with the counts of \S\ref{sec:metric} \\
\texttt{make check} \emph{(in \texttt{bench/locomo})} & Tables~\ref{tab:main} and \ref{tab:factorial}, by the original Python \\
\bottomrule
\end{tabular}
\end{center}
The predictions file carries each system's reply, its retrieved turn
identifiers, and its confidence, so the AUCs of \S\ref{sec:abstention} and the
bootstrap intervals of Table~\ref{tab:ci} are computed from a published artifact
rather than asserted. The two controls of \S\ref{sec:control-graph} and
\S\ref{sec:control-reason} are single-line edits to \texttt{graph.go}: build the
graph from the empty fact set, and skip the loop over derived facts.
Every row in the first group also takes \texttt{-factor}, which swaps
\arm{graph+edge} and \arm{oracle} for \arm{scoped} and \arm{span};
Figures~\ref{fig:reach} and \ref{fig:frontier} are drawn from both arm sets.
\texttt{-stats} reads a predictions file back rather than the memories: the file
carries each system's reply, its retrieved turn identifiers and its confidence,
so the constant-abstention score of \S\ref{sec:degenerate}, the AUCs of
\S\ref{sec:abstention} and the intervals of Table~\ref{tab:ci} are computed from
a published artifact rather than asserted beside it. Two of the three golden
files quote the dataset and so are not kept in the repository; the tests that
read them skip until \texttt{make golden} has written them.
Four things come from outside that table. The dataset's length and digest are of
the file the harness fetches, read with \texttt{shasum -a 256}; the shape of the
adversarial questions in \S\ref{sec:setup} --- which fields they carry, and what
is in them --- is read off that same file. The cost in \S\ref{sec:indexes} is
\texttt{/usr/bin/time -l} over the built binary. And the two controls of
\S\ref{sec:control-graph} and \S\ref{sec:control-reason} are single-line edits
to \texttt{graph.go} --- build the graph from the empty fact set, and skip the
loop over the derived facts --- after which the first row of the table reports
what changed.
\section{Conclusion}
\label{sec:conclusion}
@@ -895,15 +969,17 @@ graph from the empty fact set, and skip the loop over derived facts.
Three findings, of which only the last is of the kind usually reported.
The aggregate that would ordinarily be quoted for this benchmark is degenerate.
Constant abstention scores $0.228$ and beats every system we built. Any mean
Constant abstention scores $0.228$, above every system here that has to find its
own evidence and within $0.024$ of one handed the annotated evidence. Any mean
over a question set in which a quarter of the questions reward silence measures
the abstention policy first.
The mechanism the system is named for is inert. Emptying the knowledge graph
leaves every figure byte-identical, and removing the inference layer, which
derived 44 facts from 23{,}942, moves nothing at three decimals. Neither of
those is a defect in the implementation; both say where the value in this kind
of memory is not.
derived 44 facts from 23{,}942, moves no score on either half of the benchmark
and no figure anywhere by more than two thousandths. Neither of those is a
defect in the implementation; both say where the value in this kind of memory is
not.
What is left is worth stating exactly. Partitioning a retrieval index by the
subject of each sentence, and querying it with the subject of the question,
@@ -913,8 +989,8 @@ index unable to reach the trap turn: $27\%$ of adversarial evidence reachable
against $96\%$. Half of that gain is the partition applied to the unchanged
baseline; the claim-span unit on its own makes things worse; the largest single
term is their interaction, because a fine-grained unit is what makes the
partition unambiguous. An oracle with perfect retrieval does worse than the
scoped index on those questions, because there the right turn is the wrong
partition unambiguous. An oracle with perfect retrieval does worse than
\arm{graph} on those questions, because there the right turn is the wrong
answer.
The useful part of this is the shape of the intervention, which is small.
locomo-subject-scope/references.bib
+1 -1
View File
@@ -16,7 +16,7 @@
intelligence for agents, in Go}},
howpublished = {\url{https://github.com/hanzoai/semantic}},
year = {2026},
note = {MIT. Benchmark harness under \texttt{bench/locomo/}, tag v0.2.3}
note = {MIT. Benchmark harness under \texttt{bench/locomo/}, tag v0.2.4}
}
@techreport{multihop,