zeekayandHanzo Dev c99f2a73c1 bench: LoCoMo, over a graph and over a heap of turns ...

The recall benchmark run twice from these packages over the same 5,882 turns,
with the same reader and the dataset's own scorer: turns in a store.Mem
retrieved by cosine, and a kg graph of who did what retrieved by looking up the
person a question names. An oracle arm holding the annotated evidence bounds
what the reader can do at all.

The graph scores 0.223 against the flat index's 0.156 overall — level on the
1,540 answerable questions, 0.502 against 0.200 on the 446 adversarial ones —
and leads at every matched rate of declining to answer. It gets there by being
selectively blind: it reaches 58-79% of the evidence for questions that have an
answer and 27% for questions that do not, because a turn about somebody else is
not a candidate however close its wording. Where it loses is reported too.

The metric is ported from task_eval/evaluation.py, NLTK's Porter stemmer
included, and held against the Python by three golden tests and once end to
end: score.py runs the official code over the same predictions and reproduces
every figure to three decimals.

LoCoMo is CC BY-NC and this module is MIT, so neither the dataset nor their
scorer is kept here. Both are fetched on first use, the dataset through
ingest.Web.

Co-authored-by: Hanzo Dev <[email protected]>

2026-09-09 00:15:15 -07:00

..

stems.tsv

bench: LoCoMo, over a graph and over a heap of turns

2026-09-09 00:15:15 -07:00