semantic/bench/locomo/README.md

LoCoMo

LoCoMo asks what a system remembers of a conversation held over months. Ten conversations, 272 sessions, 5,882 turns, 1,986 questions, and the questions are typed: single-hop, multi-hop, temporal, open-domain, and adversarial — questions built to look answerable when they are not.

Two memories are built here over the same turns and asked the same questions by the same reader, marked by the dataset's own scorer.

• vector — every turn as a weighted bag of stemmed terms in store.Mem, retrieved by cosine. This is how the benchmark is usually attempted.
• graph — every turn read for who did what, then indexed under the person it was read as being about. Retrieval is still cosine over store.Mem; what changes is that the question's named subject filters the candidates first.

That second description is deliberate and was corrected after review. An earlier version of this file said retrieval here is "a lookup on a node, not a nearest neighbour." That was false, and the ablation is one line: empty the kg graph entirely and every number below is byte-identical. The graph is where the read turns are kept; it is not what answers a question. The measured effect comes from scoping retrieval by subject, not from graph structure and not from inference — reason derives 44 facts from 23,942, which is decoration at this scale.

A third memory does read it. path stands on the nodes a question names, takes the assertions made out of them, gathers those into the relations they belong to, and cites the turns behind the relation that fits best. It is the one arm here whose answers come out of the structure, and the same emptying that leaves every other column untouched leaves this one silent on all 1,986 questions. It is reported apart, under -arms walk.

Two more columns keep the first two readable. graph+edge is the same graph read differently: the answer comes off the matched edge rather than out of the turn it was read from. oracle is handed the turns the dataset itself names as the evidence — perfect retrieval, everything else unchanged — so the distance between an arm and the oracle is what better remembering is still worth, and the distance from the oracle to 1.0 is what the reader costs everybody equally.

The result

go run ./bench/locomo

k=5 floor=0.30 span=4 band=0.90 parts=2

                               graph            graph+edge                oracle                vector
category           n      F1 recall  quiet      F1 recall  quiet      F1 recall  quiet      F1 recall  quiet
single hop       841   0.121  0.507    24%   0.056  0.507    21%   0.199  1.000    42%   0.118  0.546    20%
multi hop        282   0.020  0.228    18%   0.025  0.228    17%   0.068  0.973    44%   0.023  0.194    17%
temporal         321   0.338  0.532    23%   0.336  0.532    22%   0.379  0.997    33%   0.357  0.581    16%
open domain       96   0.018  0.238    56%   0.018  0.238    55%   0.032  0.951    79%   0.016  0.258    45%
adversarial      446   0.502  0.135    50%   0.484  0.135    48%   0.426  1.000    43%   0.200  0.511    20%
answerable      1540   0.142  0.444    25%   0.107  0.444    23%   0.202  0.991    43%   0.144  0.471    20%
overall         1986   0.223  0.375    31%   0.191  0.375    29%   0.252  0.993    43%   0.156  0.480    20%

F1 is the official metric. recall is the share of a question's evidence turns the memory returned, which owes nothing to the reader. quiet is how often it declined to answer.

Read the overall row as arithmetic, not as a result. A system that replies "No information available" to all 1,986 questions scores 0.228 on this metric, above every arm in the table that has to find its own evidence — the best of those is 0.223 — and within 0.024 of oracle, which is handed the annotated evidence. A quarter of the questions are adversarial and the scorer rewards declining them, so any aggregate over the whole set rewards silence. The two halves have to be read apart, and only the split below means anything.

The two memories are level on the 1,540 answerable questions — 0.142 against 0.144, 95% CI [-0.010, +0.006] — and separate on the 446 adversarial ones, 0.502 against 0.200, CI [+0.247, +0.347]. Both intervals are the percentile bootstrap over the ten conversations, resampling conversations rather than questions because questions within one conversation share a memory and are not independent; there are only 92,378 ways to draw ten from ten with replacement, so -stats enumerates them all and the interval is exact. The graph declined on half the adversarial questions and a quarter of the answerable ones. The vector index declined on a fifth of each. It declines at the same rate on both because it cannot tell them apart. Abstention here is a threshold on one number, the confidence in the evidence returned, and how well that number separates an answerable question from an adversarial one is a property of the memory that owes nothing to where the threshold sits. It separates them at AUC 0.514 for vector, which is chance, and 0.698 for graph. The confidence goes out with the predictions and -stats reads it back, so this is checkable rather than asserted.

Two limits on how far that separation generalizes. Half of it is the subject filter alone rather than anything read from the turns; the 2x2 below takes that apart. And "level on answerable" is weaker than it sounds: the extractive reader tops out at 0.306 there even with perfect retrieval, so that column is nearly insensitive to retrieval quality and could move once a model reader sits behind both arms.

Which of the two changes carries it

The graph arm changes two things at once against the baseline: it indexes claim spans rather than whole turns, and it answers only out of the pieces of the person a question names rather than out of everything. Two cells cannot tell those apart, so here are the other two.

go run ./bench/locomo -arms factor

                               graph                scoped                  span                vector
category           n      F1 recall  quiet      F1 recall  quiet      F1 recall  quiet      F1 recall  quiet
single hop       841   0.121  0.507    24%   0.129  0.533    21%   0.129  0.541    16%   0.118  0.546    20%
multi hop        282   0.020  0.228    18%   0.021  0.219    18%   0.023  0.209    15%   0.023  0.194    17%
temporal         321   0.338  0.532    23%   0.328  0.559    23%   0.362  0.553    16%   0.357  0.581    16%
open domain       96   0.018  0.238    56%   0.016  0.239    47%   0.016  0.225    43%   0.016  0.258    45%
adversarial      446   0.502  0.135    50%   0.352  0.207    35%   0.170  0.571    17%   0.200  0.511    20%
answerable      1540   0.142  0.444    25%   0.144  0.463    23%   0.151  0.463    17%   0.144  0.471    20%

scoped indexes whole turns and files each under every person it was read as being about; span is the graph's own index with the subject filter taken off. Adversarial F1 across the four:

	all reachable	only the subject's
whole turn	0.200	0.352
claim span	0.170	0.502

Scoping the unchanged turn index is worth +0.152. The claim span on its own is worth -0.029 — it is worse than the baseline, because a narrower unit with everything still reachable only makes the trap turn easier to find, and span does retrieve more adversarial evidence than the baseline does, recall 0.571 against 0.511. The two together are worth +0.303, so +0.179 of the result is in neither factor but in their product: narrowing the unit is what sharpens the partition, since a turn belongs to every person spoken of in it and a span belongs to one. (Differences are computed from the unrounded scores, so they do not always match the differences of the rounded cells.) On the 1,540 answerable questions no cell moves — 0.142, 0.144, 0.151, 0.144.

Why the adversarial questions separate them

An adversarial question in LoCoMo is not nonsense. It is a question about the wrong person. From conv-43:

D20:2 John: Hi Tim! Congrats on your success! … I'm also trying out yoga to get a little extra strength and flexibility.

Q What is Tim trying out to improve his strength and flexibility after recovery from ankle injury?

John does yoga. Tim asked him about it. The question carries no reference answer — its adversarial_answer field holds the plausible wrong one, "yoga", and the scorer reads neither — so the reply that scores is one that declines. What the run produced:

vector     : 'yoga get little extra'          ctx=[D20:2 D20:4 D7:16 D19:1 D19:6]
graph      : 'No information available'       ctx=[D7:16 D9:9 D18:9 D6:4 D18:13]
oracle     : 'yoga get little extra'          ctx=[D20:2]

D20:2 is the nearest turn in the conversation to those words, and the vector index is right about that and wrong about the question. The graph never sees D20:2, because D20:2 is not a turn about Tim; the sentence carrying yoga has John as its subject, so the edge hangs off John. Asked about Tim, the graph finds nothing above the floor and says so.

The oracle answers wrongly for the same reason as the vector index, from better evidence. Perfect retrieval is a liability on an adversarial question: the trap turn is the annotated evidence.

What each index can reach at all

Run with the top-k lifted past the number of turns and the recall column stops measuring ranking and starts measuring reach — what the index is capable of returning, however hard you ask.

go run ./bench/locomo -k 2000 -floor 0

category         n    graph    vector
single hop     841    0.787     0.995
multi hop      282    0.688     0.985
temporal       321    0.785     0.997
open domain     96    0.584     0.966
adversarial    446    0.269     0.964

The flat index reaches essentially everything. The graph reaches 58–79% of the evidence for questions that have an answer, and 27% for questions that do not. It is selectively blind, and blind in the right place. The 21–42% it loses on answerable questions is the cost of a rule-based extractor: a sentence whose subject it fails to resolve produces no edge for that person, and the turn becomes unreachable through them.

The trade, at matched silence

Declining to answer is worth a point on an adversarial question and costs one everywhere else, so any comparison at a single threshold is arguable. The threshold is swept instead.

go run ./bench/locomo -sweep floor

                               graph                  graph+edge                      oracle                      vector
   floor     1-4    adv    all quiet     1-4    adv    all quiet     1-4    adv    all quiet     1-4    adv    all quiet
       0   0.152  0.047  0.129    5%   0.117  0.016  0.094    2%   0.306  0.045  0.247    3%   0.149  0.092  0.136    8%
     0.1   0.152  0.070  0.134    6%   0.117  0.038  0.099    3%   0.270  0.182  0.251   19%   0.149  0.101  0.138    9%
     0.2   0.151  0.217  0.166   13%   0.114  0.195  0.133   11%   0.238  0.285  0.248   30%   0.148  0.132  0.145   12%
     0.3   0.142  0.502  0.223   31%   0.107  0.484  0.191   29%   0.202  0.426  0.252   43%   0.144  0.200  0.156   20%
     0.4   0.126  0.693  0.253   48%   0.094  0.682  0.226   47%   0.164  0.554  0.251   55%   0.131  0.323  0.174   33%
     0.5   0.103  0.845  0.270   63%   0.078  0.841  0.249   62%   0.121  0.697  0.251   67%   0.115  0.491  0.199   48%
     0.6   0.080  0.948  0.275   75%   0.058  0.946  0.257   74%   0.095  0.791  0.251   75%   0.092  0.668  0.222   62%
     0.8   0.038  0.975  0.248   91%   0.030  0.973  0.242   90%   0.042  0.922  0.240   91%   0.043  0.879  0.231   86%

Reading across for equal quiet — each arm at whichever floor makes it as silent as the other:

silence	graph 1–4	graph adv	vector 1–4	vector adv
~12%	0.151	0.217	0.148	0.132
~32%	0.142	0.502	0.131	0.323
~48%	0.126	0.693	0.115	0.491
~62%	0.103	0.845	0.092	0.668

(graph at floors 0.2, 0.3, 0.4, 0.5 against vector at 0.2, 0.4, 0.5, 0.6.)

At each of those four rates the graph is ahead of the flat index on both halves of the benchmark. The lead belongs to the middle of the range: matched at 8% silence — vector declines no less often than that even with the floor at zero — the adversarial scores are 0.094 against 0.092, and below that rate vector cannot be matched at all. Silence alone does not buy the result: either arm can be made to fall silent as often as you like, and over the range where declining is worth anything only one of them falls silent on the right questions.

Reading the graph instead of ranking text

go run ./bench/locomo -arms walk

                               graph                oracle            oracle+set                  path              path+set                vector            vector+set
category           n      F1 recall  quiet      F1 recall  quiet      F1 recall  quiet      F1 recall  quiet      F1 recall  quiet      F1 recall  quiet      F1 recall  quiet
single hop       841   0.121  0.507    24%   0.199  1.000    42%   0.203  1.000    42%   0.111  0.467    27%   0.049  0.467    24%   0.118  0.546    20%   0.089  0.546    12%
multi hop        282   0.020  0.228    18%   0.068  0.973    44%   0.119  0.973    44%   0.032  0.222    26%   0.071  0.222    25%   0.023  0.194    17%   0.066  0.194    13%
temporal         321   0.338  0.532    23%   0.379  0.997    33%   0.379  0.997    33%   0.353  0.512    21%   0.354  0.512    21%   0.357  0.581    16%   0.356  0.581    15%
open domain       96   0.018  0.238    56%   0.032  0.951    79%   0.036  0.951    78%   0.016  0.213    61%   0.024  0.213    60%   0.016  0.258    45%   0.017  0.258    36%
adversarial      446   0.502  0.135    50%   0.426  1.000    43%   0.426  1.000    43%   0.576  0.130    58%   0.563  0.130    56%   0.200  0.511    20%   0.128  0.511    13%
answerable      1540   0.142  0.444    25%   0.202  0.991    43%   0.214  0.991    43%   0.141  0.415    28%   0.115  0.415    26%   0.144  0.471    20%   0.136  0.471    15%
overall         1986   0.223  0.375    31%   0.252  0.993    43%   0.262  0.993    43%   0.239  0.351    34%   0.216  0.351    33%   0.156  0.480    20%   0.134  0.480    14%

Two changes are crossed here, and they are the two halves of what a multi-hop question needs. path is the memory: relations instead of nearest turns. +set is the reader: Gather states every piece of evidence the memory returned rather than the best one or two, so the width of an answer is decided by the memory alone. Both are available to every arm, which is what makes the columns comparable.

The memory. On the multi-hop questions path scores 0.032 against graph's 0.020 and vector's 0.023, with the same reader and near-identical evidence recall — 0.222 against 0.228. It is not finding more of the evidence. It is citing a different five turns out of the same reach, and the difference is that the five best turns by wording are five sayings of one thing while the five best relations are five things. It is also better at silence: 0.576 adversarial against 0.502, because an assertion about the wrong person is not reached from the right person's node at all.

The reader. +set is worth more than the memory and is worth it in exactly one category. Multi-hop goes 0.023 → 0.066 on vector, 0.032 → 0.071 on path, and — the number that says what kind of bound this is — 0.068 → 0.119 on oracle, which is handed the annotated evidence. Perfect retrieval read the old way scored 0.068; perfect retrieval read this way scores 0.119. The ceiling moved, so a retrieval result can now show up underneath it, and path's does.

And it is a trade, not a gain. Single-hop falls 0.111 → 0.049 on path and 0.118 → 0.089 on vector; adversarial falls 0.200 → 0.128 on vector. The metric is asymmetric and that is the whole of the explanation: a category 1 answer is marked part by part, so a reply part that matches nothing costs nothing, while every other answerable category is marked as one bag of tokens, where it costs precision. Listing everything you have is free where the answer is a list and expensive where it is not.

The control

go run ./bench/locomo -arms walk -blind

-blind empties every graph after building it and changes nothing else — same claims, same pieces, same vectors, same subject resolution. graph, vector, oracle and both +set columns come back byte-identical. path and path+set go to 100% quiet, 0.000 evidence recall, 0.004 on the answerable questions and 0.228 overall, which is what constant abstention scores. That is the whole difference between a report that mentions a graph and a report that reads one, and it is a flag rather than a paragraph so that anybody can check it.

How far to walk

Two and three hops were built and measured, guided — a hop taken only through a node the question mentions something of — and unguided. Every arm below is read with Gather, including graph, so that the only thing varying across the columns is how far the walk goes; that is why the figures differ from the table above, where graph and path are read with Reply.

                               graph                  path                 path2                 path3                 walk2                 walk3
multi hop        282   0.064  0.228    17%   0.071  0.222    25%   0.071  0.223    25%   0.071  0.223    25%   0.074  0.229    20%   0.071  0.226    19%
adversarial      446   0.484  0.135    48%   0.563  0.130    56%   0.534  0.150    53%   0.534  0.150    53%   0.305  0.351    30%   0.258  0.404    26%
overall         1986   0.194  0.375    29%   0.216  0.351    33%   0.209  0.357    32%   0.209  0.357    32%   0.159  0.404    25%   0.148  0.420    23%

The second hop moves multi-hop by 0.000 guided and 0.003 unguided, and costs 0.029 and 0.258 on the adversarial questions. The third moves nothing at three decimals either way. Reach was never the constraint. Over all ten conversations, one hop from the node a question names reaches 90.1% of the multi-hop evidence and 84.9–88.0% of the evidence for the other answerable categories, against 34.0% of the adversarial evidence — and two hops reach 95.4% of the adversarial evidence, which is exactly the trap those questions are built around. reach_test.go asserts that shape on the first two conversations, where it is the same and runs in six seconds. Going further out only makes more to choose from. So the arm walks one hop and the hop setting is gone; this table is the reason, and the code says it rather than carrying a knob nobody should turn.

Where the graph does not win

Multi-hop. The expectation going in was that multi-hop favours a graph, since one person's several answers were said months apart and are one node's several edges here. Retrieval bears that out — recall 0.228 against 0.194 at k=5, and 0.419 against 0.388 under -k 20 -parts 4 -band 0.5, the settings that let an answer be a list. F1 does not move with it: 0.020 against 0.023, and 0.053 against 0.057. Gathering the turns is not the hard part of a multi-hop answer; assembling four phrases from four sessions into one list is.

Both halves of that were then taken apart, and the section above has the result. 169 of the 282 multi-hop answers are comma-separated lists, and Reply states at most two pieces of evidence, so the reader was throwing the shape of the answer away before retrieval got a say: handed the annotated evidence and allowed to state all of it, the oracle goes 0.068 → 0.119. Underneath that raised ceiling path scores 0.032 against 0.020 by citing relations rather than turns, and 0.071 with the wider reader. The ceiling is still overwhelmingly the reader — 0.119 with perfect evidence means 88% of the distance to 1.0 is what an extractive reader cannot write, and the last column of -arms read is where a model reader has to answer for that.

Two things that looked like they should help do not. Time: extract.Link carries From, and every claim here is dated by its session, but only 26 of the 282 multi-hop questions contain a word about time at all, so there is no surface for the temporal machinery to work on — the strength of the temporal category (0.338) does not transfer. Coreference: extract.Coref over each session, so that "she" reaches the name an earlier turn gave it, was wired in and measured. It reaches 3.1% of sentences and moves temporal by +0.009, adversarial by -0.006, multi-hop by 0.000 and the overall figure by -0.001. A name already folds to one node whichever session it was said in, so the cross-session aggregation those questions need was there before the pronouns were. The code was taken back out.

Answering off an edge. graph+edge reads the answer from the matched edge — the object phrase the extractor produced — instead of quoting the turn. It is worse on single-hop by more than half, 0.056 against 0.121. The extracted object is a lossy paraphrase of the sentence, and the sentence is what the reference answer was written from. The graph is a better index than it is a store of answers, at least when the extractor is rules rather than a model.

Inference. reason runs over the extracted triples with one rule, that what a thing is called is also what its owner has. Across all ten conversations it derived 44 facts from 23,942. Conversational memory is dominated by aggregation over identity — the same person mentioned in two sessions is one key, the folded name each piece is filed under, which needs no inference at all — and there is very little left for a reasoner to add. That is a finding, not a defect: it says where the value in this kind of graph actually sits. Deleting the derived facts moves nothing by more than two thousandths: single-hop F1 rises 0.121 → 0.122, single-hop recall falls 0.507 → 0.505, and no F1 on either half moves at three decimals.

Absolute numbers. Everything here is low. The reader is deterministic and extractive: it picks the sentence covering most of the question, drops the words the question already contained, and keeps the rarest few of what is left. No model is called. The oracle row bounds every arm — 0.306 on answerable questions with perfect retrieval and no abstention — and that bound is the reader's, not the memories'. A model reader would raise every column; it would also make the comparison a comparison of two prompts.

How it is put together

locomo.go   the dataset: sessions in order, dates parsed, evidence ids
talk.go     reading dialogue: who a sentence is about, and what it claims
graph.go    the graph arm: claims -> kg -> reason -> a store filtered by person
vector.go   the baseline: one weighted vector per turn
path.go     the walk arm: relations off the nodes a question names
oracle.go   the arm that cheats, to bound the reader
terms.go    one vocabulary, weighed once, shared by everything
read.go     the readers, shared by every arm
metric.go   the official metric, ported
porter.go   the stemmer the official metric runs
factor.go   the other two cells: turns scoped, spans unscoped
bench.go    the experiment and the table
stats.go    what the table cannot show, read back off the predictions

Reading dialogue. The library's extract.Rules knows how to find a company being founded; a friend saying "I went to a support group yesterday" is a different grammar. talk.go is its reader, and one observation does most of the work: in a two-party conversation the subject of a sentence is settled by who is speaking. I is the speaker, you is the other one, a name is that person, and a sentence naming nobody is about the speaker. Two rules keep false claims out, and both are grammar rather than tuning:

• A question asserts nothing. "How's it going with yoga?" does not make the asker someone who does yoga. This is what stops a topic leaking from the person who raised it to the person who answered, and it is exactly what the adversarial category punishes.
• A negation is kept. "I don't eat dairy" is stored with the negation on the predicate, so it is findable and is not mistaken for its opposite.

98.9% of LoCoMo questions (1,964 of 1,986) name one of the two speakers, which is what makes a lookup on a person the right query for them. The resolver takes the longest name it knows that the question contains, speakers and third parties alike, and so resolves 1,975 of them (99.45%); the remaining 11 name nobody it knows and fall back to searching every piece. The run prints both counts.

What is indexed. The baseline indexes turns. The graph indexes pieces — what one person said in one turn — so a turn holds as many pieces as it has people spoken of, and a question reaches only the pieces of the person it names. The claims decide which sentences belong to a piece; the sentences are what it is ranked on. Ranking on the extracted claims alone would measure how well the extractor paraphrases rather than what the graph knows about whom.

Both arms rank by cosine over the same weighted terms through store.Mem, and the subject restriction is a metadata filter (store.Filter{}.Eq(store.Space, who)) rather than a second store. The comparison is therefore an ablation of the index, not of two systems.

Confidence is computed once, in Sure, outside the arms: how much of the question's terms the best turn a memory returned accounts for, weighted by how rare each term is. Computing it over turns rather than over whatever an arm happens to index is what keeps the number meaning the same thing in every column — a memory indexing short pieces would otherwise look less sure of the same evidence, and the report would be measuring text length.

The reader never sees the question's category and never sees which arm produced the evidence. It reads a date off the session for a when question, resolving a month named in the text against it — "back in October" in a May session is the previous October — and otherwise quotes the sentence. Its three settings barely matter: swept over all ten conversations, span moves the graph's overall score from 0.215 to 0.224, band from 0.220 to 0.223 and parts from 0.220 to 0.223. The two settings that do matter, k and floor, both trade answering against declining, and both are swept above.

The metric is the official one

metric.go and porter.go port task_eval/evaluation.py and the NLTK stemmer it calls. That port is held against the Python three ways:

• TestStem — NLTK's stem for all 6,597 words in the corpus.
• TestNormalise — the official normalize_answer over all 10,382 distinct questions, answers and turns.
• TestGrade — eval_question_answering's score for 2,400 replies spanning every category and every branch of the metric.

And then end to end: score.py runs the official Python over the same predictions file the Go wrote.

make check

                   n                 graph            graph+edge                oracle                vector
category                  F1 recall             F1 recall             F1 recall             F1 recall
single hop       841   0.121  0.507          0.056  0.507          0.199  1.000          0.118  0.546
multi hop        282   0.020  0.228          0.025  0.228          0.068  0.973          0.023  0.194
temporal         321   0.338  0.532          0.336  0.532          0.379  0.997          0.357  0.581
open domain       96   0.018  0.238          0.018  0.238          0.032  0.951          0.016  0.258
adversarial      446   0.502  0.135          0.484  0.135          0.426  1.000          0.200  0.511
answerable      1540   0.142  0.444          0.107  0.444          0.202  0.991          0.144  0.471
overall         1986   0.223  0.375          0.191  0.375          0.252  0.993          0.156  0.480

make check scores both predictions files this way, and every figure matches the Go to three decimals, across 1,986 questions and all six arms.

One adaptation lives in score.py rather than in the vendored file: the official recall reads context[0] to tell a session id from a dialogue id, and an arm that returned nothing has no context[0], so a placeholder matching no evidence is substituted. Without it the official code awards recall 1.0 to an arm that returned nothing.

Two details of the official scorer are worth stating, because they shape what the numbers mean. An adversarial answer is scored by looking for the phrases no information available or not mentioned in the reply — the reference answer is not consulted. And a multi-hop answer is scored by splitting both sides on commas and giving each reference part the best-matching reply part, so recall is rewarded and a wrong extra part costs nothing. parts is bounded at 2 here for that reason; the bound is ours, not the scorer's.

Running it

go run ./bench/locomo                        # the table above
go run ./bench/locomo -show 5                # answers, side by side, per category
go run ./bench/locomo -sweep floor           # or k, span, band, parts
go run ./bench/locomo -arms factor           # index unit against subject scope
go run ./bench/locomo -arms walk             # reading the graph against ranking text
go run ./bench/locomo -arms walk -blind      # the same, over an emptied graph
go run ./bench/locomo -k 2000 -floor 0       # what each index can reach
go run ./bench/locomo -out bench/locomo/out/answers.json
go run ./bench/locomo -stats bench/locomo/out/answers.json
go test ./bench/locomo/

-stats reads a predictions file back and reports the three things the table cannot show: what constant abstention scores, how far each arm's confidence tells an answerable question from an adversarial one, and the paired difference between every pair of arms with its bootstrap interval. -arms chooses which memories a run builds and every other flag applies to whichever set it names.

Run from the repository root, and with GOWORK=off if a parent go.work is in the way. Reading ten conversations into both memories and answering 1,986 questions four ways takes 5.2s of CPU (10.8s wall on a busy laptop), measured with /usr/bin/time -l.

The dataset is fetched on first use, through the library's own ingest.Web, to bench/locomo/data/locomo10.json — 2,805,274 bytes, SHA-256 79fa87e90f04081343b8c8debecb80a9a6842b76a7aa537dc9fdf651ea698ff4. Take it from GitHub and not from HuggingFace: adymaharana/locomo there holds a README and .gitattributes and no data at all, which is a good way to lose an afternoon.

For the Python cross-check:

cd bench/locomo
make golden    # writes the two golden files that quote the dataset
make check     # scores both predictions files with the official evaluation.py
make stats     # constant abstention, the AUCs, the intervals

Both fetch what they need. LoCoMo is CC BY-NC 4.0 and this repository is MIT, so neither the dataset nor task_eval/evaluation.py is kept here; .gitignore lists what is fetched. testdata/stems.tsv is a word list and stays.

What was built, per conversation

conv-26     419 turns   1230 terms    1822 claims   1226 nodes    1682 edges     1 derived
conv-30     369 turns   1010 terms    1482 claims    928 nodes    1301 edges     0 derived
conv-41     663 turns   1389 terms    2714 claims   1650 nodes    2434 edges     3 derived
conv-42     629 turns   1390 terms    2394 claims   1510 nodes    2144 edges     8 derived
conv-43     680 turns   1539 terms    2931 claims   1752 nodes    2529 edges     0 derived
conv-44     675 turns   1345 terms    2964 claims   1636 nodes    2578 edges     7 derived
conv-47     689 turns   1543 terms    2664 claims   1685 nodes    2451 edges    22 derived
conv-48     681 turns   1491 terms    2417 claims   1575 nodes    2182 edges     3 derived
conv-49     509 turns   1336 terms    1899 claims   1310 nodes    1725 edges     0 derived
conv-50     568 turns   1274 terms    2655 claims   1538 nodes    2331 edges     0 derived

Caveats

The vector arm weights terms lexically — inverse document frequency, stemmed, unit length, cosine — rather than with a learned embedding, because nothing here reaches a network. On this data that is the harder baseline, not the easier one: LoCoMo questions are written from the turns they ask about and repeat their words, so lexical overlap is strong. A dense retriever generalises across wording. It does not fix the failure this benchmark turns on, which is that a turn near the question's words is not the same thing as a turn about the person the question names — that is a property of what is indexed, not of how the vectors are made.

No LLM was called, in either arm or in the reader. Absolute F1 is therefore well below published model numbers on this benchmark and is not comparable to them. What is comparable is the two columns against each other, since they differ in exactly one thing.