mirror of
https://github.com/hanzoai/semantic.git
synced 2026-10-30 15:04:19 +00:00
zeekayandHanzo Dev
c99f2a73c1
The recall benchmark run twice from these packages over the same 5,882 turns, with the same reader and the dataset's own scorer: turns in a store.Mem retrieved by cosine, and a kg graph of who did what retrieved by looking up the person a question names. An oracle arm holding the annotated evidence bounds what the reader can do at all. The graph scores 0.223 against the flat index's 0.156 overall — level on the 1,540 answerable questions, 0.502 against 0.200 on the 446 adversarial ones — and leads at every matched rate of declining to answer. It gets there by being selectively blind: it reaches 58-79% of the evidence for questions that have an answer and 27% for questions that do not, because a turn about somebody else is not a candidate however close its wording. Where it loses is reported too. The metric is ported from task_eval/evaluation.py, NLTK's Porter stemmer included, and held against the Python by three golden tests and once end to end: score.py runs the official code over the same predictions and reproduces every figure to three decimals. LoCoMo is CC BY-NC and this module is MIT, so neither the dataset nor their scorer is kept here. Both are fetched on first use, the dataset through ingest.Web. Co-authored-by: Hanzo Dev <[email protected]>
70 lines
2.3 KiB
Go
70 lines
2.3 KiB
Go
package main
|
|
|
|
import (
|
|
"context"
|
|
|
|
"github.com/hanzoai/semantic/store"
|
|
)
|
|
|
|
// The baseline: remember a conversation as a heap of turns and find the ones
|
|
// whose words sit closest to the question's. This is the approach the
|
|
// benchmark is usually attempted with, so it is built to be as strong as it
|
|
// honestly can be rather than as a foil — the retrieved unit is one turn with
|
|
// its date and speaker attached, which is what the official RAG harness
|
|
// embeds, and the terms are stemmed and weighted by how rare they are before
|
|
// the cosine is taken.
|
|
//
|
|
// The weighting is lexical, not a learned embedding: nothing here reaches a
|
|
// network, and on this data lexical retrieval is the harder baseline, because
|
|
// LoCoMo questions are written from the turns they are about and repeat their
|
|
// words. A dense retriever generalises across wording; it does not fix the
|
|
// failure this benchmark turns on, which is that a turn near the question's
|
|
// words is not the same thing as a turn about the person the question names.
|
|
|
|
// Vector is a conversation kept as vectors, one per turn.
|
|
type Vector struct {
|
|
turns []Turn
|
|
at map[string]int
|
|
words *Words
|
|
mem *store.Mem
|
|
}
|
|
|
|
// NewVector indexes every turn.
|
|
func NewVector(ctx context.Context, turns []Turn, words *Words) (*Vector, error) {
|
|
v := &Vector{
|
|
turns: turns,
|
|
at: make(map[string]int, len(turns)),
|
|
words: words,
|
|
mem: &store.Mem{},
|
|
}
|
|
// Weigh returns unit vectors, so the inner product is the cosine and the
|
|
// two lengths do not have to be computed for every comparison.
|
|
v.mem.Rank(store.Inner)
|
|
for i, t := range turns {
|
|
v.at[t.ID] = i
|
|
if err := v.mem.Put(ctx, t.ID, words.Weigh(t.Line()), nil); err != nil {
|
|
return nil, err
|
|
}
|
|
}
|
|
return v, nil
|
|
}
|
|
|
|
// Name is what this arm is called in the results.
|
|
func (v *Vector) Name() string { return "vector" }
|
|
|
|
// Recall returns the k turns nearest the question.
|
|
func (v *Vector) Recall(ctx context.Context, question string, k int) ([]Cite, error) {
|
|
near, err := v.mem.Near(ctx, v.words.Weigh(question), k)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
out := make([]Cite, 0, len(near))
|
|
for _, m := range near {
|
|
if m.Score <= 0 {
|
|
break // a turn with no word in common with the question is not evidence
|
|
}
|
|
out = append(out, Cite{Turn: v.turns[v.at[m.ID]], Score: m.Score})
|
|
}
|
|
return out, nil
|
|
}
|