mirror of
https://github.com/hanzoai/semantic.git
synced 2026-10-30 15:04:19 +00:00
zeekayandHanzo Dev
c99f2a73c1
The recall benchmark run twice from these packages over the same 5,882 turns, with the same reader and the dataset's own scorer: turns in a store.Mem retrieved by cosine, and a kg graph of who did what retrieved by looking up the person a question names. An oracle arm holding the annotated evidence bounds what the reader can do at all. The graph scores 0.223 against the flat index's 0.156 overall — level on the 1,540 answerable questions, 0.502 against 0.200 on the 446 adversarial ones — and leads at every matched rate of declining to answer. It gets there by being selectively blind: it reaches 58-79% of the evidence for questions that have an answer and 27% for questions that do not, because a turn about somebody else is not a candidate however close its wording. Where it loses is reported too. The metric is ported from task_eval/evaluation.py, NLTK's Porter stemmer included, and held against the Python by three golden tests and once end to end: score.py runs the official code over the same predictions and reproduces every figure to three decimals. LoCoMo is CC BY-NC and this module is MIT, so neither the dataset nor their scorer is kept here. Both are fetched on first use, the dataset through ingest.Web. Co-authored-by: Hanzo Dev <[email protected]>
58 lines
2.1 KiB
Go
58 lines
2.1 KiB
Go
package main
|
|
|
|
import "context"
|
|
|
|
// What the two arms have in common. Both are asked a question and both answer
|
|
// with turns of the conversation, best first. Everything downstream — the
|
|
// confidence, the reader that writes the answer, the scorer that marks it, the
|
|
// recall against the dataset's own evidence — sees only that, so the arms are
|
|
// compared on which turns they remember and on nothing else.
|
|
|
|
// Cite is one turn a memory offers, and how well it matched.
|
|
type Cite struct {
|
|
Turn Turn
|
|
Score float64
|
|
// Say is the phrase the memory matched inside the turn, when it has one.
|
|
// A memory that indexes whole turns has nothing to put here; a memory that
|
|
// indexes claims does, and the answer can be read straight off it.
|
|
Say string
|
|
}
|
|
|
|
// Memory is a conversation, remembered somehow, and asked about.
|
|
type Memory interface {
|
|
// Name is what the arm is called in the results.
|
|
Name() string
|
|
// Recall returns up to k turns bearing on the question, best first.
|
|
Recall(ctx context.Context, question string, k int) ([]Cite, error)
|
|
}
|
|
|
|
// Sure is how much of what a question asks about the turns a memory returned
|
|
// actually account for, weighted so a rare word counts for more than a common
|
|
// one, from nothing to all of it.
|
|
//
|
|
// It is computed here rather than inside a memory, over the turns themselves
|
|
// rather than over whatever a memory happens to index, so that the number
|
|
// means the same thing in every column. A memory that indexes short pieces
|
|
// would otherwise look less sure of the same evidence than one that indexes
|
|
// whole turns, and the report would be measuring text length.
|
|
func Sure(w *Words, question string, cites []Cite) float64 {
|
|
focus := w.Focus(question, "")
|
|
best := 0.0
|
|
for _, c := range cites {
|
|
if cover := w.Cover(focus, Terms(c.Turn.Text)); cover > best {
|
|
best = cover
|
|
}
|
|
}
|
|
return best
|
|
}
|
|
|
|
// ids is the dialogue identifiers of a list of cites, in order, which is what
|
|
// recall against the dataset's evidence is measured over.
|
|
func ids(cites []Cite) []string {
|
|
out := make([]string, 0, len(cites))
|
|
for _, c := range cites {
|
|
out = append(out, c.Turn.ID)
|
|
}
|
|
return out
|
|
}
|