mirror of
https://github.com/hanzoai/semantic.git
synced 2026-10-30 15:04:19 +00:00
zeekayandHanzo Dev
0c787e9f4d
Two changes that only mean something together. The graph was never read: emptying it left every number identical, so the subject filter was doing all the work and the structure none. The path arm stands on the nodes a question names, gathers the assertions out of them into relations, and cites the turns behind the one that fits — multi-hop 0.020 to 0.032, adversarial 0.502 to 0.576. Run it with the graph emptied and it goes to nothing at all: recall 0.000, quiet on every question, overall 0.228, which is the score for answering nothing. That is the control the old arms could not fail. The reader was the other half. Sixty percent of multi-hop answers are lists and the reader stopped at two parts, so it discarded the shape of the answer before retrieval had a say. Gather states what the memory returned and the oracle moves 0.068 to 0.119, which is what lets a retrieval gain show underneath it at all. Eighty-eight percent of the distance to one is still what an extractive reader cannot write. What was measured and taken back out: coreference touches three percent of sentences and moves multi-hop by nothing; a second hop reaches more evidence and costs adversarial questions dearly, because reaching the trap turn is how those are lost; scoring whole relations has an arity bias that settles on the loose large one over the exact small one. Folding predicates by stem stays, at 3,472 relations to 2,866. Losses: single hop falls 0.121 to 0.111 walking, and to 0.049 when the reader states everything, because category one is marked part by part and every other answerable category is one bag of tokens where an extra part costs precision. Co-authored-by: Hanzo Dev <[email protected]>
76 lines
2.4 KiB
Go
76 lines
2.4 KiB
Go
package main
|
|
|
|
import (
|
|
"context"
|
|
"fmt"
|
|
"testing"
|
|
)
|
|
|
|
// What one hop of the graph reaches, which is the fact the walk arm is built
|
|
// on and the fact that decides how far it is worth walking.
|
|
//
|
|
// A LoCoMo speaker is the subject of about a thousand assertions, so standing
|
|
// on the node a question names and taking the assertions out of it already
|
|
// reaches most of the turns the dataset calls that question's evidence. Reach
|
|
// is therefore not what limits the arm; choosing among what is reached is. And
|
|
// the adversarial questions are the other half of the same fact: their
|
|
// evidence is about somebody else, so one hop does not reach it and the memory
|
|
// correctly has nothing to say. A second hop reaches it, which is why the arm
|
|
// does not take one.
|
|
//
|
|
// Two conversations rather than ten, because the property is structural and
|
|
// the whole set takes a minute and a quarter to walk.
|
|
func TestOneHopReachesTheEvidence(t *testing.T) {
|
|
ctx := context.Background()
|
|
samples, err := Load("data/locomo10.json")
|
|
if err != nil {
|
|
t.Skip(err)
|
|
}
|
|
hit := map[int]map[int]float64{1: {}, 2: {}}
|
|
want := map[int]float64{}
|
|
for _, s := range samples[:2] {
|
|
words := Vocabulary(s.Turns)
|
|
k, err := NewKnowledge(ctx, s, s.Turns, words)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
for _, ask := range s.Asks {
|
|
at := k.fold(ask.Text)
|
|
if at == "" {
|
|
continue
|
|
}
|
|
for _, hops := range []int{1, 2} {
|
|
got := map[string]bool{}
|
|
for _, step := range k.graph.Walk(at, hops) {
|
|
for _, doc := range step.Docs {
|
|
got[doc] = true
|
|
}
|
|
}
|
|
for _, e := range ask.Evidence {
|
|
if hops == 1 {
|
|
want[ask.Kind]++
|
|
}
|
|
if got[trim(e)] {
|
|
hit[hops][ask.Kind]++
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
reach := func(hops, kind int) float64 { return hit[hops][kind] / want[kind] }
|
|
for _, kind := range Answerable {
|
|
if got := reach(1, kind); got < 0.80 {
|
|
t.Errorf("one hop reaches %.3f of the evidence for category %d, which is too little to be choosing from", got, kind)
|
|
}
|
|
}
|
|
if got := reach(1, 5); got > 0.55 {
|
|
t.Errorf("one hop reaches %.3f of the adversarial evidence; the whole point is that it does not", got)
|
|
}
|
|
if got := reach(2, 5); got < 0.85 {
|
|
t.Errorf("two hops reach %.3f of the adversarial evidence, so the trade the arm refuses is not the trade it thinks", got)
|
|
}
|
|
for _, kind := range []int{4, 1, 2, 3, 5} {
|
|
fmt.Printf("category %d: one hop reaches %.3f, two %.3f\n", kind, reach(1, kind), reach(2, kind))
|
|
}
|
|
}
|