mirror of
https://github.com/hanzoai/semantic.git
synced 2026-10-30 15:04:19 +00:00
zeekayandHanzo Dev
c99f2a73c1
The recall benchmark run twice from these packages over the same 5,882 turns, with the same reader and the dataset's own scorer: turns in a store.Mem retrieved by cosine, and a kg graph of who did what retrieved by looking up the person a question names. An oracle arm holding the annotated evidence bounds what the reader can do at all. The graph scores 0.223 against the flat index's 0.156 overall — level on the 1,540 answerable questions, 0.502 against 0.200 on the 446 adversarial ones — and leads at every matched rate of declining to answer. It gets there by being selectively blind: it reaches 58-79% of the evidence for questions that have an answer and 27% for questions that do not, because a turn about somebody else is not a candidate however close its wording. Where it loses is reported too. The metric is ported from task_eval/evaluation.py, NLTK's Porter stemmer included, and held against the Python by three golden tests and once end to end: score.py runs the official code over the same predictions and reproduces every figure to three decimals. LoCoMo is CC BY-NC and this module is MIT, so neither the dataset nor their scorer is kept here. Both are fetched on first use, the dataset through ingest.Web. Co-authored-by: Hanzo Dev <[email protected]>
37 lines
1.1 KiB
Python
37 lines
1.1 KiB
Python
"""The official LoCoMo scorer, fetched rather than vendored.
|
|
|
|
snap-research/locomo is CC BY-NC 4.0 and this repository is MIT, so their
|
|
evaluation.py is downloaded on first use and kept out of git. It is imported
|
|
unmodified: the one thing done to it is standing a stub in for the bert_score
|
|
import it makes at module scope and never uses in eval_question_answering.
|
|
"""
|
|
|
|
import sys
|
|
import types
|
|
import urllib.request
|
|
from pathlib import Path
|
|
|
|
HERE = Path(__file__).parent
|
|
SOURCE = "https://raw.githubusercontent.com/snap-research/locomo/main/task_eval/evaluation.py"
|
|
|
|
|
|
def official():
|
|
"""The path to evaluation.py, downloading it if this is the first ask."""
|
|
path = HERE / "evaluation.py"
|
|
if not path.exists():
|
|
print(f"fetching {SOURCE}", file=sys.stderr)
|
|
with urllib.request.urlopen(SOURCE) as response:
|
|
path.write_bytes(response.read())
|
|
return path
|
|
|
|
|
|
official()
|
|
|
|
if "bert_score" not in sys.modules:
|
|
stub = types.ModuleType("bert_score")
|
|
stub.score = None
|
|
sys.modules["bert_score"] = stub
|
|
|
|
sys.path.insert(0, str(HERE))
|
|
from evaluation import eval_question_answering, f1, f1_score, normalize_answer # noqa: E402,F401
|