mirror of
https://github.com/hanzoai/semantic.git
synced 2026-10-30 15:04:19 +00:00
zeekayandHanzo Dev
0c787e9f4d
Two changes that only mean something together. The graph was never read: emptying it left every number identical, so the subject filter was doing all the work and the structure none. The path arm stands on the nodes a question names, gathers the assertions out of them into relations, and cites the turns behind the one that fits — multi-hop 0.020 to 0.032, adversarial 0.502 to 0.576. Run it with the graph emptied and it goes to nothing at all: recall 0.000, quiet on every question, overall 0.228, which is the score for answering nothing. That is the control the old arms could not fail. The reader was the other half. Sixty percent of multi-hop answers are lists and the reader stopped at two parts, so it discarded the shape of the answer before retrieval had a say. Gather states what the memory returned and the oracle moves 0.068 to 0.119, which is what lets a retrieval gain show underneath it at all. Eighty-eight percent of the distance to one is still what an extractive reader cannot write. What was measured and taken back out: coreference touches three percent of sentences and moves multi-hop by nothing; a second hop reaches more evidence and costs adversarial questions dearly, because reaching the trap turn is how those are lost; scoring whole relations has an arity bias that settles on the loose large one over the exact small one. Folding predicates by stem stays, at 3,472 relations to 2,866. Losses: single hop falls 0.121 to 0.111 walking, and to 0.049 when the reader states everything, because category one is marked part by part and every other answerable category is one bag of tokens where an extra part costs precision. Co-authored-by: Hanzo Dev <[email protected]>
53 lines
1.7 KiB
Makefile
53 lines
1.7 KiB
Makefile
# The benchmark, and the cross-check against the Python it was ported from.
|
|
#
|
|
# make the table
|
|
# make sweeps how the answer moves with every setting
|
|
# make stats constant abstention, the AUCs, the intervals
|
|
# make check the same predictions, scored by the official Python
|
|
# make golden the files the metric tests hold themselves against
|
|
|
|
GO = GOWORK=off go
|
|
ROOT = ../..
|
|
OUT = out/answers.json
|
|
CELLS = out/factor.json
|
|
|
|
.PHONY: bench sweeps test stats check golden venv clean
|
|
|
|
bench: $(OUT)
|
|
|
|
$(OUT):
|
|
cd $(ROOT) && $(GO) run ./bench/locomo -out bench/locomo/$(OUT)
|
|
|
|
$(CELLS):
|
|
cd $(ROOT) && $(GO) run ./bench/locomo -arms factor -out bench/locomo/$(CELLS)
|
|
|
|
sweeps:
|
|
cd $(ROOT) && for s in floor k span band parts; do $(GO) run ./bench/locomo -sweep $$s; done
|
|
|
|
test:
|
|
cd $(ROOT) && $(GO) test ./bench/locomo/
|
|
|
|
# What a table of means cannot show, read back out of both predictions files.
|
|
stats: $(OUT) $(CELLS)
|
|
cd $(ROOT) && $(GO) run ./bench/locomo -stats bench/locomo/$(OUT)
|
|
cd $(ROOT) && $(GO) run ./bench/locomo -stats bench/locomo/$(CELLS)
|
|
|
|
# The official scorer over the same answers, both arm sets. It fetches
|
|
# evaluation.py itself: LoCoMo is CC BY-NC and this repository is MIT, so none
|
|
# of it is kept here.
|
|
check: venv $(OUT) $(CELLS)
|
|
.venv/bin/python score.py $(OUT)
|
|
.venv/bin/python score.py $(CELLS)
|
|
|
|
# The golden files the metric tests hold themselves against. Two of them quote
|
|
# the dataset, so they are written here rather than kept in the repository.
|
|
golden: venv
|
|
.venv/bin/python ref/golden.py
|
|
|
|
venv:
|
|
@test -d .venv || uv venv --python 3.11 .venv
|
|
@test -f .venv/bin/nltk || (VIRTUAL_ENV=$(PWD)/.venv uv pip install nltk regex numpy && touch .venv/bin/nltk)
|
|
|
|
clean:
|
|
rm -rf out
|