mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-29 07:34:31 +00:00
zooqueen
67816f32ac
The engine's ablation on all ten conversations, the test split run once after the dev freeze, the same rows over all-MiniLM-L6-v2 at k=10, and the frozen configuration by k are generated by hanzoai/cloud bench/brain/tables.mjs. LoCoMo-Conv is unreleased, so that section now says so and reports the protocol check on the original questions instead of a number nobody can reproduce.
23 lines
1.5 KiB
TeX
23 lines
1.5 KiB
TeX
% generated by hanzoai/cloud bench/brain/tables.mjs — test split (conversations 3–9), k=20, frozen at the dev sweep
|
||
\begin{tabular}{lrrrrrrrrrr}
|
||
\toprule
|
||
& \multicolumn{2}{c}{multi-hop} & single & temporal & open & all & & & pool & p50 \\
|
||
\cmidrule(lr){2-3}
|
||
Configuration & ALL@20 & ANY@20 & ALL@20 & ALL@20 & ALL@20 & R@20 & nDCG & supported & cands & ms \\
|
||
\midrule
|
||
cosine over turns & 18.8 & 80.3 & 78.3 & 72.3 & 35.6 & 70.9 & 48.8 & 80.5 & 40 & 0.55 \\
|
||
+ BM25 & 21.2 & 78.4 & 82.8 & 73.6 & 38.4 & 73.9 & 52.9 & 83.2 & 53 & 0.62 \\
|
||
+ atomic fact index & 35.1 & 88.5 & 83.0 & 78.8 & 43.8 & 77.9 & 58.7 & 86.4 & 71 & 1.15 \\
|
||
+ canonical entities & 35.1 & 88.5 & 83.0 & 78.8 & 43.8 & 77.9 & 58.7 & 86.4 & 71 & 1.10 \\
|
||
+ timeline & 34.6 & 88.5 & 83.3 & 79.7 & 45.2 & 78.2 & 59.9 & 86.8 & 71 & 1.04 \\
|
||
+ adjacency & 37.5 & 88.5 & 84.6 & 80.1 & 45.2 & 79.1 & 60.5 & 87.5 & 75 & 1.03 \\
|
||
+ typed expansion & 37.5 & 88.5 & 84.6 & 80.1 & 45.2 & 79.1 & 60.5 & 87.5 & 75 & 0.98 \\
|
||
+ second hop from resolved state & 37.5 & 88.0 & 87.7 & 81.4 & 45.2 & 80.7 & 61.6 & 87.8 & 77 & 1.67 \\
|
||
\midrule
|
||
pseudo-relevance feedback & 19.7 & 75.5 & 72.4 & 69.3 & 35.6 & 66.4 & 46.3 & 75.8 & 634 & 1.10 \\
|
||
chain search (top turn as query) & 37.5 & 88.0 & 83.8 & 78.8 & 45.2 & 78.5 & 56.7 & 86.3 & 79 & 1.64 \\
|
||
surface-entity expansion & 35.6 & 88.0 & 83.0 & 78.8 & 43.8 & 77.9 & 58.2 & 86.2 & 82 & 0.98 \\
|
||
global reciprocal-rank fusion & 32.2 & 88.5 & 86.7 & 80.1 & 38.4 & 79.6 & 55.5 & 88.4 & 139 & 1.20 \\
|
||
\bottomrule
|
||
\end{tabular}
|