mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-22 17:20:17 +00:00
zooqueen
89bf7a051f
RepoBench-R Python, test split, retrieval only: typed links over imports and definitions, with no model, find the needed snippet first 31.4% of the time where dense similarity finds it 18.8%; together 33.2%, MRR 40.8 to 52.3. The same table shape as the memory lane, the same freeze.
17 lines
1.0 KiB
TeX
17 lines
1.0 KiB
TeX
% generated by hanzoai/cloud bench/brain/tables.mjs — RepoBench-R Python, test split (test_easy[0:250] + test_hard[0:250] per setting), all-MiniLM-L6-v2 for the dense generator, weights frozen on dev
|
|
\begin{tabular}{lrrrrrrrr}
|
|
\toprule
|
|
& \multicolumn{4}{c}{cross-file-first (500)} & \multicolumn{4}{c}{cross-file-random (500)} \\
|
|
\cmidrule(lr){2-5}\cmidrule(lr){6-9}
|
|
Configuration & R@1 & R@5 & MRR & hard R@1 & R@1 & R@5 & MRR & hard R@1 \\
|
|
\midrule
|
|
dense only & 18.8 & 72.8 & 40.8 & 12.8 & 24.6 & 74.6 & 45.0 & 19.6 \\
|
|
BM25 only & 19.4 & 67.6 & 40.4 & 15.2 & 26.2 & 74.0 & 45.6 & 22.4 \\
|
|
typed links only (no model) & 31.4 & 75.4 & 50.5 & 25.6 & 29.6 & 79.0 & 49.9 & 24.0 \\
|
|
dense + BM25 & 21.2 & 73.8 & 42.3 & 17.2 & 26.8 & 77.2 & 46.9 & 22.8 \\
|
|
dense + typed links & 30.2 & 78.8 & 50.5 & 22.0 & 31.2 & 77.8 & 51.0 & 26.8 \\
|
|
BM25 + typed links (no model) & 31.4 & 75.4 & 50.5 & 25.6 & 29.6 & 79.6 & 49.9 & 24.4 \\
|
|
full: dense + BM25 + typed links & 33.2 & 79.4 & 52.3 & 26.4 & 30.2 & 79.8 & 50.2 & 24.4 \\
|
|
\bottomrule
|
|
\end{tabular}
|