mirror of
https://github.com/hanzoai/benchmarks.git
synced 2026-10-30 14:59:11 +00:00
zeekay
7e23047ddb
The Python suite is deleted; hanzoai/decision's bench builds the same
cases, answers them with Laya on its native runtime, Jev through OpenRouter
and a Kai checkpoint, and merges the same results/<suite>.json keys.
bench speed writes results/questions.json (Kai and Laya interleaved on
Metal F32, Metal BF16 and CPU F32 on ra; Jev over the network), bench
score reproduces harness/merge.py's scores.json, bench tex writes the
paper's tables from these files.
Checked against the Python runs this replaces (c9ed09f): every case set
equal, Laya's keys equal to four decimals outside timings, Jev's within
0.03. Not reproduced, by design: WordNet at K >= 1,000 (its labels
depended on Python's string hash; the port's label space shares 795 of the
kept 1,000) and conformal's joint population (the Python read a 2-state
smoke case file). Jev's reruns cost $0.70 over 13,325 calls.
Claude-Session: https://claude.ai/code/session_018Gt5TvrE2rxXuZikWSauv3