mirror of
https://github.com/hanzoai/benchmarks.git
synced 2026-10-30 14:59:11 +00:00
zeekay
7e23047ddb
The Python suite is deleted; hanzoai/decision's bench builds the same
cases, answers them with Laya on its native runtime, Jev through OpenRouter
and a Kai checkpoint, and merges the same results/<suite>.json keys.
bench speed writes results/questions.json (Kai and Laya interleaved on
Metal F32, Metal BF16 and CPU F32 on ra; Jev over the network), bench
score reproduces harness/merge.py's scores.json, bench tex writes the
paper's tables from these files.
Checked against the Python runs this replaces (c9ed09f): every case set
equal, Laya's keys equal to four decimals outside timings, Jev's within
0.03. Not reproduced, by design: WordNet at K >= 1,000 (its labels
depended on Python's string hash; the port's label space shares 795 of the
kept 1,000) and conformal's joint population (the Python read a 2-state
smoke case file). Jev's reruns cost $0.70 over 13,325 calls.
Claude-Session: https://claude.ai/code/session_018Gt5TvrE2rxXuZikWSauv3
1 line
1.8 KiB
JSON
1 line
1.8 KiB
JSON
{"cap_usd":5,"spent_usd":0.699395,"calls":13326,"refused":0,"by_suite":{"joint":{"usd":0.017648,"calls":400},"cardinality.k4":{"usd":0.006198,"calls":400},"cardinality.k16":{"usd":0.009654,"calls":400},"cardinality.k77":{"usd":0.027234,"calls":400},"cardinality.k150":{"usd":0.040971,"calls":400},"cardinality.k1000":{"usd":0.0,"calls":0},"cardinality.k10000":{"usd":0.0,"calls":0},"cardinality.k100000":{"usd":0.0,"calls":0},"cardinality.sl.k77":{"usd":0.011044,"calls":400},"cardinality.bi.k77":{"usd":0.010851,"calls":400},"cardinality.sl.k150":{"usd":0.010185,"calls":400},"cardinality.bi.k150":{"usd":0.009705,"calls":400},"cardinality.sl.k1000":{"usd":0.006791,"calls":200},"cardinality.bi.k1000":{"usd":0.007075,"calls":200},"cardinality.sl.k10000":{"usd":0.006916,"calls":200},"cardinality.bi.k10000":{"usd":0.007035,"calls":200},"cardinality.sl.k100000":{"usd":0.001728,"calls":50},"cardinality.bi.k100000":{"usd":0.001674,"calls":50},"media.har":{"usd":0.05043,"calls":600},"media.cmapss":{"usd":0.016278,"calls":200},"media.mimii":{"usd":0.011434,"calls":200},"invariance.perm0":{"usd":0.009622,"calls":400},"invariance.perm1":{"usd":0.009622,"calls":400},"invariance.perm2":{"usd":0.009622,"calls":400},"invariance.perm3":{"usd":0.009622,"calls":400},"invariance.perm4":{"usd":0.009622,"calls":400},"invariance.swap0":{"usd":0.018558,"calls":800},"invariance.swap.desc":{"usd":0.018558,"calls":800},"invariance.swap.neg":{"usd":0.018676,"calls":800},"invariance.alias.text":{"usd":0.009622,"calls":400},"invariance.alias.snake":{"usd":0.010072,"calls":400},"invariance.alias.code":{"usd":0.009512,"calls":400},"refresh.reask":{"usd":0.009145,"calls":200},"deploy":{"usd":0.00003,"calls":2},"questions":{"usd":0.230464,"calls":165},"questions.batch":{"usd":0.073797,"calls":1859}}} |