mirror of
https://github.com/hanzoai/benchmarks.git
synced 2026-10-30 14:59:11 +00:00
zeekay
a3c7859112
README.md, SUITE.md, capability/README.md and results/kai-1 compare Kai and Jev and name no other model; the harness's origin is the upstream builders, pinned in harness/three_way.py. merge.py's table.md shows Kai and Jev; its scores.json keeps every backend under the keys and paths hanzo-inc/kai's bench reads, so results/kai-1/scores.json is unchanged byte for byte. results/scores.json and table.md, merge.py's default output, are no longer committed. trained/ measured a baseline the documents no longer show and is removed. The jailbreak preregistration describes its base encoder instead of naming it. Bare python commands are python3, which is what a stock Linux has.
19 lines
1.0 KiB
Python
19 lines
1.0 KiB
Python
"""The harness's evaluation cases with their states: results/states.json.gz.
|
|
|
|
{suite: [[state, questions, gold], ...]}, from three_way.py suites(). Dropping the states gives
|
|
results/questions.json.gz exactly; the script checks that. Kai's Rust bench reads this file
|
|
(`bench preds`, `bench speed`). LAYA_SRC is the upstream checkout three_way.upstream() names.
|
|
|
|
LAYA_SRC=/path/to/checkout uv run --python 3.11 --with torch --with pandas --with pyarrow \\
|
|
--with datasets --with "$LAYA_SRC" python harness/states.py
|
|
"""
|
|
import gzip, json, os
|
|
from three_way import suites
|
|
RESULTS = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "results")
|
|
out = suites()
|
|
with gzip.open(os.path.join(RESULTS, "states.json.gz"), "wt") as f:
|
|
json.dump(out, f, ensure_ascii=False)
|
|
frozen = json.load(gzip.open(os.path.join(RESULTS, "questions.json.gz")))
|
|
mine = {n: [[g, qs] for _, qs, g in rows] for n, rows in out.items()}
|
|
print("suites", len(mine), "match", json.dumps(mine, sort_keys=True) == json.dumps(frozen, sort_keys=True))
|