mirror of
https://github.com/hanzoai/benchmarks.git
synced 2026-10-30 14:59:11 +00:00
zeekay
4f88ad2f75
The suite preregistered on 2026-10-14 and read that night: prompt injection, jailbreak, multi-turn, harmful content and AI-generated text, each item asked under the harness's jailbreak question and under the paper's detector for its task. fetch.py downloads the 46 files build.py reads at their pinned revisions and checks each SHA-256; build.py draws the items and checks every text against manifest.jsonl.gz, which carries labels, text digests and each item's clean-for-Kai decision, made against Kai's private training registry. No dataset text is committed. read.py asks any server; score.py reproduces results/report.txt and results.json byte for byte from the recorded answers, Jev's always recorded. In-the-wild benign prompts flagged: Kai 24 of 600, Jev 126 of 600. On the real-attack pool Kai flags 3.6% of benign prompts against 13.0% and catches 26.9% of attacks against 77.4%. The preregistration's host names, local paths and private file and dataset names are replaced by descriptions; nothing else in it changed.