decision/capability
What a decision model can do beyond the frozen 62 suites, measured on Kai and Jev with the same cases. Every suite is bench cap in hanzo-inc/kai (private; Rust, no Python): it builds its cases, puts them to each backend, and merges results/<suite>.json, {"meta", "keys", "detail"}, every key cap/<suite>/<who>/<metric>. bench speed writes results/questions.json; bench tex writes the paper's tables from these files.
| suite | measures |
|---|---|
orig |
the 62 frozen suites for a Kai checkpoint beside Jev, scored as harness/merge.py scores (bench score writes its scores.json) |
cardinality |
choice over K = 4 … 100,000: accuracy, recall@k, rejection, truncation, time; two wide-choice recipes (wide., sl.) and a bi-encoder shortlist (bi.) for every backend |
joint |
five dependent questions with known rules: per-variable accuracy, exact match, rule violations, and after projection onto the rules |
media |
human activity (UCI HAR), turbofan remaining life (C-MAPSS FD001), valve sound (MIMII): text renderings for Jev, the signals as evidence for Kai |
invariance |
option order, noul side swap, label aliases |
refresh |
a program re-run after one evidence change: questions asked, time against a cold re-run |
deploy |
offline in a no-network sandbox, data leaving the machine, self-hosting, fee, pinning |
conformal |
ECE, Brier, risk–coverage, split conformal coverage at 90/95% over every answer on disk |
questions |
bench speed: 1 … 1,000 questions on one state and 1 … 128 cases a call, latency p50/p95 and questions/s; Kai on one device and dtype; Jev over the network |
B=bench # hanzo-inc/kai: cargo build --release -p bench --features cuda (or rocm)
$B cap --harness decision --who kai,jev --kai <checkpoint> # every suite
$B cap --harness decision --suites cardinality --who kai --kai <checkpoint> # one suite, one backend; the others' results stay
$B speed --states decision/results/states.json.gz --results decision/capability/results \
--who kai --device rocm --dtype bf16 --kai <checkpoint> # --who jev once
$B score --harness decision --kai <preds.json.gz> --out scores.json # merge.py's scores.json
$B tex --harness decision --out <paper>/kai/tables # harness, languages, capability, speed
Jev is typesafe/jev-1.13 through OpenRouter's Decisions API (OPENROUTER_API_KEY, else the file OR_KEY_FILE names; never tracked); every call is ledgered in results/jev_spend.json and refused past --jev-cap (USD 5). A failure (refusal, truncation, error, timeout) is counted in unanswered/rejected, never dropped; a backend or suite that fails is recorded and the rest still run. Cases are drawn with CPython's random and numpy's PCG64 permutation; --limit 5 --results <dir> is a smoke run. --data (default ~/scratch/capability/data) holds the sources (WordNet 3.0, UCI HAR, C-MAPSS, MIMII), fetched and checked against their pinned hashes when absent; cases/*.json.gz (MIMII audio among them, used, not redistributed) are never tracked.
Each result file's meta names every run's host, checkpoint, weights, device and dtype.
cardinality.py
cardinality.json holds kai-1 (weights 0834a74f…, CUDA F32, 2026-10-03) beside Jev, and results/cardinality/kai.* its answers. cardinality.py reads the choice sizes where every option is scored, 4, 16, 77 and 150, in Python and over HTTP, with no Rust:
python3 capability/cardinality.py --score # kai-1 and Jev, from their recorded answers
uv run --with datasets python capability/cardinality.py --check # the rebuilt cases reproduce both records question by question
HANZO_API_KEY=... uv run --with datasets python capability/cardinality.py --url https://api.hanzo.ai --out runs/kai [--limit 40]
It builds the cases as bench cap does (Banking77's 400 harness cases at 4, 16 and 77 options, CLINC150 test at 150), asks Kai one request a case, and prints its accuracy beside the recorded kai-1 and Jev. Jev is never called (--jev recorded). WordNet at 1,000 options and more and the shortlist variants stay as recorded.