mirror of https://github.com/hanzoai/benchmarks.git synced 2026-10-30 14:59:11 +00:00

Files

T

zeekay a3c7859112 decision: the documents read Kai against Jev alone

README.md, SUITE.md, capability/README.md and results/kai-1 compare Kai
and Jev and name no other model; the harness's origin is the upstream
builders, pinned in harness/three_way.py. merge.py's table.md shows Kai
and Jev; its scores.json keeps every backend under the keys and paths
hanzo-inc/kai's bench reads, so results/kai-1/scores.json is unchanged
byte for byte. results/scores.json and table.md, merge.py's default
output, are no longer committed. trained/ measured a baseline the
documents no longer show and is removed. The jailbreak preregistration
describes its base encoder instead of naming it.

Bare python commands are python3, which is what a stock Linux has.

2026-10-15 11:57:00 -07:00

8.8 KiB

Raw Permalink Blame History

decision: Kai and Jev on the frozen harness

Written by harness/merge.py from the committed results; every backend is scored by the same functions.

kai: model kai-1, weights 0834a74f2d140642a453373da09e5a128e3d2c8d9e8bb1dcaf86d7210a4ecdfc, device cuda, dtype bf16
jev: model typesafe/jev-1.13, served ['typesafe/jev-1.13-20260917']

Every suite

suite	backend	answered	acc	macro F1	ECE	Brier	log loss	AURC	acc@50%	P(gold)≈0	score MAE
jev.ag_news	kai	400/400	0.943	0.936	0.033	0.091	0.182	0.008	1.000	0.000	–
jev.ag_news	jev	400/400	0.860	0.849	0.102	0.232	1.805	0.083	0.925	0.058	–
jev.emotion	kai	400/400	0.935	0.896	0.018	0.080	0.144	0.005	1.000	0.000	–
jev.emotion	jev	400/400	0.603	0.515	0.280	0.646	4.683	0.231	0.785	0.142	–
jev.banking77_full	kai	400/400	0.907	0.347	0.074	0.165	0.748	0.026	0.980	0.015	–
jev.banking77_full	jev	400/400	0.835	0.314	0.073	0.257	1.347	0.058	0.965	0.035	–
app.support_triage	kai	400/400	0.440	0.419	0.444	0.977	3.781	0.401	0.565	0.025	–
app.support_triage	jev	400/400	0.365	0.368	0.483	1.056	10.120	0.487	0.480	0.335	–
app.email_spam	kai	400/400	0.998	0.998	0.003	0.005	0.009	0.000	1.000	0.000	–
app.email_spam	jev	400/400	0.978	0.977	0.060	0.038	0.097	0.001	1.000	0.000	–
app.phishing	kai	400/400	0.990	0.989	0.010	0.020	0.071	0.000	1.000	0.000	–
app.phishing	jev	400/400	0.900	0.887	0.039	0.152	0.249	0.021	0.995	0.000	–
app.guardrails_jailbreak	kai	400/400	0.917	0.871	0.074	0.148	0.419	0.024	0.975	0.000	–
app.guardrails_jailbreak	jev	400/400	0.940	0.918	0.047	0.104	0.189	0.012	0.995	0.000	–
app.moderation_toxicity	kai	400/400	0.802	0.798	0.195	0.386	1.805	0.134	0.840	0.025	–
app.moderation_toxicity	jev	400/400	0.662	0.628	0.177	0.432	0.619	0.138	0.860	0.000	–
app.rag_relevance	kai	400/400	0.680	0.678	0.028	0.419	0.607	0.232	0.755	0.000	–
app.rag_relevance	jev	400/400	0.620	0.592	0.279	0.628	1.113	0.331	0.670	0.000	–
app.model_routing_domain	kai	399/399	1.000	1.000	0.000	0.000	0.000	0.000	1.000	0.000	–
app.model_routing_domain	jev	399/399	0.977	0.592	0.014	0.040	0.085	0.003	1.000	0.000	–
typed_decisions	kai	2000/2000	0.759	0.742	0.171	0.388	0.688	0.119	0.890	0.000	0.250
typed_decisions	jev	2000/2000	0.736	0.738	0.047	0.366	0.707	0.128	0.885	0.003	0.391
MASSIVE, 51 langs, macro	kai	5100/5100	0.910	0.901	0.060	0.136	0.377	0.011	0.998	0.000	–
MASSIVE, 51 langs, macro	jev	5099/5100	0.890	0.886	0.064	0.161	0.775	0.020	0.995	0.018	–

Risk at coverage

Error rate of the most confident answers, ties in confidence taken together.

suite	backend	10%	25%	50%	75%	90%	100%
jev.ag_news	kai	0.000	0.000	0.000	0.010	0.025	0.058
jev.ag_news	jev	0.077	0.077	0.077	0.078	0.103	0.140
jev.emotion	kai	0.000	0.000	0.000	0.000	0.017	0.065
jev.emotion	jev	0.127	0.127	0.217	0.331	0.375	0.398
jev.banking77_full	kai	0.023	0.010	0.020	0.027	0.053	0.092
jev.banking77_full	jev	0.025	0.025	0.032	0.097	0.116	0.165
app.support_triage	kai	0.150	0.290	0.435	0.487	0.533	0.560
app.support_triage	jev	0.263	0.371	0.522	0.610	0.639	0.635
app.email_spam	kai	0.000	0.000	0.000	0.000	0.000	0.003
app.email_spam	jev	0.000	0.000	0.000	0.000	0.000	0.022
app.phishing	kai	0.000	0.000	0.000	0.000	0.000	0.010
app.phishing	jev	0.000	0.000	0.005	0.050	0.075	0.100
app.guardrails_jailbreak	kai	0.000	0.000	0.025	0.027	0.056	0.083
app.guardrails_jailbreak	jev	0.000	0.000	0.004	0.021	0.037	0.060
app.moderation_toxicity	kai	0.065	0.100	0.160	0.147	0.172	0.198
app.moderation_toxicity	jev	0.000	0.005	0.142	0.239	0.311	0.338
app.rag_relevance	kai	0.175	0.160	0.245	0.290	0.311	0.320
app.rag_relevance	jev	0.282	0.296	0.324	0.367	0.385	0.380
app.model_routing_domain	kai	0.000	0.000	0.000	0.000	0.000	0.000
app.model_routing_domain	jev	0.000	0.000	0.000	0.006	0.008	0.023
typed_decisions	kai	0.040	0.046	0.110	0.183	0.213	0.241
typed_decisions	jev	0.025	0.077	0.114	0.186	0.231	0.265
MASSIVE, macro	kai	0.000	0.002	0.002	0.009	0.038	0.090
MASSIVE, macro	jev	0.001	0.001	0.005	0.031	0.062	0.110

MASSIVE by language

language	kai acc	jev acc	kai ECE	jev ECE	kai log loss	jev log loss
af	0.900	0.870	0.056	0.087	0.334	0.934
am	0.850	0.850	0.085	0.096	0.586	1.004
ar	0.920	0.870	0.064	0.076	0.320	0.916
az	0.930	0.880	0.045	0.048	0.207	1.120
bn	0.920	0.860	0.062	0.067	0.510	0.632
cy	0.850	0.760	0.074	0.088	0.496	1.776
da	0.920	0.880	0.051	0.044	0.335	0.811
de	0.910	0.939	0.047	0.057	0.314	0.421
el	0.940	0.940	0.041	0.057	0.248	0.431
en	0.940	0.920	0.044	0.025	0.291	0.689
es	0.920	0.930	0.049	0.063	0.344	0.493
fa	0.940	0.950	0.032	0.053	0.252	0.414
fi	0.930	0.890	0.051	0.038	0.330	0.784
fr	0.910	0.900	0.074	0.065	0.380	0.498
he	0.910	0.940	0.063	0.043	0.450	0.430
hi	0.920	0.950	0.059	0.029	0.367	0.408
hu	0.900	0.920	0.079	0.039	0.513	1.054
hy	0.900	0.840	0.062	0.090	0.454	0.629
id	0.960	0.930	0.052	0.045	0.188	0.727
is	0.850	0.840	0.081	0.073	0.486	0.978
it	0.910	0.960	0.054	0.050	0.353	0.357
ja	0.970	0.970	0.037	0.059	0.210	0.375
jv	0.890	0.750	0.081	0.086	0.578	1.835
ka	0.830	0.880	0.085	0.108	0.674	0.962
km	0.840	0.810	0.084	0.073	0.532	1.260
kn	0.910	0.870	0.076	0.070	0.432	0.555
ko	0.910	0.940	0.051	0.042	0.265	0.397
lv	0.890	0.850	0.078	0.075	0.455	0.935
ml	0.870	0.910	0.103	0.050	0.633	0.552
mn	0.880	0.840	0.097	0.083	0.560	0.691
ms	0.930	0.920	0.045	0.068	0.341	1.028
my	0.920	0.890	0.070	0.057	0.299	0.557
nb	0.940	0.860	0.037	0.071	0.260	0.620
nl	0.920	0.900	0.049	0.055	0.271	0.827
pl	0.930	0.920	0.041	0.036	0.266	0.478
pt	0.930	0.900	0.036	0.099	0.358	1.078
ro	0.910	0.940	0.051	0.082	0.235	0.753
ru	0.970	0.940	0.030	0.033	0.099	0.408
sl	0.910	0.890	0.048	0.058	0.305	0.596
sq	0.940	0.760	0.058	0.094	0.338	1.175
sv	0.930	0.940	0.037	0.074	0.216	0.531
sw	0.840	0.700	0.087	0.112	0.721	2.696
ta	0.890	0.920	0.073	0.092	0.405	0.876
te	0.870	0.910	0.083	0.037	0.486	0.498
th	0.930	0.940	0.039	0.047	0.230	0.422
tl	0.910	0.850	0.067	0.085	0.478	1.125
tr	0.900	0.920	0.075	0.061	0.581	0.834
ur	0.900	0.900	0.059	0.064	0.359	0.564
vi	0.910	0.890	0.070	0.052	0.360	0.520
zh-CN	0.940	0.940	0.059	0.036	0.268	0.426
zh-TW	0.950	0.930	0.033	0.062	0.225	0.464
macro	0.910	0.890	0.060	0.064	0.377	0.775

MASSIVE	kai	jev
English	0.940	0.920
other languages, macro	0.909	0.890
languages above 3× random	51	51

Jev: 50 sequential calls, p50 227.3 ms, p95 333.9 ms; the whole run cost $0.2486.

Metrics

acc, macro F1, ECE (15 bins, top-label confidence), Brier (summed over options), log loss (natural log, p floored at 1e-12), acc@50% (accuracy of the more confident half): upstream bench_local.py, verbatim.
AURC: area under the risk-coverage curve, the mean error rate over coverages k/n; lower is better.
P(gold)≈0: share of answers that give the gold option a probability under 1e-6.
score MAE: typed decisions' score questions, |expected level − the teacher's mean level|.
answered: questions with a probability vector; an unanswered question is not scored as wrong.

8.8 KiB Raw Permalink Blame History Unescape Escape