benchmarks/decision/results/kai-1.9/table.md

decision: Kai and Jev on the frozen harness

Written by harness/merge.py from the committed results; every backend is scored by the same functions.

• kai: model kai, weights 8069bf5cc5e897ce28399d3a5d2a17097e3859d85e5a4cda50c7f19f1e60e9f4, device cpu, dtype as served
• jev: model typesafe/jev-1.13, served ['typesafe/jev-1.13-20260917']

Every suite

suite	backend	answered	acc	macro F1	ECE	Brier	log loss	AURC	acc@50%	P(gold)≈0	score MAE
jev.ag_news	kai	397/400	0.942	0.936	0.038	0.090	0.175	0.008	1.000	0.000	–
jev.ag_news	jev	400/400	0.860	0.849	0.102	0.232	1.805	0.083	0.925	0.058	–
jev.emotion	kai	399/400	0.937	0.895	0.025	0.080	0.145	0.005	1.000	0.000	–
jev.emotion	jev	400/400	0.603	0.515	0.280	0.646	4.683	0.231	0.785	0.142	–
jev.banking77_full	kai	400/400	0.912	0.392	0.079	0.167	0.804	0.026	0.975	0.015	–
jev.banking77_full	jev	400/400	0.835	0.314	0.073	0.257	1.347	0.058	0.965	0.035	–
app.support_triage	kai	400/400	0.432	0.414	0.461	0.995	4.165	0.414	0.560	0.050	–
app.support_triage	jev	400/400	0.365	0.368	0.483	1.056	10.120	0.487	0.480	0.335	–
app.email_spam	kai	399/400	0.995	0.995	0.005	0.009	0.019	0.000	1.000	0.000	–
app.email_spam	jev	400/400	0.978	0.977	0.060	0.038	0.097	0.001	1.000	0.000	–
app.phishing	kai	400/400	0.988	0.987	0.013	0.025	0.091	0.000	1.000	0.000	–
app.phishing	jev	400/400	0.900	0.887	0.039	0.152	0.249	0.021	0.995	0.000	–
app.guardrails_jailbreak	kai	399/400	0.920	0.875	0.077	0.149	0.421	0.025	0.975	0.000	–
app.guardrails_jailbreak	jev	400/400	0.940	0.918	0.047	0.104	0.189	0.012	0.995	0.000	–
app.moderation_toxicity	kai	400/400	0.800	0.795	0.195	0.386	1.803	0.140	0.840	0.025	–
app.moderation_toxicity	jev	400/400	0.662	0.628	0.177	0.432	0.619	0.138	0.860	0.000	–
app.rag_relevance	kai	400/400	0.672	0.670	0.023	0.420	0.609	0.233	0.760	0.000	–
app.rag_relevance	jev	400/400	0.620	0.592	0.279	0.628	1.113	0.331	0.670	0.000	–
app.model_routing_domain	kai	399/399	1.000	1.000	0.000	0.000	0.000	0.000	1.000	0.000	–
app.model_routing_domain	jev	399/399	0.977	0.592	0.014	0.040	0.085	0.003	1.000	0.000	–
typed_decisions	kai	1635/2000	0.756	0.725	0.157	0.381	0.671	0.112	0.901	0.000	0.248
typed_decisions	jev	2000/2000	0.736	0.738	0.047	0.366	0.707	0.128	0.885	0.003	0.391
MASSIVE, 51 langs, macro	kai	5100/5100	0.911	0.902	0.061	0.137	0.383	0.011	0.997	0.000	–
MASSIVE, 51 langs, macro	jev	5099/5100	0.890	0.886	0.064	0.161	0.775	0.020	0.995	0.018	–

Risk at coverage

Error rate of the most confident answers, ties in confidence taken together.

suite	backend	10%	25%	50%	75%	90%	100%
jev.ag_news	kai	0.000	0.000	0.000	0.007	0.025	0.058
jev.ag_news	jev	0.077	0.077	0.077	0.078	0.103	0.140
jev.emotion	kai	0.000	0.000	0.000	0.000	0.019	0.063
jev.emotion	jev	0.127	0.127	0.217	0.331	0.375	0.398
jev.banking77_full	kai	0.025	0.010	0.025	0.027	0.056	0.087
jev.banking77_full	jev	0.025	0.025	0.032	0.097	0.116	0.165
app.support_triage	kai	0.200	0.310	0.440	0.493	0.536	0.568
app.support_triage	jev	0.263	0.371	0.522	0.610	0.639	0.635
app.email_spam	kai	0.000	0.000	0.000	0.000	0.000	0.005
app.email_spam	jev	0.000	0.000	0.000	0.000	0.000	0.022
app.phishing	kai	0.000	0.000	0.000	0.000	0.000	0.013
app.phishing	jev	0.000	0.000	0.005	0.050	0.075	0.100
app.guardrails_jailbreak	kai	0.000	0.010	0.025	0.027	0.056	0.080
app.guardrails_jailbreak	jev	0.000	0.000	0.004	0.021	0.037	0.060
app.moderation_toxicity	kai	0.100	0.100	0.160	0.153	0.172	0.200
app.moderation_toxicity	jev	0.000	0.005	0.142	0.239	0.311	0.338
app.rag_relevance	kai	0.175	0.180	0.240	0.287	0.308	0.328
app.rag_relevance	jev	0.282	0.296	0.324	0.367	0.385	0.380
app.model_routing_domain	kai	0.000	0.000	0.000	0.000	0.000	0.000
app.model_routing_domain	jev	0.000	0.000	0.000	0.006	0.008	0.023
typed_decisions	kai	0.018	0.039	0.099	0.180	0.217	0.244
typed_decisions	jev	0.025	0.077	0.114	0.186	0.231	0.265
MASSIVE, macro	kai	0.000	0.001	0.003	0.009	0.037	0.089
MASSIVE, macro	jev	0.001	0.001	0.005	0.031	0.062	0.110

MASSIVE by language

language	kai acc	jev acc	kai ECE	jev ECE	kai log loss	jev log loss
af	0.900	0.870	0.058	0.087	0.331	0.934
am	0.860	0.850	0.085	0.096	0.580	1.004
ar	0.910	0.870	0.073	0.076	0.318	0.916
az	0.940	0.880	0.038	0.048	0.209	1.120
bn	0.910	0.860	0.061	0.067	0.507	0.632
cy	0.860	0.760	0.069	0.088	0.507	1.776
da	0.920	0.880	0.058	0.044	0.333	0.811
de	0.910	0.939	0.051	0.057	0.316	0.421
el	0.940	0.940	0.063	0.057	0.246	0.431
en	0.940	0.920	0.051	0.025	0.312	0.689
es	0.920	0.930	0.067	0.063	0.348	0.493
fa	0.940	0.950	0.038	0.053	0.255	0.414
fi	0.920	0.890	0.056	0.038	0.337	0.784
fr	0.910	0.900	0.074	0.065	0.391	0.498
he	0.920	0.940	0.059	0.043	0.469	0.430
hi	0.920	0.950	0.078	0.029	0.372	0.408
hu	0.900	0.920	0.080	0.039	0.517	1.054
hy	0.900	0.840	0.064	0.090	0.459	0.629
id	0.960	0.930	0.048	0.045	0.192	0.727
is	0.860	0.840	0.078	0.073	0.499	0.978
it	0.920	0.960	0.060	0.050	0.361	0.357
ja	0.970	0.970	0.036	0.059	0.209	0.375
jv	0.890	0.750	0.087	0.086	0.579	1.835
ka	0.830	0.880	0.076	0.108	0.685	0.962
km	0.850	0.810	0.096	0.073	0.542	1.260
kn	0.900	0.870	0.077	0.070	0.436	0.555
ko	0.910	0.940	0.044	0.042	0.268	0.397
lv	0.890	0.850	0.083	0.075	0.460	0.935
ml	0.880	0.910	0.094	0.050	0.643	0.552
mn	0.890	0.840	0.094	0.083	0.578	0.691
ms	0.930	0.920	0.047	0.068	0.349	1.028
my	0.920	0.890	0.066	0.057	0.316	0.557
nb	0.950	0.860	0.040	0.071	0.255	0.620
nl	0.920	0.900	0.054	0.055	0.281	0.827
pl	0.930	0.920	0.048	0.036	0.266	0.478
pt	0.930	0.900	0.029	0.099	0.358	1.078
ro	0.920	0.940	0.053	0.082	0.244	0.753
ru	0.970	0.940	0.016	0.033	0.099	0.408
sl	0.900	0.890	0.044	0.058	0.318	0.596
sq	0.940	0.760	0.050	0.094	0.333	1.175
sv	0.940	0.940	0.048	0.074	0.220	0.531
sw	0.840	0.700	0.083	0.112	0.743	2.696
ta	0.880	0.920	0.081	0.092	0.417	0.876
te	0.880	0.910	0.076	0.037	0.491	0.498
th	0.930	0.940	0.043	0.047	0.228	0.422
tl	0.910	0.850	0.060	0.085	0.498	1.125
tr	0.890	0.920	0.090	0.061	0.601	0.834
ur	0.900	0.900	0.053	0.064	0.375	0.564
vi	0.910	0.890	0.070	0.052	0.362	0.520
zh-CN	0.940	0.940	0.053	0.036	0.272	0.426
zh-TW	0.950	0.930	0.027	0.062	0.223	0.464
macro	0.911	0.890	0.061	0.064	0.383	0.775

MASSIVE	kai	jev
English	0.940	0.920
other languages, macro	0.910	0.890
languages above 3× random	51	51

Jev: 50 sequential calls, p50 227.3 ms, p95 333.9 ms; the whole run cost $0.2486.

Metrics

• acc, macro F1, ECE (15 bins, top-label confidence), Brier (summed over options), log loss (natural log, p floored at 1e-12), acc@50% (accuracy of the more confident half): upstream bench_local.py, verbatim.
• AURC: area under the risk-coverage curve, the mean error rate over coverages k/n; lower is better.
• P(gold)≈0: share of answers that give the gold option a probability under 1e-6.
• score MAE: typed decisions' score questions, |expected level − the teacher's mean level|.
• answered: questions with a probability vector; an unanswered question is not scored as wrong.