hanzoai/papers
2
0
Fork 0
mirror of https://github.com/hanzoai/papers.git synced 2026-10-22 17:20:17 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Kai: harness and language tables from stage a5 (macro 0.851 over eleven suites, best of three in 8 of 12), which reads typed decisions' train split whole.

Browse Source 
Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
zeekay
2026-09-27 14:08:18 -07:00
co-authored by Hanzo Dev
parent 8ed5a61078
commit 4b15b53d35
4 changed files with 52 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
kai/sections/05-findings.tex
+3 -2
View File
@@ -104,8 +104,9 @@ irreversible action, gold \emph{harmful}, \emph{human review}, risk 3; in test
none of either and two tool errors, gold \emph{partial}, \emph{observe}, risk 1.
So Laya's typed-decisions checkpoint saw same-wording siblings of half the test
set, with different numbers and different gold, and no near-copy of a test
answer. Kai's builds drop the 547, so Kai trains on 638 of the states the
builder reads. Whether knowing the templates helps is measured, not assumed:
answer. Stage A's builds drop the 547, so it trains on 638 of the states the
builder reads; from stage a5 an official train split is checked only for exact
copies and keys, so all 1{,}185 train. Whether knowing the templates helps is measured, not assumed:
\texttt{harness/siblings.py} in the benchmark repository scores the 235 test
states without a sibling and the 165 with one apart
(Table~\ref{tab:siblings}). Laya's typed checkpoint scores alike on both
kai/sections/08-evaluation.tex
+4 -3
View File
@@ -4,13 +4,14 @@
Every table in this section is written by \texttt{bench tex} in
\texttt{hanzoai/decision} from the kept results of the runs it names, into
\texttt{tables/}; no number in a table is typed by hand. The capability and speed tables
are stubs until the final stage-A checkpoint is timed; each says so. Each table's last row
are stubs until the scored checkpoint is timed; each says so. Each table's last row
names the Kai revision, the device and the harness and benchmark commits it
reports.
\paragraph{Backends.}
\textbf{Kai}: the final stage-A checkpoint (Section~\ref{sec:collapse}) on the
native runtime. \textbf{Laya}: Laya 0.3.20's published weights served natively
\textbf{Kai}: stage a5, one epoch on from the final stage-A checkpoint
(Section~\ref{sec:collapse}) with typed decisions' train split read whole
(Section~\ref{sec:overlap}), on the native runtime. \textbf{Laya}: Laya 0.3.20's published weights served natively
by the Rust port of its joint head (parity in Section~\ref{sec:checkpoints}),
each case on the checkpoint Laya's router picks, except typed decisions, which
are answered by Laya's typed-decisions checkpoint: the router sends them to the
kai/tables/harness.tex
+17 -17
View File
@@ -1,26 +1,26 @@
% harness.tex: written by `bench tex` (hanzoai/decision); Kai runs/a4, weights dcd35aa6, cpu f32; harness 735e240; benchmarks c68a38b
% harness.tex: written by `bench tex` (hanzoai/decision); Kai runs/a5, weights df1c16a2, metal bf16; harness 735e240; benchmarks c68a38b
% Laya: the checkpoint its router picks per case; typed decisions on laya-agent (kai-1-agent), the typed-decisions checkpoint Laya ships and its router does not pick.
\begin{tabular}{@{}l@{\quad}ccc@{\quad}ccc@{\quad}ccc@{}}
\toprule
& \multicolumn{3}{c}{Accuracy} & \multicolumn{3}{c}{Brier} & \multicolumn{3}{c}{ECE} \\
Suite & Kai & Laya & Jev & Kai & Laya & Jev & Kai & Laya & Jev \\
\midrule
AG News & 0.935 & \textbf{0.950} & 0.860 & 0.100 & 0.081 & 0.232 & 0.020 & 0.032 & 0.102 \\
DAIR Emotion & \textbf{0.938} & 0.595 & 0.603 & 0.092 & 0.696 & 0.646 & 0.030 & 0.306 & 0.280 \\
Banking77 & \textbf{0.885} & 0.425 & 0.835 & 0.173 & 0.925 & 0.257 & 0.079 & 0.381 & 0.073 \\
Support triage & \textbf{0.453} & 0.502$^\dagger$ & 0.365 & 0.719 & 0.644 & 1.056 & 0.203 & 0.091 & 0.483 \\
Email spam & 0.995 & \textbf{0.998} & 0.978 & 0.009 & 0.005 & 0.038 & 0.006 & 0.011 & 0.060 \\
Phishing & \textbf{0.990} & 0.983$^\dagger$ & 0.900 & 0.021 & 0.026 & 0.152 & 0.011 & 0.010 & 0.039 \\
Jailbreak & 0.907 & 0.705 & \textbf{0.940} & 0.143 & 0.543 & 0.104 & 0.061 & 0.262 & 0.047 \\
Toxicity & \textbf{0.833} & 0.530 & 0.662 & 0.304 & 0.602 & 0.432 & 0.145 & 0.297 & 0.177 \\
RAG relevance & \textbf{0.672} & 0.625 & 0.620 & 0.407 & 0.466 & 0.628 & 0.041 & 0.117 & 0.279 \\
Model routing & \textbf{1.000} & 0.639 & 0.977 & 0.000 & 0.494 & 0.040 & 0.000 & 0.089 & 0.014 \\
Typed decisions & 0.523 & \textbf{0.766} & 0.736 & 0.564 & 0.400 & 0.366 & 0.042 & 0.213 & 0.047 \\
MASSIVE (51 languages, macro) & 0.858 & 0.382 & \textbf{0.890} & 0.210 & 0.987 & 0.161 & 0.079 & 0.418 & 0.064 \\
AG News & \textbf{0.950} & \textbf{0.950} & 0.860 & 0.099 & 0.081 & 0.232 & 0.070 & 0.032 & 0.102 \\
DAIR Emotion & \textbf{0.925} & 0.595 & 0.603 & 0.089 & 0.696 & 0.646 & 0.041 & 0.306 & 0.280 \\
Banking77 & \textbf{0.907} & 0.425 & 0.835 & 0.155 & 0.925 & 0.257 & 0.065 & 0.381 & 0.073 \\
Support triage & \textbf{0.490} & 0.502$^\dagger$ & 0.365 & 0.705 & 0.644 & 1.056 & 0.161 & 0.091 & 0.483 \\
Email spam & 0.995 & \textbf{0.998} & 0.978 & 0.004 & 0.005 & 0.038 & 0.004 & 0.011 & 0.060 \\
Phishing & \textbf{0.990} & 0.983$^\dagger$ & 0.900 & 0.020 & 0.026 & 0.152 & 0.010 & 0.010 & 0.039 \\
Jailbreak & 0.915 & 0.705 & \textbf{0.940} & 0.143 & 0.543 & 0.104 & 0.055 & 0.262 & 0.047 \\
Toxicity & \textbf{0.820} & 0.530 & 0.662 & 0.339 & 0.602 & 0.432 & 0.167 & 0.297 & 0.177 \\
RAG relevance & \textbf{0.667} & 0.625 & 0.620 & 0.418 & 0.466 & 0.628 & 0.037 & 0.117 & 0.279 \\
Model routing & \textbf{1.000} & 0.639 & 0.977 & 0.000 & 0.494 & 0.040 & 0.002 & 0.089 & 0.014 \\
Typed decisions & 0.705 & \textbf{0.766} & 0.736 & 0.449 & 0.400 & 0.366 & 0.171 & 0.213 & 0.047 \\
MASSIVE (51 languages, macro) & 0.888 & 0.382 & \textbf{0.890} & 0.157 & 0.987 & 0.161 & 0.066 & 0.418 & 0.064 \\
\midrule
Macro, 11 suites & 0.830 & 0.702 & 0.770 & 0.230 & 0.444 & 0.359 & 0.058 & 0.164 & 0.145 \\
Clean macro, 9 suites & 0.854 & 0.693 & 0.801 & & & & & & \\
Best of three, 12 suites & 7 & 3 & 2 & & & & & & \\
Macro, 11 suites & 0.851 & 0.702 & 0.770 & 0.220 & 0.444 & 0.359 & 0.071 & 0.164 & 0.145 \\
Clean macro, 9 suites & 0.876 & 0.693 & 0.801 & & & & & & \\
Best of three, 12 suites & 8 & 3 & 2 & & & & & & \\
\bottomrule
\multicolumn{10}{@{}l@{}}{\footnotesize\begin{tabular}[t]{@{}l@{}}Kai runs/a4, weights dcd35aa6, cpu f32 \\ harness 735e240, benchmarks c68a38b\end{tabular}} \\
\multicolumn{10}{@{}l@{}}{\footnotesize\begin{tabular}[t]{@{}l@{}}Kai runs/a5, weights df1c16a2, metal bf16 \\ harness 735e240, benchmarks c68a38b\end{tabular}} \\
\end{tabular}
kai/tables/languages.tex
+28 -28
View File
@@ -1,35 +1,35 @@
% languages.tex: written by `bench tex` (hanzoai/decision); Kai runs/a4, weights dcd35aa6, cpu f32; harness 735e240; benchmarks c68a38b
% languages.tex: written by `bench tex` (hanzoai/decision); Kai runs/a5, weights df1c16a2, metal bf16; harness 735e240; benchmarks c68a38b
% MASSIVE intent accuracy by language, 20 options, 100 utterances each; Laya as its router picks.
\begin{tabular}{@{}lccc@{\qquad}lccc@{}}
\toprule
Language & Kai & Laya & Jev & Language & Kai & Laya & Jev \\
\midrule
af & 0.870 & 0.340 & 0.870 & ko & 0.890 & 0.470 & 0.940 \\
am & 0.770 & 0.150 & 0.850 & lv & 0.850 & 0.300 & 0.850 \\
ar & 0.900 & 0.460 & 0.870 & ml & 0.810 & 0.280 & 0.910 \\
az & 0.870 & 0.360 & 0.880 & mn & 0.740 & 0.160 & 0.840 \\
bn & 0.880 & 0.450 & 0.860 & ms & 0.930 & 0.270 & 0.920 \\
cy & 0.740 & 0.160 & 0.760 & my & 0.840 & 0.160 & 0.890 \\
da & 0.880 & 0.460 & 0.880 & nb & 0.900 & 0.500 & 0.860 \\
de & 0.910 & 0.530 & 0.939 & nl & 0.910 & 0.400 & 0.900 \\
el & 0.860 & 0.440 & 0.940 & pl & 0.900 & 0.470 & 0.920 \\
en & 0.890 & 0.820 & 0.920 & pt & 0.920 & 0.440 & 0.900 \\
es & 0.890 & 0.530 & 0.930 & ro & 0.820 & 0.370 & 0.940 \\
fa & 0.910 & 0.510 & 0.950 & ru & 0.920 & 0.570 & 0.940 \\
fi & 0.860 & 0.300 & 0.890 & sl & 0.860 & 0.330 & 0.890 \\
fr & 0.900 & 0.600 & 0.900 & sq & 0.850 & 0.290 & 0.760 \\
he & 0.840 & 0.370 & 0.940 & sv & 0.910 & 0.480 & 0.940 \\
hi & 0.860 & 0.460 & 0.950 & sw & 0.800 & 0.130 & 0.700 \\
hu & 0.820 & 0.340 & 0.920 & ta & 0.830 & 0.310 & 0.920 \\
hy & 0.760 & 0.250 & 0.840 & te & 0.800 & 0.220 & 0.910 \\
id & 0.910 & 0.360 & 0.930 & th & 0.890 & 0.480 & 0.940 \\
is & 0.780 & 0.330 & 0.840 & tl & 0.860 & 0.300 & 0.850 \\
it & 0.910 & 0.470 & 0.960 & tr & 0.880 & 0.410 & 0.920 \\
ja & 0.950 & 0.640 & 0.970 & ur & 0.850 & 0.420 & 0.900 \\
jv & 0.820 & 0.160 & 0.750 & vi & 0.830 & 0.330 & 0.890 \\
ka & 0.750 & 0.150 & 0.880 & zh-CN & 0.930 & 0.650 & 0.940 \\
km & 0.760 & 0.200 & 0.810 & zh-TW & 0.950 & 0.610 & 0.930 \\
kn & 0.770 & 0.300 & 0.870 & & & & \\
af & 0.890 & 0.340 & 0.870 & ko & 0.920 & 0.470 & 0.940 \\
am & 0.840 & 0.150 & 0.850 & lv & 0.850 & 0.300 & 0.850 \\
ar & 0.880 & 0.460 & 0.870 & ml & 0.820 & 0.280 & 0.910 \\
az & 0.920 & 0.360 & 0.880 & mn & 0.880 & 0.160 & 0.840 \\
bn & 0.870 & 0.450 & 0.860 & ms & 0.920 & 0.270 & 0.920 \\
cy & 0.820 & 0.160 & 0.760 & my & 0.900 & 0.160 & 0.890 \\
da & 0.910 & 0.460 & 0.880 & nb & 0.910 & 0.500 & 0.860 \\
de & 0.920 & 0.530 & 0.939 & nl & 0.920 & 0.400 & 0.900 \\
el & 0.900 & 0.440 & 0.940 & pl & 0.940 & 0.470 & 0.920 \\
en & 0.920 & 0.820 & 0.920 & pt & 0.920 & 0.440 & 0.900 \\
es & 0.900 & 0.530 & 0.930 & ro & 0.900 & 0.370 & 0.940 \\
fa & 0.930 & 0.510 & 0.950 & ru & 0.960 & 0.570 & 0.940 \\
fi & 0.920 & 0.300 & 0.890 & sl & 0.900 & 0.330 & 0.890 \\
fr & 0.920 & 0.600 & 0.900 & sq & 0.880 & 0.290 & 0.760 \\
he & 0.870 & 0.370 & 0.940 & sv & 0.880 & 0.480 & 0.940 \\
hi & 0.930 & 0.460 & 0.950 & sw & 0.810 & 0.130 & 0.700 \\
hu & 0.870 & 0.340 & 0.920 & ta & 0.840 & 0.310 & 0.920 \\
hy & 0.840 & 0.250 & 0.840 & te & 0.830 & 0.220 & 0.910 \\
id & 0.920 & 0.360 & 0.930 & th & 0.930 & 0.480 & 0.940 \\
is & 0.860 & 0.330 & 0.840 & tl & 0.860 & 0.300 & 0.850 \\
it & 0.940 & 0.470 & 0.960 & tr & 0.880 & 0.410 & 0.920 \\
ja & 0.940 & 0.640 & 0.970 & ur & 0.860 & 0.420 & 0.900 \\
jv & 0.830 & 0.160 & 0.750 & vi & 0.890 & 0.330 & 0.890 \\
ka & 0.790 & 0.150 & 0.880 & zh-CN & 0.930 & 0.650 & 0.940 \\
km & 0.800 & 0.200 & 0.810 & zh-TW & 0.970 & 0.610 & 0.930 \\
kn & 0.870 & 0.300 & 0.870 & & & & \\
\bottomrule
\multicolumn{8}{@{}l@{}}{\footnotesize\begin{tabular}[t]{@{}l@{}}Kai runs/a4, weights dcd35aa6, cpu f32 \\ harness 735e240, benchmarks c68a38b\end{tabular}} \\
\multicolumn{8}{@{}l@{}}{\footnotesize\begin{tabular}[t]{@{}l@{}}Kai runs/a5, weights df1c16a2, metal bf16 \\ harness 735e240, benchmarks c68a38b\end{tabular}} \\
\end{tabular}