Kai: harness and language tables from the final stage-A checkpoint (runs/a4), the caption defines bold, dagger and the clean macro, capability and speed stay stubs until that checkpoint is timed.

Co-authored-by: Hanzo Dev <[email protected]>

kai/sections/01-introduction.tex

@@ -67,8 +67,7 @@ dependent questions independently, Laya breaks one of the program's rules in
 64\% of states and Jev in 25\%; reordering 16 options changes Laya's top
 answer on 25.5\% of questions \meas{} (\texttt{hanzoai/benchmarks}
 \texttt{4e57fb2}). The final checkpoint's numbers enter only through the
-tables of Section~\ref{sec:evaluation}, which print ``pending'' until it is
-scored; no sentence states them, and the first stage-A run's numbers appear
+tables of Section~\ref{sec:evaluation}; no sentence states them, and the first stage-A run's numbers appear
 only where Section~\ref{sec:findings} explains what went wrong. Kai trains on the
 train splits of the harness's sources behind a barrier against its test
 states, so a lead on a suite whose baseline never saw its source is

kai/sections/08-evaluation.tex

@@ -3,8 +3,8 @@
 
 Every table in this section is written by \texttt{bench tex} in
 \texttt{hanzoai/decision} from the kept results of the runs it names, into
-\texttt{tables/}; no number in a table is typed by hand. Until the final
-checkpoint is scored, each table is a stub that says so. Each table's last row
+\texttt{tables/}; no number in a table is typed by hand. The capability and speed tables
+are stubs until the final stage-A checkpoint is timed; each says so. Each table's last row
 names the Kai revision, the device and the harness and benchmark commits it
 reports.
 
@@ -51,7 +51,11 @@ and is read as such.
 \caption{Accuracy, Brier score and ECE on the frozen harness, per suite, for
 Kai, Laya and Jev. MASSIVE is the macro over its 51 languages
 (Table~\ref{tab:languages}); the macro row is over the other eleven suites.
-Written by \texttt{bench tex}; the last row names the revisions.}
+Bold: the best accuracy of the three in a row among backends that did not
+train on that suite's test rows. $^\dagger$: Laya's card lists the suite's
+dataset in its mixture and the harness draws the suite from that dataset's
+train split; the clean macro omits those suites. Written by \texttt{bench
+tex}; the last row names the revisions.}
 \label{tab:harness}
 \end{table}
 

kai/sections/09-limits.tex

@@ -2,8 +2,8 @@
 \label{sec:limits}
 
 \paragraph{What is not measured.}
-Until \texttt{bench tex} writes Section~\ref{sec:evaluation}'s tables for the
-final checkpoint, this paper claims no accuracy, calibration or speed for Kai.
+Until \texttt{bench tex} writes the capability and speed tables for the final
+checkpoint, this paper claims no capability or speed figure for it.
 Stage B's adapters and stage C's decoder have code, data and stage files but no
 trained checkpoint, so the media rows measure a text rendering unless the
 scored checkpoint reads evidence, and the joint rows measure refinement over

kai/tables/harness.tex

@@ -1,3 +1,26 @@
-% tables/harness.tex: stub. `bench tex --results <benchmarks>/decision --out <paper>/kai/tables`
-% (hanzoai/decision) overwrites this file with the table once the final checkpoint is scored.
-\pending{Accuracy, Brier score and ECE per harness suite (AG News, DAIR Emotion, Banking77, support triage, email spam, phishing, jailbreak, toxicity, RAG relevance, model routing, typed decisions, MASSIVE) for Kai, Laya and Jev, and the macro over the eleven non-MASSIVE suites.}
+% harness.tex: written by `bench tex` (hanzoai/decision); Kai runs/a4, weights dcd35aa6, cpu f32; harness 735e240; benchmarks c68a38b
+% Laya: the checkpoint its router picks per case; typed decisions on laya-agent (kai-1-agent), the typed-decisions checkpoint Laya ships and its router does not pick.
+\begin{tabular}{@{}l@{\quad}ccc@{\quad}ccc@{\quad}ccc@{}}
+\toprule
+ & \multicolumn{3}{c}{Accuracy} & \multicolumn{3}{c}{Brier} & \multicolumn{3}{c}{ECE} \\
+Suite & Kai & Laya & Jev & Kai & Laya & Jev & Kai & Laya & Jev \\
+\midrule
+AG News & 0.935 & \textbf{0.950} & 0.860 & 0.100 & 0.081 & 0.232 & 0.020 & 0.032 & 0.102 \\
+DAIR Emotion & \textbf{0.938} & 0.595 & 0.603 & 0.092 & 0.696 & 0.646 & 0.030 & 0.306 & 0.280 \\
+Banking77 & \textbf{0.885} & 0.425 & 0.835 & 0.173 & 0.925 & 0.257 & 0.079 & 0.381 & 0.073 \\
+Support triage & \textbf{0.453} & 0.502$^\dagger$ & 0.365 & 0.719 & 0.644 & 1.056 & 0.203 & 0.091 & 0.483 \\
+Email spam & 0.995 & \textbf{0.998} & 0.978 & 0.009 & 0.005 & 0.038 & 0.006 & 0.011 & 0.060 \\
+Phishing & \textbf{0.990} & 0.983$^\dagger$ & 0.900 & 0.021 & 0.026 & 0.152 & 0.011 & 0.010 & 0.039 \\
+Jailbreak & 0.907 & 0.705 & \textbf{0.940} & 0.143 & 0.543 & 0.104 & 0.061 & 0.262 & 0.047 \\
+Toxicity & \textbf{0.833} & 0.530 & 0.662 & 0.304 & 0.602 & 0.432 & 0.145 & 0.297 & 0.177 \\
+RAG relevance & \textbf{0.672} & 0.625 & 0.620 & 0.407 & 0.466 & 0.628 & 0.041 & 0.117 & 0.279 \\
+Model routing & \textbf{1.000} & 0.639 & 0.977 & 0.000 & 0.494 & 0.040 & 0.000 & 0.089 & 0.014 \\
+Typed decisions & 0.523 & \textbf{0.766} & 0.736 & 0.564 & 0.400 & 0.366 & 0.042 & 0.213 & 0.047 \\
+MASSIVE (51 languages, macro) & 0.858 & 0.382 & \textbf{0.890} & 0.210 & 0.987 & 0.161 & 0.079 & 0.418 & 0.064 \\
+\midrule
+Macro, 11 suites & 0.830 & 0.702 & 0.770 & 0.230 & 0.444 & 0.359 & 0.058 & 0.164 & 0.145 \\
+Clean macro, 9 suites & 0.854 & 0.693 & 0.801 & & & & & & \\
+Best of three, 12 suites & 7 & 3 & 2 & & & & & & \\
+\bottomrule
+\multicolumn{10}{@{}l@{}}{\footnotesize\begin{tabular}[t]{@{}l@{}}Kai runs/a4, weights dcd35aa6, cpu f32 \\ harness 735e240, benchmarks c68a38b\end{tabular}} \\
+\end{tabular}

kai/tables/languages.tex

@@ -1,3 +1,35 @@
-% tables/languages.tex: stub. `bench tex --results <benchmarks>/decision --out <paper>/kai/tables`
-% (hanzoai/decision) overwrites this file with the table once the final checkpoint is scored.
-\pending{MASSIVE intent accuracy for each of the 51 languages, for Kai, Laya and Jev.}
+% languages.tex: written by `bench tex` (hanzoai/decision); Kai runs/a4, weights dcd35aa6, cpu f32; harness 735e240; benchmarks c68a38b
+% MASSIVE intent accuracy by language, 20 options, 100 utterances each; Laya as its router picks.
+\begin{tabular}{@{}lccc@{\qquad}lccc@{}}
+\toprule
+Language & Kai & Laya & Jev & Language & Kai & Laya & Jev \\
+\midrule
+af & 0.870 & 0.340 & 0.870 & ko & 0.890 & 0.470 & 0.940 \\
+am & 0.770 & 0.150 & 0.850 & lv & 0.850 & 0.300 & 0.850 \\
+ar & 0.900 & 0.460 & 0.870 & ml & 0.810 & 0.280 & 0.910 \\
+az & 0.870 & 0.360 & 0.880 & mn & 0.740 & 0.160 & 0.840 \\
+bn & 0.880 & 0.450 & 0.860 & ms & 0.930 & 0.270 & 0.920 \\
+cy & 0.740 & 0.160 & 0.760 & my & 0.840 & 0.160 & 0.890 \\
+da & 0.880 & 0.460 & 0.880 & nb & 0.900 & 0.500 & 0.860 \\
+de & 0.910 & 0.530 & 0.939 & nl & 0.910 & 0.400 & 0.900 \\
+el & 0.860 & 0.440 & 0.940 & pl & 0.900 & 0.470 & 0.920 \\
+en & 0.890 & 0.820 & 0.920 & pt & 0.920 & 0.440 & 0.900 \\
+es & 0.890 & 0.530 & 0.930 & ro & 0.820 & 0.370 & 0.940 \\
+fa & 0.910 & 0.510 & 0.950 & ru & 0.920 & 0.570 & 0.940 \\
+fi & 0.860 & 0.300 & 0.890 & sl & 0.860 & 0.330 & 0.890 \\
+fr & 0.900 & 0.600 & 0.900 & sq & 0.850 & 0.290 & 0.760 \\
+he & 0.840 & 0.370 & 0.940 & sv & 0.910 & 0.480 & 0.940 \\
+hi & 0.860 & 0.460 & 0.950 & sw & 0.800 & 0.130 & 0.700 \\
+hu & 0.820 & 0.340 & 0.920 & ta & 0.830 & 0.310 & 0.920 \\
+hy & 0.760 & 0.250 & 0.840 & te & 0.800 & 0.220 & 0.910 \\
+id & 0.910 & 0.360 & 0.930 & th & 0.890 & 0.480 & 0.940 \\
+is & 0.780 & 0.330 & 0.840 & tl & 0.860 & 0.300 & 0.850 \\
+it & 0.910 & 0.470 & 0.960 & tr & 0.880 & 0.410 & 0.920 \\
+ja & 0.950 & 0.640 & 0.970 & ur & 0.850 & 0.420 & 0.900 \\
+jv & 0.820 & 0.160 & 0.750 & vi & 0.830 & 0.330 & 0.890 \\
+ka & 0.750 & 0.150 & 0.880 & zh-CN & 0.930 & 0.650 & 0.940 \\
+km & 0.760 & 0.200 & 0.810 & zh-TW & 0.950 & 0.610 & 0.930 \\
+kn & 0.770 & 0.300 & 0.870 &  & & & \\
+\bottomrule
+\multicolumn{8}{@{}l@{}}{\footnotesize\begin{tabular}[t]{@{}l@{}}Kai runs/a4, weights dcd35aa6, cpu f32 \\ harness 735e240, benchmarks c68a38b\end{tabular}} \\
+\end{tabular}