Kai: the typed-decisions siblings measured apart; Laya scores alike on both halves, Kai falls on the states whose siblings its barrier removed.

Co-authored-by: Hanzo Dev <[email protected]>

kai/sections/05-findings.tex

@@ -104,8 +104,33 @@ irreversible action, gold \emph{harmful}, \emph{human review}, risk 3; in test
 none of either and two tool errors, gold \emph{partial}, \emph{observe}, risk 1.
 So Laya's typed-decisions checkpoint saw same-wording siblings of half the test
 set, with different numbers and different gold, and no near-copy of a test
-answer. Its typed-decisions score measures a model that knows the templates;
-Kai's builds drop the 547, so Kai trains on 638 of the states the builder reads.
+answer. Kai's builds drop the 547, so Kai trains on 638 of the states the
+builder reads. Whether knowing the templates helps is measured, not assumed:
+\texttt{harness/siblings.py} in the benchmark repository scores the 235 test
+states without a sibling and the 165 with one apart
+(Table~\ref{tab:siblings}). Laya's typed checkpoint scores alike on both
+halves, so its lead is not memorized from the siblings; Jev, which never saw
+them, is 8 points better without them; and Kai falls on exactly the states
+whose siblings the barrier removed from its training. The lead is the head and
+the training, and this section is a caveat about Kai's data, not Laya's score.
+
+\begin{table}[h]
+  \centering
+  \small
+  \begin{tabular}{lccc}
+    \toprule
+    typed decisions & all 400 & without a sibling (235) & with a sibling (165) \\
+    \midrule
+    Kai (stage a4) & 0.523 & 0.605 & 0.406 \\
+    Laya typed & 0.766 & 0.773 & 0.756 \\
+    Jev & 0.736 & 0.770 & 0.686 \\
+    \bottomrule
+  \end{tabular}
+  \caption{Accuracy over every question of the state, split by whether the
+    a3 build's barrier caught a same-wording training record for it
+    (hanzoai/benchmarks \texttt{611059b}).}
+  \label{tab:siblings}
+\end{table}
 
 \subsection{Other defects, each fixed}
 \label{sec:other}

kai/sections/08-evaluation.tex

@@ -34,7 +34,8 @@ Kai trains on the train split of every harness source, less what the barrier
 catches (Section~\ref{sec:barrier}). Laya's English checkpoint reports AG News,
 Enron spam, phishing, support tickets and MS~MARCO in its mixture \reported, and
 its typed-decisions checkpoint was fine-tuned on siblings of half the
-typed-decisions test set (Section~\ref{sec:overlap}); no Laya checkpoint reports
+typed-decisions test set and scores alike with and without them
+(Section~\ref{sec:overlap}); no Laya checkpoint reports
 DAIR emotion, Banking77, jailbreak, toxicity, model routing or MASSIVE. Jev's
 training data are unpublished. A Kai lead on a suite whose baseline did not see
 its source is in-distribution for Kai and out of distribution for the baseline,

kai/sections/99-conclusion.tex

@@ -16,8 +16,9 @@ round's delta seven times, over a mixture this broad, cost the encoder its
 ability to relate two texts while every single-text task improved; only
 measurements aimed at the encoder itself found it. A barrier against the test
 set is only as good as its notion of ``the same'': half of one baseline's test
-set had same-wording siblings in its training data. Both are written down here
-with the runs that show them.
+set had same-wording siblings in its training data, and measuring the halves
+apart showed the siblings did not help it. Both are written down here with the
+runs that show them.
 
 Kai's case does not rest on winning every benchmark: Laya leads where it was
 trained and trails Jev off its mixture, and Kai enters this paper only through