papers/kai/sections/08-evaluation.tex

\section{Evaluation}
\label{sec:evaluation}

Every table in this section is written by \texttt{bench tex} in
\texttt{hanzoai/decision} from the kept results of the runs it names, into
\texttt{tables/}; no number in a table is typed by hand. Each table's last row
names the Kai revision, the device and the harness and benchmark commits it
reports.

\paragraph{Backends.}
\textbf{Kai}: stage a7, three stages on from the final stage-A checkpoint
(Section~\ref{sec:collapse}): a5 read the suites a4 trailed with more rows and
typed decisions' train split whole (Section~\ref{sec:overlap}); a6 answers
score questions as a choice over their level texts (Section~\ref{sec:score});
a7 reads AG News whole and repeats support triage and jailbreak twice; one
epoch each, on the native runtime. \textbf{Laya}: Laya 0.3.20's published weights served natively
by the Rust port of its joint head (parity in Section~\ref{sec:checkpoints}),
each case on the checkpoint Laya's router picks, except typed decisions, which
are answered by Laya's typed-decisions checkpoint: the router sends them to the
English one, which scores 0.361 there \meas. \textbf{Jev}:
\op{typesafe/jev-1.13} through OpenRouter. All three answer identical questions;
a question a backend refuses or fails is counted, never dropped.

\paragraph{The harness.}
\texttt{hanzoai/benchmarks} \texttt{decision/}, frozen at \texttt{e95c6ee}
(its questions unchanged through \texttt{4e57fb2}), is Laya's own suite
builders, unchanged, seed 13: ten application suites of 400 cases, typed
decisions~\cite{typeddecisions} (400 states, 2{,}000 decisions), and MASSIVE
intent in 51 languages of 100 utterances with 20 options each; 62 suites,
9{,}499 states, 11{,}099 questions. Accuracy is top-1; Brier sums over options;
ECE uses 15 bins of top-label confidence. Scored with upstream's metric code,
the harness reproduces upstream's own typed-decisions benchmark for all three
Laya checkpoints to the fourth decimal \meas.

\paragraph{Training overlap.}
Kai trains on the train split of every harness source, less what the barrier
catches (Section~\ref{sec:barrier}). Laya's English checkpoint reports AG News,
Enron spam, phishing, support tickets and MS~MARCO in its mixture \reported, and
its typed-decisions checkpoint was fine-tuned on siblings of half the
typed-decisions test set and scores alike with and without them
(Section~\ref{sec:overlap}); no Laya checkpoint reports
DAIR emotion, Banking77, jailbreak, toxicity, model routing or MASSIVE. Jev's
training data are unpublished. A Kai lead on a suite whose baseline did not see
its source is in-distribution for Kai and out of distribution for the baseline,
and is read as such.

\subsection{Accuracy on the frozen harness}
\label{sec:accuracy}

\begin{table}[tbp]
\centering
\small
\setlength{\tabcolsep}{4pt}
\input{tables/harness}
\caption{Accuracy, Brier score and ECE on the frozen harness, per suite, for
Kai, Laya and Jev. MASSIVE is the macro over its 51 languages
(Table~\ref{tab:languages}); the macro row is over the other eleven suites.
Bold: the best accuracy of the three in a row among backends that did not
train on that suite's test rows. $^\dagger$: Laya's card lists the suite's
dataset in its mixture and the harness draws the suite from that dataset's
train split; the clean macro omits those suites. Written by \texttt{bench
tex}; the last row names the revisions.}
\label{tab:harness}
\end{table}

\paragraph{What the table says.}
Stage a7 leads on seven suites: emotion 0.935, Banking77 0.907, phishing
0.990, toxicity 0.802, RAG relevance 0.675, model routing 1.000 and MASSIVE
0.909 across 51 languages against Jev's 0.890 and Laya's 0.382. It ties email
spam (0.998 to 0.998). It trails on four, each by less than a6 did: AG News
0.943 to Laya's 0.950, jailbreak 0.920 to Jev's 0.940, typed decisions 0.759 to
Laya's typed checkpoint at 0.766 (above Jev's 0.736), and support triage 0.443
to Laya's 0.502. Typed decisions' score questions now lead Laya's typed
checkpoint, 0.729 to 0.723 (Section~\ref{sec:score}); choice (0.723 to 0.733)
and \op{noul} (0.835 to 0.857) trail it. Support triage is the widest gap:
four of its queues overlap, Kai's recall on them is 0.17 to 0.44, and Laya
answers the largest queue for 58\% of tickets against its 30\% share. Every
figure here is \texttt{hanzoai/benchmarks} \texttt{decision/results/kai-a7}
\meas.

\paragraph{The gate.}
A checkpoint replaces a baseline in a deployment only when \texttt{bench gate}
accepts it: on up to 100 drawn cases per harness and extension suite and their
robustness transforms, no compared suite may lose more than 0.02 accuracy or
gain more than 0.03 ECE, whatever the macro average, and for Laya the native
runtime's parity must hold. The first stage-A checkpoint was rejected: macro
accuracy 0.639 over the 62 suites against Laya's 0.438, parity held, and seven
suites regressed, RAG relevance, typed decisions, Banking77 and jailbreak among
them \meas{} (\texttt{hanzoai/benchmarks} \texttt{9aa6a46}). Stage a7 is
rejected against Laya on one suite, support triage, and against Jev on typed
decisions, toxicity (its ECE on the drawn cases, at 0.802 accuracy to Jev's
0.662 over the suite) and eleven of the 51 MASSIVE languages \meas; so it does
not replace either baseline in a deployment, and this paper claims no such
thing.

\subsection{Capabilities the baselines lack by construction}
\label{sec:capabilities}

These suites are \texttt{hanzoai/benchmarks} \texttt{decision/capability}, in
the harness's case format; every backend gets the same cases. The baselines'
kept runs are at \texttt{4e57fb2}.

\begin{description}[leftmargin=1.5em,style=nextline]
  \item[Cardinality.] One choice among $K$ options: 4 and 16 (the gold Banking77
    label with its nearest labels by word overlap), 77 (Banking77), 150
    (CLINC150), and 1{,}000 to 100{,}000 (WordNet: a definition to be matched to
    its concept among co-hyponyms and filler). An unanswered question counts
    as wrong. Laya rejects a question whose option markers fall past its
    context; Jev refuses more than 255 options. Laya's authors' recipe for many
    options is a top-20 shortlist from an encoder, which caps accuracy at the
    shortlist's recall; the table reports every model on the full option set,
    where Kai's retrieval keeps the mass it cuts (Section~\ref{sec:factor}).
  \item[Joint coherence.] Two families of synthetic programs, 200 states each,
    ask five dependent questions of one state (a vehicle's fault, severity,
    readiness, action and review; a source selection's option, evidence, risk,
    readiness and review), with gold computed from a latent world by a policy
    the state spells out, so every gold answer keeps every rule. The baselines
    answer each question independently; Kai runs the refinement of
    Section~\ref{sec:diffusion} over a decoder stage a6 has not trained, and
    loses this suite to Jev (0.306 per variable to 0.837, and a rule broken in
    every state before projection): the decoder is what stage C trains. Per-variable accuracy, the share of states with
    all five right, and the share whose answers break a rule, as answered and
    after the same exact projection applied to every backend's marginals.
  \item[Invariance.] 200 Banking77 questions of 16 options asked in five
    orders: the share whose top answer changes and the largest probability
    change; 400 \op{noul} questions asked with the two sides' descriptions
    exchanged, and again with the statement negated: the share whose answer
    mirrors. Proposition~\ref{prop:symmetry} predicts no flips and exact
    mirrors under the exchange for Kai, up to rounding; under negation a
    mirror is learned, not guaranteed, for every backend.
  \item[Media.] Human activity from smartphone inertial signals, a turbofan's
    remaining life from C-MAPSS telemetry, and a valve's condition from its
    sound (MIMII): the baselines read text renderings of the signals, Kai the
    records through its adapters when the checkpoint reads evidence, else the
    same rendering, which the table records.
  \item[Conformal sets.] Split-conformal prediction sets~\cite{vovk2005} at a
    90\% target over every answer on the frozen suites: coverage and mean set
    size.
  \item[Refresh.] A study program re-run after one evidence record changes:
    the questions put to a model, against a cold run.
  \item[Deployment.] Whether the backend answers inside a sandbox with the
    network denied: Laya does and Jev cannot, and every Jev call sends the state
    to a third party \meas.
\end{description}

\begin{table}[tbp]
\centering
\small
\setlength{\tabcolsep}{4pt}
\input{tables/capability}
\caption{Capabilities, Kai against Laya and Jev: cardinality, joint coherence,
invariance, media, conformal sets, refresh and offline operation. Written by
\texttt{bench tex}; the last row names the revisions.}
\label{tab:capability}
\end{table}

\subsection{Speed}
\label{sec:speed}

A call is one state with $N$ questions, timed from tokenizing to probabilities
on the host, the first call at each $N$ discarded, percentiles nearest-rank
(\texttt{bench speed}). Kai and Laya run on the same Rust runtime, device and
batch sizes, their calls interleaved, in BF16 on a production backend, the
Radeon 8060S of an AMD Strix Halo under ROCm; Jev is one Decisions API call over
the network, split into the largest chunks it accepts,
eight in flight, when it refuses the whole. The table's last row names the Laya
checkpoint timed: Laya's multilingual checkpoint shares Kai's encoder,
mmBERT-base, so against it the difference is the head and the layout, while its
English and typed-decisions checkpoints are ModernBERT-large, 28 layers of width
1{,}024 against 22 of width 768. Kai's advantage, where it has one, comes from what
Laya's layout cannot share: each question's sequence holds its question and the
state but no options, each distinct option is encoded once for every question
that lists it, and each question's head costs the options it scores, not a
sequence holding all of them. The stage-A checkpoints use the per-question
layout (Section~\ref{sec:evidence}), so these timings do not include encoding
the state once for all $N$ questions, which the shared layout adds. Kai
leads Laya from $N = 10$ (44.7\,ms to 116\,ms) and by 4.9$\times$ at $N =
1{,}000$ (1.98\,s to 9.77\,s); Laya's single question is faster (16.5\,ms to
19.8\,ms). Jev, whose remote accelerators batch a whole call, is slower than
Kai up to $N = 100$ and faster from $N = 500$ \meas.

\begin{table}[tbp]
\centering
\small
\setlength{\tabcolsep}{4pt}
\input{tables/speed}
\caption{Latency per call (ms, p50 and p95) and questions per second, by
questions per state, on each production backend timed. Written by
\texttt{bench tex}; the last row names the machine and revisions.}
\label{tab:speed}
\end{table}

% Flush the result tables here, so none drifts past the section.
\clearpage