papers/kai/sections/04-results.tex

\section{Results}
\label{sec:results}

\subsection{Decision tasks}
\label{sec:harness}

The frozen decision harness asks 11,099 questions over 9,499 states drawn from
twelve tasks: news topics~\cite{zhang2015agnews}, emotions~\cite{saravia2018carer},
banking intents~\cite{casanueva2020banking77}, support triage, spam, phishing,
jailbreaks~\cite{lin2023toxicchat}, toxicity, retrieval relevance, routing,
typed decisions, and intents in 51 languages~\cite{fitzgerald2023massive}.
Table~\ref{tab:harness} gives accuracy. Kai averages 0.857 against Jev's 0.780
and Laya's 0.675. Against the better of the two baselines, within 0.005 a tie,
Kai wins seven tasks, ties one and trails on four.

\begin{table}[t]
\centering
\small
\begin{tabular}{lccc}
\toprule
task & Kai & Laya & Jev \\
\midrule
AG News & 0.943 & \best{0.950} & 0.860 \\
DAIR emotion & \best{0.935} & 0.595 & 0.603 \\
Banking77 & \best{0.907} & 0.425 & 0.835 \\
support triage & 0.443 & \best{0.502}$^\dagger$ & 0.365 \\
email spam & 0.998 & 0.998 & 0.978 \\
phishing & \best{0.990} & 0.983$^\dagger$ & 0.900 \\
jailbreak & 0.920 & 0.705 & \best{0.940} \\
toxicity & \best{0.802} & 0.530 & 0.662 \\
RAG relevance & \best{0.675} & 0.625 & 0.620 \\
routing & \best{1.000} & 0.639 & 0.977 \\
typed decisions & 0.759 & \best{0.766} & 0.736 \\
MASSIVE, 51 languages, macro & \best{0.909} & 0.382 & 0.890 \\
\midrule
mean of the twelve & \best{0.857} & 0.675 & 0.780 \\
\bottomrule
\end{tabular}
\caption{The frozen decision harness, accuracy. Kai is the production base;
its yes/no rows were read on the base's own yes/no answer, and on 951 held-out
items of those tasks the production model reads 0.883 against Jev's 0.875.
$^\dagger$Laya's support-triage and phishing items are drawn from data its
builder marks as trained on.}
\label{tab:harness}
\end{table}

\subsection{Many labels, and labels never seen}
\label{sec:labels}

A decision often has more than a handful of answers. Table~\ref{tab:card}
reads accuracy as the label set grows, from four labels to Banking77's 77 and
CLINC150's 150~\cite{casanueva2020banking77,larson2019clinc}. Kai holds 0.908
at 77 and 150 labels, ahead of Jev up to 77 and behind it at 150. On a public
zero-shot pilot, 100 items a dataset with Jev's own request shape, Kai reads AG
News at 0.940 and emotions at 0.750 against Jev's published 0.910 and 0.480,
and Banking77 with 72 sentence-long labels at 0.810 against 0.870; 35 of the
pilot's 300 items also appear in the harness.

\begin{table}[t]
\centering
\small
\begin{tabular}{lcccc}
\toprule
labels & 4 & 16 & 77 & 150 \\
\midrule
Kai & \best{0.968} & \best{0.935} & \best{0.908} & 0.908 \\
Jev & 0.905 & 0.853 & 0.843 & \best{0.940} \\
\bottomrule
\end{tabular}
\caption{Accuracy as the label set grows (400 questions a cell; 77 is
Banking77, 150 is CLINC150).}
\label{tab:card}
\end{table}

\subsection{Long states}
\label{sec:long}

A planted-fact task hides one fact at a swept position in real text of a given
length, with decoys in 60\% of states, and asks a choice, a score or a yes/no
question about it: 120 states a cell, chance about 0.33. Kai's long-context
release reads it through the served CPU path (Table~\ref{tab:long}). A model of
the same size trained on states of up to 1,024 tokens reads at chance from the
first cell.

\begin{table}[t]
\centering
\small
\begin{tabular}{lccccc}
\toprule
state length (tokens) & 900 & 4,000 & 8,000 & 16,000 & 31,000 \\
\midrule
Kai, long-context release & 0.800 & 0.758 & 0.733 & 0.792 & 0.733 \\
the same base trained to 1,024 tokens & 0.333 & 0.308 & 0.375 & 0.350 & 0.300 \\
\bottomrule
\end{tabular}
\caption{Planted-fact accuracy by state length. The long-context release is
read on an x86-64 CPU; the short base was read on a GPU.}
\label{tab:long}
\end{table}

\subsection{Web agents}
\label{sec:web}

Mind2Web~\cite{deng2023mind2web} asks, at each step of a real website task,
which of the page's candidate elements to act on and how. Kai answers each step
as a choice over the 50 candidates the benchmark's ranker supplies, with a task
head trained on Mind2Web's training split. Table~\ref{tab:web} reads its element
accuracy and step success on the three test splits. Kai's element accuracy is
above the published Flan-T5-Base and GPT-4 numbers on every split and below
Flan-T5-XL's. Kai writes no text, so a step that types a value cannot count as
a full success, which holds its step success below Flan-T5-Base's. On the same
300-step samples, Claude Haiku 5.5 and Claude Opus 5.5 given the same candidates
read 47.3 / 43.3 / 50.0 and 59.7 / 58.0 / 60.7 element accuracy against Kai's
51.3 / 41.3 / 40.7, at \$0.0003 and \$0.011 a step against Kai's \$0.00004.

\begin{table}[t]
\centering
\small
\begin{tabular}{lcccccc}
\toprule
& \multicolumn{3}{c}{element accuracy} & \multicolumn{3}{c}{step success} \\
\cmidrule(lr){2-4}\cmidrule(lr){5-7}
& task & website & domain & task & website & domain \\
\midrule
Kai, with a task head & 48.8 & 41.5 & 39.1 & 37.5 & 28.6 & 29.2 \\
Flan-T5-Base (published) & 43.6 & 32.1 & 33.9 & 41.0 & 29.5 & 31.6 \\
Flan-T5-XL (published) & \best{55.1} & \best{42.0} & \best{42.1} & \best{52.0} & \best{38.9} & \best{39.6} \\
GPT-4, 50 tasks (published) & 41.6 & 35.8 & 37.1 & 36.2 & 30.1 & 26.4 \\
\bottomrule
\end{tabular}
\caption{Mind2Web cross-task, cross-website and cross-domain splits, percent.
Kai's cross-domain split is its first 2,000 steps.}
\label{tab:web}
\end{table}

\subsection{Tools}
\label{sec:tools}

An agent must pick the right tool for a request, and know when no tool fits.
On held-out lists of candidate tools drawn from public tool-use datasets and
Hanzo's own catalog, 5,658 ranked cases, Kai's agentic release ranks the right
tool with nDCG@10 0.894 [0.889, 0.898] and puts it first on 0.758 [0.747,
0.769] of cases, against BM25's 0.781 and 0.541. Asked whether to call a tool
at all, it calls on 0.122 [0.087, 0.168] of the cases where none applies while
keeping 0.919 of the cases where one does.

\subsection{Safety}
\label{sec:safety}

Table~\ref{tab:safety} reads the production model on two sets. On
InjecAgent~\cite{zhan2024injecagent}, instructions planted in a tool's output
for an agent its user never gave, Kai catches 55\% of the attacks at 2.4\%
false alarms. On a pool of 1,163 real jailbreak and injection attacks and
1,006 human-written benign prompts, Jev ranks attacks better and catches far
more of them at 0.5; Kai raises under a third of Jev's false alarms.

\begin{table}[t]
\centering
\small
\begin{tabular}{lcc}
\toprule
& Kai & Jev \\
\midrule
InjecAgent: attacks caught, of 250 & 138 & -- \\
InjecAgent: benign flagged, of 207 & 5 & -- \\
real attacks: AUROC & 0.725 & \best{0.896} \\
real attacks: caught at 0.5 & 26.9\% & \best{77.4\%} \\
human-written benign: flagged at 0.5 & \best{3.6\%} & 13.0\% \\
\bottomrule
\end{tabular}
\caption{Safety reads of the production model.}
\label{tab:safety}
\end{table}

\subsection{Consequences under policy}
\label{sec:grounding}

Under the harm policy, Kai reads for each course of an action what lies on its
path and how many people, and Nio decides. This grounding was qualified once, on
a sealed suite of 14,832 generated scenes in two settings no candidate had
trained on, at the policy's floor of 0.9 (Table~\ref{tab:ground}). 5,680 of the
scenes state a harm with no exact count (a crowd past the numerals, a hedged
count, infrastructure others depend on, a vague harm) and must escalate; the
rest must ground. Of the scenes it grounded Kai was right on 99.94\%, with 5
wrong courses in 14,832 scenes, and it grounded 2 of the 5,680 it should not
have. On 9,600 further scenes built only of unusable evidence (two possible
counts, ranges, corrupted figures, crowds, hedges and dependent infrastructure)
it grounded none. One person read the same whether described as a stranger, an
employee, a chief executive or the system's own developer, on 480 of 480
variants, and a server that keeps someone alive escalated on 400 of 400 scenes
where an isolated server read as property.

\begin{table}[t]
\centering
\small
\begin{tabular}{lr}
\toprule
scenes & 14,832 \\
grounded (both courses read at the floor) & 60.8\% \\
right, of the grounded & 99.94\% \\
wrong courses, of all scenes & 5 (0.03\%) \\
grounded where the harm has no exact count & 2 of 5,680 \\
unusable-evidence scenes grounded & 0 of 9,600 \\
one person read alike across four roles & 480 of 480 \\
life-support infrastructure escalated & 400 of 400 \\
\bottomrule
\end{tabular}
\caption{The sealed qualification of the production model's typed grounding,
at the floor of 0.9.}
\label{tab:ground}
\end{table}

On a public trolley-problem benchmark of 5,240 setups, Kai's own yes/no answer
does not weigh the stakes: it pulls the lever in 650 of the 672 setups where
pulling kills more than it saves, against Jev's 69 and GPT-6 Luna's 143. Asked
through the harm policy instead, Kai takes such a dominated course in 17 of
the 672 and escalates to a person wherever a count is not established, 2,291 of
the 5,240 setups.

\subsection{Serving}
\label{sec:serving}

The production model serves on CPUs in full precision. Its release was read on
an arm64 host and an x86-64 host before it rolled (Table~\ref{tab:serving}),
then canaried through the public API: every answer matched the x86-64
reference to $1.5\times10^{-6}$.

\begin{table}[t]
\centering
\small
\begin{tabular}{lr}
\toprule
yes/no answers across CPU families: largest difference & $1.8\times10^{-5}$ \\
yes/no answers across CPU families: decisions that differ & 0 of 955 \\
typed policy requests across CPU families: outcomes that differ & 0 of 400 \\
production canary: answers within $1.5\times10^{-6}$ & 16 of 16 \\
production request of two questions: server time, median & 209 ms \\
yes/no request, x86-64, four threads: median / 95th / 99th & 172 / 1,122 / 1,570 ms \\
policy request of seven questions, x86-64: median & 947 ms \\
peak resident memory of a server & 3.31 GB \\
\bottomrule
\end{tabular}
\caption{Serving the production model.}
\label{tab:serving}
\end{table}

\subsection{Where Kai trails}
\label{sec:trails}

On a public decision index of 38 benchmarks, Kai beats neither Jev nor Clef on
any of the 32 rows it was read on. By area, Kai, Jev and Clef read arts 0.35,
0.61 and 0.63; knowledge 0.24, 0.66 and 0.66; language 0.38, 0.81 and 0.83;
retrieval 0.51, 0.72 and 0.79; tools 0.14, 0.77 and 0.83. The knowledge and
language rows ask what a compact decision model does not hold, and Hanzo answers them
with its generative models instead. On sealed transfer suites of decision tasks
Kai never trained toward, its pooled accuracy is 0.440 against Laya's 0.576 and
Von's 0.646 and 0.711; the hardest tier, long policies and multi-hop
reasoning, is at chance for every model. On jailbreaks Jev catches far more at
0.5 (Table~\ref{tab:safety}), and on the trolley benchmark Kai's own answer
does not read the stakes (Section~\ref{sec:grounding}).