papers/kai/sections/03-evaluation.tex

\section{Evaluation}
\label{sec:evaluation}

\paragraph{Protocol.} Every number is a kept run read through the same path
production serves, \op{kai serve} on a CPU, unless a table says otherwise. A
benchmark's test items never enter training: every item of every benchmark in
this paper, its normalized text and its near duplicates are held out of every
training set by an exact and near-duplicate barrier. A sealed suite is written
and hashed before the model it measures is frozen, read once, and spent after
its read. Comparisons are paired on the same items: accuracy by McNemar's exact
test, other statistics by a paired bootstrap, with Holm's correction over a
table's family of tests. Intervals are 95\%.

\paragraph{Baselines.} \emph{Jev} 1.13 is a hosted decision model served
through OpenRouter's Decisions API~\cite{openrouterdecisions}; its numbers are
its own answers to the same requests, recorded once. \emph{Laya} is an open
decision model~\cite{laya}, run from its released weights. \emph{Clef}'s and
\emph{Von}'s numbers are as their authors recorded them on the same suites.
Where a public paper reports a baseline, we cite it and say so.

\paragraph{Releases.} Kai is one model line served under one endpoint. A table
reports the release that was read on it: the production model, its
long-context release, or the production base with a task head added for one
benchmark. A task head reads the base's own pass of a request and changes no
other answer.