papers/kai/sections/02-decisions.tex

\section{Typed decisions}
\label{sec:decisions}

\begin{definition}[Typed question]
A typed question is a triple $q = (\tau, \iota, C)$ of a type
$\tau \in \{\op{choice}, \op{score}, \op{noul}\}$, an instruction $\iota$ in
natural language, and an ordered list of options
$C = (c_1, \dots, c_k)$, each a label with a description.
A \op{choice} selects one of $k \ge 1$ unordered labels; a \op{score} selects
one of $k$ ordered levels $0, \dots, k-1$; a \op{noul} is a predicate, $k = 2$,
with sides $(\op{false}, \op{true})$ whose descriptions the caller may
override.
\end{definition}

\begin{definition}[Answer]
An answer to $q$ over evidence $E$ is a distribution $p \in \Delta^{k-1}$ over
$C$. A \op{choice} reports $\arg\max_i p_i$, a \op{score} the expected level
$\bar\ell = \sum_i i\, p_i$ with a legend, a \op{noul} $p_{\op{true}}$; every
answer also carries $p$.
\end{definition}

$(q, E)$ and the model revision determine $p$, so a decision replays.

\subsection{What an answer is sure of}
\label{sec:confidence}

Three summaries of $p$ are in use under the name \op{confidence}, and they are
different numbers: the top probability $\pmax = \max_i p_i$; the rescaled
maximum
\begin{equation}
  \kappa = \frac{k\,\pmax - 1}{k - 1}, \qquad k \ge 2,
  \label{eq:kappa}
\end{equation}
which is 0 at the uniform distribution and 1 at a point mass, reported by Jev
and OpenRouter~\cite{laya,openrouterdecisions}; and one minus normalised
entropy, $h = 1 - H(p)/\ln k$, reported by the Laya runtime~\cite{laya}.

\begin{proposition}
\label{prop:kappa}
For fixed $k$, $\kappa \ge t \iff \pmax \ge \bigl(1 + t(k-1)\bigr)/k$.
\end{proposition}
\begin{proof}
Equation~\eqref{eq:kappa} is affine and increasing in $\pmax$ for $k \ge 2$.
\end{proof}

So $\kappa \ge 0.5$ asks for $\pmax \ge 0.75$ on a \op{noul}, $0.67$ on a
three-way and $0.55$ on a ten-way choice: one threshold, a different risk per
arity; $h$ orders answers differently again. For $p = (0.05, 0.15, 0.80)$, $\kappa = 0.70$ and
$h = 0.44$; a gate at 0.5 passes under one and abstains under the other.

\paragraph{The convention we fix.}
Every backend behind \op{/v1/decisions} reports \op{confidence} as $\kappa$,
so a threshold written for Jev means the same when Kai answers, and adds
\op{answer\_confidence} $= \pmax$. A Decision Program writes abstention
thresholds against $\kappa$ and \op{noul} thresholds against $p_{\op{true}}$,
and names the calibration they were set against (Section~\ref{sec:programs})
\shipped.