mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-22 17:20:17 +00:00
Kai is one model against Laya and Jev, with a designed Enso systems evaluation.
The Laya-identical checkpoints are the Laya baseline, run in its Python reference and natively in Rust; Kai is a multilingual encoder, evidence adapters and a typed-variable decoder on one checkpoint line, initialized from Laya. A new section tests the claim that Kai makes Zen generate less: Enso conditions B0 to K5, routing regret, context, skills, early stop, reasoning budget, KV retention and concurrency, keyed enso/<study>/<condition>/<metric> and printing XXX until runs land. Capability tables use the keys decision/capability writes. Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
Hanzo Dev
+1
-1
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"source": "Laya 0.3.20 (github.com/NandhaKishorM/laya) as its authors describe it: the English checkpoint's training mixture, and laya-typed-decisions (kai-1-agent) fine-tuned on the typed-decisions train split. Reported, not verified by us.",
|
||||
"source": "Laya 0.3.20 (github.com/NandhaKishorM/laya) as its authors describe it: the English checkpoint's training mixture, and laya-typed-decisions fine-tuned on the typed-decisions train split. Reported, not verified by us.",
|
||||
"laya": {
|
||||
"agnews": "mix",
|
||||
"emotion": "no",
|
||||
|
||||
+3
-2
@@ -4,6 +4,7 @@
|
||||
"td": "$BENCH/decision/results/laya/typed_decisions.json",
|
||||
"manifest": "$DECISION/train/manifest.json",
|
||||
"mix": "$PAPER/data/mix.json",
|
||||
"parity": "$BENCH/decision/results/kai1/parity.json",
|
||||
"cap": "$BENCH/decision/capability/results"
|
||||
"parity": "$BENCH/decision/results/laya/parity.json",
|
||||
"cap": "$BENCH/decision/capability/results",
|
||||
"enso": "$BENCH/decision/enso/results"
|
||||
}
|
||||
|
||||
+4
-4
@@ -1,4 +1,4 @@
|
||||
% Kai's data flow. Solid boxes are in the code; dashed boxes are proposed.
|
||||
% Kai: encoder, evidence adapters, decoder. Solid boxes are in the code; dashed are proposed.
|
||||
\begin{figure}[t]
|
||||
\centering
|
||||
\begin{tikzpicture}[
|
||||
@@ -17,7 +17,7 @@
|
||||
\node[prop, src, below=of sen] (aud) {audio};
|
||||
\node[prop, src, below=of aud] (img) {image, video};
|
||||
\node[box, enc, right=of txt] (etxt) {text encoder};
|
||||
\node[prop, enc, right=of sen] (esen) {rate-aware encoder};
|
||||
\node[prop, enc, right=of sen] (esen) {rate-aware adapter};
|
||||
\node[prop, enc, right=of aud] (eaud) {Whisper, frozen};
|
||||
\node[prop, enc, right=of img] (eimg) {SigLIP, frozen};
|
||||
\foreach \a/\b in {txt/etxt, sen/esen, aud/eaud, img/eimg} \draw[arr] (\a) -- (\b);
|
||||
@@ -54,8 +54,8 @@
|
||||
\draw[arr] (wave) -- (proj);
|
||||
\draw[arr] (sel) -- (proj);
|
||||
\end{tikzpicture}
|
||||
\caption{Kai. Evidence records are encoded once per change; options are
|
||||
encoded alone, narrowed by retrieval above $k$, and scored against the
|
||||
\caption{Kai: a text encoder, evidence adapters and a typed-variable decoder.
|
||||
Evidence records are encoded once per change; options are encoded alone, narrowed by retrieval above $k$, and scored against the
|
||||
evidence by two cross-attention blocks under the question type's law; a
|
||||
program's dependent questions are refined in waves and projected onto its
|
||||
rules. Solid: in the code. Dashed: proposed.}
|
||||
|
||||
+20
-18
@@ -32,7 +32,7 @@
|
||||
\newcommand{\setres}[2]{\expandafter\def\csname res:#1\endcsname{#2}}
|
||||
\newcommand{\res}[1]{\ifcsname res:#1\endcsname\csname res:#1\endcsname\else\ph\fi}
|
||||
\input{results-data}
|
||||
% One metric for Kai, kai-1 and Jev: \three{kj/agnews}{accuracy}.
|
||||
% One metric for Kai, Laya and Jev: \three{kj/agnews}{accuracy}.
|
||||
\newcommand{\three}[2]{\res{#1/kai/#2} & \res{#1/laya/#2} & \res{#1/jev/#2}}
|
||||
|
||||
\newcommand{\op}[1]{\texttt{#1}}
|
||||
@@ -63,7 +63,7 @@ as versioned programs under deterministic policy}}
|
||||
San Francisco, California \\
|
||||
\texttt{https://hanzo.ai}
|
||||
}
|
||||
\date{2026-09-25 \quad Code at \texttt{hanzoai/decision} \texttt{20742d5}}
|
||||
\date{2026-09-25 \quad Code at \texttt{hanzoai/decision} \texttt{ac32882}}
|
||||
|
||||
\begin{document}
|
||||
|
||||
@@ -75,22 +75,23 @@ as versioned programs under deterministic policy}}
|
||||
Agents, controllers and engineering studies ask many small questions of
|
||||
explicit state. Kai answers them as typed decisions, a \op{choice},
|
||||
\op{score} or \op{noul} over options the caller lists, returned as a
|
||||
calibrated distribution with no generated text. We give its final
|
||||
architecture. All input is evidence of one record type (text, sensor streams
|
||||
with their metadata, audio, images, video), encoded once per change. Options
|
||||
are encoded alone and scored against the evidence by two cross-attention
|
||||
blocks, so option count, order and question count are not bounded by one
|
||||
sequence; \op{noul} is label-symmetric by construction, \op{score} a
|
||||
cumulative-link model, and large choices are narrowed by retrieval that keeps
|
||||
the mass it cuts. A program's dependent questions are typed variables, refined
|
||||
in waves bounded by dependency depth and projected onto the program's rules.
|
||||
Programs are versioned DAGs whose verdicts join a deterministic policy on
|
||||
calibrated distribution with no generated text. Kai is one model: a
|
||||
multilingual encoder, adapters that bring sensor streams with their metadata,
|
||||
audio, images and video into it, and a typed-variable decoder. Options are
|
||||
encoded alone and scored against the evidence by two cross-attention blocks, so
|
||||
option count, order and question count are not bounded by one sequence;
|
||||
\op{noul} is label-symmetric by construction, \op{score} a cumulative-link
|
||||
model, and large choices are narrowed by retrieval that keeps the mass it cuts.
|
||||
A program's dependent questions are typed variables, refined in waves bounded
|
||||
by dependency depth and projected onto the program's rules. Programs are
|
||||
versioned DAGs whose verdicts join a deterministic policy on
|
||||
$\allow < \ask < \deny$, so no model loosens policy, and each run is a hashed
|
||||
package that replays, diffs and refreshes. We report capabilities existing
|
||||
decision models lack, the native runtime's speed, and accuracy on a frozen
|
||||
62-suite harness against kai-1 (the Laya checkpoints) and TypeSafe's Jev. Every
|
||||
claim carries its status, and every number is written by a script from a kept
|
||||
run.
|
||||
decision models lack, speed, and accuracy on a frozen 62-suite harness against
|
||||
Laya and TypeSafe's Jev, and design a systems evaluation of Kai controlling
|
||||
Enso, whose thesis is that Kai does not make Zen generate faster but makes it
|
||||
generate less. Every claim carries its status, and every number is written by a
|
||||
script from a kept run.
|
||||
\end{abstract}
|
||||
|
||||
\clearpage
|
||||
@@ -103,8 +104,9 @@ run.
|
||||
\input{sections/04-authority}
|
||||
\input{sections/05-programs}
|
||||
\input{sections/06-evaluation}
|
||||
\input{sections/07-limits}
|
||||
\input{sections/08-related}
|
||||
\input{sections/07-enso}
|
||||
\input{sections/08-limits}
|
||||
\input{sections/09-related}
|
||||
\input{sections/99-conclusion}
|
||||
|
||||
{\small
|
||||
|
||||
@@ -25,7 +25,6 @@
|
||||
\setres{kj/modelroutingdomain/cases}{399}
|
||||
\setres{kj/typeddecisions/options}{2--5}
|
||||
\setres{kj/typeddecisions/cases}{400}
|
||||
\setres{kj/laya/bundle}{hanzoai/kai-1}
|
||||
\setres{kj/laya/revision}{b50502c2}
|
||||
\setres{kj/laya/runtime}{laya 0.3.20}
|
||||
\setres{kj/laya/device}{cpu}
|
||||
@@ -3036,5 +3035,6 @@
|
||||
\setres{td/agent/qt/choice}{0.733}
|
||||
\setres{td/agent/qt/noul}{0.857}
|
||||
\setres{td/agent/qt/score}{0.723}
|
||||
% absent: $BENCH/decision/results/kai1/parity.json
|
||||
% absent: $BENCH/decision/results/laya/parity.json
|
||||
% absent: $BENCH/decision/capability/results
|
||||
% absent: $BENCH/decision/enso/results
|
||||
|
||||
+18
-13
@@ -13,16 +13,17 @@ re-scores a prediction. Plain python3; no dependencies.
|
||||
|
||||
Keys:
|
||||
kj/harness/... the question set: suites, states, questions
|
||||
kj/<suite>/<who>/<metric> who: kai, laya (kai-1 as Laya's router picks), jev, and the
|
||||
kai-1 checkpoints one, multi, agent; suite as in the harness,
|
||||
kj/<suite>/<who>/<metric> who: kai, laya (as Laya's router picks), jev, and Laya's
|
||||
checkpoints one, multi, agent; suite as in the harness,
|
||||
lowercased, [a-z0-9] only (jev.ag_news -> agnews)
|
||||
kj/massive/<lang>/<who>/... one MASSIVE language; kj/massive/<who>/... their summary
|
||||
kj/<suite>/<who>/trained training overlap: Kai from train/manifest.json, Laya reported
|
||||
kj/macro, kj/wins macro accuracy over the 11 non-MASSIVE suites, suites won
|
||||
kj/guard/... the training data's guard against the harness
|
||||
td/<ckpt>/... upstream's typed-decisions benchmark rerun on kai-1
|
||||
par/... native kai-1 against the reference runtime
|
||||
td/<ckpt>/... upstream's typed-decisions benchmark rerun on Laya's weights
|
||||
par/... Laya on the native runtime against its Python reference
|
||||
cap/<suite>/<who>/<metric> hanzoai/benchmarks decision/capability, as its files name them
|
||||
enso/<study>/<cond>/<metric> Kai controlling Enso; the ladder's study is its workload
|
||||
"""
|
||||
import gzip
|
||||
import json
|
||||
@@ -74,7 +75,7 @@ def name(s):
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ the frozen harness
|
||||
# scores.json's backends: Kai, kai-1 routed as Laya's router picks, each kai-1 checkpoint, Jev
|
||||
# scores.json's backends: Kai, Laya as its router picks, each Laya checkpoint, Jev
|
||||
WHO = {"kai": "kai", "laya": "laya", "jev": "jev",
|
||||
"laya:english": "one", "laya:multilingual": "multi", "laya:typed-decisions": "agent"}
|
||||
|
||||
@@ -97,6 +98,8 @@ def harness(runs):
|
||||
return
|
||||
for who, src in d["sources"].items():
|
||||
for f, v in src.items():
|
||||
if f == "bundle": # the repository name, not a fact the paper states
|
||||
continue
|
||||
if f == "revision":
|
||||
v = v[:8]
|
||||
elif f == "weights":
|
||||
@@ -196,7 +199,7 @@ def overlap(runs):
|
||||
|
||||
# ------------------------------------------------------------------ typed-decisions rerun
|
||||
def td(runs):
|
||||
"""Upstream's Part B (typed decisions) on the three kai-1 checkpoints."""
|
||||
"""Upstream's Part B (typed decisions) on Laya's three checkpoints."""
|
||||
d = load(runs["td"])
|
||||
if d is None:
|
||||
return
|
||||
@@ -235,13 +238,13 @@ def parity(runs):
|
||||
put("par/" + f.replace("max_", ""), sci(r[f]))
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ capability suites
|
||||
def capability(runs):
|
||||
"""decision/capability/results/<suite>.json: {"meta", "keys": {cap/<suite>/<who>/<metric>}},
|
||||
each key written as the file names it."""
|
||||
root = path(runs["cap"])
|
||||
# ------------------------------------------------------------------ keyed results
|
||||
def keyed(runs, part):
|
||||
"""<results dir>/<suite>.json: {"meta", "keys": {<key>: value}}, each key written as the
|
||||
file names it: decision/capability (cap/...) and the Enso evaluation (enso/...)."""
|
||||
root = path(runs[part])
|
||||
if not os.path.isdir(root):
|
||||
print("%% absent: %s" % runs["cap"])
|
||||
print("%% absent: %s" % runs[part])
|
||||
return
|
||||
for f in sorted(os.listdir(root)):
|
||||
if f.endswith(".json"):
|
||||
@@ -261,8 +264,10 @@ def capability(runs):
|
||||
def main():
|
||||
runs = json.load(open(sys.argv[1]))
|
||||
print("% Generated by scripts/results.py from data/runs.json; do not edit.")
|
||||
for part in (harness, overlap, td, parity, capability):
|
||||
for part in (harness, overlap, td, parity):
|
||||
part(runs)
|
||||
for part in ("cap", "enso"):
|
||||
keyed(runs, part)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -17,18 +17,18 @@ compute, policy and people hold authority, and each run leaves a package.
|
||||
|
||||
\input{figures/program}
|
||||
|
||||
The baseline is kai-1, three checkpoints byte-identical to the open Laya
|
||||
checkpoints~\cite{laya}, which pack the question, every option and the state
|
||||
into one encoder sequence. The comparison is TypeSafe's hosted
|
||||
The baselines are Laya~\cite{laya}, open encoder checkpoints that pack the
|
||||
question, every option and the state into one sequence, and TypeSafe's hosted
|
||||
Jev~\cite{typesafejev}. Status tags: \meas{} run by us and written into the
|
||||
paper by a script from the run's artifact; \reported{} published by others;
|
||||
\shipped{} on the main branch of \texttt{hanzoai/decision} at the revision on
|
||||
the title page; \proposed{} designed, not built. A number no run has produced
|
||||
prints \ph. Contributions:
|
||||
\begin{enumerate}[leftmargin=1.5em]
|
||||
\item \textbf{Evidence} as one record type for text, sensor streams with
|
||||
their metadata, audio, images and video, encoded once per change
|
||||
\proposed{} (Section~\ref{sec:evidence}).
|
||||
\item \textbf{One model}: a multilingual encoder, adapters for sensor
|
||||
streams with their metadata, audio, images and video, and a typed-variable
|
||||
decoder, on one checkpoint line trained in stages; the adapters
|
||||
\proposed{} (Section~\ref{sec:model}).
|
||||
\item \textbf{Factorized choice}: options encoded alone and scored against
|
||||
shared evidence by two cross-attention blocks, a law per type, and
|
||||
retrieval that keeps the mass it cuts, so option count, order and question
|
||||
@@ -44,18 +44,22 @@ prints \ph. Contributions:
|
||||
\shipped{} (Sections~\ref{sec:authority} and~\ref{sec:programs}).
|
||||
\item \textbf{Benchmarks} of capability, speed and accuracy, each cell wired
|
||||
to the artifact that fills it (Section~\ref{sec:evaluation}).
|
||||
\item \textbf{Kai in the loop}: a systems evaluation of Kai controlling
|
||||
Enso's model, context, tools, reasoning budget and stopping, whose thesis
|
||||
is that Kai makes Zen generate less, not faster \proposed{}
|
||||
(Section~\ref{sec:enso}).
|
||||
\end{enumerate}
|
||||
|
||||
\paragraph{Results.}
|
||||
At 100{,}000 options Kai's accuracy is
|
||||
\res{cap/cardinality/kai/k100000.accall} (Section~\ref{sec:capabilities}). One
|
||||
question costs \res{cap/questions/kai/n1.p50ms}\,ms natively,
|
||||
\res{cap/questions/laya/n1.p50ms}\,ms for kai-1 in Python and
|
||||
\res{cap/questions/laya/n1.p50ms}\,ms for Laya in Python and
|
||||
\res{cap/questions/jev/n1.p50ms}\,ms for Jev over the network; 1{,}000
|
||||
questions on one state, \res{cap/questions/kai/n1000.p50ms}\,ms natively
|
||||
(p50, Section~\ref{sec:speed}). Macro accuracy over the harness's
|
||||
\res{kj/wins/of} non-MASSIVE suites is \res{kj/macro/kai/accuracy} (Kai),
|
||||
\res{kj/macro/laya/accuracy} (kai-1) and \res{kj/macro/jev/accuracy} (Jev), Kai
|
||||
\res{kj/macro/laya/accuracy} (Laya) and \res{kj/macro/jev/accuracy} (Jev), Kai
|
||||
leading Jev on \res{kj/wins/kai/jev}; over MASSIVE's
|
||||
\res{kj/harness/languages} languages, \res{kj/massive/kai/macroaccuracy},
|
||||
\res{kj/massive/laya/macroaccuracy} and \res{kj/massive/jev/macroaccuracy}
|
||||
|
||||
+55
-37
@@ -1,14 +1,15 @@
|
||||
\section{Kai}
|
||||
\label{sec:model}
|
||||
|
||||
Kai maps evidence $E$, a typed question and, inside a program, other questions'
|
||||
answers to a distribution over the question's options: an evidence encoder, an
|
||||
option encoder, a cross-attention head with a law per type, and a refinement
|
||||
schedule over dependent questions (Figure~\ref{fig:kai}).
|
||||
Kai is one model: a multilingual text encoder, adapters that bring other
|
||||
evidence into its width, and a typed-variable decision decoder
|
||||
(Figure~\ref{fig:kai}). It maps evidence $E$, a typed question and, inside a
|
||||
program, other questions' answers to a distribution over the question's
|
||||
options.
|
||||
|
||||
\input{figures/kai}
|
||||
|
||||
\subsection{Evidence}
|
||||
\subsection{Evidence and adapters}
|
||||
\label{sec:evidence}
|
||||
|
||||
\begin{definition}[Evidence]
|
||||
@@ -27,11 +28,12 @@ Sensor metadata (unit, rate, calibration, error bound, quality) is input, and
|
||||
temporal summaries (instant, window, trend, event, baseline, forecast) are
|
||||
records too, produced by code or a forecasting
|
||||
model~\cite{das2024timesfm,ansari2024chronos}; a new sensor needs an adapter,
|
||||
not a new model. Each modality has one encoder into width $d$: the text encoder;
|
||||
one rate-aware encoder for every time series, embedding each sample with its
|
||||
timestamp, rate and unit; and frozen Whisper~\cite{radford2023whisper} and
|
||||
SigLIP~\cite{zhai2023siglip} with trained projections. Latents are keyed by
|
||||
content hash and computed once per changed evidence state \proposed.
|
||||
not a new model. Each non-text modality has one adapter into the encoder's
|
||||
width $d$: one rate-aware adapter for every time series, embedding each sample
|
||||
with its timestamp, rate and unit; frozen Whisper~\cite{radford2023whisper} and
|
||||
SigLIP~\cite{zhai2023siglip} front ends with trained projections for audio,
|
||||
images and video. Latents are keyed by content hash and computed once per
|
||||
changed evidence state \proposed.
|
||||
|
||||
Today evidence is JSON read as text: each question's instruction and the
|
||||
serialised state form one context,
|
||||
@@ -39,7 +41,7 @@ serialised state form one context,
|
||||
cut at the context length by dropping the state's tail and encoded by
|
||||
ModernBERT~\cite{modernbert} or mmBERT~\cite{mmbert} \shipped.
|
||||
|
||||
\subsection{Factorized choice}
|
||||
\subsection{Decoder: factorized choice}
|
||||
\label{sec:factor}
|
||||
|
||||
Let $H \in \mathbb{R}^{L\times d}$ be the context's token states plus a learned
|
||||
@@ -80,7 +82,7 @@ the exchange swaps $s_{\op{true}}$ and $s_{\op{false}}$, and
|
||||
$\sigma(-x) = 1 - \sigma(x)$.
|
||||
\end{proof}
|
||||
|
||||
kai-1 has neither: its options share one sequence, in order.
|
||||
Laya has neither: its options share one sequence, in order.
|
||||
|
||||
\paragraph{Retrieval.}
|
||||
A choice with more than $k = 32$ options is narrowed first. The context's first
|
||||
@@ -99,13 +101,13 @@ costs $O(k)$ per question for any option count \shipped.
|
||||
\paragraph{Training.}
|
||||
Log loss plus Brier score against each question's target distribution, both
|
||||
strictly proper~\cite{gneiting2007}, plus 0.2 times the retrieval log loss;
|
||||
bf16 over f32 masters on Metal, AdamW, token-budgeted batches, encoder from
|
||||
kai-1. Data are the train splits of the harness's sources at pinned revisions,
|
||||
bf16 over f32 masters on Metal, AdamW, token-budgeted batches. Data are the
|
||||
train splits of the harness's sources at pinned revisions,
|
||||
minus any example sharing a content unit (the state's text, or the first 200
|
||||
letters and digits of a field of 200 or more characters) with a harness
|
||||
evaluation state \shipped.
|
||||
|
||||
\subsection{Typed-variable diffusion}
|
||||
\subsection{Decoder: typed-variable diffusion}
|
||||
\label{sec:diffusion}
|
||||
|
||||
A program's questions depend on each other, and answered independently they
|
||||
@@ -155,34 +157,50 @@ and is re-asked by pass $d+1$. Each variable is asked in pass 1 and at most once
|
||||
more.
|
||||
\end{proof}
|
||||
|
||||
Steps 1, 2 and 4 are in the code with the factorized model as the conditional:
|
||||
Steps 1, 2 and 4 are in the code with Kai's factorized head as the conditional:
|
||||
a re-asked variable sees its parents' answers and probabilities as typed facts
|
||||
added to its state, each pass is one batched call, and the projection is an
|
||||
exact branch and bound bounded by the sum of per-variable maxima \shipped. The
|
||||
trained slot model, the lattice and selector, and distilling the passes into
|
||||
one or two are \proposed.
|
||||
|
||||
\subsection{Checkpoints and runtime}
|
||||
\subsection{One checkpoint line, provenance, runtime}
|
||||
\label{sec:checkpoints}
|
||||
|
||||
\textbf{kai-1} is \texttt{hanzoai/kai-1} (\texttt{b50502c2}),
|
||||
\texttt{kai-1-multilingual} (\texttt{3119843b}) and \texttt{kai-1-agent}
|
||||
(\texttt{e3e41649}), whose weights have the SHA-256 of the Laya 0.3.20
|
||||
checkpoints~\cite{laya}: ModernBERT-large at context 512, mmBERT-base at 1024,
|
||||
and ModernBERT-large fine-tuned on typed decisions~\cite{typeddecisions} at
|
||||
1024. Their joint head packs the question, each option after a marker token,
|
||||
and the state into one sequence, within a 192--256-token option budget; a script
|
||||
detector routes non-English states to the multilingual checkpoint. Every
|
||||
measurement of kai-1 is a measurement of Laya \shipped. \textbf{Factorized
|
||||
checkpoints} have the architecture above, start from kai-1's encoders, and are
|
||||
named by the run that evaluates them (\res{kj/kai/model}, weights
|
||||
\res{kj/kai/weights}).
|
||||
Kai has one checkpoint line, \texttt{hanzoai/kai}, with a revision per training
|
||||
stage: A, typed text decisions (the harness's train splits, MASSIVE in 51
|
||||
locales, agentic, coding and skill decisions, CLINC150, XNLI, SST-5, extreme
|
||||
choice, robustness); B, the evidence adapters with the encoder frozen (sensors,
|
||||
time series, audio, image, video, graph and SysML); C, joint masked decoding
|
||||
over whole programs. Stage A's training code is on the main branch \shipped;
|
||||
the stages and their revisions are \proposed{} until each has a harness run.
|
||||
The encoder is mmBERT-base~\cite{mmbert}, multilingual, at an 8,192-token
|
||||
context \proposed; the model code runs any ModernBERT-family
|
||||
encoder~\cite{modernbert} \shipped. The revision evaluated here is
|
||||
\res{kj/kai/model} (weights \res{kj/kai/weights}).
|
||||
|
||||
The runtime is Rust on Hanzo's tensor library, \texttt{hanzo-ml}. The
|
||||
\texttt{kai} crate holds the model, question parsing, layout, token-budget
|
||||
batching, decoding and refinement, on CPU in f32 and Metal in bf16 \shipped,
|
||||
and on CUDA and ROCm where the library builds them \proposed. The
|
||||
\texttt{decision} crate serves \op{POST /v1/decisions} over a registry:
|
||||
factorized checkpoints and kai-1 in process, Jev through
|
||||
OpenRouter~\cite{openrouterdecisions} \proposed. Native kai-1 matches the Python
|
||||
reference to \res{par/logit} per logit (Section~\ref{sec:repro}).
|
||||
\paragraph{Provenance.}
|
||||
Kai initializes from Laya's weights~\cite{laya}, released under Apache-2.0: the
|
||||
encoder and type embedding as they are, Laya's two self-attention head layers
|
||||
split into the two cross-attention blocks, and its marker scorer as the option
|
||||
scorer; retrieval and the ordinal head start fresh \shipped.
|
||||
|
||||
\paragraph{Laya, the baseline.}
|
||||
Laya 0.3.20's three checkpoints are not Kai revisions: English on
|
||||
ModernBERT-large at a 512-token context, multilingual on mmBERT-base at 1,024,
|
||||
and Laya-TD, ModernBERT-large fine-tuned on typed
|
||||
decisions~\cite{typeddecisions} at 1,024. Their joint head packs the question,
|
||||
each option after a marker token, and the state into one sequence within a
|
||||
192--256-token option budget, and a script detector routes non-English states
|
||||
to the multilingual checkpoint. We run their byte-identical weights in Laya's
|
||||
Python reference runtime and natively in Rust through a port of the joint head
|
||||
\proposed, whose parity with the reference is \res{par/logit} per logit
|
||||
(Section~\ref{sec:repro}).
|
||||
|
||||
\paragraph{Runtime.}
|
||||
Rust on Hanzo's tensor library, \texttt{hanzo-ml}. The \texttt{kai} crate holds
|
||||
the model, question parsing, layout, token-budget batching, decoding and
|
||||
refinement, on CPU in f32 and Metal in bf16 \shipped, and on CUDA and ROCm where
|
||||
the library builds them \proposed. The \texttt{decision} crate serves
|
||||
\op{POST /v1/decisions} over a registry: Kai and Laya in process, Jev through
|
||||
OpenRouter~\cite{openrouterdecisions} \proposed.
|
||||
|
||||
@@ -148,7 +148,7 @@ world from what changed in the record \proposed.
|
||||
|
||||
\op{POST /v1/decisions} speaks OpenRouter's Decisions
|
||||
API~\cite{openrouterdecisions} field for field (Appendix~\ref{app:wire}). The
|
||||
\op{model} field selects the backend: Kai and kai-1 run in process;
|
||||
\op{model} field selects the backend: Kai and Laya run in process;
|
||||
\op{typesafe/jev-1.13} is forwarded to OpenRouter, each answer checked for its
|
||||
type and each choice restored to the caller's order. Hanzo adds
|
||||
\op{answer\_confidence}, \op{routing}, and a program form,
|
||||
|
||||
@@ -2,12 +2,12 @@
|
||||
\label{sec:evaluation}
|
||||
|
||||
\paragraph{Backends.}
|
||||
\textbf{Kai}: factorized checkpoint \res{kj/kai/model} on the native runtime
|
||||
\textbf{Kai}: revision \res{kj/kai/model} on the native runtime
|
||||
(\res{kj/kai/device}, \res{kj/kai/dtype}), run by \texttt{bench}.
|
||||
\textbf{kai-1}: the Laya 0.3.20 reference runtime on \texttt{hanzoai/kai-1}
|
||||
at \res{kj/laya/revision}, CPU f32, each case on the checkpoint Laya's router
|
||||
picks; the router sends typed-decision states to the English checkpoint, so
|
||||
kai-1-agent, which Laya ships for that benchmark, is reported beside it.
|
||||
\textbf{Laya}: Laya 0.3.20's own Python reference runtime on its published
|
||||
weights, CPU f32, each case on the checkpoint Laya's router picks; the router
|
||||
sends typed-decision states to the English checkpoint, so Laya-TD, the
|
||||
typed-decisions checkpoint Laya ships, is reported beside it.
|
||||
\textbf{Jev}: \res{kj/jev/model} through OpenRouter, served as
|
||||
\res{kj/jev/served}. All three answer identical questions.
|
||||
|
||||
@@ -27,7 +27,7 @@ this section are \meas{} unless marked.
|
||||
|
||||
These suites are built by \texttt{hanzoai/benchmarks}
|
||||
\texttt{decision/capability} in the harness's case format and scored by its
|
||||
functions; every backend gets the same cases, and kai-1 is Laya's reference
|
||||
functions; every backend gets the same cases, and Laya is Laya's reference
|
||||
runtime.
|
||||
|
||||
\paragraph{Cardinality.}
|
||||
@@ -35,9 +35,9 @@ One choice among $K$ options: $K$ = 4, 16 and 77 from Banking77 (at 4 and 16
|
||||
the gold label with its nearest labels by word overlap), 150 from CLINC150, and
|
||||
1{,}000 to 100{,}000 from WordNet, a definition to be matched to its concept
|
||||
among co-hyponyms and filler. Accuracy counts an unanswered question as wrong.
|
||||
kai-1 fits all options in one sequence and \emph{rejects} a question whose
|
||||
Laya fits all options in one sequence and \emph{rejects} a question whose
|
||||
option markers fall past its context; its authors' recipe for many options,
|
||||
a top-20 shortlist from kai-1's own encoder, is reported as \emph{shortlist},
|
||||
a top-20 shortlist from Laya's own encoder, is reported as \emph{shortlist},
|
||||
a gold label outside the shortlist being a miss. Jev's refusals count as
|
||||
rejected.
|
||||
|
||||
@@ -51,11 +51,11 @@ Options $K$ & 4 & 16 & 77 & 150 & 1{,}000 & 10{,}000 & 100{,}000 \\
|
||||
\midrule
|
||||
\multicolumn{8}{@{}l}{\emph{Accuracy}} \\
|
||||
Kai & \res{cap/cardinality/kai/k4.accall} & \res{cap/cardinality/kai/k16.accall} & \res{cap/cardinality/kai/k77.accall} & \res{cap/cardinality/kai/k150.accall} & \res{cap/cardinality/kai/k1000.accall} & \res{cap/cardinality/kai/k10000.accall} & \res{cap/cardinality/kai/k100000.accall} \\
|
||||
kai-1 & \res{cap/cardinality/laya/k4.accall} & \res{cap/cardinality/laya/k16.accall} & \res{cap/cardinality/laya/k77.accall} & \res{cap/cardinality/laya/k150.accall} & \res{cap/cardinality/laya/k1000.accall} & \res{cap/cardinality/laya/k10000.accall} & \res{cap/cardinality/laya/k100000.accall} \\
|
||||
kai-1, shortlist & \res{cap/cardinality/laya/k4.sl.accall} & \res{cap/cardinality/laya/k16.sl.accall} & \res{cap/cardinality/laya/k77.sl.accall} & \res{cap/cardinality/laya/k150.sl.accall} & \res{cap/cardinality/laya/k1000.sl.accall} & \res{cap/cardinality/laya/k10000.sl.accall} & \res{cap/cardinality/laya/k100000.sl.accall} \\
|
||||
Laya & \res{cap/cardinality/laya/k4.accall} & \res{cap/cardinality/laya/k16.accall} & \res{cap/cardinality/laya/k77.accall} & \res{cap/cardinality/laya/k150.accall} & \res{cap/cardinality/laya/k1000.accall} & \res{cap/cardinality/laya/k10000.accall} & \res{cap/cardinality/laya/k100000.accall} \\
|
||||
Laya, shortlist & \res{cap/cardinality/laya/k4.sl.accall} & \res{cap/cardinality/laya/k16.sl.accall} & \res{cap/cardinality/laya/k77.sl.accall} & \res{cap/cardinality/laya/k150.sl.accall} & \res{cap/cardinality/laya/k1000.sl.accall} & \res{cap/cardinality/laya/k10000.sl.accall} & \res{cap/cardinality/laya/k100000.sl.accall} \\
|
||||
Jev & \res{cap/cardinality/jev/k4.accall} & \res{cap/cardinality/jev/k16.accall} & \res{cap/cardinality/jev/k77.accall} & \res{cap/cardinality/jev/k150.accall} & \res{cap/cardinality/jev/k1000.accall} & \res{cap/cardinality/jev/k10000.accall} & \res{cap/cardinality/jev/k100000.accall} \\
|
||||
\multicolumn{8}{@{}l}{\emph{Rejected}} \\
|
||||
kai-1 & \res{cap/cardinality/laya/k4.rejected} & \res{cap/cardinality/laya/k16.rejected} & \res{cap/cardinality/laya/k77.rejected} & \res{cap/cardinality/laya/k150.rejected} & \res{cap/cardinality/laya/k1000.rejected} & \res{cap/cardinality/laya/k10000.rejected} & \res{cap/cardinality/laya/k100000.rejected} \\
|
||||
Laya & \res{cap/cardinality/laya/k4.rejected} & \res{cap/cardinality/laya/k16.rejected} & \res{cap/cardinality/laya/k77.rejected} & \res{cap/cardinality/laya/k150.rejected} & \res{cap/cardinality/laya/k1000.rejected} & \res{cap/cardinality/laya/k10000.rejected} & \res{cap/cardinality/laya/k100000.rejected} \\
|
||||
Jev & \res{cap/cardinality/jev/k4.rejected} & \res{cap/cardinality/jev/k16.rejected} & \res{cap/cardinality/jev/k77.rejected} & \res{cap/cardinality/jev/k150.rejected} & \res{cap/cardinality/jev/k1000.rejected} & \res{cap/cardinality/jev/k10000.rejected} & \res{cap/cardinality/jev/k100000.rejected} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
@@ -86,13 +86,13 @@ Section~\ref{sec:diffusion} (\texttt{bench refine}) at one to three passes.
|
||||
Kai, 1 pass & \res{cap/joint/kai/pass1.acc} & & & \res{cap/joint/kai/pass1.proj.acc} & \\
|
||||
Kai, 2 passes & \res{cap/joint/kai/pass2.acc} & & & \res{cap/joint/kai/pass2.proj.acc} & \\
|
||||
Kai, 3 passes & \res{cap/joint/kai/pass3.acc} & & & \res{cap/joint/kai/pass3.proj.acc} & \\
|
||||
kai-1-agent & \res{cap/joint/laya/acc} & \res{cap/joint/laya/exact} & \res{cap/joint/laya/violate} & \res{cap/joint/laya/proj.acc} & \res{cap/joint/laya/proj.exact} \\
|
||||
kai-1 English & \res{cap/joint/laya/en.acc} & \res{cap/joint/laya/en.exact} & \res{cap/joint/laya/en.violate} & \res{cap/joint/laya/en.proj.acc} & \res{cap/joint/laya/en.proj.exact} \\
|
||||
Laya-TD & \res{cap/joint/laya/acc} & \res{cap/joint/laya/exact} & \res{cap/joint/laya/violate} & \res{cap/joint/laya/proj.acc} & \res{cap/joint/laya/proj.exact} \\
|
||||
Laya English & \res{cap/joint/laya/en.acc} & \res{cap/joint/laya/en.exact} & \res{cap/joint/laya/en.violate} & \res{cap/joint/laya/en.proj.acc} & \res{cap/joint/laya/en.proj.exact} \\
|
||||
Jev & \res{cap/joint/jev/acc} & \res{cap/joint/jev/exact} & \res{cap/joint/jev/violate} & \res{cap/joint/jev/proj.acc} & \res{cap/joint/jev/proj.exact} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Joint coherence on dependent questions. kai-1-agent is the
|
||||
typed-decisions checkpoint Laya ships for such workflows; kai-1 English is the
|
||||
\caption{Joint coherence on dependent questions. Laya-TD is the
|
||||
typed-decisions checkpoint Laya ships for such workflows; Laya English is the
|
||||
one its router picks. Kai's exact and violation rates need its per-state
|
||||
answers, which \texttt{bench refine} does not yet emit.}
|
||||
\label{tab:joint}
|
||||
@@ -120,7 +120,7 @@ up to rounding.
|
||||
& Flip & TV & Max & Mirror & Gap & Snake & Code \\
|
||||
\midrule
|
||||
Kai & \res{cap/invariance/kai/perm.flip} & \res{cap/invariance/kai/perm.tv} & \res{cap/invariance/kai/perm.maxdp} & \res{cap/invariance/kai/swap.mirror} & \res{cap/invariance/kai/swap.gap} & \res{cap/invariance/kai/alias.snake.flip} & \res{cap/invariance/kai/alias.code.flip} \\
|
||||
kai-1 & \res{cap/invariance/laya/perm.flip} & \res{cap/invariance/laya/perm.tv} & \res{cap/invariance/laya/perm.maxdp} & \res{cap/invariance/laya/swap.mirror} & \res{cap/invariance/laya/swap.gap} & \res{cap/invariance/laya/alias.snake.flip} & \res{cap/invariance/laya/alias.code.flip} \\
|
||||
Laya & \res{cap/invariance/laya/perm.flip} & \res{cap/invariance/laya/perm.tv} & \res{cap/invariance/laya/perm.maxdp} & \res{cap/invariance/laya/swap.mirror} & \res{cap/invariance/laya/swap.gap} & \res{cap/invariance/laya/alias.snake.flip} & \res{cap/invariance/laya/alias.code.flip} \\
|
||||
Jev & \res{cap/invariance/jev/perm.flip} & \res{cap/invariance/jev/perm.tv} & \res{cap/invariance/jev/perm.maxdp} & \res{cap/invariance/jev/swap.mirror} & \res{cap/invariance/jev/swap.gap} & \res{cap/invariance/jev/alias.snake.flip} & \res{cap/invariance/jev/alias.code.flip} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
@@ -130,13 +130,13 @@ Jev & \res{cap/invariance/jev/perm.flip} & \res{cap/invariance/jev/perm.tv} & \r
|
||||
|
||||
\paragraph{Media, refresh, conformal sets, deployment.}
|
||||
\emph{Media}: decisions whose evidence is a time series with its metadata,
|
||||
audio, an image or video; Kai reads the records through its encoders, kai-1
|
||||
audio, an image or video; Kai reads the records through its encoders, Laya
|
||||
and Jev text renderings of them. \emph{Refresh}: a study program is run, one
|
||||
evidence record changes, and it is run again; \emph{asked} counts the questions
|
||||
the second run puts to a model, \emph{full} those a fresh run puts.
|
||||
\emph{Conformal}: split-conformal sets~\cite{vovk2005} at a 90\% target;
|
||||
coverage and mean set size. These suites are specified and not yet built, so
|
||||
Table~\ref{tab:media} prints \ph. Deployment needs no suite: Kai and kai-1 run
|
||||
Table~\ref{tab:media} prints \ph. Deployment needs no suite: Kai and Laya run
|
||||
offline on the caller's hardware, send no state off the host and charge no
|
||||
per-call fee \shipped; Jev needs the network, sends each state to OpenRouter,
|
||||
and cost \$\res{kj/jev/costk} per 1{,}000 calls on the harness
|
||||
@@ -152,7 +152,7 @@ and cost \$\res{kj/jev/costk} per 1{,}000 calls on the harness
|
||||
& Sensor & Audio & Image & Video & Asked & Full & Cov. & Size \\
|
||||
\midrule
|
||||
Kai & \res{cap/media/kai/sensor.accuracy} & \res{cap/media/kai/audio.accuracy} & \res{cap/media/kai/image.accuracy} & \res{cap/media/kai/video.accuracy} & \res{cap/refresh/kai/asked} & \res{cap/refresh/kai/full} & \res{cap/conformal/kai/cov90} & \res{cap/conformal/kai/size90} \\
|
||||
kai-1 & \res{cap/media/laya/sensor.accuracy} & \res{cap/media/laya/audio.accuracy} & \res{cap/media/laya/image.accuracy} & \res{cap/media/laya/video.accuracy} & \res{cap/refresh/laya/asked} & \res{cap/refresh/laya/full} & \res{cap/conformal/laya/cov90} & \res{cap/conformal/laya/size90} \\
|
||||
Laya & \res{cap/media/laya/sensor.accuracy} & \res{cap/media/laya/audio.accuracy} & \res{cap/media/laya/image.accuracy} & \res{cap/media/laya/video.accuracy} & \res{cap/refresh/laya/asked} & \res{cap/refresh/laya/full} & \res{cap/conformal/laya/cov90} & \res{cap/conformal/laya/size90} \\
|
||||
Jev & \res{cap/media/jev/sensor.accuracy} & \res{cap/media/jev/audio.accuracy} & \res{cap/media/jev/image.accuracy} & \res{cap/media/jev/video.accuracy} & \res{cap/refresh/jev/asked} & \res{cap/refresh/jev/full} & \res{cap/conformal/jev/cov90} & \res{cap/conformal/jev/size90} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
@@ -167,7 +167,7 @@ A call is one state with $N$ questions, from tokenising to probabilities on
|
||||
the host. States are the harness's typed-decision states in turn, questions its
|
||||
distinct question definitions in order, wrapping; at each $N$ the first call is
|
||||
discarded and the next are timed one at a time, percentiles nearest-rank.
|
||||
Kai runs natively (\texttt{bench speed}); kai-1 runs Laya's reference runtime
|
||||
Kai runs natively (\texttt{bench speed}); Laya runs Laya's reference runtime
|
||||
in Python, every question its own sequence in one forward pass, on the
|
||||
typed-decisions checkpoint; Jev is one Decisions API call, split into the
|
||||
largest chunks it accepts, eight in flight, when it refuses the whole. All run
|
||||
@@ -179,8 +179,8 @@ from one Apple M1 Max.
|
||||
\setlength{\tabcolsep}{4pt}
|
||||
\begin{tabular}{@{}r@{\quad}cc@{\quad}cc@{\quad}cc@{\quad}ccc@{}}
|
||||
\toprule
|
||||
& \multicolumn{2}{c}{Kai, native} & \multicolumn{2}{c}{kai-1, Python} & \multicolumn{2}{c}{Jev, network} & \multicolumn{3}{c}{Questions/s} \\
|
||||
$N$ & p50 & p95 & p50 & p95 & p50 & p95 & Kai & kai-1 & Jev \\
|
||||
& \multicolumn{2}{c}{Kai, native} & \multicolumn{2}{c}{Laya, Python} & \multicolumn{2}{c}{Jev, network} & \multicolumn{3}{c}{Questions/s} \\
|
||||
$N$ & p50 & p95 & p50 & p95 & p50 & p95 & Kai & Laya & Jev \\
|
||||
\midrule
|
||||
1 & \res{cap/questions/kai/n1.p50ms} & \res{cap/questions/kai/n1.p95ms} & \res{cap/questions/laya/n1.p50ms} & \res{cap/questions/laya/n1.p95ms} & \res{cap/questions/jev/n1.p50ms} & \res{cap/questions/jev/n1.p95ms} & \res{cap/questions/kai/n1.qpers} & \res{cap/questions/laya/n1.qpers} & \res{cap/questions/jev/n1.qpers} \\
|
||||
10 & \res{cap/questions/kai/n10.p50ms} & \res{cap/questions/kai/n10.p95ms} & \res{cap/questions/laya/n10.p50ms} & \res{cap/questions/laya/n10.p95ms} & \res{cap/questions/jev/n10.p50ms} & \res{cap/questions/jev/n10.p95ms} & \res{cap/questions/kai/n10.qpers} & \res{cap/questions/laya/n10.qpers} & \res{cap/questions/jev/n10.qpers} \\
|
||||
@@ -195,7 +195,7 @@ $N$ & p50 & p95 & p50 & p95 & p50 & p95 & Kai & kai-1 & Jev \\
|
||||
\end{table}
|
||||
|
||||
Kai's peak memory at $N = 1{,}000$ is \res{cap/questions/kai/n1000.peakmb}\,MB.
|
||||
kai-1's English checkpoint answers one question in
|
||||
Laya's English checkpoint answers one question in
|
||||
\res{cap/questions/laya/n1.large.p50ms}\,ms at p50. Jev's cost per call at
|
||||
$N = 1$ is \$\res{cap/questions/jev/n1.usdpercall}; on the harness run,
|
||||
\res{kj/jevlat/n} sequential single calls took \res{kj/jevlat/p50}\,ms at p50 and
|
||||
@@ -215,9 +215,9 @@ languages of 100 utterances and 20 options; \res{kj/harness/suites} suites,
|
||||
Kai's training data are the train splits of the harness's sources, minus
|
||||
examples sharing a content unit with its \res{kj/guard/units} evaluation units
|
||||
(\res{kj/guard/excluded} dropped, \res{kj/guard/train} kept; harness
|
||||
\res{kj/guard/commit}). Per suite the tables mark whether Kai and kai-1 trained
|
||||
\res{kj/guard/commit}). Per suite the tables mark whether Kai and Laya trained
|
||||
on its source:
|
||||
\emph{split} for its train split, \emph{mix} for kai-1's upstream training
|
||||
\emph{split} for its train split, \emph{mix} for Laya's upstream training
|
||||
mixture \reported, \emph{no} otherwise. Jev's training data are unpublished.
|
||||
A lead where Kai says \emph{split} and the other \emph{no} is
|
||||
in-distribution.
|
||||
@@ -229,7 +229,7 @@ in-distribution.
|
||||
\begin{tabular}{@{}lccc@{\quad}ccc@{\quad}ccc@{}}
|
||||
\toprule
|
||||
& & \multicolumn{2}{c}{Trained} & \multicolumn{3}{c}{Accuracy} & \multicolumn{3}{c}{Brier} \\
|
||||
Suite & $K$ & Kai & kai-1 & Kai & kai-1 & Jev & Kai & kai-1 & Jev \\
|
||||
Suite & $K$ & Kai & Laya & Kai & Laya & Jev & Kai & Laya & Jev \\
|
||||
\midrule
|
||||
AG News & \res{kj/agnews/options} & \res{kj/agnews/kai/trained} & \res{kj/agnews/laya/trained} & \three{kj/agnews}{accuracy} & \three{kj/agnews}{brier} \\
|
||||
DAIR Emotion & \res{kj/emotion/options} & \res{kj/emotion/kai/trained} & \res{kj/emotion/laya/trained} & \three{kj/emotion}{accuracy} & \three{kj/emotion}{brier} \\
|
||||
@@ -242,13 +242,13 @@ Toxicity & \res{kj/moderationtoxicity/options} & \res{kj/moderationtoxicit
|
||||
RAG relevance & \res{kj/ragrelevance/options} & \res{kj/ragrelevance/kai/trained} & \res{kj/ragrelevance/laya/trained} & \three{kj/ragrelevance}{accuracy} & \three{kj/ragrelevance}{brier} \\
|
||||
Model routing & \res{kj/modelroutingdomain/options} & \res{kj/modelroutingdomain/kai/trained} & \res{kj/modelroutingdomain/laya/trained} & \three{kj/modelroutingdomain}{accuracy} & \three{kj/modelroutingdomain}{brier} \\
|
||||
Typed decisions & \res{kj/typeddecisions/options} & \res{kj/typeddecisions/kai/trained} & \res{kj/typeddecisions/laya/trained} & \three{kj/typeddecisions}{accuracy} & \three{kj/typeddecisions}{brier} \\
|
||||
\quad kai-1-agent & & & \res{kj/typeddecisions/agent/trained} & & \res{kj/typeddecisions/agent/accuracy} & & & \res{kj/typeddecisions/agent/brier} & \\
|
||||
\quad Laya-TD & & & \res{kj/typeddecisions/agent/trained} & & \res{kj/typeddecisions/agent/accuracy} & & & \res{kj/typeddecisions/agent/brier} & \\
|
||||
\midrule
|
||||
Macro, 11 suites & & & & \res{kj/macro/kai/accuracy} & \res{kj/macro/laya/accuracy} & \res{kj/macro/jev/accuracy} & & & \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Accuracy and Brier score on the frozen harness. $K$: options per
|
||||
question. kai-1 is Laya's router's pick per case; kai-1-agent is the
|
||||
question. Laya is Laya's router's pick per case; Laya-TD is the
|
||||
typed-decisions checkpoint Laya ships, which the router does not pick.}
|
||||
\label{tab:acc}
|
||||
\end{table}
|
||||
@@ -260,7 +260,7 @@ typed-decisions checkpoint Laya ships, which the router does not pick.}
|
||||
\begin{tabular}{@{}l@{\quad}ccc@{\quad}ccc@{\quad}ccc@{}}
|
||||
\toprule
|
||||
& \multicolumn{3}{c}{ECE} & \multicolumn{3}{c}{Zero-p (\%)} & \multicolumn{3}{c}{AURC} \\
|
||||
Suite & Kai & kai-1 & Jev & Kai & kai-1 & Jev & Kai & kai-1 & Jev \\
|
||||
Suite & Kai & Laya & Jev & Kai & Laya & Jev & Kai & Laya & Jev \\
|
||||
\midrule
|
||||
AG News & \three{kj/agnews}{ece} & \three{kj/agnews}{zeropct} & \three{kj/agnews}{aurc} \\
|
||||
DAIR Emotion & \three{kj/emotion}{ece} & \three{kj/emotion}{zeropct} & \three{kj/emotion}{aurc} \\
|
||||
@@ -281,8 +281,8 @@ option probability below $10^{-6}$.}
|
||||
\end{table}
|
||||
|
||||
Of the \res{kj/wins/of} suites, Kai beats Jev on \res{kj/wins/kai/jev} and
|
||||
kai-1 on \res{kj/wins/kai/laya}; Jev beats kai-1 on \res{kj/wins/jev/laya}.
|
||||
On typed decisions kai-1-agent scores \res{kj/typeddecisions/agent/accuracy},
|
||||
Laya on \res{kj/wins/kai/laya}; Jev beats Laya on \res{kj/wins/jev/laya}.
|
||||
On typed decisions Laya-TD scores \res{kj/typeddecisions/agent/accuracy},
|
||||
Jev \res{kj/typeddecisions/jev/accuracy} and Kai
|
||||
\res{kj/typeddecisions/kai/accuracy} (by workflow: Appendix~\ref{app:results}).
|
||||
|
||||
@@ -291,7 +291,7 @@ Jev \res{kj/typeddecisions/jev/accuracy} and Kai
|
||||
\small
|
||||
\begin{tabular}{@{}lccccc@{}}
|
||||
\toprule
|
||||
& Kai & kai-1 & kai-1 English & kai-1 multilingual & Jev \\
|
||||
& Kai & Laya & Laya English & Laya multilingual & Jev \\
|
||||
\midrule
|
||||
English & \res{kj/massive/kai/english} & \res{kj/massive/laya/english} & \res{kj/massive/one/english} & \res{kj/massive/multi/english} & \res{kj/massive/jev/english} \\
|
||||
Other languages, macro & \res{kj/massive/kai/nonenglishmacro} & \res{kj/massive/laya/nonenglishmacro} & \res{kj/massive/one/nonenglishmacro} & \res{kj/massive/multi/nonenglishmacro} & \res{kj/massive/jev/nonenglishmacro} \\
|
||||
@@ -302,7 +302,7 @@ Zero-p (\%), macro & \res{kj/massive/kai/zeropct} & \res{kj/massive/laya/z
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{MASSIVE intent, 20 options, \res{kj/massive/jev/languages} languages;
|
||||
random is 0.05. Trained: Kai \res{kj/massive/kai/trained}, kai-1
|
||||
random is 0.05. Trained: Kai \res{kj/massive/kai/trained}, Laya
|
||||
\res{kj/massive/laya/trained}. Per language: Table~\ref{tab:bylang}.}
|
||||
\label{tab:massive}
|
||||
\end{table}
|
||||
@@ -311,12 +311,11 @@ random is 0.05. Trained: Kai \res{kj/massive/kai/trained}, kai-1
|
||||
\label{sec:repro}
|
||||
|
||||
Upstream's own typed-decisions harness (Part~B of \texttt{bench\_local.py}),
|
||||
run unchanged on \texttt{hanzoai/kai-1} at \res{td/rev} with
|
||||
\res{td/runtime}, gives kai-1-agent \res{td/agent/accuracy} accuracy and the
|
||||
run unchanged on Laya's weights with \res{td/runtime}, gives Laya-TD \res{td/agent/accuracy} accuracy and the
|
||||
English checkpoint \res{td/one/accuracy} (Table~\ref{tab:td}); the frozen
|
||||
harness, scoring its own predictions, gives \res{kj/typeddecisions/agent/accuracy}
|
||||
and \res{kj/typeddecisions/one/accuracy}.
|
||||
Native kai-1 against the Python reference, on \res{par/states} recorded states
|
||||
Laya on our native Rust runtime against its Python reference, on \res{par/states} recorded states
|
||||
and \res{par/checkpoints} checkpoints: labels identical on \res{par/labels},
|
||||
largest logit difference \res{par/logit}, largest probability difference
|
||||
\res{par/prob}, against an enforced bound of \res{par/bound} on logits.
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
\section{Kai in the loop: Enso systems evaluation}
|
||||
\label{sec:enso}
|
||||
|
||||
This section is a design \proposed: every cell prints \ph{} until its run
|
||||
lands, under the same rules as Section~\ref{sec:evaluation}.
|
||||
|
||||
\paragraph{Thesis.}
|
||||
Kai does not make Zen generate faster; it makes Zen generate less. Enso is
|
||||
Hanzo's adaptive model family, served by the Zen engine, and Kai is its
|
||||
controller, in process: at each step of an agent task it decides which model
|
||||
answers, what context it sees, which tools and skills it is shown, how long it
|
||||
may reason, whether the task is done, and whether to escalate, each a Decision
|
||||
Program under policy in shadow, advisory or enforced mode.
|
||||
|
||||
\paragraph{Cost.}
|
||||
Per task,
|
||||
\begin{equation}
|
||||
C = C_{\mathrm{Kai}} + C_{\mathrm{prefill}} + C_{\mathrm{decode}}
|
||||
+ C_{\mathrm{retrieval}} + C_{\mathrm{tools}},
|
||||
\label{eq:cost}
|
||||
\end{equation}
|
||||
where prefill and decode sum over the task's generative calls, each the chosen
|
||||
model's per-token cost times its prompt or decode tokens, and $C_{\mathrm{Kai}}$
|
||||
is one encoder pass per decision. Kai spends $C_{\mathrm{Kai}}$ to shrink the
|
||||
prefill and decode terms with a smaller model, fewer prompt tokens, fewer decode
|
||||
tokens and fewer steps; the thesis holds where that saving exceeds
|
||||
$C_{\mathrm{Kai}}$ at equal success. Every metric is per successful task, so no
|
||||
condition looks cheap by failing: success rate, GPU-seconds, prompt and decode
|
||||
tokens, KV GB-seconds $\int \mathrm{KV}(t)\,dt$ over the bytes of KV cache the
|
||||
task holds, peak VRAM, latency p50 and p95, and dollars per successful task.
|
||||
|
||||
\paragraph{Conditions.}
|
||||
\textbf{B0} Zen only: the largest Zen model for every step. \textbf{B1} a
|
||||
static router by task type. \textbf{R1} Zen as router: the same large Zen model
|
||||
routes itself. \textbf{K1} Kai routes the model. \textbf{K2} K1 plus context
|
||||
pruning. \textbf{K3} K2 plus a tool and skill shortlist. \textbf{K4} K3 plus
|
||||
progress and early stop. \textbf{K5} the full decision plane: model, context,
|
||||
tools, reasoning budget, completion and escalation.
|
||||
|
||||
\paragraph{Workloads.}
|
||||
Kept separate, never pooled: synthetic controlled tasks whose cheapest
|
||||
successful model per step is known; public agent benchmarks; and real Hanzo
|
||||
traces replayed from shadow records. Table~\ref{tab:enso} gives the synthetic
|
||||
workload; the other two are in Appendix~\ref{app:results}.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\small
|
||||
\setlength{\tabcolsep}{4pt}
|
||||
\begin{tabular}{@{}l@{\quad}ccccccccc@{}}
|
||||
\toprule
|
||||
& Success & GPU-s & Prompt & Decode & KV GB$\cdot$s & VRAM & p50 & p95 & \$ \\
|
||||
\midrule
|
||||
B0 & \res{enso/synthetic/B0/success} & \res{enso/synthetic/B0/gpus} & \res{enso/synthetic/B0/prompt} & \res{enso/synthetic/B0/decode} & \res{enso/synthetic/B0/kvgbs} & \res{enso/synthetic/B0/vram} & \res{enso/synthetic/B0/p50} & \res{enso/synthetic/B0/p95} & \res{enso/synthetic/B0/usd} \\
|
||||
B1 & \res{enso/synthetic/B1/success} & \res{enso/synthetic/B1/gpus} & \res{enso/synthetic/B1/prompt} & \res{enso/synthetic/B1/decode} & \res{enso/synthetic/B1/kvgbs} & \res{enso/synthetic/B1/vram} & \res{enso/synthetic/B1/p50} & \res{enso/synthetic/B1/p95} & \res{enso/synthetic/B1/usd} \\
|
||||
R1 & \res{enso/synthetic/R1/success} & \res{enso/synthetic/R1/gpus} & \res{enso/synthetic/R1/prompt} & \res{enso/synthetic/R1/decode} & \res{enso/synthetic/R1/kvgbs} & \res{enso/synthetic/R1/vram} & \res{enso/synthetic/R1/p50} & \res{enso/synthetic/R1/p95} & \res{enso/synthetic/R1/usd} \\
|
||||
K1 & \res{enso/synthetic/K1/success} & \res{enso/synthetic/K1/gpus} & \res{enso/synthetic/K1/prompt} & \res{enso/synthetic/K1/decode} & \res{enso/synthetic/K1/kvgbs} & \res{enso/synthetic/K1/vram} & \res{enso/synthetic/K1/p50} & \res{enso/synthetic/K1/p95} & \res{enso/synthetic/K1/usd} \\
|
||||
K2 & \res{enso/synthetic/K2/success} & \res{enso/synthetic/K2/gpus} & \res{enso/synthetic/K2/prompt} & \res{enso/synthetic/K2/decode} & \res{enso/synthetic/K2/kvgbs} & \res{enso/synthetic/K2/vram} & \res{enso/synthetic/K2/p50} & \res{enso/synthetic/K2/p95} & \res{enso/synthetic/K2/usd} \\
|
||||
K3 & \res{enso/synthetic/K3/success} & \res{enso/synthetic/K3/gpus} & \res{enso/synthetic/K3/prompt} & \res{enso/synthetic/K3/decode} & \res{enso/synthetic/K3/kvgbs} & \res{enso/synthetic/K3/vram} & \res{enso/synthetic/K3/p50} & \res{enso/synthetic/K3/p95} & \res{enso/synthetic/K3/usd} \\
|
||||
K4 & \res{enso/synthetic/K4/success} & \res{enso/synthetic/K4/gpus} & \res{enso/synthetic/K4/prompt} & \res{enso/synthetic/K4/decode} & \res{enso/synthetic/K4/kvgbs} & \res{enso/synthetic/K4/vram} & \res{enso/synthetic/K4/p50} & \res{enso/synthetic/K4/p95} & \res{enso/synthetic/K4/usd} \\
|
||||
K5 & \res{enso/synthetic/K5/success} & \res{enso/synthetic/K5/gpus} & \res{enso/synthetic/K5/prompt} & \res{enso/synthetic/K5/decode} & \res{enso/synthetic/K5/kvgbs} & \res{enso/synthetic/K5/vram} & \res{enso/synthetic/K5/p50} & \res{enso/synthetic/K5/p95} & \res{enso/synthetic/K5/usd} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Conditions on the synthetic workload, per successful task: GPU-seconds, prompt and decode tokens, KV GB-seconds, peak VRAM (GB), latency (s) and dollars.}
|
||||
\label{tab:enso}
|
||||
\end{table}
|
||||
|
||||
\paragraph{Studies.}
|
||||
Each isolates one decision on the synthetic workload (Table~\ref{tab:ensostudies}).
|
||||
\emph{Routing} against the oracle, the cheapest model that succeeds on the
|
||||
step; regret is the chosen model's cost minus the oracle's. \emph{Context}:
|
||||
tokens kept and success with pruning. \emph{Tools and skills}: a shortlist
|
||||
from Hanzo Skills (900+) against listing them all. \emph{Early stop}: steps and
|
||||
decode tokens saved, and stops made before the task was done. \emph{Reasoning
|
||||
budget}: decode tokens and success under Kai's budget. \emph{KV retention}: Kai
|
||||
predicts which contexts return, against LRU. The \emph{full system} runs B0 and
|
||||
K5 on the replayed traces under 1 to 128 concurrent agents
|
||||
(Table~\ref{tab:ensoload}). Keys are \texttt{enso/<study>/<condition>/<metric>};
|
||||
for the condition ladder the study names the workload.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\small
|
||||
\begin{tabular}{@{}llcccc@{}}
|
||||
\toprule
|
||||
Study & Metric & Baseline & & Kai & \\
|
||||
\midrule
|
||||
Routing & regret, \$ & B1 & \res{enso/routing/B1/regret} & K1 & \res{enso/routing/K1/regret} \\
|
||||
& regret, \$ & R1 & \res{enso/routing/R1/regret} & K1 & \res{enso/routing/K1/regret} \\
|
||||
& success & R1 & \res{enso/routing/R1/success} & K1 & \res{enso/routing/K1/success} \\
|
||||
Context & prompt tokens & K1 & \res{enso/context/K1/prompt} & K2 & \res{enso/context/K2/prompt} \\
|
||||
& success & K1 & \res{enso/context/K1/success} & K2 & \res{enso/context/K2/success} \\
|
||||
Tools, skills & prompt tokens & K2 & \res{enso/tools/K2/prompt} & K3 & \res{enso/tools/K3/prompt} \\
|
||||
& needed skill in top 5 & & & K3 & \res{enso/tools/K3/recall5} \\
|
||||
Early stop & decode tokens & K3 & \res{enso/stop/K3/decode} & K4 & \res{enso/stop/K4/decode} \\
|
||||
& premature stops & & & K4 & \res{enso/stop/K4/premature} \\
|
||||
Reasoning budget & decode tokens & K4 & \res{enso/budget/K4/decode} & K5 & \res{enso/budget/K5/decode} \\
|
||||
& success & K4 & \res{enso/budget/K4/success} & K5 & \res{enso/budget/K5/success} \\
|
||||
KV retention & KV GB$\cdot$s & LRU & \res{enso/cache/lru/kvgbs} & Kai & \res{enso/cache/kai/kvgbs} \\
|
||||
& cache hits & LRU & \res{enso/cache/lru/hit} & Kai & \res{enso/cache/kai/hit} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Studies, per successful task; regret in dollars over the oracle.}
|
||||
\label{tab:ensostudies}
|
||||
\end{table}
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\small
|
||||
\setlength{\tabcolsep}{4pt}
|
||||
\begin{tabular}{@{}r@{\quad}cccc@{\quad}cccc@{}}
|
||||
\toprule
|
||||
& \multicolumn{4}{c}{B0, Zen only} & \multicolumn{4}{c}{K5, full decision plane} \\
|
||||
Agents & Tasks/h & p95 & GPU-s & VRAM & Tasks/h & p95 & GPU-s & VRAM \\
|
||||
\midrule
|
||||
1 & \res{enso/concurrency/B0/c1.tph} & \res{enso/concurrency/B0/c1.p95} & \res{enso/concurrency/B0/c1.gpus} & \res{enso/concurrency/B0/c1.vram} & \res{enso/concurrency/K5/c1.tph} & \res{enso/concurrency/K5/c1.p95} & \res{enso/concurrency/K5/c1.gpus} & \res{enso/concurrency/K5/c1.vram} \\
|
||||
8 & \res{enso/concurrency/B0/c8.tph} & \res{enso/concurrency/B0/c8.p95} & \res{enso/concurrency/B0/c8.gpus} & \res{enso/concurrency/B0/c8.vram} & \res{enso/concurrency/K5/c8.tph} & \res{enso/concurrency/K5/c8.p95} & \res{enso/concurrency/K5/c8.gpus} & \res{enso/concurrency/K5/c8.vram} \\
|
||||
32 & \res{enso/concurrency/B0/c32.tph} & \res{enso/concurrency/B0/c32.p95} & \res{enso/concurrency/B0/c32.gpus} & \res{enso/concurrency/B0/c32.vram} & \res{enso/concurrency/K5/c32.tph} & \res{enso/concurrency/K5/c32.p95} & \res{enso/concurrency/K5/c32.gpus} & \res{enso/concurrency/K5/c32.vram} \\
|
||||
128 & \res{enso/concurrency/B0/c128.tph} & \res{enso/concurrency/B0/c128.p95} & \res{enso/concurrency/B0/c128.gpus} & \res{enso/concurrency/B0/c128.vram} & \res{enso/concurrency/K5/c128.tph} & \res{enso/concurrency/K5/c128.p95} & \res{enso/concurrency/K5/c128.gpus} & \res{enso/concurrency/K5/c128.vram} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{The full system under concurrent agents on replayed traces: tasks per
|
||||
hour, p95 latency (s), GPU-seconds per successful task, peak VRAM (GB).}
|
||||
\label{tab:ensoload}
|
||||
\end{table}
|
||||
@@ -2,14 +2,16 @@
|
||||
\label{sec:limits}
|
||||
|
||||
\paragraph{What is not measured.}
|
||||
Each \ph{} is a number no kept run has produced: the factorized checkpoint on
|
||||
the harness (\texttt{bench preds}, then \texttt{merge.py}), native kai-1
|
||||
parity, and each capability suite for each backend, speed included. Until they land, this paper claims no
|
||||
accuracy for the factorized architecture.
|
||||
Each \ph{} is a number no kept run has produced: a Kai revision on
|
||||
the harness (\texttt{bench preds}, then \texttt{merge.py}), native Laya
|
||||
parity, each capability suite for each backend, speed included, and the whole
|
||||
Enso evaluation, which is a design until its runs exist. Until they land, this paper claims no
|
||||
accuracy for Kai.
|
||||
|
||||
\paragraph{What is not built.}
|
||||
The evidence encoders, caching across calls, the masked-slot model with its
|
||||
lattice, selector and distillation, refinement inside the program executor, the
|
||||
The evidence adapters, the 8K-context encoder, caching across calls, the
|
||||
masked-slot decoder with its lattice, selector and distillation, refinement
|
||||
inside the program executor, the Rust port of Laya's joint head, the
|
||||
\op{/v1/decisions} service, the joint weight-change bound and sampled flip
|
||||
probabilities are \proposed{} until they land on the main branch. Today's refinement writes
|
||||
parents' answers into a question's state as text, a weaker conditional than a
|
||||
@@ -17,7 +19,7 @@ trained slot model. The media rows compare Kai with baselines reading text
|
||||
renderings, which measures the renderings as much as the models.
|
||||
|
||||
\paragraph{What is not won.}
|
||||
Off its training mixture kai-1 trails Jev: on Banking77 (\res{kj/banking77full/laya/accuracy} against
|
||||
Off its training mixture Laya trails Jev: on Banking77 (\res{kj/banking77full/laya/accuracy} against
|
||||
\res{kj/banking77full/jev/accuracy}), on jailbreak detection
|
||||
(\res{kj/guardrailsjailbreak/laya/accuracy} against
|
||||
\res{kj/guardrailsjailbreak/jev/accuracy}), and on MASSIVE outside English
|
||||
@@ -29,7 +31,7 @@ harness source, so its leads over a model that did not are in-distribution;
|
||||
with Jev's training data unknown, no table here measures generalisation to a
|
||||
new task.
|
||||
|
||||
\paragraph{Known failure modes of kai-1.}
|
||||
\paragraph{Known failure modes of Laya.}
|
||||
Its authors report that boolean-word labels can be followed instead of their
|
||||
descriptions, negated requests answered as unnegated, \op{noul} answers driven
|
||||
by the labels rather than the state, and a score's first level rarely picked
|
||||
@@ -6,10 +6,10 @@ TypeSafe's Jev~\cite{typesafejev} introduced the typed interface adopted
|
||||
here, served through OpenRouter~\cite{openrouterdecisions}; its design
|
||||
notes~\cite{jevnote} argue for typed state and permissions as decisions.
|
||||
Laya~\cite{laya} is an open encoder implementation trained with proper scoring
|
||||
rules~\cite{gneiting2007}; kai-1's weights are Laya's.
|
||||
rules~\cite{gneiting2007}.
|
||||
|
||||
\paragraph{Encoders that score options.}
|
||||
kai-1's joint head is a cross-encoder, as in passage
|
||||
Laya's joint head is a cross-encoder, as in passage
|
||||
re-ranking~\cite{nogueira2019} and entailment-based zero-shot
|
||||
classification~\cite{yin2019}, after BERT~\cite{devlin2019}. Encoding
|
||||
candidates independently, caching them, and letting them attend to a context is
|
||||
@@ -10,9 +10,11 @@ and projected onto rules, makes their answers coherent; joining every model
|
||||
verdict with a deterministic policy makes safety the policy's, with the cost of
|
||||
trusting the model counted exactly in shadow.
|
||||
|
||||
Kai's case does not rest on winning every benchmark: kai-1 leads where it was
|
||||
trained and trails Jev off its mixture, and the factorized checkpoints enter
|
||||
Kai's case does not rest on winning every benchmark: Laya leads where it was
|
||||
trained and trails Jev off its mixture, and Kai's revisions enter
|
||||
this paper only through the runs that measure them. It rests on what a hosted
|
||||
model cannot offer: a pinned revision, deployment where the data live, no
|
||||
per-call fee, a runtime linked into the system it governs, and a package per
|
||||
run that says what was decided, from what, and what would change it.
|
||||
per-call fee, a runtime linked into the system it governs, a package per run
|
||||
that says what was decided, from what, and what would change it, and, in Enso,
|
||||
less generation per successful task, which Section~\ref{sec:enso} is built to
|
||||
measure.
|
||||
|
||||
@@ -5,7 +5,7 @@ A request to \op{POST /v1/decisions}, in OpenRouter's Decisions schema:
|
||||
|
||||
\begin{lstlisting}[style=wire]
|
||||
{
|
||||
"model": "kai-1",
|
||||
"model": "kai",
|
||||
"state": "Hi, we were billed twice for March. Please refund the duplicate today or we will cancel our plan.",
|
||||
"questions": {
|
||||
"department": {
|
||||
@@ -37,7 +37,7 @@ other Hanzo fields are omitted. The values are illustrative, not measured.
|
||||
\begin{lstlisting}[style=wire]
|
||||
{
|
||||
"id": "dec_...",
|
||||
"model": "kai-1",
|
||||
"model": "kai",
|
||||
"provider": "hanzo",
|
||||
"answers": {
|
||||
"department": {
|
||||
@@ -66,13 +66,15 @@ Jev through OpenRouter; nothing else changes.
|
||||
\section{A Decision Program}
|
||||
\label{app:program}
|
||||
|
||||
\op{agent.command-risk@1}, from \texttt{program/programs} \shipped:
|
||||
\op{agent.command-risk@1}, as in \texttt{program/programs} \shipped, whose
|
||||
copy names Laya's typed-decisions checkpoint until a Kai revision is calibrated
|
||||
for it:
|
||||
|
||||
\begin{lstlisting}[style=wire]
|
||||
{
|
||||
"id": "agent.command-risk@1",
|
||||
"description": "Before an agent runs a shell command: allow it, ask the user, or deny it.",
|
||||
"model": "kai-1-agent",
|
||||
"model": "kai",
|
||||
"mode": "advisory",
|
||||
"calibration": "cal_6dbb072f8109a222",
|
||||
"questions": {
|
||||
|
||||
@@ -11,7 +11,7 @@ Section~\ref{sec:evaluation}.
|
||||
\begin{tabular}{@{}l@{\quad}ccc@{\quad}ccc@{\quad}cc@{}}
|
||||
\toprule
|
||||
& \multicolumn{3}{c}{Macro F1} & \multicolumn{3}{c}{Unanswered} & \multicolumn{2}{c}{Jev, ms} \\
|
||||
Suite & Kai & kai-1 & Jev & Kai & kai-1 & Jev & p50 & p95 \\
|
||||
Suite & Kai & Laya & Jev & Kai & Laya & Jev & p50 & p95 \\
|
||||
\midrule
|
||||
AG News & \three{kj/agnews}{macrof1} & \three{kj/agnews}{unanswered} & \res{kj/agnews/jev/p50} & \res{kj/agnews/jev/p95} \\
|
||||
DAIR Emotion & \three{kj/emotion}{macrof1} & \three{kj/emotion}{unanswered} & \res{kj/emotion/jev/p50} & \res{kj/emotion/jev/p95} \\
|
||||
@@ -36,7 +36,7 @@ per call under eight concurrent calls.}
|
||||
\small
|
||||
\begin{tabular}{@{}lcccc@{}}
|
||||
\toprule
|
||||
Typed decisions & Kai & kai-1 & kai-1-agent & Jev \\
|
||||
Typed decisions & Kai & Laya & Laya-TD & Jev \\
|
||||
\midrule
|
||||
Agent-trace observability & \res{kj/typeddecisions/kai/wf/agenttraceobservability} & \res{kj/typeddecisions/laya/wf/agenttraceobservability} & \res{kj/typeddecisions/agent/wf/agenttraceobservability} & \res{kj/typeddecisions/jev/wf/agenttraceobservability} \\
|
||||
Customer service & \res{kj/typeddecisions/kai/wf/customerservice} & \res{kj/typeddecisions/laya/wf/customerservice} & \res{kj/typeddecisions/agent/wf/customerservice} & \res{kj/typeddecisions/jev/wf/customerservice} \\
|
||||
@@ -65,13 +65,13 @@ Score within one level & \res{kj/typeddecisions/kai/within1level} & \res{kj/type
|
||||
\toprule
|
||||
Checkpoint & Accuracy & Soft acc. & Brier (soft) & ECE & Score MAE & ms/case \\
|
||||
\midrule
|
||||
kai-1-agent & \res{td/agent/accuracy} & \res{td/agent/softaccuracy} & \res{td/agent/briervssoft} & \res{td/agent/ece} & \res{td/agent/scoremae} & \res{td/agent/mspercase} \\
|
||||
kai-1 & \res{td/one/accuracy} & \res{td/one/softaccuracy} & \res{td/one/briervssoft} & \res{td/one/ece} & \res{td/one/scoremae} & \res{td/one/mspercase} \\
|
||||
kai-1-multilingual & \res{td/multi/accuracy} & \res{td/multi/softaccuracy} & \res{td/multi/briervssoft} & \res{td/multi/ece} & \res{td/multi/scoremae} & \res{td/multi/mspercase} \\
|
||||
Laya-TD & \res{td/agent/accuracy} & \res{td/agent/softaccuracy} & \res{td/agent/briervssoft} & \res{td/agent/ece} & \res{td/agent/scoremae} & \res{td/agent/mspercase} \\
|
||||
Laya & \res{td/one/accuracy} & \res{td/one/softaccuracy} & \res{td/one/briervssoft} & \res{td/one/ece} & \res{td/one/scoremae} & \res{td/one/mspercase} \\
|
||||
Laya multilingual & \res{td/multi/accuracy} & \res{td/multi/softaccuracy} & \res{td/multi/briervssoft} & \res{td/multi/ece} & \res{td/multi/scoremae} & \res{td/multi/mspercase} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Upstream's typed-decisions harness run unchanged on
|
||||
\texttt{hanzoai/kai-1} at \res{td/rev}, \res{td/runtime}, CPU f32,
|
||||
\caption{Upstream's typed-decisions harness run unchanged on Laya's weights,
|
||||
\res{td/runtime}, CPU f32,
|
||||
\res{td/agent/n} decisions.}
|
||||
\label{tab:td}
|
||||
\end{table}
|
||||
@@ -82,7 +82,7 @@ kai-1-multilingual & \res{td/multi/accuracy} & \res{td/multi/softaccuracy} & \re
|
||||
\small
|
||||
\begin{tabular}{@{}lccc@{\qquad}lccc@{}}
|
||||
\toprule
|
||||
Language & Kai & kai-1 & Jev & Language & Kai & kai-1 & Jev \\
|
||||
Language & Kai & Laya & Jev & Language & Kai & Laya & Jev \\
|
||||
\midrule
|
||||
\langrow{af}{af} & \langrow{ko}{ko} \\
|
||||
\langrow{am}{am} & \langrow{lv}{lv} \\
|
||||
@@ -116,3 +116,47 @@ Language & Kai & kai-1 & Jev & Language & Kai & kai-1 & Jev \\
|
||||
each, 20 options.}
|
||||
\label{tab:bylang}
|
||||
\end{table}
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\small
|
||||
\setlength{\tabcolsep}{4pt}
|
||||
\begin{tabular}{@{}l@{\quad}ccccccccc@{}}
|
||||
\toprule
|
||||
& Success & GPU-s & Prompt & Decode & KV GB$\cdot$s & VRAM & p50 & p95 & \$ \\
|
||||
\midrule
|
||||
B0 & \res{enso/public/B0/success} & \res{enso/public/B0/gpus} & \res{enso/public/B0/prompt} & \res{enso/public/B0/decode} & \res{enso/public/B0/kvgbs} & \res{enso/public/B0/vram} & \res{enso/public/B0/p50} & \res{enso/public/B0/p95} & \res{enso/public/B0/usd} \\
|
||||
B1 & \res{enso/public/B1/success} & \res{enso/public/B1/gpus} & \res{enso/public/B1/prompt} & \res{enso/public/B1/decode} & \res{enso/public/B1/kvgbs} & \res{enso/public/B1/vram} & \res{enso/public/B1/p50} & \res{enso/public/B1/p95} & \res{enso/public/B1/usd} \\
|
||||
R1 & \res{enso/public/R1/success} & \res{enso/public/R1/gpus} & \res{enso/public/R1/prompt} & \res{enso/public/R1/decode} & \res{enso/public/R1/kvgbs} & \res{enso/public/R1/vram} & \res{enso/public/R1/p50} & \res{enso/public/R1/p95} & \res{enso/public/R1/usd} \\
|
||||
K1 & \res{enso/public/K1/success} & \res{enso/public/K1/gpus} & \res{enso/public/K1/prompt} & \res{enso/public/K1/decode} & \res{enso/public/K1/kvgbs} & \res{enso/public/K1/vram} & \res{enso/public/K1/p50} & \res{enso/public/K1/p95} & \res{enso/public/K1/usd} \\
|
||||
K2 & \res{enso/public/K2/success} & \res{enso/public/K2/gpus} & \res{enso/public/K2/prompt} & \res{enso/public/K2/decode} & \res{enso/public/K2/kvgbs} & \res{enso/public/K2/vram} & \res{enso/public/K2/p50} & \res{enso/public/K2/p95} & \res{enso/public/K2/usd} \\
|
||||
K3 & \res{enso/public/K3/success} & \res{enso/public/K3/gpus} & \res{enso/public/K3/prompt} & \res{enso/public/K3/decode} & \res{enso/public/K3/kvgbs} & \res{enso/public/K3/vram} & \res{enso/public/K3/p50} & \res{enso/public/K3/p95} & \res{enso/public/K3/usd} \\
|
||||
K4 & \res{enso/public/K4/success} & \res{enso/public/K4/gpus} & \res{enso/public/K4/prompt} & \res{enso/public/K4/decode} & \res{enso/public/K4/kvgbs} & \res{enso/public/K4/vram} & \res{enso/public/K4/p50} & \res{enso/public/K4/p95} & \res{enso/public/K4/usd} \\
|
||||
K5 & \res{enso/public/K5/success} & \res{enso/public/K5/gpus} & \res{enso/public/K5/prompt} & \res{enso/public/K5/decode} & \res{enso/public/K5/kvgbs} & \res{enso/public/K5/vram} & \res{enso/public/K5/p50} & \res{enso/public/K5/p95} & \res{enso/public/K5/usd} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Enso conditions on public agent benchmarks, per successful task; units as in Table~\ref{tab:enso}.}
|
||||
\label{tab:ensopublic}
|
||||
\end{table}
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\small
|
||||
\setlength{\tabcolsep}{4pt}
|
||||
\begin{tabular}{@{}l@{\quad}ccccccccc@{}}
|
||||
\toprule
|
||||
& Success & GPU-s & Prompt & Decode & KV GB$\cdot$s & VRAM & p50 & p95 & \$ \\
|
||||
\midrule
|
||||
B0 & \res{enso/traces/B0/success} & \res{enso/traces/B0/gpus} & \res{enso/traces/B0/prompt} & \res{enso/traces/B0/decode} & \res{enso/traces/B0/kvgbs} & \res{enso/traces/B0/vram} & \res{enso/traces/B0/p50} & \res{enso/traces/B0/p95} & \res{enso/traces/B0/usd} \\
|
||||
B1 & \res{enso/traces/B1/success} & \res{enso/traces/B1/gpus} & \res{enso/traces/B1/prompt} & \res{enso/traces/B1/decode} & \res{enso/traces/B1/kvgbs} & \res{enso/traces/B1/vram} & \res{enso/traces/B1/p50} & \res{enso/traces/B1/p95} & \res{enso/traces/B1/usd} \\
|
||||
R1 & \res{enso/traces/R1/success} & \res{enso/traces/R1/gpus} & \res{enso/traces/R1/prompt} & \res{enso/traces/R1/decode} & \res{enso/traces/R1/kvgbs} & \res{enso/traces/R1/vram} & \res{enso/traces/R1/p50} & \res{enso/traces/R1/p95} & \res{enso/traces/R1/usd} \\
|
||||
K1 & \res{enso/traces/K1/success} & \res{enso/traces/K1/gpus} & \res{enso/traces/K1/prompt} & \res{enso/traces/K1/decode} & \res{enso/traces/K1/kvgbs} & \res{enso/traces/K1/vram} & \res{enso/traces/K1/p50} & \res{enso/traces/K1/p95} & \res{enso/traces/K1/usd} \\
|
||||
K2 & \res{enso/traces/K2/success} & \res{enso/traces/K2/gpus} & \res{enso/traces/K2/prompt} & \res{enso/traces/K2/decode} & \res{enso/traces/K2/kvgbs} & \res{enso/traces/K2/vram} & \res{enso/traces/K2/p50} & \res{enso/traces/K2/p95} & \res{enso/traces/K2/usd} \\
|
||||
K3 & \res{enso/traces/K3/success} & \res{enso/traces/K3/gpus} & \res{enso/traces/K3/prompt} & \res{enso/traces/K3/decode} & \res{enso/traces/K3/kvgbs} & \res{enso/traces/K3/vram} & \res{enso/traces/K3/p50} & \res{enso/traces/K3/p95} & \res{enso/traces/K3/usd} \\
|
||||
K4 & \res{enso/traces/K4/success} & \res{enso/traces/K4/gpus} & \res{enso/traces/K4/prompt} & \res{enso/traces/K4/decode} & \res{enso/traces/K4/kvgbs} & \res{enso/traces/K4/vram} & \res{enso/traces/K4/p50} & \res{enso/traces/K4/p95} & \res{enso/traces/K4/usd} \\
|
||||
K5 & \res{enso/traces/K5/success} & \res{enso/traces/K5/gpus} & \res{enso/traces/K5/prompt} & \res{enso/traces/K5/decode} & \res{enso/traces/K5/kvgbs} & \res{enso/traces/K5/vram} & \res{enso/traces/K5/p50} & \res{enso/traces/K5/p95} & \res{enso/traces/K5/usd} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Enso conditions on replayed Hanzo traces, per successful task; units as in Table~\ref{tab:enso}.}
|
||||
\label{tab:ensotraces}
|
||||
\end{table}
|
||||
|
||||
Reference in New Issue
Block a user