Kai: the speed section says which precision each runtime serves on CPU and why the Laya checkpoint timed matters.

Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
2026-09-27 09:21:53 -07:00
co-authored by Hanzo Dev
parent 6df3d4401c
commit c1d6c0dff3
+8 -3
View File
@@ -131,9 +131,14 @@ invariance, media, conformal sets, refresh and offline operation. Written by
A call is one state with $N$ questions, timed from tokenizing to probabilities
on the host, the first call at each $N$ discarded, percentiles nearest-rank
(\texttt{bench speed}). Kai and Laya run on the same Rust runtime, device and
batch sizes, on Metal and on CPU; Jev is one Decisions API call over the
network, split into the largest chunks it accepts, eight in flight, when it
refuses the whole. Kai's advantage, where it has one, has three sources that
batch sizes, their calls interleaved, on Metal in F32 and on CPU, where the
runtime serves Kai's encoder projections in Q8\_0 and Laya's in F32; Jev is one
Decisions API call over the network, split into the largest chunks it accepts,
eight in flight, when it refuses the whole. The table's last row names the Laya
checkpoint timed: Laya's multilingual checkpoint shares Kai's encoder,
mmBERT-base, so against it the difference is the head and the layout, while its
English and typed-decisions checkpoints are ModernBERT-large, 28 layers of width
1{,}024 against 22 of width 768. Kai's advantage, where it has one, has three sources that
Laya's layout cannot share: the state is encoded once for all $N$ questions, an
option shared by several questions is encoded once, and each question's head
costs the options it scores, not a sequence holding all of them. For