mirror of
https://github.com/hanzoai/papers.git
synced 2026-10-22 17:20:17 +00:00
Kai: the speed section says which precision each runtime serves on CPU and why the Laya checkpoint timed matters.
Co-authored-by: Hanzo Dev <[email protected]>
This commit is contained in:
@@ -131,9 +131,14 @@ invariance, media, conformal sets, refresh and offline operation. Written by
|
||||
A call is one state with $N$ questions, timed from tokenizing to probabilities
|
||||
on the host, the first call at each $N$ discarded, percentiles nearest-rank
|
||||
(\texttt{bench speed}). Kai and Laya run on the same Rust runtime, device and
|
||||
batch sizes, on Metal and on CPU; Jev is one Decisions API call over the
|
||||
network, split into the largest chunks it accepts, eight in flight, when it
|
||||
refuses the whole. Kai's advantage, where it has one, has three sources that
|
||||
batch sizes, their calls interleaved, on Metal in F32 and on CPU, where the
|
||||
runtime serves Kai's encoder projections in Q8\_0 and Laya's in F32; Jev is one
|
||||
Decisions API call over the network, split into the largest chunks it accepts,
|
||||
eight in flight, when it refuses the whole. The table's last row names the Laya
|
||||
checkpoint timed: Laya's multilingual checkpoint shares Kai's encoder,
|
||||
mmBERT-base, so against it the difference is the head and the layout, while its
|
||||
English and typed-decisions checkpoints are ModernBERT-large, 28 layers of width
|
||||
1{,}024 against 22 of width 768. Kai's advantage, where it has one, has three sources that
|
||||
Laya's layout cannot share: the state is encoded once for all $N$ questions, an
|
||||
option shared by several questions is encoded once, and each question's head
|
||||
costs the options it scores, not a sequence holding all of them. For
|
||||
|
||||
Reference in New Issue
Block a user