mirror of
https://github.com/hanzoai/laya.git
synced 2026-10-30 19:33:31 +00:00
The old comparison table carried figures that this repo's own benchmark run does not support (83.8% in-task macro accuracy, "+16% higher accuracy than Jev"). Replaced with what was actually measured on identical questions across all three checkpoints, plus the limits. - Speed: T4-measured, 32.8 ms for 1 question and 7.2 ms/question at batch 10. - Jev: only third-party published figures, labelled as such, since there is no TypeSafe API access here. Laya leads on AG News (0.947 vs 0.910) and DAIR Emotion (0.573 vs 0.480); fine-tuned leads typed-decisions (0.766 vs 0.727). - Multilingual: 51-language sweep. The English checkpoint macro-averages 0.227 with macro ECE 0.733 and only clears 3x random on 23 of 51 languages; Khmer is 0.000 accuracy at 0.952 confidence. That is the case for routing. - Honest limits section: the base checkpoints are near chance on typed-decisions zero-shot (0.362 / 0.352 vs a 0.318 random and 0.461 majority-class baseline). The 0.766 is the fine-tuned checkpoint on that benchmark's own training split, and the README now says so. - Calibration: both ship over-confident; refitting temperature moves mean ECE 0.466 -> 0.081 and 0.314 -> 0.106. - Fine-tuning: keep only the 2xT4 Kaggle notebook, drop the two Colab ones. Co-Authored-By: Claude Opus 5 <[email protected]>