Files
NandakishorandClaude Opus 5 7c3a480da3 docs: replace benchmark claims with measured numbers; trim notebooks
The old comparison table carried figures that this repo's own benchmark run
does not support (83.8% in-task macro accuracy, "+16% higher accuracy than
Jev"). Replaced with what was actually measured on identical questions across
all three checkpoints, plus the limits.

- Speed: T4-measured, 32.8 ms for 1 question and 7.2 ms/question at batch 10.
- Jev: only third-party published figures, labelled as such, since there is no
  TypeSafe API access here. Laya leads on AG News (0.947 vs 0.910) and DAIR
  Emotion (0.573 vs 0.480); fine-tuned leads typed-decisions (0.766 vs 0.727).
- Multilingual: 51-language sweep. The English checkpoint macro-averages 0.227
  with macro ECE 0.733 and only clears 3x random on 23 of 51 languages; Khmer
  is 0.000 accuracy at 0.952 confidence. That is the case for routing.
- Honest limits section: the base checkpoints are near chance on
  typed-decisions zero-shot (0.362 / 0.352 vs a 0.318 random and 0.461
  majority-class baseline). The 0.766 is the fine-tuned checkpoint on that
  benchmark's own training split, and the README now says so.
- Calibration: both ship over-confident; refitting temperature moves mean ECE
  0.466 -> 0.081 and 0.314 -> 0.106.
- Fine-tuning: keep only the 2xT4 Kaggle notebook, drop the two Colab ones.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-19 11:30:34 +05:30
..