Files
zeekay 35bbf78aea ModernBERT runs quantized inference packed on the CPU and its attention fused on Metal.
quantize turns the projections the CPU tiles into Q8_0 blocks. A quantized backbone then runs
each sequence's real tokens only: embeddings, norms and projections over the packed tokens,
attention per (sequence, head) straight from the QKV projection (attention::cpu_flash::packed:
rotation, window, softmax and both products in one parallel pass), GeGLU in one vectorized pass
(ops::geglu, erf within 1.5e-7 of exact, the composite wherever a gradient flows); pad positions
come back zero. On Metal, inference takes one SDPA kernel for scores, mask, softmax and values.
Training's path is unchanged.

Claude-Session: https://claude.ai/code/session_018Gt5TvrE2rxXuZikWSauv3
2026-09-27 05:00:26 -07:00
..
2026-07-28 06:03:51 -07:00