mirror of
https://github.com/hanzoai/ml.git
synced 2026-11-19 23:50:51 +00:00
quantize turns the projections the CPU tiles into Q8_0 blocks. A quantized backbone then runs each sequence's real tokens only: embeddings, norms and projections over the packed tokens, attention per (sequence, head) straight from the QKV projection (attention::cpu_flash::packed: rotation, window, softmax and both products in one parallel pass), GeGLU in one vectorized pass (ops::geglu, erf within 1.5e-7 of exact, the composite wherever a gradient flows); pad positions come back zero. On Metal, inference takes one SDPA kernel for scores, mask, softmax and values. Training's path is unchanged. Claude-Session: https://claude.ai/code/session_018Gt5TvrE2rxXuZikWSauv3