mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 23:45:50 +00:00
The fused indexed-MoE decode kernel computed 1 output row per block with a 4-warp shared-memory reduction, leaving the GB10's SMs underutilized and reduction-bound. Templatize rows_per_cuda_block and have each block compute a tile of INDEXED_MOE_ROWS_PER_BLOCK (4) rows, reading the q8_1 input once and reusing it across the tile; the launchers set gridDim.x = ceil(n/4). Dot-product math is unchanged (mirrors mul_mat_vec_q); weight bandwidth is unchanged. Covers all wired dtypes (Q4_0/Q4_1/Q5_0/Q5_1/Q8_0/Q2K-Q6K). Co-authored-by: Hanzo Dev <[email protected]>