Files
zeekayandHanzo Dev 54e770669d perf(cuda): tile indexed-MoE rows per block for higher GB10 occupancy
The fused indexed-MoE decode kernel computed 1 output row per block with a
4-warp shared-memory reduction, leaving the GB10's SMs underutilized and
reduction-bound. Templatize rows_per_cuda_block and have each block compute
a tile of INDEXED_MOE_ROWS_PER_BLOCK (4) rows, reading the q8_1 input once
and reusing it across the tile; the launchers set gridDim.x = ceil(n/4).
Dot-product math is unchanged (mirrors mul_mat_vec_q); weight bandwidth is
unchanged. Covers all wired dtypes (Q4_0/Q4_1/Q5_0/Q5_1/Q8_0/Q2K-Q6K).

Co-authored-by: Hanzo Dev <[email protected]>
2026-06-02 14:30:42 -07:00
..