Files
zeekayandHanzo Dev 4e43d6e9cc fix(attn): FlashInfer instantiates GQA group 6
Qwen3.5-27B runs 24 query heads over 4 KV heads. DISPATCH_GQA_GROUP_SIZE
had no arm for a group of 6, so FlashInfer refused the layer. Refusing it
also gave the model the Standard cache layout, and the paged prefill kernel
cannot read that layout, so every prefill fell through to eager attention.

Add the arm and keep FLASHINFER_DECODE_GROUP_SIZES in step with it. The
layer then gets the FlashInfer layout: prefill runs flashinfer_prefill,
which never builds the score matrix, and decode has a kernel for the group.
Taken from mistral.rs d1ffbe7e43 (#2373). Our decode.cuh differs from theirs
only in how the shared-memory attribute is set, so the kernel the new arm
instantiates is the one they ship.


Co-authored-by: Hanzo Dev <[email protected]>
2026-09-15 15:46:08 -07:00
..