mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-21 02:36:58 +00:00
Qwen3.5-27B runs 24 query heads over 4 KV heads. DISPATCH_GQA_GROUP_SIZE had no arm for a group of 6, so FlashInfer refused the layer. Refusing it also gave the model the Standard cache layout, and the paged prefill kernel cannot read that layout, so every prefill fell through to eager attention. Add the arm and keep FLASHINFER_DECODE_GROUP_SIZES in step with it. The layer then gets the FlashInfer layout: prefill runs flashinfer_prefill, which never builds the score matrix, and decode has a kernel for the group. Taken from mistral.rs d1ffbe7e43 (#2373). Our decode.cuh differs from theirs only in how the shared-memory attribute is set, so the kernel the new arm instantiates is the one they ship. Co-authored-by: Hanzo Dev <[email protected]>