Files
zeekayandHanzo Dev 3b46de2c60 perf(quant): NVFP4 prefill on block-scaled tensor cores
The WMMA kernel dequantizes weights to bf16 and runs the MMA there, which
caps it at the bf16 tensor-core rate. A GB10 measures 123.5 TFLOP/s on a
register-resident bf16 WMMA chain and this kernel reached 18.5 of it; the
block-scaled FP4 MMA reaches 357 on the same shapes, so the ceiling is the
dtype, not the tiling.

Both operands go in as E2M1: packed weights feed the MMA untouched, and
activations are quantized against the `input_scale` the checkpoint
calibrated for exactly this. A layer takes the path only when the device
carries the MMA, the shape meets its alignment and that scale is present;
any other layer keeps the dequantizing kernel. Decode stays on the matvec
path.

The GEMM is upstream's block-scaled FP4 kernel (0a29442f9b). Upstream
reaches it only through cuTile, which needs CUDA 13.2, and pins its build
to 13.3. The kernel compiles and runs correctly on 13.0, so activations
are quantized by a native CUDA kernel here and the build accepts the
version measured to work.

In-engine on a GB10, Qwen3.8-27B-NVFP4, prefill T/s:

    prompt tokens    before    after
        178            297      665
      1,291            358      954
     12,449            310      657

Decode is unchanged at 7.01 / 6.95 / 6.60.

Co-authored-by: Hanzo Dev <[email protected]>
2026-09-15 11:57:40 -07:00
..