Files
zeekayandHanzo Dev d51a119438 quant: the FP8 conversion kernels report a failed launch and skip an empty tensor
A conversion launch that failed printed to stderr and exited the process, from
inside a serving engine; it returns the CUDA error to the caller, who reports
it with the tensor's shape. A tensor with no elements launches nothing, since a
zero-block grid is itself the error.

(cherry picked from commit f29516e09cd6c1777575ec27481bf57efbff4003)

Co-authored-by: Hanzo Dev <[email protected]>
2026-09-17 20:36:44 -07:00
..

hanzo-quant

An advanced and highly diverse set of quantization techniques. This crate supports both quantization and optimized inference.

It has grown beyon simply quantization and is used by hanzo to power:

  • ISQ
  • Imatrix collection
  • General quantization features
  • Specific CUDA and Metal features
  • cuBLASlt integration

Currently supported:

  • AFQ: GgufMatMul (2-8 bit quantization optimized for Metal and compatible with MLX)
  • GGUF: GgufMatMul (2-8 bit quantization, with imatrix)
  • Gptq/Awq: GptqAwqLayer (with CUDA marlin kernel)
  • Hqq: HqqLayer (4, 8 bit quantization)
  • FP8: FP8Linear
  • F8Q8: F8Q8Linear
  • Unquantized (used for ISQ): UnquantLinear
  • Bnb: BnbLinear (int8, fp4, nf4)

Some kernels are copied or based on implementations in: