Files

3.5 KiB

Quantization in Hanzo Engine

Hanzo Engine supports the following quantization:

  • ⭐ ISQ (read more detail)
    • Supported in all plain/multimodal and adapter models
    • Works on all supported devices
    • Automatic selection to use the fastest and most accurate method
    • Supports:
      • Q, K type GGUF quants
      • AFQ
      • HQQ
      • FP8
      • F8Q8
      • MXFP4
  • GGUF/GGML
    • Q, K type
    • Supported in GGUF/GGML and GGUF/GGML adapter models
    • Supported in all plain/multimodal and adapter models
    • Imatrix quantization is supported
    • I-quants (IQ1/IQ2/IQ3/IQ4) decode natively + bit-exact on ROCm (RDNA3.5 APUs)
    • CPU, CUDA, Metal, ROCm (all supported devices)
    • 1, 2, 3, 4, 5, 6, 8 bit
    • On gfx1151 the full GGUF zoo (22 types: Q/K, legacy, IQ, TQ) decodes resident through one unified compute core -- see ROCm GGUF decode on RDNA3.5 APUs
  • GPTQ (convert with this script)
    • Supported in all plain/multimodal and adapter models
    • CUDA only
    • 2, 3, 4, 8 bit
    • Marlin kernel support in 4-bit and 8-bit.
  • AWQ (convert with this script)
    • Supported in all plain/multimodal and adapter models
    • CUDA only
    • 4 and 8 bit
    • Marlin kernel support in 4-bit and 8-bit.
  • HQQ
    • Supported in all plain/multimodal and adapter models via ISQ
    • 4, 8 bit
    • CPU, CUDA, Metal (all supported devices)
  • FP8
    • Supported in all plain/multimodal and adapter models
    • CPU, CUDA, Metal (all supported devices)
  • BNB
    • Supported in all plain/multimodal and adapter models
    • bitsandbytes int8, fp4, nf4 support
  • AFQ
    • 2, 3, 4, 6, 8 bit
    • 🔥 Designed to be fast on Metal!
    • Only supported on Metal.
  • MLX prequantized
    • Supported in all plain/multimodal and adapter models

Using a GGUF quantized model

  • Use the gguf (cli) / GGUF (Python) model selector
  • Provide the GGUF file
hanzo run --format gguf -f my-gguf-file.gguf

Using ISQ

See the docs

hanzo run --isq 4 -m microsoft/Phi-3-mini-4k-instruct

Using a GPTQ quantized model

  • Provide the model ID for the GPTQ model
  • Hanzo Engine will automatically detect and use GPTQ quantization for plain and multimodal models!
  • The Marlin kernel will automatically be used for 4-bit and 8-bit.
hanzo run -m kaitchup/Phi-3-mini-4k-instruct-gptq-4bit

You can create your own GPTQ model using scripts/convert_to_gptq.py:

pip install gptqmodel transformers datasets

python3 scripts/convert_to_gptq.py --src path/to/model --dst output/model/path --bits 4

Using a MLX prequantized model (on Metal)

  • Provide the model ID for the MLX prequantized model
  • Hanzo Engine will automatically detect and use quantization for plain and multimodal models!
  • Specialized kernels will be used to accelerate inference!
hanzo run -m mlx-community/Llama-3.8-1B-8bit

See Also