mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
3.5 KiB
3.5 KiB
Quantization in Hanzo Engine
Hanzo Engine supports the following quantization:
- ⭐ ISQ (read more detail)
- Supported in all plain/multimodal and adapter models
- Works on all supported devices
- Automatic selection to use the fastest and most accurate method
- Supports:
- Q, K type GGUF quants
- AFQ
- HQQ
- FP8
- F8Q8
- MXFP4
- GGUF/GGML
- Q, K type
- Supported in GGUF/GGML and GGUF/GGML adapter models
- Supported in all plain/multimodal and adapter models
- Imatrix quantization is supported
- I-quants (IQ1/IQ2/IQ3/IQ4) decode natively + bit-exact on ROCm (RDNA3.5 APUs)
- CPU, CUDA, Metal, ROCm (all supported devices)
- 1, 2, 3, 4, 5, 6, 8 bit
- On
gfx1151the full GGUF zoo (22 types: Q/K, legacy, IQ, TQ) decodes resident through one unified compute core -- see ROCm GGUF decode on RDNA3.5 APUs
- GPTQ (convert with this script)
- Supported in all plain/multimodal and adapter models
- CUDA only
- 2, 3, 4, 8 bit
- Marlin kernel support in 4-bit and 8-bit.
- AWQ (convert with this script)
- Supported in all plain/multimodal and adapter models
- CUDA only
- 4 and 8 bit
- Marlin kernel support in 4-bit and 8-bit.
- HQQ
- Supported in all plain/multimodal and adapter models via ISQ
- 4, 8 bit
- CPU, CUDA, Metal (all supported devices)
- FP8
- Supported in all plain/multimodal and adapter models
- CPU, CUDA, Metal (all supported devices)
- BNB
- Supported in all plain/multimodal and adapter models
- bitsandbytes int8, fp4, nf4 support
- AFQ
- 2, 3, 4, 6, 8 bit
- 🔥 Designed to be fast on Metal!
- Only supported on Metal.
- MLX prequantized
- Supported in all plain/multimodal and adapter models
Using a GGUF quantized model
- Use the
gguf(cli) /GGUF(Python) model selector - Provide the GGUF file
hanzo run --format gguf -f my-gguf-file.gguf
Using ISQ
See the docs
hanzo run --isq 4 -m microsoft/Phi-3-mini-4k-instruct
Using a GPTQ quantized model
- Provide the model ID for the GPTQ model
- Hanzo Engine will automatically detect and use GPTQ quantization for plain and multimodal models!
- The Marlin kernel will automatically be used for 4-bit and 8-bit.
hanzo run -m kaitchup/Phi-3-mini-4k-instruct-gptq-4bit
You can create your own GPTQ model using scripts/convert_to_gptq.py:
pip install gptqmodel transformers datasets
python3 scripts/convert_to_gptq.py --src path/to/model --dst output/model/path --bits 4
Using a MLX prequantized model (on Metal)
- Provide the model ID for the MLX prequantized model
- Hanzo Engine will automatically detect and use quantization for plain and multimodal models!
- Specialized kernels will be used to accelerate inference!
hanzo run -m mlx-community/Llama-3.8-1B-8bit
See Also
- Performance Guide: Choosing the right quantization for your hardware
- ISQ (In-Situ Quantization): Quantize any model at load time
- UQFF Format: Pre-quantized model format
- Topology: Per-layer quantization control