Unbrand the pre-existing HANZO_* env vars (multinode MN_*, SANDBOX, VK_*, NO_MMAP/NO_MLA/NO_NCCL, IGPU_MEMORY_FRACTION, ...) to bare functional names, keeping their meaning. No HANZO_ env var branding remains.
7.4 KiB
Performance Guide
This guide helps you choose the right performance settings for your hardware and workload. Each section gives you the key decision, then links to the detailed reference page.
Don't want to read all this? Run
hanzo tune -m <model>and it will recommend the best configuration for your hardware automatically.
Quick Recommendations
| Hardware | Suggested command |
|---|---|
| NVIDIA GPU (16+ GB VRAM) | hanzo serve --isq 4 -m <model> |
| NVIDIA GPU (8 GB VRAM) | hanzo serve --isq 4 -m <7B-model> |
| Apple Silicon (16+ GB RAM) | hanzo serve --isq 4 -m <model> |
| CPU only | hanzo serve --isq 4 --cpu -m <small-model> |
| Multi-GPU NVIDIA | hanzo serve -m <model> (auto tensor parallelism) |
For precise recommendations, use hanzo tune -m <model> which analyzes your available memory and model size.
The Performance Stack
hanzo has several independent optimization layers. You can use any combination:
- Quantization: Reduces model weight memory (ISQ, GGUF, UQFF)
- FlashAttention: Accelerates prompt processing (prefill phase)
- PagedAttention: Efficiently manages KV cache memory during generation
- MLA: Compresses KV cache for DeepSeek/GLM models
- Speculative Decoding: Uses a draft model to propose multiple tokens at once
Quantization: Choosing a Method
Quantization reduces model size by using lower-precision weights. hanzo offers several approaches:
| Method | When to use | Details |
|---|---|---|
ISQ (--isq <level>) |
Default choice. Quantize any model at load time. Levels: 2-8 | ISQ docs |
GGUF (--format gguf) |
Load pre-quantized GGUF files from the community | GGUF section |
UQFF (--from-uqff) |
Load pre-quantized UQFF files (faster startup than ISQ) | UQFF docs |
Topology (--topology) |
Per-layer control: different quantization per layer | Topology docs |
| GPTQ/AWQ | Use GPTQ or AWQ models from HF (auto-detected, CUDA only) | GPTQ/AWQ section |
ISQ levels at a glance:
| Level | Quality | Notes |
|---|---|---|
--isq 8 (Q8_0/AFQ8) |
Near-lossless | Use when you have VRAM headroom |
--isq 6 (Q6K/AFQ6) |
Good | Balanced quality and size |
--isq 5 (Q5K) |
Good | Slightly better quality than Q4K |
--isq 4 (Q4K/AFQ4) |
Acceptable | Most common choice |
--isq 3 (Q3K/AFQ3) |
Degraded | Tight VRAM budget |
--isq 2 (Q2K/AFQ2) |
Significantly degraded | Extreme constraint |
On Metal, ISQ will prefer using AFQ (optimized for Apple Silicon). On CUDA/CPU, it uses Q/K quantization.
To improve ISQ accuracy, use an importance matrix: --calibration-file calibration_data/calibration_datav3_small.txt. See Importance Matrix.
MoQE: Smarter Quantization for MoE Models
For Mixture of Experts models (DeepSeek V2/V3, Qwen3 MoE, Qwen3-VL MoE, GLM-4.7), MoQE quantizes only the expert layers while keeping attention and other layers at higher precision. This preserves more quality than uniform quantization since expert layers make up the bulk of parameters in MoE models.
hanzo run --isq 4 --isq-organization moqe -m deepseek-ai/DeepSeek-R1
MoQE is available in the CLI (--isq-organization moqe), Python SDK (organization="MoQE"), and Rust SDK. See ISQ docs for the full list of supported models.
Full quantization reference: Quantization Overview
Memory Management: PagedAttention
PagedAttention manages KV cache memory efficiently, enabling longer contexts and higher throughput through continuous batching.
Default behavior:
- CUDA: Enabled automatically
- Metal: Disabled by default (enable with
--paged-attn on) - CPU: Not supported
Tuning KV cache allocation:
# Allocate for a specific context length (recommended)
hanzo serve -m <model> --pa-context-len 8192
# Or set a fixed memory budget
hanzo serve -m <model> --pa-memory-mb 4096
# Or use a fraction of available VRAM
hanzo serve -m <model> --pa-memory-fraction 0.8
FP8 KV cache quantization halves KV cache memory, allowing ~2x longer contexts:
hanzo serve -m <model> --pa-cache-type f8e4m3
Full reference: PagedAttention
Attention Acceleration: FlashAttention
FlashAttention accelerates the prefill phase (processing the prompt). It requires CUDA and a compatible GPU.
| GPU generation | Feature flag | Notes |
|---|---|---|
| Ampere (RTX 30xx, A100) | --features flash-attn |
FlashAttention V2 |
| Ada Lovelace (RTX 40xx) | --features flash-attn |
FlashAttention V2 |
| Hopper (H100) | --features flash-attn-v3 |
FlashAttention V3 |
| Blackwell (RTX 50xx) | --features flash-attn |
FlashAttention V2 |
FlashAttention V2 and V3 are mutually exclusive. If compiled with FlashAttention and PagedAttention is enabled, both work together automatically.
Full reference: FlashAttention
Multi-head Latent Attention (MLA)
MLA compresses the KV cache for DeepSeek V2/V3 and GLM-4.7-Flash models, reducing memory usage and enabling longer contexts. It activates automatically when:
- The model supports it (DeepSeek V2, V3, GLM-4.7-Flash)
- PagedAttention is enabled
- Running on CUDA
Disable with NO_MLA=1 if needed.
Full reference: MLA
Multi-GPU and Distributed Inference
Single machine, multiple GPUs:
hanzo auto-detects multiple CUDA GPUs and uses tensor parallelism via NCCL. No configuration needed:
# Uses all available GPUs automatically
hanzo serve -m <large-model>
# Or specify GPU count
MN_LOCAL_WORLD_SIZE=2 hanzo serve -m <large-model>
If the model doesn't fit on GPUs even with parallelism, disable NCCL to use automatic device mapping (GPU + CPU offloading):
NO_NCCL=1 hanzo serve --isq 4 -m <large-model>
Multiple machines: Use the Ring backend for cross-machine inference over TCP.
Full reference: Device Mapping | NCCL | Ring
Speculative Decoding
Speculative decoding uses a small "draft" model to propose tokens, which the large "target" model validates in parallel. This can significantly speed up generation when the draft model is accurate.
When it helps:
- Long generation tasks (100+ tokens)
- Draft model from the same family (e.g., Llama 3.2-1B drafting for Llama 3.1-8B)
- Low batch sizes
When to skip it:
- High-throughput batched serving (standard decoding is more efficient)
- No good draft model available
# Configure via TOML
hanzo run --from-toml speculative.toml
Full reference: Speculative Decoding
Benchmarking Your Setup
Quick benchmark:
hanzo bench -m <model> --isq 4
This measures prefill speed (tokens/sec processing the prompt) and decode speed (tokens/sec generating output). Key metrics:
- Prefill T/s: Higher is better. Affected by FlashAttention, GPU compute.
- Decode T/s: Higher is better. Affected by memory bandwidth, quantization, PagedAttention.
- TTFT (Time to First Token): The prefill latency in ms.
Automated recommendations:
hanzo tune -m <model>
Shows a table of quantization options ranked by fit, quality, and performance for your hardware. Use --profile quality or --profile fast to shift the ranking.