crates.io `hanzo-cli` is the Hanzo CLI (hanzoai/cli: bins hanzo, driftgate, genproduct, genspec), not this repo's engine CLI, so `cargo install hanzo-cli --features cuda` installed the wrong crate or failed on the feature. Every such line now reads `cargo install --locked --git https://github.com/hanzoai/engine hanzo-cli`, which builds `hanzo-engine` from this repo as the root install.sh does. site/install.sh (engine.hanzo.ai/install.sh) fell back to `cargo install hanzo-engine`; crates.io hanzo-engine 0.6.1 is a library with no binary. It now uses the same git install.
4.8 KiB
Frequently Asked Questions
General
How much VRAM do I need?
It depends on the model, quantization level, context length, and batch size. The best way to find out is to let hanzo analyze your hardware:
hanzo tune -m <model>
This shows a table of quantization options with estimated memory usage, context headroom, and quality trade-offs, all specific to your GPU. Quantization (--isq 4) typically reduces memory by 3-4x compared to FP16.
Which quantization should I use?
- Just want it to work? Use
--isq 4. hanzo picks the best format for your hardware. - Have plenty of VRAM? Use
--isq 8for near-lossless quality. - Want the fastest loading? Use a pre-quantized UQFF or GGUF model. No quantization happens at load time.
- Need per-layer control? Use a topology file to set different quantization per layer.
- On Apple Silicon?
--isq 4uses AFQ4, which is optimized for Metal.
See the Quantization Overview for all options.
What's the difference between ISQ, GGUF, and UQFF?
- ISQ quantizes at load time. Any HF model works, but loading is slower because quantization happens on the fly.
- GGUF is a pre-quantized single-file format, widely published on the Hugging Face Hub. Download a GGUF file and load it directly.
- UQFF is hanzo's own pre-quantized format. Supports more quantization types than GGUF (HQQ, FP8, AFQ, etc.) and loads instantly.
Can I use models from local files?
Yes. Pass a local path instead of a Hugging Face model ID:
hanzo run -m /path/to/model
How do I use gated models (Llama, etc.)?
- Accept the model license on Hugging Face
- Authenticate:
hanzo login - Run normally:
hanzo run -m meta-llama/Llama-3.2-3B-Instruct
Alternatively, use --token-source env:HF_TOKEN with a Hugging Face token in your environment.
How do I update hanzo?
Re-run the install script to build the latest version:
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/hanzoai/engine/master/install.sh | sh
Or if installed via cargo:
cargo install --locked --git https://github.com/hanzoai/engine hanzo-cli --features "<your-features>" --force
Does hanzo work on WSL?
Yes. WSL2 with CUDA support works well. Install the NVIDIA CUDA toolkit for WSL and build with --features cuda.
Is Windows supported?
Yes, but WSL2 is recommended for the best experience. Native Windows builds work but PagedAttention is only available on Unix-like platforms.
Performance
Why is the first request slow?
hanzo performs a warmup run when loading a model. It sends a short dummy request to initialize CUDA kernels and caches. You'll see "Beginning dummy run..." and "Dummy run completed" in the logs. After this, subsequent requests are much faster.
How do I increase throughput?
- Enable PagedAttention (on by default for CUDA)
- Increase
--max-seqsfor more concurrent batching (default: 32) - Use FlashAttention (compile with
--features flash-attn) - Use quantization to free VRAM for larger batches
See the Performance Guide for detailed optimization recipes.
PagedAttention is using too much VRAM
By default, PagedAttention allocates 90% of free VRAM for the KV cache. Reduce it:
# Allocate for a specific context length
hanzo serve -m <model> --pa-context-len 4096
# Or set a memory cap
hanzo serve -m <model> --pa-memory-mb 2048
# Or use a fraction
hanzo serve -m <model> --pa-memory-fraction 0.5
How do I reduce Time to First Token (TTFT)?
- Compile with FlashAttention (
--features flash-attn) to accelerate prefill - Enable prefix caching (
--prefix-cache-n 16, on by default) to reuse KV cache across turns - Use quantization to reduce memory bandwidth pressure
Compatibility
Can I use OpenAI client libraries?
Yes. hanzo provides an OpenAI-compatible HTTP API. Use the standard openai Python package:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:1234/v1/", api_key="foobar")
The api_key can be any non-empty string. hanzo doesn't validate it.
Can I use LangChain or LlamaIndex?
Yes. Point them at your hanzo server as an OpenAI-compatible endpoint:
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(base_url="http://localhost:1234/v1/", api_key="foobar")
Which models are supported?
See Supported Models for the complete list. Architecture is auto-detected, so just point hanzo at the model and it works.
Can I run multiple models at once?
Yes. Use a TOML config file with multiple [[models]] entries:
hanzo from-config --file config.toml
See Multi-Model Support for details.