Every mention of another inference engine is rewritten to say what our code does, or the math it computes. Reference values are "the reference value from the published config" or "the recorded reference output", line citations into another engine's source become the rule itself, comparisons with other engines are dropped, and asides about other engines' bugs go. Identifiers follow: - CUDA: hanzo-paged-attn's `namespace vllm` is `pagedattn`, hanzo-engine's MoE helpers `moe_dtype` and `moe_detail`, the rotary kernels' `hanzo_rotary`; the shuffle/ldg macros are HANZO_*. - GPTQ: the shuffled-layout path (`reconstruct_exllama`, `use_exllama`) is `reconstruct_shuffled` / `is_shuffled`, on the kernel and the FFI side. - Tests: `*_matches_vllm` and friends name what they compare against (the reference fixture, the tile rule). - MiniCPM-o's vision-LLM embedding is `vlm_embedding`; the rotary custom op is `rotary-embedding`. - hanzo-bench's board scores ours against a "rival" whose name comes from the manifest: `rival_<tag>.json`, `--rival*` manifest flags, `rival t/s` columns; dossier.sh and spec_decode.sh take `--rival-bench`. - install.ps1 installs this repository's hanzo-cli, as install.sh does, with HANZO_INSTALL_NCCL / HANZO_INSTALL_NO_NCCL. - bench_http.py reads only the hanzo: metric families. Kept, because they are what they are: LICENSE, LICENSE-APACHE, NOTICE and tools/upstream-map.sh; each ported file's copyright/licence/provenance header; Cargo.toml dependency names; data identifiers (Hugging Face repo IDs, fixture file names and fixture metadata keys the tests read, on-disk paths, executable names the GPU and engine guards match on a host); model and format names. LLM.md says the rule once, under "One name: Hanzo Engine". Co-authored-by: Hanzo Dev <[email protected]>
4.8 KiB
Frequently Asked Questions
General
How much VRAM do I need?
It depends on the model, quantization level, context length, and batch size. The best way to find out is to let hanzo analyze your hardware:
hanzo tune -m <model>
This shows a table of quantization options with estimated memory usage, context headroom, and quality trade-offs, all specific to your GPU. Quantization (--isq 4) typically reduces memory by 3-4x compared to FP16.
Which quantization should I use?
- Just want it to work? Use
--isq 4. hanzo picks the best format for your hardware. - Have plenty of VRAM? Use
--isq 8for near-lossless quality. - Want the fastest loading? Use a pre-quantized UQFF or GGUF model. No quantization happens at load time.
- Need per-layer control? Use a topology file to set different quantization per layer.
- On Apple Silicon?
--isq 4uses AFQ4, which is optimized for Metal.
See the Quantization Overview for all options.
What's the difference between ISQ, GGUF, and UQFF?
- ISQ quantizes at load time. Any HF model works, but loading is slower because quantization happens on the fly.
- GGUF is a pre-quantized single-file format, widely published on the Hugging Face Hub. Download a GGUF file and load it directly.
- UQFF is hanzo's own pre-quantized format. Supports more quantization types than GGUF (HQQ, FP8, AFQ, etc.) and loads instantly.
Can I use models from local files?
Yes. Pass a local path instead of a Hugging Face model ID:
hanzo run -m /path/to/model
How do I use gated models (Llama, etc.)?
- Accept the model license on Hugging Face
- Authenticate:
hanzo login - Run normally:
hanzo run -m meta-llama/Llama-3.2-3B-Instruct
Alternatively, use --token-source env:HF_TOKEN with a Hugging Face token in your environment.
How do I update hanzo?
Re-run the install script to build the latest version:
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/hanzoai/engine/master/install.sh | sh
Or if installed via cargo:
cargo install hanzo-cli --features "<your-features>" --force
Does hanzo work on WSL?
Yes. WSL2 with CUDA support works well. Install the NVIDIA CUDA toolkit for WSL and build with --features cuda.
Is Windows supported?
Yes, but WSL2 is recommended for the best experience. Native Windows builds work but PagedAttention is only available on Unix-like platforms.
Performance
Why is the first request slow?
hanzo performs a warmup run when loading a model. It sends a short dummy request to initialize CUDA kernels and caches. You'll see "Beginning dummy run..." and "Dummy run completed" in the logs. After this, subsequent requests are much faster.
How do I increase throughput?
- Enable PagedAttention (on by default for CUDA)
- Increase
--max-seqsfor more concurrent batching (default: 32) - Use FlashAttention (compile with
--features flash-attn) - Use quantization to free VRAM for larger batches
See the Performance Guide for detailed optimization recipes.
PagedAttention is using too much VRAM
By default, PagedAttention allocates 90% of free VRAM for the KV cache. Reduce it:
# Allocate for a specific context length
hanzo serve -m <model> --pa-context-len 4096
# Or set a memory cap
hanzo serve -m <model> --pa-memory-mb 2048
# Or use a fraction
hanzo serve -m <model> --pa-memory-fraction 0.5
How do I reduce Time to First Token (TTFT)?
- Compile with FlashAttention (
--features flash-attn) to accelerate prefill - Enable prefix caching (
--prefix-cache-n 16, on by default) to reuse KV cache across turns - Use quantization to reduce memory bandwidth pressure
Compatibility
Can I use OpenAI client libraries?
Yes. hanzo provides an OpenAI-compatible HTTP API. Use the standard openai Python package:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:1234/v1/", api_key="foobar")
The api_key can be any non-empty string. hanzo doesn't validate it.
Can I use LangChain or LlamaIndex?
Yes. Point them at your hanzo server as an OpenAI-compatible endpoint:
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(base_url="http://localhost:1234/v1/", api_key="foobar")
Which models are supported?
See Supported Models for the complete list. Architecture is auto-detected, so just point hanzo at the model and it works.
Can I run multiple models at once?
Yes. Use a TOML config file with multiple [[models]] entries:
hanzo from-config --file config.toml
See Multi-Model Support for details.