Replace all remaining references to the upstream project/author/framework brands with hanzo / hanzoai across source, comments, docs, examples, and build scripts. - Project spellings mistral.rs / mistral-rs / mistralrs -> hanzo (and the hanzo-* crate/binary names in Dockerfiles, Makefile, install.ps1, .gitignore). - Author EricLBuehler / EricBuehler / EricB / Eric Buehler -> hanzoai / Hanzo AI (incl. Cargo `authors` and LICENSE copyright). - URLs: github.com/EricLBuehler/mistral.rs -> github.com/hanzoai/engine; ericlbuehler.github.io/mistral.rs -> hanzoai.github.io/engine; github.com/huggingface/candle -> github.com/hanzoai/ml; docs.rs/candle-* -> docs.rs/hanzo-ml; ghcr.io image -> ghcr.io/hanzoai/engine. - Env var prefix MISTRALRS_ -> HANZO_ (producers and consumers kept in sync, incl. the build-script-injected HANZO_GIT_REVISION). - ROCm static lib libmistralrsrocm.a / mistralrsrocm -> libhanzorocm.a / hanzorocm. - localStorage keys mistralrs_settings -> hanzo_settings. Deliberately preserved: real model architectures (struct Mistral, models/mistral.rs file, Mixtral/Llama/Gemma/Qwen/Phi), real model-org HF repo-ids (mistralai/, meta-llama/, google/, Qwen/, microsoft/), the `use hanzo_ml as candle` code aliases and `candle::` identifiers, and the upstream-author HF data repo-id suffix in "hanzoai/mistralrs_tests". Both `cargo check` and `cargo check --features vulkan` pass with 0 errors. Co-authored-by: Hanzo Dev <[email protected]>
4.8 KiB
Frequently Asked Questions
General
How much VRAM do I need?
It depends on the model, quantization level, context length, and batch size. The best way to find out is to let hanzo analyze your hardware:
hanzo tune -m <model>
This shows a table of quantization options with estimated memory usage, context headroom, and quality trade-offs, all specific to your GPU. Quantization (--isq 4) typically reduces memory by 3-4x compared to FP16.
Which quantization should I use?
- Just want it to work? Use
--isq 4. hanzo picks the best format for your hardware. - Have plenty of VRAM? Use
--isq 8for near-lossless quality. - Want the fastest loading? Use a pre-quantized UQFF or GGUF model. No quantization happens at load time.
- Need per-layer control? Use a topology file to set different quantization per layer.
- On Apple Silicon?
--isq 4uses AFQ4, which is optimized for Metal.
See the Quantization Overview for all options.
What's the difference between ISQ, GGUF, and UQFF?
- ISQ quantizes at load time. Any HF model works, but loading is slower because quantization happens on the fly.
- GGUF is a pre-quantized format from the llama.cpp ecosystem. Download a GGUF file and load it directly.
- UQFF is hanzo's own pre-quantized format. Supports more quantization types than GGUF (HQQ, FP8, AFQ, etc.) and loads instantly.
Can I use models from local files?
Yes. Pass a local path instead of a Hugging Face model ID:
hanzo run -m /path/to/model
How do I use gated models (Llama, etc.)?
- Accept the model license on Hugging Face
- Authenticate:
hanzo login - Run normally:
hanzo run -m meta-llama/Llama-3.2-3B-Instruct
Alternatively, use --token-source env:HF_TOKEN with a Hugging Face token in your environment.
How do I update hanzo?
Re-run the install script to build the latest version:
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/hanzoai/engine/master/install.sh | sh
Or if installed via cargo:
cargo install hanzo-cli --features "<your-features>" --force
Does hanzo work on WSL?
Yes. WSL2 with CUDA support works well. Install the NVIDIA CUDA toolkit for WSL and build with --features cuda.
Is Windows supported?
Yes, but WSL2 is recommended for the best experience. Native Windows builds work but PagedAttention is only available on Unix-like platforms.
Performance
Why is the first request slow?
hanzo performs a warmup run when loading a model. It sends a short dummy request to initialize CUDA kernels and caches. You'll see "Beginning dummy run..." and "Dummy run completed" in the logs. After this, subsequent requests are much faster.
How do I increase throughput?
- Enable PagedAttention (on by default for CUDA)
- Increase
--max-seqsfor more concurrent batching (default: 32) - Use FlashAttention (compile with
--features flash-attn) - Use quantization to free VRAM for larger batches
See the Performance Guide for detailed optimization recipes.
PagedAttention is using too much VRAM
By default, PagedAttention allocates 90% of free VRAM for the KV cache. Reduce it:
# Allocate for a specific context length
hanzo serve -m <model> --pa-context-len 4096
# Or set a memory cap
hanzo serve -m <model> --pa-memory-mb 2048
# Or use a fraction
hanzo serve -m <model> --pa-memory-fraction 0.5
How do I reduce Time to First Token (TTFT)?
- Compile with FlashAttention (
--features flash-attn) to accelerate prefill - Enable prefix caching (
--prefix-cache-n 16, on by default) to reuse KV cache across turns - Use quantization to reduce memory bandwidth pressure
Compatibility
Can I use OpenAI client libraries?
Yes. hanzo provides an OpenAI-compatible HTTP API. Use the standard openai Python package:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:1234/v1/", api_key="foobar")
The api_key can be any non-empty string. hanzo doesn't validate it.
Can I use LangChain or LlamaIndex?
Yes. Point them at your hanzo server as an OpenAI-compatible endpoint:
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(base_url="http://localhost:1234/v1/", api_key="foobar")
Which models are supported?
See Supported Models for the complete list. Architecture is auto-detected, so just point hanzo at the model and it works.
Can I run multiple models at once?
Yes. Use a TOML config file with multiple [[models]] entries:
hanzo from-config --file config.toml
See Multi-Model Support for details.