Files
zeekayandHanzo Dev 02b3f00cef chore: complete brand scrub of upstream mistral.rs lineage to hanzo/hanzoai
Replace all remaining references to the upstream project/author/framework
brands with hanzo / hanzoai across source, comments, docs, examples, and
build scripts.

- Project spellings mistral.rs / mistral-rs / mistralrs -> hanzo (and the
  hanzo-* crate/binary names in Dockerfiles, Makefile, install.ps1, .gitignore).
- Author EricLBuehler / EricBuehler / EricB / Eric Buehler -> hanzoai / Hanzo AI
  (incl. Cargo `authors` and LICENSE copyright).
- URLs: github.com/EricLBuehler/mistral.rs -> github.com/hanzoai/engine;
  ericlbuehler.github.io/mistral.rs -> hanzoai.github.io/engine;
  github.com/huggingface/candle -> github.com/hanzoai/ml;
  docs.rs/candle-* -> docs.rs/hanzo-ml; ghcr.io image -> ghcr.io/hanzoai/engine.
- Env var prefix MISTRALRS_ -> HANZO_ (producers and consumers kept in sync,
  incl. the build-script-injected HANZO_GIT_REVISION).
- ROCm static lib libmistralrsrocm.a / mistralrsrocm -> libhanzorocm.a / hanzorocm.
- localStorage keys mistralrs_settings -> hanzo_settings.

Deliberately preserved: real model architectures (struct Mistral, models/mistral.rs
file, Mixtral/Llama/Gemma/Qwen/Phi), real model-org HF repo-ids (mistralai/,
meta-llama/, google/, Qwen/, microsoft/), the `use hanzo_ml as candle` code
aliases and `candle::` identifiers, and the upstream-author HF data repo-id
suffix in "hanzoai/mistralrs_tests".

Both `cargo check` and `cargo check --features vulkan` pass with 0 errors.

Co-authored-by: Hanzo Dev <[email protected]>
2026-06-01 11:53:12 -07:00

4.8 KiB

Frequently Asked Questions

General

How much VRAM do I need?

It depends on the model, quantization level, context length, and batch size. The best way to find out is to let hanzo analyze your hardware:

hanzo tune -m <model>

This shows a table of quantization options with estimated memory usage, context headroom, and quality trade-offs, all specific to your GPU. Quantization (--isq 4) typically reduces memory by 3-4x compared to FP16.

Which quantization should I use?

  • Just want it to work? Use --isq 4. hanzo picks the best format for your hardware.
  • Have plenty of VRAM? Use --isq 8 for near-lossless quality.
  • Want the fastest loading? Use a pre-quantized UQFF or GGUF model. No quantization happens at load time.
  • Need per-layer control? Use a topology file to set different quantization per layer.
  • On Apple Silicon? --isq 4 uses AFQ4, which is optimized for Metal.

See the Quantization Overview for all options.

What's the difference between ISQ, GGUF, and UQFF?

  • ISQ quantizes at load time. Any HF model works, but loading is slower because quantization happens on the fly.
  • GGUF is a pre-quantized format from the llama.cpp ecosystem. Download a GGUF file and load it directly.
  • UQFF is hanzo's own pre-quantized format. Supports more quantization types than GGUF (HQQ, FP8, AFQ, etc.) and loads instantly.

Can I use models from local files?

Yes. Pass a local path instead of a Hugging Face model ID:

hanzo run -m /path/to/model

How do I use gated models (Llama, etc.)?

  1. Accept the model license on Hugging Face
  2. Authenticate:
    hanzo login
    
  3. Run normally:
    hanzo run -m meta-llama/Llama-3.2-3B-Instruct
    

Alternatively, use --token-source env:HF_TOKEN with a Hugging Face token in your environment.

How do I update hanzo?

Re-run the install script to build the latest version:

curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/hanzoai/engine/master/install.sh | sh

Or if installed via cargo:

cargo install hanzo-cli --features "<your-features>" --force

Does hanzo work on WSL?

Yes. WSL2 with CUDA support works well. Install the NVIDIA CUDA toolkit for WSL and build with --features cuda.

Is Windows supported?

Yes, but WSL2 is recommended for the best experience. Native Windows builds work but PagedAttention is only available on Unix-like platforms.

Performance

Why is the first request slow?

hanzo performs a warmup run when loading a model. It sends a short dummy request to initialize CUDA kernels and caches. You'll see "Beginning dummy run..." and "Dummy run completed" in the logs. After this, subsequent requests are much faster.

How do I increase throughput?

  1. Enable PagedAttention (on by default for CUDA)
  2. Increase --max-seqs for more concurrent batching (default: 32)
  3. Use FlashAttention (compile with --features flash-attn)
  4. Use quantization to free VRAM for larger batches

See the Performance Guide for detailed optimization recipes.

PagedAttention is using too much VRAM

By default, PagedAttention allocates 90% of free VRAM for the KV cache. Reduce it:

# Allocate for a specific context length
hanzo serve -m <model> --pa-context-len 4096

# Or set a memory cap
hanzo serve -m <model> --pa-memory-mb 2048

# Or use a fraction
hanzo serve -m <model> --pa-memory-fraction 0.5

How do I reduce Time to First Token (TTFT)?

  • Compile with FlashAttention (--features flash-attn) to accelerate prefill
  • Enable prefix caching (--prefix-cache-n 16, on by default) to reuse KV cache across turns
  • Use quantization to reduce memory bandwidth pressure

Compatibility

Can I use OpenAI client libraries?

Yes. hanzo provides an OpenAI-compatible HTTP API. Use the standard openai Python package:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:1234/v1/", api_key="foobar")

The api_key can be any non-empty string. hanzo doesn't validate it.

Can I use LangChain or LlamaIndex?

Yes. Point them at your hanzo server as an OpenAI-compatible endpoint:

from langchain_openai import ChatOpenAI
llm = ChatOpenAI(base_url="http://localhost:1234/v1/", api_key="foobar")

Which models are supported?

See Supported Models for the complete list. Architecture is auto-detected, so just point hanzo at the model and it works.

Can I run multiple models at once?

Yes. Use a TOML config file with multiple [[models]] entries:

hanzo from-config --file config.toml

See Multi-Model Support for details.