Replace all remaining references to the upstream project/author/framework brands with hanzo / hanzoai across source, comments, docs, examples, and build scripts. - Project spellings mistral.rs / mistral-rs / mistralrs -> hanzo (and the hanzo-* crate/binary names in Dockerfiles, Makefile, install.ps1, .gitignore). - Author EricLBuehler / EricBuehler / EricB / Eric Buehler -> hanzoai / Hanzo AI (incl. Cargo `authors` and LICENSE copyright). - URLs: github.com/EricLBuehler/mistral.rs -> github.com/hanzoai/engine; ericlbuehler.github.io/mistral.rs -> hanzoai.github.io/engine; github.com/huggingface/candle -> github.com/hanzoai/ml; docs.rs/candle-* -> docs.rs/hanzo-ml; ghcr.io image -> ghcr.io/hanzoai/engine. - Env var prefix MISTRALRS_ -> HANZO_ (producers and consumers kept in sync, incl. the build-script-injected HANZO_GIT_REVISION). - ROCm static lib libmistralrsrocm.a / mistralrsrocm -> libhanzorocm.a / hanzorocm. - localStorage keys mistralrs_settings -> hanzo_settings. Deliberately preserved: real model architectures (struct Mistral, models/mistral.rs file, Mixtral/Llama/Gemma/Qwen/Phi), real model-org HF repo-ids (mistralai/, meta-llama/, google/, Qwen/, microsoft/), the `use hanzo_ml as candle` code aliases and `candle::` identifiers, and the upstream-author HF data repo-id suffix in "hanzoai/mistralrs_tests". Both `cargo check` and `cargo check --features vulkan` pass with 0 errors. Co-authored-by: Hanzo Dev <[email protected]>
4.8 KiB
Getting Started with hanzo
hanzo is a fast, flexible LLM inference engine. It supports 40+ model families (text, vision, audio, speech, image generation, and embeddings) with an OpenAI-compatible API, Python SDK, and Rust SDK.
This tutorial takes you from zero to running models in about 5 minutes.
Step 1: Install
Linux/macOS:
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/hanzoai/engine/master/install.sh | sh
Windows (PowerShell):
irm https://raw.githubusercontent.com/hanzoai/engine/master/install.ps1 | iex
The install script detects your hardware (CUDA, Metal, MKL) and builds with the right features automatically.
For manual installation, Docker, or building from source, see the full Installation guide.
Step 2: Run Your First Model
hanzo run -m Qwen/Qwen3-4B
That's it. hanzo auto-detects the model architecture, downloads weights from Hugging Face, and drops you into an interactive chat. Type a message and press Enter:
> What is Rust's ownership model?
Rust uses an ownership system where each value has a single owner...
> /exit
Use /exit to quit, /clear to reset the conversation, or /help for all commands.
Step 3: Reduce Memory with Quantization
Most models load in BF16 by default. If you're short on VRAM, quantize at load time with --isq:
hanzo run --isq 4 -m Qwen/Qwen3-4B
The --isq 4 flag quantizes weights to 4-bit as they load, so the full model never needs to fit in memory. hanzo picks the best quantization format for your hardware automatically (AFQ on Metal, Q4K on CUDA/CPU).
Not sure what settings to use? Run
hanzo tune -m <model>and it will analyze your hardware and recommend the best configuration.
Step 4: Start an HTTP Server
Serve any model as an OpenAI-compatible API:
hanzo serve --isq 4 -m Qwen/Qwen3-4B
The server starts on http://localhost:1234. Test it with curl:
curl http://localhost:1234/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "default",
"messages": [{"role": "user", "content": "Hello!"}]
}'
Or use the OpenAI Python client directly, no changes needed:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:1234/v1/", api_key="foobar")
response = client.chat.completions.create(
model="default",
messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)
Add --ui to get a built-in web chat interface:
hanzo serve --ui --isq 4 -m Qwen/Qwen3-4B
# Visit http://localhost:1234/ui
Step 5: Use the Python SDK
pip install hanzo # CPU
pip install hanzo-cuda # NVIDIA GPU
pip install hanzo-metal # Apple Silicon
from hanzo import Runner, Which, ChatCompletionRequest
runner = Runner(
which=Which.Plain(model_id="Qwen/Qwen3-4B"),
in_situ_quant="4",
)
res = runner.send_chat_completion_request(
ChatCompletionRequest(
model="default",
messages=[{"role": "user", "content": "Hello!"}],
max_tokens=256,
)
)
print(res.choices[0].message.content)
For the Rust SDK, see the Rust SDK guide.
Step 6: Try a Multimodal Model
hanzo handles images, video, and audio out of the box:
# Describe an image
hanzo run -m google/gemma-4-E4B-it --image photo.jpg -i "Describe this image"
# Analyze a video
hanzo run -m google/gemma-4-E4B-it --video clip.mp4 -i "What happens here?"
# Process audio
hanzo run -m google/gemma-4-E4B-it --audio recording.wav -i "Transcribe this"
In interactive mode, just paste file paths directly into your prompt. hanzo detects image, video, and audio files automatically.
Step 7: Auto-Tune Your Setup
Let hanzo recommend the best configuration for your hardware:
hanzo tune -m Qwen/Qwen3-4B
This shows a table of quantization options with estimated VRAM usage, context length headroom, and quality trade-offs. Add --emit-config config.toml to save the recommended settings:
hanzo tune -m Qwen/Qwen3-4B --emit-config config.toml
hanzo from-config -f config.toml
Next Steps
| I want to... | Read... |
|---|---|
| Learn all CLI commands and options | CLI Reference |
| Optimize performance and memory | Performance Guide |
| Build agents with tools and web search | Agentic Features Guide |
| Use the Python SDK in depth | Python SDK |
| Use the Rust SDK in depth | Rust SDK |
| Serve models over HTTP | HTTP Server |
| Connect to MCP tool servers | MCP Client |
| See all supported models | Supported Models |
| Choose a quantization method | Quantization Overview |
| Run multiple models at once | Multi-Model Support |