Files
engine/docs/GETTING_STARTED.md
zeekayandHanzo Dev 02b3f00cef chore: complete brand scrub of upstream mistral.rs lineage to hanzo/hanzoai
Replace all remaining references to the upstream project/author/framework
brands with hanzo / hanzoai across source, comments, docs, examples, and
build scripts.

- Project spellings mistral.rs / mistral-rs / mistralrs -> hanzo (and the
  hanzo-* crate/binary names in Dockerfiles, Makefile, install.ps1, .gitignore).
- Author EricLBuehler / EricBuehler / EricB / Eric Buehler -> hanzoai / Hanzo AI
  (incl. Cargo `authors` and LICENSE copyright).
- URLs: github.com/EricLBuehler/mistral.rs -> github.com/hanzoai/engine;
  ericlbuehler.github.io/mistral.rs -> hanzoai.github.io/engine;
  github.com/huggingface/candle -> github.com/hanzoai/ml;
  docs.rs/candle-* -> docs.rs/hanzo-ml; ghcr.io image -> ghcr.io/hanzoai/engine.
- Env var prefix MISTRALRS_ -> HANZO_ (producers and consumers kept in sync,
  incl. the build-script-injected HANZO_GIT_REVISION).
- ROCm static lib libmistralrsrocm.a / mistralrsrocm -> libhanzorocm.a / hanzorocm.
- localStorage keys mistralrs_settings -> hanzo_settings.

Deliberately preserved: real model architectures (struct Mistral, models/mistral.rs
file, Mixtral/Llama/Gemma/Qwen/Phi), real model-org HF repo-ids (mistralai/,
meta-llama/, google/, Qwen/, microsoft/), the `use hanzo_ml as candle` code
aliases and `candle::` identifiers, and the upstream-author HF data repo-id
suffix in "hanzoai/mistralrs_tests".

Both `cargo check` and `cargo check --features vulkan` pass with 0 errors.

Co-authored-by: Hanzo Dev <[email protected]>
2026-06-01 11:53:12 -07:00

4.8 KiB

Getting Started with hanzo

hanzo is a fast, flexible LLM inference engine. It supports 40+ model families (text, vision, audio, speech, image generation, and embeddings) with an OpenAI-compatible API, Python SDK, and Rust SDK.

This tutorial takes you from zero to running models in about 5 minutes.

Step 1: Install

Linux/macOS:

curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/hanzoai/engine/master/install.sh | sh

Windows (PowerShell):

irm https://raw.githubusercontent.com/hanzoai/engine/master/install.ps1 | iex

The install script detects your hardware (CUDA, Metal, MKL) and builds with the right features automatically.

For manual installation, Docker, or building from source, see the full Installation guide.

Step 2: Run Your First Model

hanzo run -m Qwen/Qwen3-4B

That's it. hanzo auto-detects the model architecture, downloads weights from Hugging Face, and drops you into an interactive chat. Type a message and press Enter:

> What is Rust's ownership model?
Rust uses an ownership system where each value has a single owner...
> /exit

Use /exit to quit, /clear to reset the conversation, or /help for all commands.

Step 3: Reduce Memory with Quantization

Most models load in BF16 by default. If you're short on VRAM, quantize at load time with --isq:

hanzo run --isq 4 -m Qwen/Qwen3-4B

The --isq 4 flag quantizes weights to 4-bit as they load, so the full model never needs to fit in memory. hanzo picks the best quantization format for your hardware automatically (AFQ on Metal, Q4K on CUDA/CPU).

Not sure what settings to use? Run hanzo tune -m <model> and it will analyze your hardware and recommend the best configuration.

Step 4: Start an HTTP Server

Serve any model as an OpenAI-compatible API:

hanzo serve --isq 4 -m Qwen/Qwen3-4B

The server starts on http://localhost:1234. Test it with curl:

curl http://localhost:1234/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "default",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Or use the OpenAI Python client directly, no changes needed:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:1234/v1/", api_key="foobar")
response = client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

Add --ui to get a built-in web chat interface:

hanzo serve --ui --isq 4 -m Qwen/Qwen3-4B
# Visit http://localhost:1234/ui

Step 5: Use the Python SDK

pip install hanzo        # CPU
pip install hanzo-cuda   # NVIDIA GPU
pip install hanzo-metal  # Apple Silicon
from hanzo import Runner, Which, ChatCompletionRequest

runner = Runner(
    which=Which.Plain(model_id="Qwen/Qwen3-4B"),
    in_situ_quant="4",
)

res = runner.send_chat_completion_request(
    ChatCompletionRequest(
        model="default",
        messages=[{"role": "user", "content": "Hello!"}],
        max_tokens=256,
    )
)
print(res.choices[0].message.content)

For the Rust SDK, see the Rust SDK guide.

Step 6: Try a Multimodal Model

hanzo handles images, video, and audio out of the box:

# Describe an image
hanzo run -m google/gemma-4-E4B-it --image photo.jpg -i "Describe this image"

# Analyze a video
hanzo run -m google/gemma-4-E4B-it --video clip.mp4 -i "What happens here?"

# Process audio
hanzo run -m google/gemma-4-E4B-it --audio recording.wav -i "Transcribe this"

In interactive mode, just paste file paths directly into your prompt. hanzo detects image, video, and audio files automatically.

Step 7: Auto-Tune Your Setup

Let hanzo recommend the best configuration for your hardware:

hanzo tune -m Qwen/Qwen3-4B

This shows a table of quantization options with estimated VRAM usage, context length headroom, and quality trade-offs. Add --emit-config config.toml to save the recommended settings:

hanzo tune -m Qwen/Qwen3-4B --emit-config config.toml
hanzo from-config -f config.toml

Next Steps

I want to... Read...
Learn all CLI commands and options CLI Reference
Optimize performance and memory Performance Guide
Build agents with tools and web search Agentic Features Guide
Use the Python SDK in depth Python SDK
Use the Rust SDK in depth Rust SDK
Serve models over HTTP HTTP Server
Connect to MCP tool servers MCP Client
See all supported models Supported Models
Choose a quantization method Quantization Overview
Run multiple models at once Multi-Model Support