Files
zeekayandHanzo Dev 02b3f00cef chore: complete brand scrub of upstream mistral.rs lineage to hanzo/hanzoai
Replace all remaining references to the upstream project/author/framework
brands with hanzo / hanzoai across source, comments, docs, examples, and
build scripts.

- Project spellings mistral.rs / mistral-rs / mistralrs -> hanzo (and the
  hanzo-* crate/binary names in Dockerfiles, Makefile, install.ps1, .gitignore).
- Author EricLBuehler / EricBuehler / EricB / Eric Buehler -> hanzoai / Hanzo AI
  (incl. Cargo `authors` and LICENSE copyright).
- URLs: github.com/EricLBuehler/mistral.rs -> github.com/hanzoai/engine;
  ericlbuehler.github.io/mistral.rs -> hanzoai.github.io/engine;
  github.com/huggingface/candle -> github.com/hanzoai/ml;
  docs.rs/candle-* -> docs.rs/hanzo-ml; ghcr.io image -> ghcr.io/hanzoai/engine.
- Env var prefix MISTRALRS_ -> HANZO_ (producers and consumers kept in sync,
  incl. the build-script-injected HANZO_GIT_REVISION).
- ROCm static lib libmistralrsrocm.a / mistralrsrocm -> libhanzorocm.a / hanzorocm.
- localStorage keys mistralrs_settings -> hanzo_settings.

Deliberately preserved: real model architectures (struct Mistral, models/mistral.rs
file, Mixtral/Llama/Gemma/Qwen/Phi), real model-org HF repo-ids (mistralai/,
meta-llama/, google/, Qwen/, microsoft/), the `use hanzo_ml as candle` code
aliases and `candle::` identifiers, and the upstream-author HF data repo-id
suffix in "hanzoai/mistralrs_tests".

Both `cargo check` and `cargo check --features vulkan` pass with 0 errors.

Co-authored-by: Hanzo Dev <[email protected]>
2026-06-01 11:53:12 -07:00

25 KiB

Examples

A comprehensive index of all examples in the repository, organized by SDK and category.

Python SDK

Examples using the hanzo Python package directly.

Getting Started

Example Description
plain.py Basic text generation with a plain (unquantized) model
streaming.py Streaming chat completions token-by-token with a GGUF model
gguf.py Load and run a GGUF-quantized model
embedding_gemma.py Generate text embeddings with EmbeddingGemma
qwen3_embedding.py Generate text embeddings with Qwen3-Embedding
token_source.py Configure a custom HF token source (literal, env, path, cache)

Models

Text Models

Example Description
deepseekr1.py Run DeepSeek-R1 reasoning model
deepseekv2.py Run DeepSeek-V2-Lite
qwen3.py Run Qwen3 with thinking mode toggling via /think and /no_think
qwen3_5.py Run Qwen3.5 dense or MoE variant with vision input
qwen3_next.py Run Qwen3-Coder-Next
smollm3.py Run SmolLM3 text model
granite.py Run IBM Granite 4.0 MoE Hybrid model
gpt_oss.py Run GPT-OSS MoE model with MXFP4 quantized experts
glm4_moe_lite.py Run GLM-4.7-Flash MoE model

Vision Models

Example Description
gemma3.py Run Gemma 3 multimodal model with image input
gemma3n.py Run Gemma 3n multimodal model with image input
gemma4.py Run Gemma 4 multimodal model with image input
qwen2vl.py Run Qwen2-VL vision-language model
qwen3_vl.py Run Qwen3-VL vision-language model with thinking
llama4.py Run Llama 4 Scout multimodal model with ISQ
llama_vision.py Run Llama 3.2 Vision model
mistral3.py Run Mistral Small 3.1 vision model with ISQ
phi3v.py Run Phi-3.5 Vision with a remote image URL
phi3v_base64.py Run Phi-3.5 Vision with a base64-encoded image
phi3v_local_img.py Run Phi-3.5 Vision with a local image file path
phi4mm.py Run Phi-4 Multimodal with image input
phi4mm_audio.py Run Phi-4 Multimodal with combined audio and image input
idefics2.py Run Idefics 2 vision model
idefics3.py Run Idefics 3 vision model
llava_next.py Run LLaVA-NeXT vision model
smolvlm.py Run SmolVLM vision model
minicpmo_2_6.py Run MiniCPM-o 2.6 multimodal model

Image Generation and Speech

Example Description
flux.py Generate images with Flux diffusion model
dia.py Text-to-speech synthesis with Dia and WAV output

Tools and Agents

Example Description
tool_call.py Client-side tool calling with manual tool execution loop
agentic_tools.py Agentic tool callbacks with automatic multi-round execution
custom_tool_call.py Custom tool callback for local filesystem search
web_search.py Web-search-augmented generation with WebSearchOptions
custom_search.py Custom search callback replacing the default web search
mcp_client.py MCP client connecting to an external tool server

Constrained Generation

Example Description
json_schema.py JSON schema-constrained generation
pydantic_schema.py Constrained generation from a Pydantic model schema
regex.py Regex-constrained generation
lark.py Lark grammar-constrained generation (JSON via grammar)
lark_llg.py Lark grammar with inline %json schema directive
llguidance.py llguidance grammar with named JSON schema references

Quantization

Example Description
isq.py In-situ quantization (ISQ) at load time
imatrix.py ISQ with importance-matrix calibration data
topology.py Per-layer quantization control via a topology YAML file
mixture_of_quant_experts.py MoQE: quantize only MoE expert layers differently

Adapters (LoRA / X-LoRA)

Example Description
lora_zephyr.py Load a LoRA adapter on a GGUF model and select adapters per request
lora_activation.py Programmatically activate/deactivate LoRA adapters
xlora_zephyr.py Run X-LoRA with GGUF on Zephyr
xlora_gemma.py Run X-LoRA on Gemma

AnyMoE

Example Description
anymoe.py Train AnyMoE gating layer from fine-tuned expert models
anymoe_inference.py Run AnyMoE inference with a pretrained gating model
anymoe_lora.py AnyMoE with LoRA adapters as experts

Advanced

Example Description
paged_attention.py Enable PagedAttention for efficient KV-cache memory
speculative.py Speculative decoding with a smaller GGUF draft model
speculative_xlora.py Speculative decoding combined with X-LoRA
text_auto_device_map.py Automatic multi-GPU device mapping for text models
multimodal_auto_device_map.py Automatic multi-GPU device mapping for multimodal models
multi_model_example.py Load, manage, and dispatch requests across multiple models
test_multi_model.py Multi-model management test harness

HTTP Server

Examples using the OpenAI-compatible HTTP API (start the server with hanzo serve).

Getting Started

Example Description
chat.py Interactive multi-turn chat with the OpenAI client
completion.py Text completion (non-chat) endpoint
streaming.py Streaming chat completions token-by-token
streaming_completion.py Streaming text completion endpoint
embedding.py Generate text embeddings via the embeddings endpoint

Models

Vision Models

Example Description
gemma3.py Gemma 3 multimodal chat with image input
gemma3n.py Gemma 3n multimodal chat with image input
gemma4.py Gemma 4 multimodal chat with image input
gemma4_video.py Gemma 4 with video input
qwen2vl.py Qwen2-VL vision-language chat
qwen3_vl.py Qwen3-VL vision-language chat
llama4.py Llama 4 Scout multimodal chat
llama_vision.py Llama 3.2 Vision chat
mistral3.py Mistral Small 3.1 vision chat
phi3v.py Phi-3.5 Vision with remote image URL
phi3v_base64.py Phi-3.5 Vision with base64-encoded image
phi3v_local_img.py Phi-3.5 Vision with local image file
phi4mm.py Phi-4 Multimodal with image input
phi4mm_audio.py Phi-4 Multimodal with audio and image input
idefics2.py Idefics 2 vision chat
idefics3.py Idefics 3 vision chat
llava.py LLaVA 1.5 vision chat
llava_next.py LLaVA-NeXT vision chat
minicpmo_2_6.py MiniCPM-o 2.6 multimodal chat
smollm3.py SmolLM3 chat

Text Models

Example Description
qwen3.py Qwen3 chat
qwen3_5.py Qwen3.5 chat
qwen3_next.py Qwen3-Coder-Next chat
gpt_oss.py GPT-OSS MoE chat

Image Generation and Speech

Example Description
flux.py Generate images via the images endpoint
dia.py Text-to-speech via the audio speech endpoint

OpenAI Responses API

Example Description
responses.py Multi-turn conversation using the Responses API with reasoning
responses_vision.py Responses API with vision (image) input
responses_audio.py Responses API with audio input and transcription

Tools and Agents

Example Description
tool_calling.py Client-side tool calling with the OpenAI client
streaming_tool_calling.py Streaming tool calls with multi-round execution loop
tool_dispatch.py Server-side tool dispatch via an external HTTP endpoint
agentic_tool_rounds.py Server-side agentic loop with max_tool_rounds
web_search.py Web-search-augmented generation via web_search_options
mcp_chat.py Chat with MCP tool server integration

Constrained Generation

Example Description
json_schema.py JSON schema-constrained output via response_format
openai_response_format.py Structured output with Pydantic via beta.chat.completions.parse
regex.py Regex-constrained generation and token healing
lark.py Lark grammar-constrained generation (C code grammar)
llguidance.py llguidance grammar with named JSON schema references

Advanced

Example Description
adapter_chat.py Chat with per-request LoRA adapter selection
multi_model_chat.py Multi-model server: list models, route requests, compare outputs
stream_completion_bench.py Benchmark streaming vs non-streaming latency

Rust SDK

Examples using the hanzo Rust crate (in hanzo/examples/).

Getting Started

Example Description
text_generation Basic text generation with ISQ quantization and chat
streaming Streaming text generation with token-by-token output
gguf Load and run a GGUF-quantized model from Hugging Face
gguf_locally Load and run a GGUF model from a local file path
multimodal Simple multimodal "hello world" with image input
embedding Generate text embeddings using an embedding model

Models

Example Description
text_models Unified text model example with all supported model IDs
multimodal_models Unified multimodal model example with all supported model IDs
multimodal Multimodal streaming with combined image and audio inputs
multimodal_multiturn Multi-turn conversation with a multimodal model
audio Audio input processing with a multimodal model
asr Automatic speech recognition (ASR) with Voxtral
speech Text-to-speech synthesis using a speech model
diffusion Image generation using a diffusion model

Quantization

Example Description
isq In-situ quantization with explicit and automatic type selection
imatrix ISQ with importance-matrix calibration data
topology Per-layer quantization control using a Topology
uqff Load a pre-quantized UQFF text model
uqff_multimodal Load a pre-quantized UQFF multimodal model
mixture_of_quant_experts MoQE: quantize only MoE expert layers at a different precision

Tools and Agents

Example Description
agent Agentic loop with #[tool] macro and AgentBuilder (non-streaming)
agent_streaming Agentic loop with streaming output and real-time events
tools Tool calling (function calling) with manual tool definitions
tool_callback Tool callbacks for automatic server-side tool execution
web_search Web-search-augmented generation
search_callback Custom search callback to override default web search
mcp_client MCP client connecting to an external tool server

Constrained Generation

Example Description
json_schema JSON schema-constrained generation for typed output
grammar Constrained generation using a GBNF grammar
llguidance Constrained generation using an llguidance grammar

Adapters and AnyMoE

Example Description
lora Load and run a model with a LoRA adapter
xlora X-LoRA adapter mixing
anymoe AnyMoE: create a MoE model from fine-tuned adapters
anymoe_lora AnyMoE with LoRA adapters for expert specialization

Advanced

Example Description
paged_attn Enable PagedAttention for efficient KV-cache memory management
speculative Speculative decoding with a smaller draft model
auto_device_map Automatic device mapping across multiple GPUs
multi_model Load and dispatch requests across multiple models
batching Concurrent request batching with parallel requests
embeddings Compute and compare embeddings with cosine similarity
batching_embeddings Batch multiple embedding requests for parallel encoding
logits_processor Custom logits processor to modify token probabilities
perplexity Compute perplexity of a text file
error_handling Error handling patterns with error variant matching
file_logging Log model output to a file using the tracing framework

Cookbook

Example Description
agent Code review agent using #[tool] macro and AgentBuilder
multiturn Interactive multi-turn chatbot with streaming output
rag Simple RAG with embedding retrieval and text generation
structured Structured data extraction with generate_structured<T>()

Jupyter Notebooks

Notebook Description
cookbook.ipynb Python API cookbook: loading GGUF, Plain, and X-LoRA models
tool_calling.ipynb Interactive tool calling walkthrough with Python SDK
phi3_duckduckgo_hanzo.ipynb Phi-3 tool calling with DuckDuckGo search via LangChain