mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
Replace all remaining references to the upstream project/author/framework brands with hanzo / hanzoai across source, comments, docs, examples, and build scripts. - Project spellings mistral.rs / mistral-rs / mistralrs -> hanzo (and the hanzo-* crate/binary names in Dockerfiles, Makefile, install.ps1, .gitignore). - Author EricLBuehler / EricBuehler / EricB / Eric Buehler -> hanzoai / Hanzo AI (incl. Cargo `authors` and LICENSE copyright). - URLs: github.com/EricLBuehler/mistral.rs -> github.com/hanzoai/engine; ericlbuehler.github.io/mistral.rs -> hanzoai.github.io/engine; github.com/huggingface/candle -> github.com/hanzoai/ml; docs.rs/candle-* -> docs.rs/hanzo-ml; ghcr.io image -> ghcr.io/hanzoai/engine. - Env var prefix MISTRALRS_ -> HANZO_ (producers and consumers kept in sync, incl. the build-script-injected HANZO_GIT_REVISION). - ROCm static lib libmistralrsrocm.a / mistralrsrocm -> libhanzorocm.a / hanzorocm. - localStorage keys mistralrs_settings -> hanzo_settings. Deliberately preserved: real model architectures (struct Mistral, models/mistral.rs file, Mixtral/Llama/Gemma/Qwen/Phi), real model-org HF repo-ids (mistralai/, meta-llama/, google/, Qwen/, microsoft/), the `use hanzo_ml as candle` code aliases and `candle::` identifiers, and the upstream-author HF data repo-id suffix in "hanzoai/mistralrs_tests". Both `cargo check` and `cargo check --features vulkan` pass with 0 errors. Co-authored-by: Hanzo Dev <[email protected]>
25 KiB
25 KiB
Examples
A comprehensive index of all examples in the repository, organized by SDK and category.
Python SDK
Examples using the hanzo Python package directly.
Getting Started
| Example | Description |
|---|---|
| plain.py | Basic text generation with a plain (unquantized) model |
| streaming.py | Streaming chat completions token-by-token with a GGUF model |
| gguf.py | Load and run a GGUF-quantized model |
| embedding_gemma.py | Generate text embeddings with EmbeddingGemma |
| qwen3_embedding.py | Generate text embeddings with Qwen3-Embedding |
| token_source.py | Configure a custom HF token source (literal, env, path, cache) |
Models
Text Models
| Example | Description |
|---|---|
| deepseekr1.py | Run DeepSeek-R1 reasoning model |
| deepseekv2.py | Run DeepSeek-V2-Lite |
| qwen3.py | Run Qwen3 with thinking mode toggling via /think and /no_think |
| qwen3_5.py | Run Qwen3.5 dense or MoE variant with vision input |
| qwen3_next.py | Run Qwen3-Coder-Next |
| smollm3.py | Run SmolLM3 text model |
| granite.py | Run IBM Granite 4.0 MoE Hybrid model |
| gpt_oss.py | Run GPT-OSS MoE model with MXFP4 quantized experts |
| glm4_moe_lite.py | Run GLM-4.7-Flash MoE model |
Vision Models
| Example | Description |
|---|---|
| gemma3.py | Run Gemma 3 multimodal model with image input |
| gemma3n.py | Run Gemma 3n multimodal model with image input |
| gemma4.py | Run Gemma 4 multimodal model with image input |
| qwen2vl.py | Run Qwen2-VL vision-language model |
| qwen3_vl.py | Run Qwen3-VL vision-language model with thinking |
| llama4.py | Run Llama 4 Scout multimodal model with ISQ |
| llama_vision.py | Run Llama 3.2 Vision model |
| mistral3.py | Run Mistral Small 3.1 vision model with ISQ |
| phi3v.py | Run Phi-3.5 Vision with a remote image URL |
| phi3v_base64.py | Run Phi-3.5 Vision with a base64-encoded image |
| phi3v_local_img.py | Run Phi-3.5 Vision with a local image file path |
| phi4mm.py | Run Phi-4 Multimodal with image input |
| phi4mm_audio.py | Run Phi-4 Multimodal with combined audio and image input |
| idefics2.py | Run Idefics 2 vision model |
| idefics3.py | Run Idefics 3 vision model |
| llava_next.py | Run LLaVA-NeXT vision model |
| smolvlm.py | Run SmolVLM vision model |
| minicpmo_2_6.py | Run MiniCPM-o 2.6 multimodal model |
Image Generation and Speech
| Example | Description |
|---|---|
| flux.py | Generate images with Flux diffusion model |
| dia.py | Text-to-speech synthesis with Dia and WAV output |
Tools and Agents
| Example | Description |
|---|---|
| tool_call.py | Client-side tool calling with manual tool execution loop |
| agentic_tools.py | Agentic tool callbacks with automatic multi-round execution |
| custom_tool_call.py | Custom tool callback for local filesystem search |
| web_search.py | Web-search-augmented generation with WebSearchOptions |
| custom_search.py | Custom search callback replacing the default web search |
| mcp_client.py | MCP client connecting to an external tool server |
Constrained Generation
| Example | Description |
|---|---|
| json_schema.py | JSON schema-constrained generation |
| pydantic_schema.py | Constrained generation from a Pydantic model schema |
| regex.py | Regex-constrained generation |
| lark.py | Lark grammar-constrained generation (JSON via grammar) |
| lark_llg.py | Lark grammar with inline %json schema directive |
| llguidance.py | llguidance grammar with named JSON schema references |
Quantization
| Example | Description |
|---|---|
| isq.py | In-situ quantization (ISQ) at load time |
| imatrix.py | ISQ with importance-matrix calibration data |
| topology.py | Per-layer quantization control via a topology YAML file |
| mixture_of_quant_experts.py | MoQE: quantize only MoE expert layers differently |
Adapters (LoRA / X-LoRA)
| Example | Description |
|---|---|
| lora_zephyr.py | Load a LoRA adapter on a GGUF model and select adapters per request |
| lora_activation.py | Programmatically activate/deactivate LoRA adapters |
| xlora_zephyr.py | Run X-LoRA with GGUF on Zephyr |
| xlora_gemma.py | Run X-LoRA on Gemma |
AnyMoE
| Example | Description |
|---|---|
| anymoe.py | Train AnyMoE gating layer from fine-tuned expert models |
| anymoe_inference.py | Run AnyMoE inference with a pretrained gating model |
| anymoe_lora.py | AnyMoE with LoRA adapters as experts |
Advanced
| Example | Description |
|---|---|
| paged_attention.py | Enable PagedAttention for efficient KV-cache memory |
| speculative.py | Speculative decoding with a smaller GGUF draft model |
| speculative_xlora.py | Speculative decoding combined with X-LoRA |
| text_auto_device_map.py | Automatic multi-GPU device mapping for text models |
| multimodal_auto_device_map.py | Automatic multi-GPU device mapping for multimodal models |
| multi_model_example.py | Load, manage, and dispatch requests across multiple models |
| test_multi_model.py | Multi-model management test harness |
HTTP Server
Examples using the OpenAI-compatible HTTP API (start the server with hanzo serve).
Getting Started
| Example | Description |
|---|---|
| chat.py | Interactive multi-turn chat with the OpenAI client |
| completion.py | Text completion (non-chat) endpoint |
| streaming.py | Streaming chat completions token-by-token |
| streaming_completion.py | Streaming text completion endpoint |
| embedding.py | Generate text embeddings via the embeddings endpoint |
Models
Vision Models
| Example | Description |
|---|---|
| gemma3.py | Gemma 3 multimodal chat with image input |
| gemma3n.py | Gemma 3n multimodal chat with image input |
| gemma4.py | Gemma 4 multimodal chat with image input |
| gemma4_video.py | Gemma 4 with video input |
| qwen2vl.py | Qwen2-VL vision-language chat |
| qwen3_vl.py | Qwen3-VL vision-language chat |
| llama4.py | Llama 4 Scout multimodal chat |
| llama_vision.py | Llama 3.2 Vision chat |
| mistral3.py | Mistral Small 3.1 vision chat |
| phi3v.py | Phi-3.5 Vision with remote image URL |
| phi3v_base64.py | Phi-3.5 Vision with base64-encoded image |
| phi3v_local_img.py | Phi-3.5 Vision with local image file |
| phi4mm.py | Phi-4 Multimodal with image input |
| phi4mm_audio.py | Phi-4 Multimodal with audio and image input |
| idefics2.py | Idefics 2 vision chat |
| idefics3.py | Idefics 3 vision chat |
| llava.py | LLaVA 1.5 vision chat |
| llava_next.py | LLaVA-NeXT vision chat |
| minicpmo_2_6.py | MiniCPM-o 2.6 multimodal chat |
| smollm3.py | SmolLM3 chat |
Text Models
| Example | Description |
|---|---|
| qwen3.py | Qwen3 chat |
| qwen3_5.py | Qwen3.5 chat |
| qwen3_next.py | Qwen3-Coder-Next chat |
| gpt_oss.py | GPT-OSS MoE chat |
Image Generation and Speech
| Example | Description |
|---|---|
| flux.py | Generate images via the images endpoint |
| dia.py | Text-to-speech via the audio speech endpoint |
OpenAI Responses API
| Example | Description |
|---|---|
| responses.py | Multi-turn conversation using the Responses API with reasoning |
| responses_vision.py | Responses API with vision (image) input |
| responses_audio.py | Responses API with audio input and transcription |
Tools and Agents
| Example | Description |
|---|---|
| tool_calling.py | Client-side tool calling with the OpenAI client |
| streaming_tool_calling.py | Streaming tool calls with multi-round execution loop |
| tool_dispatch.py | Server-side tool dispatch via an external HTTP endpoint |
| agentic_tool_rounds.py | Server-side agentic loop with max_tool_rounds |
| web_search.py | Web-search-augmented generation via web_search_options |
| mcp_chat.py | Chat with MCP tool server integration |
Constrained Generation
| Example | Description |
|---|---|
| json_schema.py | JSON schema-constrained output via response_format |
| openai_response_format.py | Structured output with Pydantic via beta.chat.completions.parse |
| regex.py | Regex-constrained generation and token healing |
| lark.py | Lark grammar-constrained generation (C code grammar) |
| llguidance.py | llguidance grammar with named JSON schema references |
Advanced
| Example | Description |
|---|---|
| adapter_chat.py | Chat with per-request LoRA adapter selection |
| multi_model_chat.py | Multi-model server: list models, route requests, compare outputs |
| stream_completion_bench.py | Benchmark streaming vs non-streaming latency |
Rust SDK
Examples using the hanzo Rust crate (in hanzo/examples/).
Getting Started
| Example | Description |
|---|---|
| text_generation | Basic text generation with ISQ quantization and chat |
| streaming | Streaming text generation with token-by-token output |
| gguf | Load and run a GGUF-quantized model from Hugging Face |
| gguf_locally | Load and run a GGUF model from a local file path |
| multimodal | Simple multimodal "hello world" with image input |
| embedding | Generate text embeddings using an embedding model |
Models
| Example | Description |
|---|---|
| text_models | Unified text model example with all supported model IDs |
| multimodal_models | Unified multimodal model example with all supported model IDs |
| multimodal | Multimodal streaming with combined image and audio inputs |
| multimodal_multiturn | Multi-turn conversation with a multimodal model |
| audio | Audio input processing with a multimodal model |
| asr | Automatic speech recognition (ASR) with Voxtral |
| speech | Text-to-speech synthesis using a speech model |
| diffusion | Image generation using a diffusion model |
Quantization
| Example | Description |
|---|---|
| isq | In-situ quantization with explicit and automatic type selection |
| imatrix | ISQ with importance-matrix calibration data |
| topology | Per-layer quantization control using a Topology |
| uqff | Load a pre-quantized UQFF text model |
| uqff_multimodal | Load a pre-quantized UQFF multimodal model |
| mixture_of_quant_experts | MoQE: quantize only MoE expert layers at a different precision |
Tools and Agents
| Example | Description |
|---|---|
| agent | Agentic loop with #[tool] macro and AgentBuilder (non-streaming) |
| agent_streaming | Agentic loop with streaming output and real-time events |
| tools | Tool calling (function calling) with manual tool definitions |
| tool_callback | Tool callbacks for automatic server-side tool execution |
| web_search | Web-search-augmented generation |
| search_callback | Custom search callback to override default web search |
| mcp_client | MCP client connecting to an external tool server |
Constrained Generation
| Example | Description |
|---|---|
| json_schema | JSON schema-constrained generation for typed output |
| grammar | Constrained generation using a GBNF grammar |
| llguidance | Constrained generation using an llguidance grammar |
Adapters and AnyMoE
| Example | Description |
|---|---|
| lora | Load and run a model with a LoRA adapter |
| xlora | X-LoRA adapter mixing |
| anymoe | AnyMoE: create a MoE model from fine-tuned adapters |
| anymoe_lora | AnyMoE with LoRA adapters for expert specialization |
Advanced
| Example | Description |
|---|---|
| paged_attn | Enable PagedAttention for efficient KV-cache memory management |
| speculative | Speculative decoding with a smaller draft model |
| auto_device_map | Automatic device mapping across multiple GPUs |
| multi_model | Load and dispatch requests across multiple models |
| batching | Concurrent request batching with parallel requests |
| embeddings | Compute and compare embeddings with cosine similarity |
| batching_embeddings | Batch multiple embedding requests for parallel encoding |
| logits_processor | Custom logits processor to modify token probabilities |
| perplexity | Compute perplexity of a text file |
| error_handling | Error handling patterns with error variant matching |
| file_logging | Log model output to a file using the tracing framework |
Cookbook
| Example | Description |
|---|---|
| agent | Code review agent using #[tool] macro and AgentBuilder |
| multiturn | Interactive multi-turn chatbot with streaming output |
| rag | Simple RAG with embedding retrieval and text generation |
| structured | Structured data extraction with generate_structured<T>() |
Jupyter Notebooks
| Notebook | Description |
|---|---|
| cookbook.ipynb | Python API cookbook: loading GGUF, Plain, and X-LoRA models |
| tool_calling.ipynb | Interactive tool calling walkthrough with Python SDK |
| phi3_duckduckgo_hanzo.ipynb | Phi-3 tool calling with DuckDuckGo search via LangChain |