Hanzo ML - Rust ML Framework
Category: Hanzo Ecosystem
Related Skills: hanzo/hanzo-candle.md, hanzo/hanzo-engine.md, hanzo/hanzo-kensho.md
Overview
Hanzo ML is a Rust-based minimalist ML framework forked from HuggingFace Candle, optimized for edge AI, quantization, and multimodal inference. It provides the tensor computation layer for the Hanzo ecosystem, with GPU acceleration via CUDA, Metal (Apple Silicon), and CPU optimizations (MKL, Accelerate).
The repository maintains dual crate namespaces: original candle-* crates for upstream compatibility plus hanzo-* crates as the forward-looking Hanzo-branded API. Both coexist in the workspace and share the same underlying code.
Key Differentiators from Upstream Candle
- Hanzo-branded crates:
hanzo-ml, hanzo-nn, hanzo-transformers, hanzo-datasets published at v0.9.2-alpha.2
- HANZO_INTEGRATION.md: Integration guide for Hanzo Engine (mistral-rs fork)
- Hanzo-specific kernels:
hanzo-flash-attn, hanzo-metal-kernels, hanzo-kernels
- Python bindings:
hanzo-ml-pyo3 for Python interop
- WASM examples:
hanzo-ml-wasm-examples for browser inference
When to use
- High-performance ML inference in Rust applications
- CUDA/Metal GPU acceleration without Python runtime
- Loading GGUF, safetensors, ONNX models in Rust
- Browser-based ML via WebAssembly
- Integration with Hanzo Engine for model serving
- Building custom inference pipelines with zero-cost abstractions
Hard requirements
- Rust 1.75+
- CUDA Toolkit 12+ (for CUDA backend) or macOS 13+ (for Metal)
- For WASM:
wasm-pack and compatible browser
Quick reference
| Item |
Value |
| Repo |
github.com/hanzoai/ml |
| Branch |
main |
| Language |
Rust (primary), C++/CUDA/Metal/Python |
| Version |
0.9.2-alpha.2 (hanzo crates), 0.9.2 (candle crates) |
| Build |
cargo build --workspace |
| Test |
cargo test --workspace |
| License |
BSD-3-Clause OR Apache-2.0 |
Workspace Crates
Hanzo-Branded (v0.9.2-alpha.2)
| Crate |
Purpose |
hanzo-ml |
Core tensor operations, Device, DType, quantization |
hanzo-nn |
Neural network layers (Linear, Conv, LayerNorm, Attention) |
hanzo-transformers |
Transformer model implementations (90+) |
hanzo-datasets |
Dataset loading (MNIST, CIFAR, TinyStories) |
hanzo-ml-pyo3 |
Python bindings via PyO3 |
hanzo-flash-attn |
Flash Attention v2 CUDA kernels |
hanzo-metal-kernels |
Custom Metal GPU kernels |
hanzo-kernels |
Custom CUDA kernels |
hanzo-onnx |
ONNX model evaluation |
hanzo-ug |
Universal Graph backend |
hanzo-ml-examples |
Example binaries |
hanzo-ml-wasm-examples |
Browser WASM examples |
hanzo-ml-wasm-tests |
WASM test suite |
Upstream-Compatible (v0.9.2)
| Crate |
Purpose |
candle-core |
Original tensor core (same code as hanzo-ml) |
candle-nn |
Original NN layers |
candle-transformers |
Original transformer models |
candle-datasets |
Original datasets |
candle-examples |
Original examples |
candle-pyo3 |
Original Python bindings |
GPU Backend Crates (opt-in)
| Crate |
Purpose |
hanzo-kernels |
Custom CUDA kernels (reduce, cast, affine, etc.) |
hanzo-metal-kernels |
Custom Metal kernels (Apple GPU) |
hanzo-flash-attn |
Flash Attention v2 (CUDA SM80+) |
Feature Flags
| Feature |
Effect |
cuda |
NVIDIA GPU via cudarc 0.19.1 |
cudnn |
Additional cuDNN kernels |
nccl |
Multi-GPU distribution |
mkl |
Intel Math Kernel Library |
accelerate |
Apple Accelerate framework |
metal |
Apple GPU via hanzo-metal-kernels + objc2-metal |
ug |
Universal Graph backend |
One-file quickstart
Tensor Operations
Neural Network Training
Load GGUF Model
Load safetensors
Cargo.toml Setup
Integration with Hanzo Engine
Hanzo Engine (mistral-rs fork) uses Hanzo ML as its tensor backend:
Quantization Support
| Format |
Use Case |
| GGUF/GGML |
Universal, llama.cpp compatible |
| AFQ (Affine) |
Optimized for Metal/Apple Silicon |
| GPTQ/AWQ |
GPU-optimized quantization |
| ISQ |
In-situ runtime quantization |
Supported Models (90+ via hanzo-transformers)
| Category |
Models |
| LLMs |
LLaMA 1/2/3, Falcon, Gemma 1/2, Phi 1-3, Mistral, Mixtral, Mamba/2, StarCoder/2, Qwen3 MoE, Yi, GLM4, DeepSeek v2, SmolLM3, Olmo |
| Vision |
DINOv2, ConvMixer, EfficientNet, ResNet, ViT, VGG, YOLO v3/v8, SAM, SegFormer, MobileNet v4, CLIP, SigLIP |
| Audio |
Whisper, EnCodec, MetaVoice, Parler-TTS, Mimi, Silero VAD |
| Diffusion |
Stable Diffusion 1.5/2.1/XL/3, Flux |
| Multimodal |
BLIP, LLaVA, Moondream, PaddleOCR-VL, Pixtral, PaliGemma |
| Quantized |
GGUF/GGML format, llama.cpp compatible |
Supported Formats
| Format |
Extension |
Use Case |
| GGUF |
.gguf |
Quantized models (llama.cpp compatible) |
| safetensors |
.safetensors |
HuggingFace standard (fast, safe) |
| ONNX |
.onnx |
Cross-framework interop |
| PyTorch |
.bin, .pt |
Legacy format |
Project Structure
Development Workflow
Performance Considerations
Apple Silicon (Metal)
- Use
AFQ4 quantization for best throughput
- Enable
--features "metal accelerate" for CPU fallback ops
- Group size 64 balances speed and accuracy
CUDA
- Use GPTQ or AWQ quantization
- Flash Attention enabled via
hanzo-flash-attn (SM80+)
- PagedAttention for memory efficiency in long sequences
CPU
- GGUF models with appropriate quantization level
mkl feature for Intel platforms
accelerate feature for Apple platforms
Related Skills
hanzo/hanzo-candle.md - Upstream candle documentation (planned fork)
hanzo/hanzo-engine.md - Inference engine using hanzo-ml
hanzo/hanzo-ane.md - Apple Neural Engine (complementary to Metal)
hanzo/hanzo-kensho.md - Image generation model
hanzo/hanzo-sho.md - Text diffusion engine
Last Updated: 2026-03-13
Category: Hanzo Ecosystem
Related: ml, rust, tensor, cuda, metal, inference, quantization, wasm
Prerequisites: Rust, ML fundamentals, GPU programming concepts