Files
engine/hanzo/README.md
zeekay 37ff59034b merge: upstream/master (-X ours, brand preserved)
# Conflicts:
#	hanzo-engine/src/device_map.rs
#	hanzo-engine/src/device_map/mappers.rs
#	hanzo-engine/src/device_map/mask.rs
#	hanzo-engine/src/device_map/mod.rs
#	hanzo-engine/src/device_map/peer.rs
#	hanzo-engine/src/moe/experts.rs
#	hanzo-engine/src/moe/experts/backends.rs
#	hanzo-engine/src/moe/experts/checkpoint.rs
#	hanzo-engine/src/moe/experts/config.rs
#	hanzo-engine/src/moe/experts/forward.rs
#	hanzo-engine/src/moe/experts/mod.rs
#	hanzo-engine/src/paged_attention/attention_backend.rs
#	hanzo-engine/src/perf_flags.rs
#	hanzo-engine/src/pipeline/cuda_graph.rs
#	hanzo-paged-attn/src/cuda/backend/flashinfer.rs
#	hanzo-paged-attn/src/cuda/flashinfer/attention/default_prefill_params.cuh
#	hanzo-paged-attn/src/cuda/flashinfer/attention/mask.cuh
#	hanzo-paged-attn/src/cuda/flashinfer/attention/prefill.cuh
#	hanzo-paged-attn/src/cuda/flashinfer/fp16.h
#	hanzo-paged-attn/src/cuda/flashinfer/frag_layout_swizzle.cuh
#	hanzo-paged-attn/src/cuda/flashinfer/mma.cuh
#	hanzo-paged-attn/src/cuda/flashinfer/permuted_smem.cuh
#	hanzo-paged-attn/src/cuda/flashinfer_decode.cu
#	hanzo-quant/src/metal_kernels/moe.metal
#	hanzo-quant/src/metal_kernels/rotary.metal
#	hanzo-quant/src/metal_kernels/softcap.metal
#	hanzo-server-core/src/anthropic.rs
#	hanzo-server-core/src/route_registry.rs
#	mistralrs-core/Cargo.toml
#	mistralrs-core/src/lib.rs
2026-06-02 10:21:05 -07:00

2.8 KiB

hanzo — Blazing-Fast LLM Inference in Rust

The Rust SDK for hanzo, a high-performance LLM inference engine supporting text, multimodal, speech, image generation, and embedding models.

API Docs | GitHub | Examples | Discord

Quick Start

use hanzo::{IsqBits, ModelBuilder, TextMessages, TextMessageRole};

#[tokio::main]
async fn main() -> hanzo::error::Result<()> {
    let model = ModelBuilder::new("Qwen/Qwen3-4B")
        .with_auto_isq(IsqBits::Four)
        .build()
        .await?;

    let response = model.chat("What is Rust's ownership model?").await?;
    println!("{response}");
    Ok(())
}

Capabilities

Capability Builder Example
Any model (auto-detect) ModelBuilder examples/getting_started/text_generation/
Text generation TextModelBuilder examples/getting_started/text_generation/
Multimodal (image+text) MultimodalModelBuilder examples/getting_started/multimodal/
GGUF quantized models GgufModelBuilder examples/getting_started/gguf/
Image generation DiffusionModelBuilder examples/models/diffusion/
Speech synthesis SpeechModelBuilder examples/models/speech/
Embeddings EmbeddingModelBuilder examples/getting_started/embedding/
Structured output Model::generate_structured examples/advanced/json_schema/
Tool calling Tool, ToolChoice examples/advanced/tools/
Agents AgentBuilder examples/advanced/agent/
LoRA / X-LoRA LoraModelBuilder, XLoraModelBuilder examples/advanced/lora/
AnyMoE AnyMoeModelBuilder examples/advanced/anymoe/
MCP client McpClientConfig examples/advanced/mcp_client/

Choosing a Request Type

Type Use When Sampling
TextMessages Simple text-only chat Deterministic
MultimodalMessages Prompt includes images or audio Deterministic
RequestBuilder Tools, logprobs, custom sampling, constraints, adapters, or web search Configurable

TextMessages and MultimodalMessages convert into RequestBuilder via Into<RequestBuilder>.

Feature Flags

Flag Effect
cuda CUDA GPU support
flash-attn Flash Attention 2 kernels (requires cuda)
cudnn cuDNN acceleration (requires cuda)
nccl Multi-GPU via NCCL (requires cuda and NCCL)
metal Apple Metal GPU support
accelerate Apple Accelerate framework
mkl Intel MKL acceleration

The default feature set (no flags) builds with pure Rust — no C compiler or system libraries required.

License

MIT