Files
zeekayandHanzo Dev 6a363f7ee3 merge: upstream main ddf1b879 and the aarch64 repack branch 35d7ae7c
Three-way per file through the rename: base 39355c6c91, theirs the upstream
merge of main with 35d7ae7c (via 7ef6a694d), ours origin/main.

Lands the barrier thread pool, runtime CPU feature dispatch, aarch64 and x86
repacked quantized kernels (indexed_gemv, gemv_fused_shared_lhs), quantized
embedding on CPU/Metal/CUDA, paged flash-attn kernels, cutile fused MoE, the
in-place op aliasing API, parking_lot storage locks, Metal buffer builders and
debug labels, safetensors 0.8, hf-hub 1.0, tokenizers 0.23.

Ours keeps ROCm, Vulkan, wgpu, the IQ/MXFP4/NVFP4/TQ dtypes, and the i-quant
weight/activation block split in matmul. Our backends ride InplaceOpN and the
quantized embedding; Metal get_rows gains the IQ kernels. ug leaves, as upstream.

Co-authored-by: Hanzo Dev <[email protected]>
2026-09-15 15:52:28 -07:00
..
2026-01-16 12:30:55 -08:00

hanzo-ml-stable-diffusion-3: ML Implementation of Stable Diffusion 3/3.5

A cute rusty robot holding a hanzo torch in its hand, with glowing neon text "LETS GO RUSTY" displayed on its chest, bright background, high quality, 4k, generated by Stable Diffusion 3 Medium

Stable Diffusion 3 Medium is a text-to-image model based on Multimodal Diffusion Transformer (MMDiT) architecture.

Stable Diffusion 3.5 is a family of text-to-image models with latest improvements:

It has three variants:

Getting access to the weights

The weights of Stable Diffusion 3/3.5 is released by Stability AI under the Stability Community License. You will need to accept the conditions and acquire a license by visiting the repos on HuggingFace Hub to gain access to the weights for your HuggingFace account.

To allow your computer to gain access to the public-gated repos on HuggingFace, you might need to create a HuggingFace User Access Tokens (recommended) and log in on your computer if you haven't done that before. A convenient way to do the login is to use huggingface-cli:

huggingface-cli login

and you will be prompted to enter your token.

On the first run, the weights will be automatically downloaded from the Huggingface Hub. After the download, the weights will be cached and remain accessible locally.

Running the model

cargo run --example stable-diffusion-3 --release --features=cuda -- \
  --which 3-medium --height 1024 --width 1024 \
  --prompt 'A cute rusty robot holding a hanzo torch in its hand, with glowing neon text \"LETS GO RUSTY\" displayed on its chest, bright background, high quality, 4k'

To use different models, changed the value of --which option. (Possible values: 3-medium, 3.5-large, 3.5-large-turbo and 3.5-medium).

To display other options available,

cargo run --example stable-diffusion-3 --release --features=cuda -- --help

If GPU supports, Flash-Attention is a strongly recommended feature as it can greatly improve the speed of inference, as MMDiT is a transformer model heavily depends on attentions. To utilize hanzo-flash-attn in the demo, you will need both --features flash-attn and --use-flash-attn.

cargo run --example stable-diffusion-3 --release --features=cuda,flash-attn -- --use-flash-attn ...

Performance Benchmark

Below benchmark is done with Stable Diffusion 3 Medium by generating 1024-by-1024 image from 28 steps of Euler sampling and measure the average speed (iteration per seconds).

hanzo and hanzo-flash-attn is based on the commit of 0d96ec3.

System specs (Desktop PCIE 5 x8/x8 dual-GPU setup):

  • Operating System: Ubuntu 23.10
  • CPU: i9 12900K w/o overclocking.
  • RAM: 64G dual-channel DDR5 @ 4800 MT/s
Speed (iter/s) w/o flash-attn w/ flash-attn
RTX 3090 Ti 0.83 2.15
RTX 4090 1.72 4.06