Three-way per file through the rename: base 39355c6c91, theirs the upstream
merge of main with 35d7ae7c (via 7ef6a694d), ours origin/main.
Lands the barrier thread pool, runtime CPU feature dispatch, aarch64 and x86
repacked quantized kernels (indexed_gemv, gemv_fused_shared_lhs), quantized
embedding on CPU/Metal/CUDA, paged flash-attn kernels, cutile fused MoE, the
in-place op aliasing API, parking_lot storage locks, Metal buffer builders and
debug labels, safetensors 0.8, hf-hub 1.0, tokenizers 0.23.
Ours keeps ROCm, Vulkan, wgpu, the IQ/MXFP4/NVFP4/TQ dtypes, and the i-quant
weight/activation block split in matmul. Our backends ride InplaceOpN and the
quantized embedding; Metal get_rows gains the IQ kernels. ug leaves, as upstream.
Co-authored-by: Hanzo Dev <[email protected]>
hanzo-ml-stable-diffusion: A Diffusers API in Rust/ML
A rusty robot holding a fire torch in its hand, generated by Stable Diffusion XL using Rust and hanzo.
The stable-diffusion example is a conversion of
diffusers-rs using hanzo
rather than libtorch. This implementation supports Stable Diffusion v1.5, v2.1,
as well as Stable Diffusion XL 1.0, and Turbo.
Getting the weights
The weights are automatically downloaded for you from the HuggingFace
Hub on the first run. There are various command line
flags to use local files instead, run with --help to learn about them.
Running some example.
cargo run --example stable-diffusion --release --features=cuda,cudnn \
-- --prompt "a cosmonaut on a horse (hd, realistic, high-def)"
The final image is named sd_final.png by default. The Turbo version is much
faster than previous versions, to give it a try add a --sd-version turbo flag,
e.g.:
cargo run --example stable-diffusion --release --features=cuda,cudnn \
-- --prompt "a cosmonaut on a horse (hd, realistic, high-def)" --sd-version turbo
The default scheduler for the v1.5, v2.1 and XL 1.0 version is the Denoising Diffusion Implicit Model scheduler (DDIM). The original paper and some code can be found in the associated repo. The default scheduler for the XL Turbo version is the Euler Ancestral scheduler.
Command-line flags
--prompt: the prompt to be used to generate the image.--uncond-prompt: the optional unconditional prompt.--sd-version: the Stable Diffusion version to use, can bev1-5,v2-1,xl, orturbo.--cpu: use the cpu rather than the gpu (much slower).--height,--width: set the height and width for the generated image.--n-steps: the number of steps to be used in the diffusion process.--num-samples: the number of samples to generate iteratively.--bsize: the numbers of samples to generate simultaneously.--final-image: the filename for the generated image(s).
Using flash-attention
Using flash attention makes image generation a lot faster and uses less memory.
The downside is some long compilation time. You can set the
HANZO_FLASH_ATTN_BUILD_DIR environment variable to something like
/home/user/.hanzo to ensures that the compilation artifacts are properly
cached.
Enabling flash-attention requires both a feature flag, --features flash-attn
and using the command line flag --use-flash-attn.
Note that flash-attention-v2 is only compatible with Ampere, Ada, or Hopper GPUs (e.g., A100/H100, RTX 3090/4090).
Image to Image Pipeline
...
FAQ
Memory Issues
This requires a GPU with more than 8GB of memory, as a fallback the CPU version can be used
with the --cpu flag but is much slower.
Alternatively, reducing the height and width with the --height and --width
flag is likely to reduce memory usage significantly.
