Files
zeekay efc6b22415 chore: bump engine 1.7.18 + document GGUF pipeline parallelism
Releases the cross-node pipeline-parallel path for GGUF models (ring feature),
verified on rocm+cuda+metal. Docs: RING.md pipeline-parallelism section +
DISTRIBUTED.md pointer.
2026-07-09 19:30:58 -07:00

846 B

Distributed inference in Hanzo Engine

Hanzo Engine supports distributed inference with a few strategies

What backend is best?

  • For CUDA-only system: NCCL
  • Anything else: Ring backend

The Ring backend is also heterogenous! This means that you can use the Ring backend on any set of multiple devices connected over TCP. For example, you can connect 2 Metal systems, or 2 Metal and 1 CPU system with the Ring backend!

For quantized GGUF models the Ring backend runs pipeline parallelism: the model is split by layer range across the ring so a model larger than any single node fits across the cluster. This works across mixed vendors (e.g. a ROCm head with a CUDA worker).