Files
zeekay 872895bb3c debrand: strip HANZO_ prefix from remaining product env vars
Unbrand the pre-existing HANZO_* env vars (multinode MN_*, SANDBOX, VK_*,
NO_MMAP/NO_MLA/NO_NCCL, IGPU_MEMORY_FRACTION, ...) to bare functional names,
keeping their meaning. No HANZO_ env var branding remains.
2026-06-04 15:20:41 -07:00

3.2 KiB

NCCL in Hanzo Engine

Hanzo Engine supports distributed inference on CUDA with Tensor Parallelism via NCCL.

Note: Multi-node support is coming! Distributed inference on Apple hardware is also being investigated.

Tensor Parallelism (TP) is automatically used to accelerate distributed inference when more than one CUDA GPUs are detected. The tensor parallelism size is always automatically set to the total number of GPUs.

TP splits the model into shards and benefits from fast single-node interconnects like NVLink. Both normal and multimodal models support tensor parallelism.

Important: The world size (total number of GPUs) must be a power of 2 (e.g., 1, 2, 4, 8, 16, 32, etc.). This is a requirement for optimal performance and correct operation of the distributed algorithms.

Note: In Hanzo Engine, if NCCL is enabled, then automatic device mapping will not be used.

Important: To build for NCCL, be sure to add the nccl feature flag (for example: --features nccl,cuda).

See the following environment variables:

Name Function Usage
NO_NCCL=1 Disable TP and NCCL If the model does not fit on the available CUDA devices, disabling NCCL will re-enable automatic device mapping

Single-Node Support

Set the number of ranks using MN_LOCAL_WORLD_SIZE, e.g.,

MN_LOCAL_WORLD_SIZE=2 hanzo serve -p 8000 -m Qwen/Qwen3-30B-A3B-Instruct-2507

where, if no MN_LOCAL_WORLD_SIZE env given, Hanzo Engine will split the model across all available devices.

Multi-node support

# Head node:
MN_GLOBAL_WORLD_SIZE=32 MN_HEAD_NUM_WORKERS=1 MN_HEAD_PORT=<PORT> hanzo run -m ...

# For the worker nodes:
MN_GLOBAL_WORLD_SIZE=32 MN_WORKER_ID=0 HANZO_WORKER_SERVER_ADDR=<HEAD ADDR>:<PORT> hanzo run -m ...
MN_GLOBAL_WORLD_SIZE=32 MN_WORKER_ID=1 HANZO_WORKER_SERVER_ADDR=<HEAD ADDR>:<PORT> hanzo run -m ...
MN_GLOBAL_WORLD_SIZE=32 MN_WORKER_ID=2 HANZO_WORKER_SERVER_ADDR=<HEAD ADDR>:<PORT> hanzo run -m ...

Multi-node support in Hanzo Engine divides the nodes into two groups: a "head" node, and multiple "worker" nodes. Head node choice is arbitrary. For example, if a system has 8 nodes, there will be 1 "head" node, and 7 "worker" nodes.

To enable multi-node, set the MN_GLOBAL_WORLD_SIZE=<number> environment variable to the total number of GPUs in all nodes, including "head" and "worker"s. Note: This number must be a power of 2.

It is recommended to use server mode with Hanzo Engine when in multi-node. Currently, you must send requests to every node!

The following environment variables must be set for each node:

Head node:

Name Function Usage
MN_HEAD_NUM_WORKERS=<number> The number of worker nodes which will be connected. This should be the number of nodes in the system, minus 1 for the head node.
MN_HEAD_PORT=<PORT> The port on which to communicate with the worker nodes. Worker nodes will connect to this port via TCP sockets

Worker node:

Name Function Usage
MN_WORKER_ID=<number> The 0-indexed worker ID for this worker node. If there are 4 nodes (1 head, 3 workers), then the worker ids will be 0, 1, and 2
MN_WORKER_SERVER_ADDR=<ADDR>:<PORT> The IP address and port to connect to the server. This is used to establish communication with the head node.