Files
engine/docs/DEVICE_MAPPING.md
zeekay 872895bb3c debrand: strip HANZO_ prefix from remaining product env vars
Unbrand the pre-existing HANZO_* env vars (multinode MN_*, SANDBOX, VK_*,
NO_MMAP/NO_MLA/NO_NCCL, IGPU_MEMORY_FRACTION, ...) to bare functional names,
keeping their meaning. No HANZO_ env var branding remains.
2026-06-04 15:20:41 -07:00

4.2 KiB

Device mapping

In Hanzo Engine, device mapping is automatically managed to be as performant and easy as possible. Automatic device mapping is enabled by default in the CLI/server and Python SDK and does not make any changes when the model fits entirely on the GPU.

Note

If your system has more than one CUDA device, Hanzo Engine will automatically use tensor parallelism. If the model does not completely fit on the available GPUs, or you wish to use automatic device mapping, you can disable tensor parallelism by setting NO_NCCL=1.

Automatic device mapping works by prioritizing loading models into GPU memory, and any remaining parts are loaded into CPU memory. Models architectures such as multimodal models which greatly benefit from GPU acceleration also automatically prioritize keeping those components on the GPU.

To control the mapping across devices, you can set the following maximum parameters which the model should expect in a prompt.

  • maximum sequence length (default: 4096)
  • maximum batch size (default: 1)
  • (multimodal models) maximum image length (length refers to the edge length) (default: 1024)
  • (multimodal models) maximum number of images (default: 1)

These parameters do not translate to hard limits during runtime, they only control the mapping.

Unified memory systems

On integrated GPU systems (e.g. Apple Silicon, NVIDIA Grace Blackwell, Jetson) where GPU and CPU share the same physical RAM, the auto device mapper caps the GPU memory budget to a fraction of system RAM (75% by default for CUDA iGPUs, configurable via IGPU_MEMORY_FRACTION; Metal uses the iogpu.wired_limit_mb sysctl). CPU offload capacity is limited to the remaining fraction to prevent over-subscription of shared memory. Use hanzo doctor to check whether your device is detected as unified memory.

Note

The maximum sequence length is also used to ensure that a KV cache will fit for with and without PagedAttention.

Examples


If you want to manually device map the model (not recommended), please continue reading.

Note

Manual device mapping is deprecated in favor of automatic device mapping due to the possibility for user error in manual.

Manual device mapping

There are 2 ways to do device mapping:

  1. Specify the number of layers to put on the GPU - this uses the GPU with ordinal 0.
  2. Specify the ordinals and number of layers - this allows for cross-GPU device mapping.

The format for the ordinals and number of layers is ORD:NUM;... where ORD is the unique ordinal and NUM is the number of layers for that GPU. This may be repeated as many times as necessary.

Note: We refer to GPU layers as "device layers" throughout Hanzo Engine.

Example of specifying ordinals

hanzo run -n "0:16;1:16" -m gradientai/Llama-3-8B-Instruct-262k

Note: In the Python SDK, the "0:16;1:16" string is passed as the list ["0:16", "1:16"].

Example of specifying the number of GPU layers

hanzo run -n 16 -m gradientai/Llama-3-8B-Instruct-262k

See Also