Files
zeekay efc6b22415 chore: bump engine 1.7.18 + document GGUF pipeline parallelism
Releases the cross-node pipeline-parallel path for GGUF models (ring feature),
verified on rocm+cuda+metal. Docs: RING.md pipeline-parallelism section +
DISTRIBUTED.md pointer.
2026-07-09 19:30:58 -07:00

6.3 KiB

Ring backend in Hanzo Engine

Hanzo Engine provides a TCP-based ring backend for distributed tensor-parallel inference. This backend is enabled by compiling with the ring feature and implements collective operations over a ring topology using TCP sockets.

Prerequisites

  • Build with the ring feature enable, in addition to any others:
    cargo build --release --features ring
    
  • Ensure the specified TCP ports are open and reachable between processes.
  • The world_size must be a power of 2 (2, 4, 8, 16, etc.) for correct operation.

Configuration

Create one JSON configuration file per process with the following fields:

Field Type Description
master_ip string Optional. IP address for master node.
master_port integer Optional. Port for master node.
port integer Local port to bind for incoming connections from the left neighbor.
right_port integer Port on which the right neighbor is listening (used to connect outgoing to the right).
right_ip string Optional. IP address of the right neighbor (defaults to 0.0.0.0).
rank integer Rank of this process in [0..world_size).
world_size integer Total number of processes in the ring. Must be a power of 2 (e.g., 2, 4, 8, 16, etc.).

This address and port should form a ring topology for each of the nodes. For example, the last node should point to the first node as its right neighbor.

Although all processes participate in collective communication, Rank 0 acts as the master node. For example, interactive mode or the server runs on Rank 0, while other ranks act as background workers.

Example ring topology:

+---------+         +---------+
| Rank 0  | ----->  | Rank 1  |
| IP: A   |         | IP: B   |
| Port: X |         | Port: Y |
+----+----+         +----+----+
     ^                   |
     |                   v
+----+----+         +----+----+
| Rank 3  | <-----  | Rank 2  |
| IP: D   |         | IP: C   |
| Port: W |         | Port: Z |
+---------+         +---------+

Each node connects to its right neighbor by IP and port, and the last node wraps around to the first.

Example for two processes:

  • ring_0.json:

    {
      "master_ip": "0.0.0.0",
      "master_port": 1234,
      "port": 12345,
      "right_port": 12346,
      "rank": 0,
      "world_size": 2
    }
    
  • ring_1.json:

    {
      "master_ip": "0.0.0.0",
      "master_port": 1234,
      "port": 12346,
      "right_port": 12345,
      "rank": 1,
      "world_size": 2
    }
    

Multi-Machine Example

To run on different machines, update the right_ip field in each config to the actual IP address of the neighbor process. For example, if you have two machines with IPs 192.168.1.10 and 192.168.1.11:

  • ring_0.json on Machine A (192.168.1.10):

    {
      "port": 12345,
      "right_port": 12346,
      "right_ip": "192.168.1.11",
      "rank": 0,
      "world_size": 2
    }
    
  • ring_1.json on Machine B (192.168.1.11):

    {
      "port": 12346,
      "right_port": 12345,
      "right_ip": "192.168.1.10",
      "rank": 1,
      "world_size": 2
    }
    

Make sure that the specified ports are open and that each machine can reach the other via TCP on those ports.

Usage

Set the RING_CONFIG environment variable to point to the JSON file for each process, then run your application built with the ring feature:

# Process 0 or computer 0
export RING_CONFIG=path/to/ring_0.json
cargo run --release --features ring -- ...

# Process 1 or computer 1
export RING_CONFIG=path/to/ring_1.json
cargo run --release --features ring -- ...

The ring backend will automatically handle collective communication for tensor-parallel inference.

Pipeline parallelism (GGUF / quantized models)

Quantized GGUF models run the ring as a pipeline rather than a tensor-parallel collective. This is the only way to distribute a GGUF model, and it is what lets a model larger than any single node fit across the cluster.

  • Rank 0 (the head) owns the generation loop: tokenizer, embeddings, lm_head, sampling and streaming. It runs its own layer range, sends the activation to its right neighbor, and receives the post-final-layer activation back from its left neighbor before the final norm and lm_head.
  • Every other rank is a stateless layer executor: it receives an activation from the left, runs its local layer range, and sends the result to the right. Workers never see tokens, only activations, so there is no lockstep sampling and no head-divisibility constraint. Any world_size >= 2 works.
  • Each rank loads only its own layer range from disk, so weights and the KV cache split roughly in proportion to the layer counts. A rank never materializes tensors outside its range.
  • Activations cross the wire as canonical f32 with a small per-hop header (shape + per-sequence offsets), so a heterogeneous ring (e.g. a ROCm head and a CUDA worker) agrees on the bytes. Use --dtype f32 for heterogeneous clusters.

Pipeline parallelism is the default when the ring feature is active and a GGUF model is loaded. Set RING_PP=0 to fall back to the legacy safetensors tensor-parallel behavior.

Layer assignment is proportional to per-rank memory. Override it with:

Variable Effect
RING_LAYER_SPLIT="a,b,.." Exact per-rank block counts (must sum to the model's layer count, one entry per rank).
RING_MEM_GIB="g0,g1,.." Per-rank GPU budget; layers are distributed proportionally.

Currently wired for the qwen3 and qwen3moe GGUF architectures.

Example, two heterogeneous machines A (192.168.1.10, head) and B (192.168.1.11):

# Machine A, rank 0
RING_CONFIG=ring_0.json hanzo serve -p 1234 --dtype f32 --format gguf -m /models/M -f model.gguf

# Machine B, rank 1 (auto-enters the worker loop)
RING_CONFIG=ring_1.json hanzo serve --dtype f32 --format gguf -m /models/M -f model.gguf