Files
zeekay 795188eabe fix(distributed/ring): canonical f32 wire + N-node all-reduce for heterogeneous clusters
The socket-ring collectives sent raw device-dtype bytes; the receiver reinterpreted
them as its own local dtype. Heterogeneous ranks auto-pick different compute dtypes
(determine_auto_dtype_all: CUDA->bf16, ROCm->f16), so each node misread its
neighbour's bytes as the wrong type. bf16 and f16 share a 16-bit container but have
different exponent/mantissa splits, so every all-reduce was corrupted -> garbage
tokens. Same-vendor worked (matching dtype); cross-vendor (ROCm+CUDA) garbled.

- SumAllReduce/AllGather exchange a canonical f32 wire payload (lossless superset of
  bf16/f16, plus higher-precision accumulation). One cpu_to_f32 helper + one
  ring_exchange primitive shared by both collectives.
- SumAllReduce generalized to any world_size: N-1 accumulating ring steps (was a
  single left/right exchange, complete only for N==2). Enables 3-node clusters.
- AllGather: fix nbytes double-count (elem_cnt * size_of_val(slice) spanned
  len^2*elem_size) and route through the f32 wire.
- Drop the power-of-2 world_size requirement from the ring backend and the
  `hanzo cluster` layout validator (NCCL-only constraint; the ring all-reduce is
  correct for any N >= 2).

Validated on 2 localhost ranks (Qwen3-0.6B): same-vendor f16+f16 and heterogeneous
f32+f16 both generate "The capital of France is Paris." in lockstep. Ring
all-reduce/all-gather numerics verified for world_size 2 and 3 (examples/ring_check).
2026-07-09 14:04:43 -07:00
..