mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
The socket-ring collectives sent raw device-dtype bytes; the receiver reinterpreted them as its own local dtype. Heterogeneous ranks auto-pick different compute dtypes (determine_auto_dtype_all: CUDA->bf16, ROCm->f16), so each node misread its neighbour's bytes as the wrong type. bf16 and f16 share a 16-bit container but have different exponent/mantissa splits, so every all-reduce was corrupted -> garbage tokens. Same-vendor worked (matching dtype); cross-vendor (ROCm+CUDA) garbled. - SumAllReduce/AllGather exchange a canonical f32 wire payload (lossless superset of bf16/f16, plus higher-precision accumulation). One cpu_to_f32 helper + one ring_exchange primitive shared by both collectives. - SumAllReduce generalized to any world_size: N-1 accumulating ring steps (was a single left/right exchange, complete only for N==2). Enables 3-node clusters. - AllGather: fix nbytes double-count (elem_cnt * size_of_val(slice) spanned len^2*elem_size) and route through the f32 wire. - Drop the power-of-2 world_size requirement from the ring backend and the `hanzo cluster` layout validator (NCCL-only constraint; the ring all-reduce is correct for any N >= 2). Validated on 2 localhost ranks (Qwen3-0.6B): same-vendor f16+f16 and heterogeneous f32+f16 both generate "The capital of France is Paris." in lockstep. Ring all-reduce/all-gather numerics verified for world_size 2 and 3 (examples/ring_check).