Files
zeekay 795188eabe fix(distributed/ring): canonical f32 wire + N-node all-reduce for heterogeneous clusters
The socket-ring collectives sent raw device-dtype bytes; the receiver reinterpreted
them as its own local dtype. Heterogeneous ranks auto-pick different compute dtypes
(determine_auto_dtype_all: CUDA->bf16, ROCm->f16), so each node misread its
neighbour's bytes as the wrong type. bf16 and f16 share a 16-bit container but have
different exponent/mantissa splits, so every all-reduce was corrupted -> garbage
tokens. Same-vendor worked (matching dtype); cross-vendor (ROCm+CUDA) garbled.

- SumAllReduce/AllGather exchange a canonical f32 wire payload (lossless superset of
  bf16/f16, plus higher-precision accumulation). One cpu_to_f32 helper + one
  ring_exchange primitive shared by both collectives.
- SumAllReduce generalized to any world_size: N-1 accumulating ring steps (was a
  single left/right exchange, complete only for N==2). Enables 3-node clusters.
- AllGather: fix nbytes double-count (elem_cnt * size_of_val(slice) spanned
  len^2*elem_size) and route through the f32 wire.
- Drop the power-of-2 world_size requirement from the ring backend and the
  `hanzo cluster` layout validator (NCCL-only constraint; the ring all-reduce is
  correct for any N >= 2).

Validated on 2 localhost ranks (Qwen3-0.6B): same-vendor f16+f16 and heterogeneous
f32+f16 both generate "The capital of France is Paris." in lockstep. Ring
all-reduce/all-gather numerics verified for world_size 2 and 3 (examples/ring_check).
2026-07-09 14:04:43 -07:00

36 lines
1.5 KiB
Rust

// Standalone ring-collective numerics check. Run one process per rank with RING_CONFIG set.
// Rank r contributes (r+1) to an all-reduce (global sum must be sum_{r}(r+1)) and [r] to an
// all-gather (result must be [0,1,...,world-1]). Validates any world_size >= 2.
use std::sync::Arc;
use hanzo_ml::{Device, Result, Tensor};
use hanzo_quant::distributed::AllGather;
use hanzo_quant::{Comm, Id, RingConfig, SumAllReduce};
fn main() -> Result<()> {
let cfg = RingConfig::load();
let (rank, world) = (cfg.rank, cfg.world_size);
let dev = Device::Cpu;
let comm = Arc::new(Comm::from_device(Id::new(), &dev, rank, world).unwrap());
let n = 8usize;
let local = vec![(rank + 1) as f32; n];
let xs = Tensor::from_slice(&local, (n,), &dev)?;
let reduced = SumAllReduce::new(&comm)
.sum_all_reduce(&xs)?
.to_vec1::<f32>()?;
let expected = (1..=world).sum::<usize>() as f32;
assert!(
reduced.iter().all(|&v| (v - expected).abs() < 1e-3),
"rank {rank}: all_reduce got {reduced:?}, expected all {expected}"
);
println!("[rank {rank}/{world}] sum_all_reduce OK -> {expected}");
let g = Tensor::from_slice(&[rank as f32], (1,), &dev)?;
let gathered = AllGather::new(&comm, 0).all_gather(&g)?.to_vec1::<f32>()?;
let want: Vec<f32> = (0..world).map(|r| r as f32).collect();
assert_eq!(gathered, want, "rank {rank}: all_gather mismatch");
println!("[rank {rank}/{world}] all_gather OK -> {gathered:?}");
Ok(())
}