mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
The socket-ring collectives sent raw device-dtype bytes; the receiver reinterpreted them as its own local dtype. Heterogeneous ranks auto-pick different compute dtypes (determine_auto_dtype_all: CUDA->bf16, ROCm->f16), so each node misread its neighbour's bytes as the wrong type. bf16 and f16 share a 16-bit container but have different exponent/mantissa splits, so every all-reduce was corrupted -> garbage tokens. Same-vendor worked (matching dtype); cross-vendor (ROCm+CUDA) garbled. - SumAllReduce/AllGather exchange a canonical f32 wire payload (lossless superset of bf16/f16, plus higher-precision accumulation). One cpu_to_f32 helper + one ring_exchange primitive shared by both collectives. - SumAllReduce generalized to any world_size: N-1 accumulating ring steps (was a single left/right exchange, complete only for N==2). Enables 3-node clusters. - AllGather: fix nbytes double-count (elem_cnt * size_of_val(slice) spanned len^2*elem_size) and route through the f32 wire. - Drop the power-of-2 world_size requirement from the ring backend and the `hanzo cluster` layout validator (NCCL-only constraint; the ring all-reduce is correct for any N >= 2). Validated on 2 localhost ranks (Qwen3-0.6B): same-vendor f16+f16 and heterogeneous f32+f16 both generate "The capital of France is Paris." in lockstep. Ring all-reduce/all-gather numerics verified for world_size 2 and 3 (examples/ring_check).
36 lines
1.5 KiB
Rust
36 lines
1.5 KiB
Rust
// Standalone ring-collective numerics check. Run one process per rank with RING_CONFIG set.
|
|
// Rank r contributes (r+1) to an all-reduce (global sum must be sum_{r}(r+1)) and [r] to an
|
|
// all-gather (result must be [0,1,...,world-1]). Validates any world_size >= 2.
|
|
use std::sync::Arc;
|
|
|
|
use hanzo_ml::{Device, Result, Tensor};
|
|
use hanzo_quant::distributed::AllGather;
|
|
use hanzo_quant::{Comm, Id, RingConfig, SumAllReduce};
|
|
|
|
fn main() -> Result<()> {
|
|
let cfg = RingConfig::load();
|
|
let (rank, world) = (cfg.rank, cfg.world_size);
|
|
let dev = Device::Cpu;
|
|
let comm = Arc::new(Comm::from_device(Id::new(), &dev, rank, world).unwrap());
|
|
|
|
let n = 8usize;
|
|
let local = vec![(rank + 1) as f32; n];
|
|
let xs = Tensor::from_slice(&local, (n,), &dev)?;
|
|
let reduced = SumAllReduce::new(&comm)
|
|
.sum_all_reduce(&xs)?
|
|
.to_vec1::<f32>()?;
|
|
let expected = (1..=world).sum::<usize>() as f32;
|
|
assert!(
|
|
reduced.iter().all(|&v| (v - expected).abs() < 1e-3),
|
|
"rank {rank}: all_reduce got {reduced:?}, expected all {expected}"
|
|
);
|
|
println!("[rank {rank}/{world}] sum_all_reduce OK -> {expected}");
|
|
|
|
let g = Tensor::from_slice(&[rank as f32], (1,), &dev)?;
|
|
let gathered = AllGather::new(&comm, 0).all_gather(&g)?.to_vec1::<f32>()?;
|
|
let want: Vec<f32> = (0..world).map(|r| r as f32).collect();
|
|
assert_eq!(gathered, want, "rank {rank}: all_gather mismatch");
|
|
println!("[rank {rank}/{world}] all_gather OK -> {gathered:?}");
|
|
Ok(())
|
|
}
|