Files
ml/hanzo-ml/tests/metal_concurrent_tests.rs
zeekay 5548af564d fix(metal): correct NaN logits, readback race, and SDPA vector routing
Four defects on the Metal path, each carried since the candle merge-base and each
fixed upstream; ported here with the upstream reasoning.

rms_norm produced NaN on large-magnitude F32 inputs: RMSReduceOp merged partial
sums with a Welford-style step that squared `delta` over the running sums, which
overflows to inf, and multiplied it by an integer `b.count / new_count` that is
always 0 -- inf * 0 = NaN. Merging sums of squares is a plain sum, so the field is
now named for what it holds (sum_sq) and the merge is an addition.

SDPA returned NaN logits when given an explicit mask: the mask already encodes
causality, so also enabling the in-kernel causal path masked a query row in full,
leaving softmax's normalizer at 0 and the final divide computing 0/0.

CPU readbacks could observe stale buffers under concurrent submission: they waited
via flush_and_wait, which takes the whole in-flight set, so a concurrent flush on
another thread could drain the buffer carrying this thread's blit and let the wait
return before that work ran. Readbacks now commit and wait on their own buffer.

The SDPA vector kernel reduces over a single query row, so it is only correct at
q_seq == 1, but q_seq <= 8 was routed into it.

Upstream: huggingface/candle #3477, #3625, #3595, #3479.
2026-07-16 11:56:32 -07:00

57 lines
2.2 KiB
Rust

#![cfg(feature = "metal")]
use hanzo_ml::{Device, Result, Tensor};
// A readback must wait on the command buffer holding its OWN blit, not on shared device state: waiting
// on "the last in-flight buffer" lets a concurrent flush on another thread drain the buffer carrying
// this thread's work, so the wait returns before the blit has run and the CPU reads a stale buffer.
#[test]
fn concurrent_readback() -> Result<()> {
let device = Device::new_metal(0)?;
std::thread::scope(|scope| {
for thread in 0..8usize {
let device = device.clone();
scope.spawn(move || {
for iter in 0..100usize {
let value = (thread * 1000 + iter) as f64;
let a = Tensor::full(value as f32, (64, 64), &device).unwrap();
let b = a.affine(2.0, 1.0).unwrap();
let values = b.flatten_all().unwrap().to_vec1::<f32>().unwrap();
let expected = (2.0 * value + 1.0) as f32;
assert!(
values.iter().all(|&x| x == expected),
"thread {thread} iter {iter}: expected {expected}, got {:?}",
&values[..4]
);
}
});
}
});
Ok(())
}
#[test]
fn concurrent_quantized_data_roundtrip() -> Result<()> {
use hanzo_ml::quantized::{GgmlDType, QTensor};
let device = Device::new_metal(0)?;
std::thread::scope(|scope| {
for thread in 0..8usize {
let device = device.clone();
scope.spawn(move || {
for iter in 0..25usize {
let src = Tensor::rand(-1f32, 1f32, (256, 256), &device).unwrap();
let q = QTensor::quantize(&src, GgmlDType::Q8_0).unwrap();
let bytes = q.data().unwrap();
let q2 = QTensor::quantize(&src, GgmlDType::Q8_0).unwrap();
let bytes2 = q2.data().unwrap();
assert_eq!(
bytes, bytes2,
"thread {thread} iter {iter}: data() readback mismatch"
);
}
});
}
});
Ok(())
}