mirror of
https://github.com/hanzoai/ml.git
synced 2026-11-20 23:08:29 +00:00
Four defects on the Metal path, each carried since the candle merge-base and each fixed upstream; ported here with the upstream reasoning. rms_norm produced NaN on large-magnitude F32 inputs: RMSReduceOp merged partial sums with a Welford-style step that squared `delta` over the running sums, which overflows to inf, and multiplied it by an integer `b.count / new_count` that is always 0 -- inf * 0 = NaN. Merging sums of squares is a plain sum, so the field is now named for what it holds (sum_sq) and the merge is an addition. SDPA returned NaN logits when given an explicit mask: the mask already encodes causality, so also enabling the in-kernel causal path masked a query row in full, leaving softmax's normalizer at 0 and the final divide computing 0/0. CPU readbacks could observe stale buffers under concurrent submission: they waited via flush_and_wait, which takes the whole in-flight set, so a concurrent flush on another thread could drain the buffer carrying this thread's blit and let the wait return before that work ran. Readbacks now commit and wait on their own buffer. The SDPA vector kernel reduces over a single query row, so it is only correct at q_seq == 1, but q_seq <= 8 was routed into it. Upstream: huggingface/candle #3477, #3625, #3595, #3479.
57 lines
2.2 KiB
Rust
57 lines
2.2 KiB
Rust
#![cfg(feature = "metal")]
|
|
|
|
use hanzo_ml::{Device, Result, Tensor};
|
|
|
|
// A readback must wait on the command buffer holding its OWN blit, not on shared device state: waiting
|
|
// on "the last in-flight buffer" lets a concurrent flush on another thread drain the buffer carrying
|
|
// this thread's work, so the wait returns before the blit has run and the CPU reads a stale buffer.
|
|
#[test]
|
|
fn concurrent_readback() -> Result<()> {
|
|
let device = Device::new_metal(0)?;
|
|
std::thread::scope(|scope| {
|
|
for thread in 0..8usize {
|
|
let device = device.clone();
|
|
scope.spawn(move || {
|
|
for iter in 0..100usize {
|
|
let value = (thread * 1000 + iter) as f64;
|
|
let a = Tensor::full(value as f32, (64, 64), &device).unwrap();
|
|
let b = a.affine(2.0, 1.0).unwrap();
|
|
let values = b.flatten_all().unwrap().to_vec1::<f32>().unwrap();
|
|
let expected = (2.0 * value + 1.0) as f32;
|
|
assert!(
|
|
values.iter().all(|&x| x == expected),
|
|
"thread {thread} iter {iter}: expected {expected}, got {:?}",
|
|
&values[..4]
|
|
);
|
|
}
|
|
});
|
|
}
|
|
});
|
|
Ok(())
|
|
}
|
|
|
|
#[test]
|
|
fn concurrent_quantized_data_roundtrip() -> Result<()> {
|
|
use hanzo_ml::quantized::{GgmlDType, QTensor};
|
|
let device = Device::new_metal(0)?;
|
|
std::thread::scope(|scope| {
|
|
for thread in 0..8usize {
|
|
let device = device.clone();
|
|
scope.spawn(move || {
|
|
for iter in 0..25usize {
|
|
let src = Tensor::rand(-1f32, 1f32, (256, 256), &device).unwrap();
|
|
let q = QTensor::quantize(&src, GgmlDType::Q8_0).unwrap();
|
|
let bytes = q.data().unwrap();
|
|
let q2 = QTensor::quantize(&src, GgmlDType::Q8_0).unwrap();
|
|
let bytes2 = q2.data().unwrap();
|
|
assert_eq!(
|
|
bytes, bytes2,
|
|
"thread {thread} iter {iter}: data() readback mismatch"
|
|
);
|
|
}
|
|
});
|
|
}
|
|
});
|
|
Ok(())
|
|
}
|