mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
Colocated CPU integration test loads a tiny random-weight glm-dsa GGUF (split-MLA + DSA lightning indexer + sigmoid/noaux MoE + shared expert) on Device::Cpu and asserts the forward matches a transformers reference token-for-token: teacher forcing over 32 tokens (argmax positions 11..=30) and 20-step greedy decode. With index_topk=4096 >> seq and no attn_indexer_* tensors, DSA degenerates to dense attention, validating the whole forward without the sparse path. Fixtures (glm_tiny_f32.gguf + ref_glm.json) committed; converter scripts/tiny_glm_to_gguf.py. Fix: fused-MoE forward assumed quantized (QTensor) expert banks; an all-F32 GGUF dequantizes the bank to a plain Tensor, which moe_gate_up/indexed_moe_forward do not cover -> panic. Add a dense per-expert twin (same gather->matmul->scatter contract) that activates only for unquantized banks, leaving the quantized path untouched. Token-exact F32 is required for argmax parity with the f32 oracle.