mirror of
https://github.com/hanzoai/ml.git
synced 2026-11-20 23:08:29 +00:00
Continues the ml-encoder DSL-vs-hand board (rms_norm arm) with the router-gate GEMV arm and the cubecl-Metal throughput board, all measured cold on dbc M4 Max, one GPU job at a time. Landed: - metal_dsl_board::board_gemv: DSL gemv_f_f32 (one threadgroup/row, 32-wide strided reduction) vs the shipped hand call_mlx_gemm -- the route ml's matmul takes for every dense f32 mat-vec, m=1 included -- both through ml's OWN encoder against one oracle. W is [n,k] for the DSL; the hand GEMM reads the SAME weights transposed to [k,n] (the convention the mlx_gemm test pins), transpose explicit not stride-inferred, so a wrong layout cannot pass as a fast kernel. Bit-exact ~1.5e-6 both arms. - kernel_msl seam gains gemv_f_f32 -> src/metal/dsl/gemv.metal (committed cubecl MSL artifact). - matvec-check Metal branch: extended from 4 checks to the full Qwen3 decode board (rms/gemv/router/ MoE q4k+blk+dp4a/q6k/dp4a-blk), the DSL bit-exact + throughput board AND the MSL-dump source. - tools/dsl/msl_extract.sh: brace-count the kernel block instead of stopping at the first column-0 close brace, which truncated any kernel with a mid-body loop/if -- gemv lost its reduction+store. Verified it reproduces the full committed gemv.metal byte-identical. Measured (wall-clock best-of-N through ml's encoder, ~0.15 ms submit floor; ratios are the signal): - gemv m=1: DSL ties-or-beats hand mlx_gemm -- 0.83-0.95x at 4096x4096 (above the floor, a real kernel win), 0.73-0.98x at the 128x2048 router gate (submit-bound). mlx_gemm is structurally starved at m=1 (it runs a tiled GEMM); the purpose-built GEMV is leaner -- the same lesson as the Vulkan router gate. - rms_norm: DSL LOSES at decode 1.13-1.36x (1 row) and is parity at batch 0.82-1.05x (512 rows). The fixed 128-thread tree reduce cannot use the hand kernel's up-to-1024 threads on a single row; they converge once many rows saturate. Stays hand. - cubecl-Metal throughput board (cache-warm small shapes, bit-exact): dp4a-blk 411 GFLOP/s / 206 GB/s, matvec_q8 202 GB/s, Q4_K 247 GFLOP/s, MoE dp4a 201 GFLOP/s, router 0 decode ties. Two cubecl-0.10-Metal defects found (kernels are bit-exact; the failures are in the runtime path): - sdpa_blk: client.profile errors during timing on Metal (omitted from the throughput branch; the device-timestamped attention A/B lives in flash-metal-check). - dense matvec_q4k_dp4a_blk: wgpu worker-thread panic -> zero output (scale_rel 1.0); the MoE dp4a twin dispatches fine, so it is specific to the dense kernel. Omitted; root-cause is a next-step. Anchor (device-timestamped, flash-metal-check bench, unchanged): the f16 8x8 simdgroup flash arm is 3.0-4.55x over the portable scalar arm -- the one Metal perf arm that clearly earns authoring. Its generalization to the prefill GEMM (f16-dequant tile + 8x8 simdgroup accumulators; int8 MMA impossible on Apple) is documented as the NAMED NEXT PERF ARM in the board header. Nothing flipped live: the wins are marginal or submit-bound at live decode shapes; a live gate needs an in-engine A/B. This is the measurement board plus the clean tractable arm, not a decode-path change.