mirror of
https://github.com/hanzoai/wasm2go.git
synced 2026-10-30 15:06:48 +00:00
f2b99fe737
* gcasm: AVX2 8-bit dot selection for amd64 with runtime dispatch The amd64 SIMD splices lowered the quantized integer dot literally in SSE — per 16-byte chunk, sign-extend each half of each operand to i16, PMADDWD each half, PADDD the two: ~9 xmm ops. AVX2 does the whole chunk in one 256-bit pass: VPMOVSXBW sign-extends all 16 bytes to i16 across a ymm, VPMADDWD yields 8 i32 lanes (0..3 the low-half dot, 4..7 the high), and VEXTRACTI128 + VPADDD fold the high 128 onto the low. int8*int8 fits i16 and the pair sums fit i32, so nothing saturates and every lane is bit-identical to the SSE lowering — five ops replace nine. x64Dot8Rewrite mirrors the arm64 a64Dot8Rewrite: at splice time it collapses a matched add(dot_low, dot_high) chunk into the synthetic x64OpBlockDot reading the two byte sources directly, eliding the six absorbed nodes. The x64 core emits the AVX2 sequence and one VZEROUPPER per region on the exit path. AVX2 is not the amd64 baseline (SSE4.1 / x86-64-v2 is), so it is gated at runtime, not by build tag: the amd64 archSpec entry now carries the "// avx2 dot" marker, so a body using it is emitted twice — an <fn>avx2 feature body and an <fn>sse portable twin (transformed with PortableSIMD, keeping the literal SSE dot) — behind a NOSPLIT tail-jump stub that branches on base.HasAVX2. cpufeat_amd64 detects AVX2 with the standard CPUID/XGETBV probe (OSXSAVE + AVX + XCR0 YMM + AVX2), dependency-free like the arm64 CPUDotProd detection. Unit tests cover the rewrite (fires, portable suppresses, rejects a non-single-use dot) and the emission (AVX2 mnemonics + VZEROUPPER vs the literal SSE twin). Execution correctness is validated by the amd64 CI differential/loop suites and downstream by the llama q8 token-match tests. Co-Authored-By: Claude Opus 4.8 <[email protected]> * gcasm: x64 fused-region memory addressing diet Member loads inside fused regions and loops paid per-access Module loads: every load re-fetched m.M (and every bounds check re-fetched the memSize pointer and dereferenced it), then added the base by hand — 11 instructions for one checked 16-byte load, ~60% of the q8 kernel's fused-loop body (profile: that kernel is 76% of amd64 tg). Three changes, all x64-emitter-local: - Loads finish with one indexed MOVOU (base)(R12*1) instead of the ADDQ base + plain MOVOU pair. - x64HoistMem pins m.M and the memSize VALUE in free argument-tail registers once per fused region or loop (splices contain no calls, so memory cannot grow and both stay valid); member emitters then address through the pinned registers and bounds-check with a single CMPQ. Regions with one memory op keep the per-site loads. - x64FusedEA64/x64FusedLaneAddr return the base register instead of fixing AX, so lane inserts ride the pinned base too. Co-Authored-By: Claude Opus 4.8 <[email protected]> * gcasm: amd64 fast-math 2x2 tile kernel (nrc2 row pairing) The vec_dot row/column pairing (nrows=2) already restructures the GEMM driver on every backend, but only arm64 had a native tile kernel behind it (SMMLA, pp +26% measured); the amd64 feature body kept the 4-subcall bit-exact companion, reloading every quant vector twice. x64Nrc2Kernel is the AVX2 analog: per 34-byte block the four quant vectors (rows x0/x1, cols y0/y1) load once and feed all four dots — sign-extend to i16x16 halves, VPMADDWD the four pairs, and collapse the four i32x8 sums into the 2x2 tile in one xmm with three VPHADDD and a 128-fold. The f16 d scales gather via VPINSRW, convert with VCVTPH2PS, spread with two VPERMILPS, and accumulate with VFMADD231PS. Entirely VEX-encoded with VZEROUPPER on both exits (the Intel dirty-upper hazard), bounds-checked like the arm64 kernel, both pointer widths. Like the arm64 kernel the retarget is fast-math-only (FMA regroups the float accumulate; the integer tile is exact). HasAVX2 detection now also requires FMA and F16C — every AVX2 CPU shipped has both, and the tile kernel uses them. Tests: kernel shape (loads once, 8 VPMADDWD, 6 bounds checks, VZEROUPPER on every exit, ILP32 layout) and real-assembler acceptance of the emitted body (GOOS=linux GOARCH=amd64 go build). Co-Authored-By: Claude Opus 4.8 <[email protected]> * gcasm: VEX direct-destination forms for hot fused vector ops The canned SSE paste stages every operand into X0/X1 and parks every result back out with MOVDQU — the parking copies are the hottest instructions in both decode and prompt profiles. Ops with a VEX three-operand register form (add/sub/mul families, cvtdq2ps, maddwd) now read their sources where they live and write the pool register directly, eliminating both copies. VEX-128 zeroes the ymm uppers, so no dirty-upper hazard; the portable SSE twin keeps the canned paste (VEX is above its baseline). Co-Authored-By: Claude Opus 4.8 <[email protected]> * gcasm: hoist strided window checks out of x64 fused loops A pretest fused loop with a power-of-two step runs exactly counter>>log2(Dec) iterations, so one loop-head check of base + rlo + span + (iters-1)*delta against memSize covers every iteration's window; the in-loop range-checked loads then collapse to single indexed instructions (the memory64 folded form). Trapping at the loop head instead of the precise access is the relaxation the windowed range check already makes. Guards: memory64 trees, constant strides, non-negative rlo, a zero-iteration skip, and MOVL zero-extension of 32-bit counters. Co-Authored-By: Claude Opus 4.8 <[email protected]> * gcasm: fully-VEX fused regions with one deferred upper clear Two wins over the previous emission, found by reading the hot loop: the folded unchecked loads still carried their dead two-instruction address computation (the fold decision ran after base materialization — now it short-circuits at the top of the load emitter), and every block dot paid a VZEROUPPER inside the loop. The remaining legacy-SSE emissions on the fused path (vector copies, the float pack, splats, load tails, the scalar chain's FP ops) now emit their VEX forms, and when a rewritten tree's emission is entirely VEX (x64TreeVexClean) the per-dot clears collapse into a single VZEROUPPER at the region or loop exit — VEX-128 is safe under dirty uppers, so the Intel hazard needs clearing only before returning to gc-emitted legacy code. Co-Authored-By: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: Claude Opus 4.8 <[email protected]>
77 lines
2.4 KiB
Go
77 lines
2.4 KiB
Go
//go:build amd64
|
|
|
|
package helpers
|
|
|
|
// HasAVX2 reports AVX2 (plus FMA and F16C, which every AVX2 CPU ever
|
|
// shipped carries) with OS-enabled YMM state. Feature-gated splice
|
|
// bodies — the AVX2 8-bit dot kernels and the fast-math 2x2 tile
|
|
// kernel, which uses VCVTPH2PS and VFMADD — dispatch on it at
|
|
// runtime; the portable SSE twin body runs when it is false. The
|
|
// check mirrors the standard three-step CPUID/XGETBV probe: OSXSAVE +
|
|
// AVX + FMA + F16C (leaf 1), the OS actually saving XMM+YMM state
|
|
// (XCR0), and AVX2 (leaf 7).
|
|
var HasAVX2 = detectAVX2()
|
|
|
|
// HasAVX512VNNI reports AVX-512 VNNI usable at 256-bit width (VL) with
|
|
// OS-enabled ZMM/opmask state — VPDPBUSD on ymm, the single-instruction
|
|
// int8 dot product the fast-math tile kernel branches to at runtime.
|
|
var HasAVX512VNNI = detectAVX512VNNI()
|
|
|
|
// cpuidAMD64 executes CPUID for the given leaf/subleaf.
|
|
func cpuidAMD64(eax, ecx uint32) (a, b, c, d uint32)
|
|
|
|
// xgetbvAMD64 reads the extended control register XCR0 (via XGETBV with
|
|
// ECX=0), returning its low and high halves.
|
|
func xgetbvAMD64() (lo, hi uint32)
|
|
|
|
func detectAVX2() bool {
|
|
const (
|
|
fmaBit = 1 << 12 // CPUID.1:ECX.FMA
|
|
osxsaveBit = 1 << 27 // CPUID.1:ECX.OSXSAVE
|
|
avxBit = 1 << 28 // CPUID.1:ECX.AVX
|
|
f16cBit = 1 << 29 // CPUID.1:ECX.F16C
|
|
avx2Bit = 1 << 5 // CPUID.7.0:EBX.AVX2
|
|
xcr0YMM = 1<<1 | 1<<2
|
|
)
|
|
maxLeaf, _, _, _ := cpuidAMD64(0, 0)
|
|
if maxLeaf < 7 {
|
|
return false
|
|
}
|
|
_, _, c1, _ := cpuidAMD64(1, 0)
|
|
if c1&osxsaveBit == 0 || c1&avxBit == 0 || c1&fmaBit == 0 || c1&f16cBit == 0 {
|
|
return false
|
|
}
|
|
if lo, _ := xgetbvAMD64(); lo&xcr0YMM != xcr0YMM {
|
|
return false
|
|
}
|
|
_, b7, _, _ := cpuidAMD64(7, 0)
|
|
return b7&avx2Bit != 0
|
|
}
|
|
|
|
func detectAVX512VNNI() bool {
|
|
const (
|
|
osxsaveBit = 1 << 27 // CPUID.1:ECX.OSXSAVE
|
|
avx512fBit = 1 << 16 // CPUID.7.0:EBX.AVX512F
|
|
avx512bwBit = 1 << 30 // CPUID.7.0:EBX.AVX512BW
|
|
avx512vlBit = 1 << 31 // CPUID.7.0:EBX.AVX512VL
|
|
vnniBit = 1 << 11 // CPUID.7.0:ECX.AVX512_VNNI
|
|
// XCR0: SSE+YMM plus the AVX-512 opmask/upper-ZMM/hi16-ZMM state.
|
|
xcr0AVX512 = 1<<1 | 1<<2 | 1<<5 | 1<<6 | 1<<7
|
|
)
|
|
if !HasAVX2 {
|
|
return false
|
|
}
|
|
_, _, c1, _ := cpuidAMD64(1, 0)
|
|
if c1&osxsaveBit == 0 {
|
|
return false
|
|
}
|
|
if lo, _ := xgetbvAMD64(); lo&xcr0AVX512 != xcr0AVX512 {
|
|
return false
|
|
}
|
|
_, b7, c7, _ := cpuidAMD64(7, 0)
|
|
if b7&avx512fBit == 0 || b7&avx512bwBit == 0 || b7&avx512vlBit == 0 {
|
|
return false
|
|
}
|
|
return c7&vnniBit != 0
|
|
}
|