mirror of
https://github.com/hanzoai/wasm2go.git
synced 2026-10-30 15:06:48 +00:00
f2b99fe737
* gcasm: AVX2 8-bit dot selection for amd64 with runtime dispatch The amd64 SIMD splices lowered the quantized integer dot literally in SSE — per 16-byte chunk, sign-extend each half of each operand to i16, PMADDWD each half, PADDD the two: ~9 xmm ops. AVX2 does the whole chunk in one 256-bit pass: VPMOVSXBW sign-extends all 16 bytes to i16 across a ymm, VPMADDWD yields 8 i32 lanes (0..3 the low-half dot, 4..7 the high), and VEXTRACTI128 + VPADDD fold the high 128 onto the low. int8*int8 fits i16 and the pair sums fit i32, so nothing saturates and every lane is bit-identical to the SSE lowering — five ops replace nine. x64Dot8Rewrite mirrors the arm64 a64Dot8Rewrite: at splice time it collapses a matched add(dot_low, dot_high) chunk into the synthetic x64OpBlockDot reading the two byte sources directly, eliding the six absorbed nodes. The x64 core emits the AVX2 sequence and one VZEROUPPER per region on the exit path. AVX2 is not the amd64 baseline (SSE4.1 / x86-64-v2 is), so it is gated at runtime, not by build tag: the amd64 archSpec entry now carries the "// avx2 dot" marker, so a body using it is emitted twice — an <fn>avx2 feature body and an <fn>sse portable twin (transformed with PortableSIMD, keeping the literal SSE dot) — behind a NOSPLIT tail-jump stub that branches on base.HasAVX2. cpufeat_amd64 detects AVX2 with the standard CPUID/XGETBV probe (OSXSAVE + AVX + XCR0 YMM + AVX2), dependency-free like the arm64 CPUDotProd detection. Unit tests cover the rewrite (fires, portable suppresses, rejects a non-single-use dot) and the emission (AVX2 mnemonics + VZEROUPPER vs the literal SSE twin). Execution correctness is validated by the amd64 CI differential/loop suites and downstream by the llama q8 token-match tests. Co-Authored-By: Claude Opus 4.8 <[email protected]> * gcasm: x64 fused-region memory addressing diet Member loads inside fused regions and loops paid per-access Module loads: every load re-fetched m.M (and every bounds check re-fetched the memSize pointer and dereferenced it), then added the base by hand — 11 instructions for one checked 16-byte load, ~60% of the q8 kernel's fused-loop body (profile: that kernel is 76% of amd64 tg). Three changes, all x64-emitter-local: - Loads finish with one indexed MOVOU (base)(R12*1) instead of the ADDQ base + plain MOVOU pair. - x64HoistMem pins m.M and the memSize VALUE in free argument-tail registers once per fused region or loop (splices contain no calls, so memory cannot grow and both stay valid); member emitters then address through the pinned registers and bounds-check with a single CMPQ. Regions with one memory op keep the per-site loads. - x64FusedEA64/x64FusedLaneAddr return the base register instead of fixing AX, so lane inserts ride the pinned base too. Co-Authored-By: Claude Opus 4.8 <[email protected]> * gcasm: amd64 fast-math 2x2 tile kernel (nrc2 row pairing) The vec_dot row/column pairing (nrows=2) already restructures the GEMM driver on every backend, but only arm64 had a native tile kernel behind it (SMMLA, pp +26% measured); the amd64 feature body kept the 4-subcall bit-exact companion, reloading every quant vector twice. x64Nrc2Kernel is the AVX2 analog: per 34-byte block the four quant vectors (rows x0/x1, cols y0/y1) load once and feed all four dots — sign-extend to i16x16 halves, VPMADDWD the four pairs, and collapse the four i32x8 sums into the 2x2 tile in one xmm with three VPHADDD and a 128-fold. The f16 d scales gather via VPINSRW, convert with VCVTPH2PS, spread with two VPERMILPS, and accumulate with VFMADD231PS. Entirely VEX-encoded with VZEROUPPER on both exits (the Intel dirty-upper hazard), bounds-checked like the arm64 kernel, both pointer widths. Like the arm64 kernel the retarget is fast-math-only (FMA regroups the float accumulate; the integer tile is exact). HasAVX2 detection now also requires FMA and F16C — every AVX2 CPU shipped has both, and the tile kernel uses them. Tests: kernel shape (loads once, 8 VPMADDWD, 6 bounds checks, VZEROUPPER on every exit, ILP32 layout) and real-assembler acceptance of the emitted body (GOOS=linux GOARCH=amd64 go build). Co-Authored-By: Claude Opus 4.8 <[email protected]> * gcasm: VEX direct-destination forms for hot fused vector ops The canned SSE paste stages every operand into X0/X1 and parks every result back out with MOVDQU — the parking copies are the hottest instructions in both decode and prompt profiles. Ops with a VEX three-operand register form (add/sub/mul families, cvtdq2ps, maddwd) now read their sources where they live and write the pool register directly, eliminating both copies. VEX-128 zeroes the ymm uppers, so no dirty-upper hazard; the portable SSE twin keeps the canned paste (VEX is above its baseline). Co-Authored-By: Claude Opus 4.8 <[email protected]> * gcasm: hoist strided window checks out of x64 fused loops A pretest fused loop with a power-of-two step runs exactly counter>>log2(Dec) iterations, so one loop-head check of base + rlo + span + (iters-1)*delta against memSize covers every iteration's window; the in-loop range-checked loads then collapse to single indexed instructions (the memory64 folded form). Trapping at the loop head instead of the precise access is the relaxation the windowed range check already makes. Guards: memory64 trees, constant strides, non-negative rlo, a zero-iteration skip, and MOVL zero-extension of 32-bit counters. Co-Authored-By: Claude Opus 4.8 <[email protected]> * gcasm: fully-VEX fused regions with one deferred upper clear Two wins over the previous emission, found by reading the hot loop: the folded unchecked loads still carried their dead two-instruction address computation (the fold decision ran after base materialization — now it short-circuits at the top of the load emitter), and every block dot paid a VZEROUPPER inside the loop. The remaining legacy-SSE emissions on the fused path (vector copies, the float pack, splats, load tails, the scalar chain's FP ops) now emit their VEX forms, and when a rewritten tree's emission is entirely VEX (x64TreeVexClean) the per-dot clears collapse into a single VZEROUPPER at the region or loop exit — VEX-128 is safe under dirty uppers, so the Intel hazard needs clearing only before returning to gc-emitted legacy code. Co-Authored-By: Claude Opus 4.8 <[email protected]> --------- Co-authored-by: Claude Opus 4.8 <[email protected]>
100 lines
3.2 KiB
Go
100 lines
3.2 KiB
Go
package codegen
|
|
|
|
import (
|
|
_ "embed"
|
|
"strings"
|
|
)
|
|
|
|
// The SIMD helper implementation set. Unlike the name-filtered helpers in
|
|
// helpers.go, these ship as whole files: the pure lane ops have per-arch
|
|
// native bodies (simd_asm_*.s) selected by build tags. The asm arches only
|
|
// compile the scalar bodies their assembly does not cover
|
|
// (simd_scalar_rest_<arch>.go, a few KB) — the full reference set in
|
|
// simd_scalar.go is tagged out there, since keeping generated Go small is
|
|
// the whole point of the asm backend. tools/gen-simd-asm regenerates all of
|
|
// them.
|
|
|
|
//go:embed helpers/simd_scalar.go
|
|
var simdScalarSrc string
|
|
|
|
//go:embed helpers/simd_pair.go
|
|
var simdPairSrc string
|
|
|
|
//go:embed helpers/simd_fallback.go
|
|
var simdFallbackSrc string
|
|
|
|
//go:embed helpers/simd_asm_decls_arm64.go
|
|
var simdAsmDeclsArm64Src string
|
|
|
|
//go:embed helpers/simd_asm_decls_amd64.go
|
|
var simdAsmDeclsAmd64Src string
|
|
|
|
//go:embed helpers/simd_scalar_rest_arm64.go
|
|
var simdScalarRestArm64Src string
|
|
|
|
//go:embed helpers/simd_scalar_rest_amd64.go
|
|
var simdScalarRestAmd64Src string
|
|
|
|
//go:embed helpers/simd_asm_arm64.s
|
|
var simdAsmArm64Src string
|
|
|
|
//go:embed helpers/cpufeat_arm64_darwin.go
|
|
var cpufeatArm64DarwinSrc string
|
|
|
|
//go:embed helpers/cpufeat_arm64_linux.go
|
|
var cpufeatArm64LinuxSrc string
|
|
|
|
//go:embed helpers/cpufeat_arm64_other.go
|
|
var cpufeatArm64OtherSrc string
|
|
|
|
//go:embed helpers/simd_asm_amd64.s
|
|
var simdAsmAmd64Src string
|
|
|
|
//go:embed helpers/cpufeat_amd64.go
|
|
var cpufeatAmd64Src string
|
|
|
|
//go:embed helpers/cpufeat_amd64.s
|
|
var cpufeatAmd64AsmSrc string
|
|
|
|
// appendSimdHelperFiles adds the SIMD helper files to the output tree when
|
|
// the module uses any SIMD helper. In multi-package mode the entry points
|
|
// are exported (simd_ → Simd_) to match helperRef's capitalized cross-
|
|
// package references; the internal utilities (simdToU8, simdConst*, ...)
|
|
// stay package-private either way.
|
|
func (t *translator) appendSimdHelperFiles(files map[string][]byte) {
|
|
if !t.usesSimd {
|
|
return
|
|
}
|
|
dir, pkg := "", t.opts.Package
|
|
if t.multiPackage {
|
|
dir, pkg = "base/", "base"
|
|
}
|
|
goFile := func(src string) []byte {
|
|
src = strings.Replace(src, "package helpers", "package "+pkg, 1)
|
|
if t.multiPackage {
|
|
src = strings.ReplaceAll(src, "simd_", "Simd_")
|
|
}
|
|
return []byte(src)
|
|
}
|
|
asmFile := func(src string) []byte {
|
|
if t.multiPackage {
|
|
src = strings.ReplaceAll(src, "simd_", "Simd_")
|
|
}
|
|
return []byte(src)
|
|
}
|
|
files[dir+"simd_scalar.go"] = goFile(simdScalarSrc)
|
|
files[dir+"simd_pair.go"] = goFile(simdPairSrc)
|
|
files[dir+"simd_fallback.go"] = goFile(simdFallbackSrc)
|
|
files[dir+"simd_asm_decls_arm64.go"] = goFile(simdAsmDeclsArm64Src)
|
|
files[dir+"simd_asm_decls_amd64.go"] = goFile(simdAsmDeclsAmd64Src)
|
|
files[dir+"simd_scalar_rest_arm64.go"] = goFile(simdScalarRestArm64Src)
|
|
files[dir+"simd_scalar_rest_amd64.go"] = goFile(simdScalarRestAmd64Src)
|
|
files[dir+"simd_asm_arm64.s"] = asmFile(simdAsmArm64Src)
|
|
files[dir+"simd_asm_amd64.s"] = asmFile(simdAsmAmd64Src)
|
|
files[dir+"cpufeat_arm64_darwin.go"] = goFile(cpufeatArm64DarwinSrc)
|
|
files[dir+"cpufeat_arm64_linux.go"] = goFile(cpufeatArm64LinuxSrc)
|
|
files[dir+"cpufeat_arm64_other.go"] = goFile(cpufeatArm64OtherSrc)
|
|
files[dir+"cpufeat_amd64.go"] = goFile(cpufeatAmd64Src)
|
|
files[dir+"cpufeat_amd64.s"] = asmFile(cpufeatAmd64AsmSrc)
|
|
}
|