Files
wasm2go/internal/gcasm/simdsplice_fuse_pool_test.go
f2b99fe737 gcasm: AVX2 8-bit dot selection for amd64 with runtime dispatch (#47)
* gcasm: AVX2 8-bit dot selection for amd64 with runtime dispatch

The amd64 SIMD splices lowered the quantized integer dot literally in
SSE — per 16-byte chunk, sign-extend each half of each operand to i16,
PMADDWD each half, PADDD the two: ~9 xmm ops. AVX2 does the whole chunk
in one 256-bit pass: VPMOVSXBW sign-extends all 16 bytes to i16 across a
ymm, VPMADDWD yields 8 i32 lanes (0..3 the low-half dot, 4..7 the high),
and VEXTRACTI128 + VPADDD fold the high 128 onto the low. int8*int8 fits
i16 and the pair sums fit i32, so nothing saturates and every lane is
bit-identical to the SSE lowering — five ops replace nine.

x64Dot8Rewrite mirrors the arm64 a64Dot8Rewrite: at splice time it
collapses a matched add(dot_low, dot_high) chunk into the synthetic
x64OpBlockDot reading the two byte sources directly, eliding the six
absorbed nodes. The x64 core emits the AVX2 sequence and one VZEROUPPER
per region on the exit path.

AVX2 is not the amd64 baseline (SSE4.1 / x86-64-v2 is), so it is gated
at runtime, not by build tag: the amd64 archSpec entry now carries the
"// avx2 dot" marker, so a body using it is emitted twice — an <fn>avx2
feature body and an <fn>sse portable twin (transformed with
PortableSIMD, keeping the literal SSE dot) — behind a NOSPLIT
tail-jump stub that branches on base.HasAVX2. cpufeat_amd64 detects
AVX2 with the standard CPUID/XGETBV probe (OSXSAVE + AVX + XCR0 YMM +
AVX2), dependency-free like the arm64 CPUDotProd detection.

Unit tests cover the rewrite (fires, portable suppresses, rejects a
non-single-use dot) and the emission (AVX2 mnemonics + VZEROUPPER vs
the literal SSE twin). Execution correctness is validated by the amd64
CI differential/loop suites and downstream by the llama q8 token-match
tests.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* gcasm: x64 fused-region memory addressing diet

Member loads inside fused regions and loops paid per-access Module
loads: every load re-fetched m.M (and every bounds check re-fetched
the memSize pointer and dereferenced it), then added the base by hand
— 11 instructions for one checked 16-byte load, ~60% of the q8
kernel's fused-loop body (profile: that kernel is 76% of amd64 tg).

Three changes, all x64-emitter-local:
- Loads finish with one indexed MOVOU (base)(R12*1) instead of the
  ADDQ base + plain MOVOU pair.
- x64HoistMem pins m.M and the memSize VALUE in free argument-tail
  registers once per fused region or loop (splices contain no calls,
  so memory cannot grow and both stay valid); member emitters then
  address through the pinned registers and bounds-check with a single
  CMPQ. Regions with one memory op keep the per-site loads.
- x64FusedEA64/x64FusedLaneAddr return the base register instead of
  fixing AX, so lane inserts ride the pinned base too.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* gcasm: amd64 fast-math 2x2 tile kernel (nrc2 row pairing)

The vec_dot row/column pairing (nrows=2) already restructures the
GEMM driver on every backend, but only arm64 had a native tile
kernel behind it (SMMLA, pp +26% measured); the amd64 feature body
kept the 4-subcall bit-exact companion, reloading every quant vector
twice. x64Nrc2Kernel is the AVX2 analog: per 34-byte block the four
quant vectors (rows x0/x1, cols y0/y1) load once and feed all four
dots — sign-extend to i16x16 halves, VPMADDWD the four pairs, and
collapse the four i32x8 sums into the 2x2 tile in one xmm with three
VPHADDD and a 128-fold. The f16 d scales gather via VPINSRW, convert
with VCVTPH2PS, spread with two VPERMILPS, and accumulate with
VFMADD231PS. Entirely VEX-encoded with VZEROUPPER on both exits (the
Intel dirty-upper hazard), bounds-checked like the arm64 kernel, both
pointer widths.

Like the arm64 kernel the retarget is fast-math-only (FMA regroups
the float accumulate; the integer tile is exact). HasAVX2 detection
now also requires FMA and F16C — every AVX2 CPU shipped has both, and
the tile kernel uses them.

Tests: kernel shape (loads once, 8 VPMADDWD, 6 bounds checks,
VZEROUPPER on every exit, ILP32 layout) and real-assembler
acceptance of the emitted body (GOOS=linux GOARCH=amd64 go build).

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* gcasm: VEX direct-destination forms for hot fused vector ops

The canned SSE paste stages every operand into X0/X1 and parks every
result back out with MOVDQU — the parking copies are the hottest
instructions in both decode and prompt profiles. Ops with a VEX
three-operand register form (add/sub/mul families, cvtdq2ps, maddwd)
now read their sources where they live and write the pool register
directly, eliminating both copies. VEX-128 zeroes the ymm uppers, so
no dirty-upper hazard; the portable SSE twin keeps the canned paste
(VEX is above its baseline).

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* gcasm: hoist strided window checks out of x64 fused loops

A pretest fused loop with a power-of-two step runs exactly
counter>>log2(Dec) iterations, so one loop-head check of
base + rlo + span + (iters-1)*delta against memSize covers every
iteration's window; the in-loop range-checked loads then collapse to
single indexed instructions (the memory64 folded form). Trapping at
the loop head instead of the precise access is the relaxation the
windowed range check already makes. Guards: memory64 trees, constant
strides, non-negative rlo, a zero-iteration skip, and MOVL
zero-extension of 32-bit counters.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* gcasm: fully-VEX fused regions with one deferred upper clear

Two wins over the previous emission, found by reading the hot loop:
the folded unchecked loads still carried their dead two-instruction
address computation (the fold decision ran after base
materialization — now it short-circuits at the top of the load
emitter), and every block dot paid a VZEROUPPER inside the loop.
The remaining legacy-SSE emissions on the fused path (vector copies,
the float pack, splats, load tails, the scalar chain's FP ops) now
emit their VEX forms, and when a rewritten tree's emission is
entirely VEX (x64TreeVexClean) the per-dot clears collapse into a
single VZEROUPPER at the region or loop exit — VEX-128 is safe under
dirty uppers, so the Intel hazard needs clearing only before
returning to gc-emitted legacy code.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

---------

Co-authored-by: Claude Opus 4.8 <[email protected]>
2026-08-20 16:32:40 +09:00

127 lines
4.2 KiB
Go
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
package gcasm
import (
"errors"
"regexp"
"strings"
"testing"
"github.com/goccy/wasm2go/internal/simdfuse"
)
// fusePoolClobberTree keeps a loaded vector (n0) live across a canned
// pair-table op (i8x16_shuffle) — the rotate-window shape (RoPE) that
// exposed the pool/scratch overlap: canned op bodies scratch X0–X7,
// so a value parked there was silently corrupted and every consumer
// after the shuffle computed garbage.
func fusePoolClobberTree() *simdfuse.Tree {
return &simdfuse.Tree{
Name: "simd_p_fx2",
NumScalars: 2,
NumPairs: 1,
NeedsMem: true,
Nodes: []simdfuse.Node{
{Op: "v128_load", Args: []simdfuse.Arg{
{Kind: simdfuse.ArgScalar, Index: 0},
{Kind: simdfuse.ArgConst, Const: 0},
}},
{Op: "v128_load", Args: []simdfuse.Arg{
{Kind: simdfuse.ArgScalar, Index: 1},
{Kind: simdfuse.ArgConst, Const: 0},
}},
{Op: "v128_load", Args: []simdfuse.Arg{
{Kind: simdfuse.ArgScalar, Index: 1},
{Kind: simdfuse.ArgConst, Const: 16},
}},
{Op: "i8x16_shuffle", Args: []simdfuse.Arg{
{Kind: simdfuse.ArgNode, Index: 1},
{Kind: simdfuse.ArgNode, Index: 2},
{Kind: simdfuse.ArgPairIn, Index: 0},
}},
{Op: "f32x4_mul", Args: []simdfuse.Arg{
{Kind: simdfuse.ArgNode, Index: 0},
{Kind: simdfuse.ArgNode, Index: 3},
}},
},
Roots: []int{4},
}
}
// A value parked in X4–X7 that must survive a canned pair-table op
// (whose pasted body uses X0–X7 as inputs and scratch) is relocated
// to X8+ before the paste, and every later consumer reads the new
// home. Without this, the shuffle's scratch writes corrupted the
// parked load and every consumer after it computed garbage.
func TestX64FusedCannedOpRelocatesLiveValues(t *testing.T) {
var b strings.Builder
spliced, _, err := x64SpliceFused(&b, fusePoolClobberTree(), &ConstPool{}, fuseTestOffs, false, 0)
if err != nil || !spliced {
t.Fatalf("spliced=%v err=%v", spliced, err)
}
body := b.String()
// n0 parks in X4 (first pool register); the shuffle must move it
// to X8+ before its canned lines run.
if !regexp.MustCompile(`VMOVDQU X4, X(8|9|1[0-4])\b`).MatchString(body) {
t.Fatalf("live value not relocated out of canned scratch:\n%s", body)
}
// The mul consumes the relocated home, never stale X4.
if strings.Contains(body, "VMOVDQU X4, X0") {
t.Fatalf("consumer read the clobbered X4 park:\n%s", body)
}
}
// A tree over the shrunken pool budget must surface errFusedCapacity
// (the transform then falls the whole function back to pure Go) —
// never a silent partial splice.
func TestX64FusedPoolCapacity(t *testing.T) {
// Twelve loads all live until a final chain of muls: more parked
// values than the X4..X14 pool can hold.
tree := &simdfuse.Tree{
Name: "simd_p_fx3",
NumScalars: 1,
NeedsMem: true,
}
const n = 12
for i := 0; i < n; i++ {
tree.Nodes = append(tree.Nodes, simdfuse.Node{Op: "v128_load", Args: []simdfuse.Arg{
{Kind: simdfuse.ArgScalar, Index: 0},
{Kind: simdfuse.ArgConst, Const: int32(16 * i)},
}})
}
prev := 0
for i := 1; i < n; i++ {
tree.Nodes = append(tree.Nodes, simdfuse.Node{Op: "f32x4_mul", Args: []simdfuse.Arg{
{Kind: simdfuse.ArgNode, Index: prev},
{Kind: simdfuse.ArgNode, Index: i},
}})
prev = len(tree.Nodes) - 1
}
tree.Roots = []int{prev}
var b strings.Builder
spliced, _, err := x64SpliceFused(&b, tree, &ConstPool{}, fuseTestOffs, false, 0)
if spliced {
t.Fatal("over-budget tree spliced")
}
if err == nil || !errors.Is(err, errFusedCapacity) {
t.Fatalf("want errFusedCapacity, got %v", err)
}
}
// Registrations from a transform attempt that later fell back must
// leave the jump-table spec set: their stubs are absent from the
// emitted asm, and a stale spec sends the generated init's signature
// scan past the end of the text segment.
func TestJTTableDrop(t *testing.T) {
jt := &JTTable{}
jt.fn("Fn1").Sites = append(jt.fn("Fn1").Sites, JTSite{TabVar: "Fn1_jt0"})
jt.fn("Fn2").Sites = append(jt.fn("Fn2").Sites, JTSite{TabVar: "Fn2_jt0"})
jt.Drop("Fn1")
if len(jt.Fns) != 1 || jt.Fns[0].Sym != "Fn2" {
t.Fatalf("Drop(Fn1) left %+v", jt.Fns)
}
jt.Drop("Fn2")
if got := jt.EmitGo("amd64"); got != "" {
t.Fatalf("empty table emitted Go:\n%s", got)
}
}