Files
wasm2go/internal/gcasm/simdsplice_fuse_loop_thresh_test.go
743b14bbfa Fuse goto-form countdown loops with per-iteration scale chains (#52)
* asmgen: emit the branch-based EH ops in direct-asm bodies

Functions containing exception handling fell back to the listing
transform because the direct emitters rejected every exc op. The
branch-based EH state is plain Module fields, so the whole family
lowers to inline MOVs against the module pointer, mirroring the
inlined-global path on both architectures:

- OpExcPending / OpExcTag / OpExcVal: loads into the value's slot
  (OpExcVal narrows the uint64 operand slot by loading the low
  bytes, or the raw bits for floats, at the slot's base address.
- OpExcRaise: store the tag, widen each operand into its slot
  (zero-extend for 32-bit values, raw bits for floats), set the
  pending flag.
- OpExcRearm / OpExcClear: flag stores.
- OpCatchArg: narrow the dispatch's int64-typed snapshot.

The exception-state fields move into the Module struct's
fixed-offset leading region (right after the outline pack) so the
offsets stay trivially modeled instead of requiring the whole
struct tail: codegen computes them (moduleExcOffsets), they ride
Result.DirectAsmExc -> gcasm.Config -> asmgen FuncOptions.Exc, and
the generated compile-time layout pins assert them — including,
now, for memory-less modules. The bundle passes Config to
emitDirectAsmBody directly: offsets previously rode the SIMD memory
probe, which does not exist for modules without SIMD memory ops.

Two adjacent defects fixed along the way:
- The pure-Go emitter produced math.Float64frombits(int64) — a
  compile error — when a catch handler read an f64 operand, because
  narrowExcSlot assumed a uint64-typed slot while OpCatchArg feeds
  it the dispatch's int64 snapshot.
- i64_reinterpret_f64 / f64_reinterpret_i64 were missing from the
  always-inline helper set, forcing a per-function fallback under
  ForbidCalls; amd64 already carried (dead) inline coverage and
  arm64 gains the two-instruction FMOV lowering.

New gates: an EH fixture whose seven functions all emit direct-asm
on both architectures, verified differentially against the normal
backend plus pinned values across all four operand widths, executed
on amd64 and natively on arm64; a unit test pins the offset model.

Co-Authored-By: Claude Fable 5 <[email protected]>
EOF
)

* helpers: fix memoryInit64 disabling shared-image CoW for threads builds

memoryInit64 saturated dataEnd to MaxUint32 on EVERY install - a
leftover from before memory64 images were shareable. A threads build
makes every data segment passive (__wasm_init_memory installs them
via memory.init at start-up), so the very first install saturated the
extent and the sharing layer refused the image: every wasm64-threads
engine silently lost copy-on-write model sharing and fell back to
private loads.

Raise dataEnd to the real extent like the 32-bit memoryInit, and
saturate only when a segment genuinely lands beyond the uint32
bookkeeping range. Also adopt the 32-bit variant's compare-then-write
path so an image-backed instance re-running the start section over a
copy-on-write map does not fault identical pages private.

Co-Authored-By: Claude Fable 5 <[email protected]>

* codegen: carry the source SSA value on SIMD call marks

The fusion pass will use it to report each fused window's member and
root values back to direct-asm retention, so the asm emitter can
splice the same windows straight from the retained SSA instead of
re-deriving the match.

Co-Authored-By: Claude Fable 5 <[email protected]>

* codegen: record SSA provenance for fused-signature parameters

Each fused-tree parameter (scalar/float/pair) now carries where it
came from: a compile-time constant, or a member SSA value's argument
position. Chased compound expressions and named-local sum bases get
no provenance — a window carrying one will keep per-op splices in
direct-asm bodies rather than guess. The chase-rollback path
truncates the provenance array in lockstep with the parameters.

Pure bookkeeping: fusion decisions and emitted output are unchanged;
the direct-asm window retention that consumes this lands next.

Co-Authored-By: Claude Fable 5 <[email protected]>

* codegen: track member SSA values per fused-tree node

nodeVals parallels the builder's node list with the SSA value each
member call was built from (nil for synthesized scalar-chain nodes),
remapped through scheduleNodes and rolled back by restoreChase in
lockstep. Together with the parameter provenance this gives window
retention everything it needs to name a window's members, roots and
arguments in the retained SSA.

Co-Authored-By: Claude Fable 5 <[email protected]>

* codegen: report committed fused windows to direct-asm retention

At both fusion commit points (single-tree and window), a retained
function now gets a DirectAsmWindow descriptor: the interned tree,
the member and root SSA values, and the fused signature's parameter
sources. Recording is strict — any parameter without provenance or
root without a member value skips the window (FuseDebug logs why),
leaving the direct body on per-op splices.

Verified against the llama bundle: the mechanism records windows
with full provenance end to end (23 windows in one retained
function); the asm-bundle consumer lands next.

Co-Authored-By: Claude Fable 5 <[email protected]>

* asmgen: plumb fused-window descriptors to the direct-asm emitters

Canonical FusedWindow/FusedParamSrc types live in asmgen; transpile
converts the codegen retention's descriptors, gcasm carries them on
DirectAsmFn, and emitDirectAsmBody hands them to FuncOptions. No
consumer yet — emission still takes the per-op path for every
window member.

Co-Authored-By: Claude Fable 5 <[email protected]>

* asmgen: define the optional FusedSplicer interface

Per-window splice entry point the emitters will call: operands in
fused-signature order as immediates or frame slots, root result
slots for the epilogue, false to keep per-op emission. Type-asserted
from the existing Splicer so per-op-only implementations are
unaffected.

Co-Authored-By: Claude Fable 5 <[email protected]>

* asmgen: plan fused windows against the retained SSA

planFusedWindows validates each codegen-claimed region — members in
one block and unclaimed, non-root members consumed only inside the
window — and resolves the fused signature's parameter sources to
frame-slot/immediate operands plus root slot targets. Any mismatch
is a quiet per-op fallback. Emission consumption lands next.

Co-Authored-By: Claude Fable 5 <[email protected]>

* asmgen: emit validated fused windows at their last member

The whole window body emits at the LAST member — every interleaved
scalar input is materialized by then — and earlier members emit
nothing. Plan validation makes the deferral observation-free: values
between the members must be pure scalar glue (no memory ops, no
effects, no reads of a root), non-root members must have no
consumers outside the window, and planning only runs when the
splicer implements FusedSplicer. A splicer decline at emission is a
per-function fallback to the listing transform, never a half-emitted
body.

Co-Authored-By: Claude Fable 5 <[email protected]>

* gcasm: emit fused-window bodies inside direct-asm functions

SpliceFused stages the fused ABIInternal signature exactly as gc's
call site would — m in R0 from the emitter's R24 cache, scalars and
pair halves in R1.., float32 leaves in F0.. — drops in the same
prolog+core+epilogue the capture path splices, lands each root's
result register pair in the root value's slot, and re-primes the
m cache and hoisted module state the fused body clobbers.

Two defects the SIMD differential gate caught on the way in:
- The fused pool registers (v16..v30) overlap the direct-asm text
  passes' cache and home ranges, and the passes' line model does
  not understand window interiors: window-bearing bodies skip the
  four passes (a marker comment keys it; integrating the passes
  around windows is tracked separately).
- Unpacked parameters are FP-resident and their RSP slots are never
  written: operand resolution now emits FP-relative references for
  them, exactly like the per-op operand resolvers, instead of
  reading an unwritten slot.

WASM2GO_FUSEDWIN_MAX (env) caps windows per function for bisection.

Co-Authored-By: Claude Fable 5 <[email protected]>

* asmgen: allow interleaved plain loads across load-only fused windows

The codegen fusion proved exactly this hoist when it claimed a
region past scalar loads (f16 scale reads sit between the gather
members in every hot attention window); a window that writes no
memory can be deferred past them without changing any observation.
Storing windows still reject interleaved loads. Plan rejects now
report their reason under WASM2GO_FUSEDWIN_DEBUG.

Co-Authored-By: Claude Fable 5 <[email protected]>

* asmgen: pin window operands and disable splice coalesce around windows

Two llama-scale correctness fixes the NLL parity gate demanded:
window operand sources keep dedicated slots (their reads happen at
the window's deferred emission point, past the reuse model's
lifetimes), and the splice coalesce pass is off for window-bearing
bodies — a coalesced v128 loop carry lives in a home register and
never materializes the slot the window stages from, and the fused
pool clobbers the home ranges regardless.

With these the full go-llama suite (native NLL parity included)
passes on a bundle emitting the hottest attention function through
five fused windows. The configuration is slower than the gc path
for now: skipping the text passes and the coalesce costs the
function's per-op remainder more than the windows gain, so the
feature stays opt-in while pass integration around windows lands.

Co-Authored-By: Claude Fable 5 <[email protected]>

* gcasm: run the text passes around fused windows instead of skipping

The register-rewriting passes (slot forwarding, spare-register cache)
operate per outside-window segment with state flushed at each window
boundary; the read-counting dead-store pass runs whole-body so a
store whose only reader sits inside a window survives; windows are
bracketed by begin/end marker comments and passed through verbatim.

Full go-llama gates (NLL parity included) pass on the hottest
attention function emitted with five fused windows. Perf remains
behind the gc path for the whole function (the direct body's per-op
remainder still lacks gc-grade register allocation), so the feature
stays opt-in.

Co-Authored-By: Claude Fable 5 <[email protected]>

* gcasm: convert f16 scales with FCVT instead of the f32 table lookup

The guest code converts f16 scale factors through ggml's f32 table
(load16_u -> <<2 -> + table base -> f32 load); the arm64 host has
hardware half-to-single conversion, so the fused scalar-chain
splicer now emits one H load plus FCVT where it recognized the
lookup, dropping the dependent second load. The conversion is
bit-exact (IEEE f16->f32 is exact, which is precisely what the
table encodes).

Method per the asm-first rule: the sequence was first patched by
hand into a built bundle and measured (~6% faster tg on the q8_0
routing bench, NLL parity green) before being wired into
generation; the generated form reproduces the measurement (80 sites
across the dot kernels and the attention windows vs 40 in the hand
patch). A companion hand experiment (eliding the mov feeding scvtf)
measured no additional win and is intentionally not implemented.

Co-Authored-By: Claude Fable 5 <[email protected]>

* codegen: rewrite f16 table gathers to vector loads in fused windows

The guest converts f16 values (attention KV, activations) through
ggml's f32 lookup table one element at a time: scalar load16_u
reads feed shifted indices into a load32_zero + 3x load32_lane
chain over the table — eight dependent loads per four elements.
The sources are consecutive halfwords, so the whole group equals
f16x4_cvt(v128_load16x4_u(base)): one D-register load plus FCVTL,
the exact shape the native kernel compiles to.

The SSA-level gather pass cannot claim this form (the shift amount
rides a single-assignment local, constant only through definition
chasing), so the normalization runs on the built fused tree, where
the chase machinery lives: each lane's index expression is chased
to a scalar load16_u, lane adjacency (+0,+2,+4,+6) is proven
structurally with the delta carried by one constant arm
(argOffsetBy), the table is verified byte-for-byte as the IEEE
conversion, and the chain is rewritten with dead nodes and retired
parameters compacted. Supporting pieces: a function-wide
single-assignment definition map consulted by the chase (gated to
this rewrite so ordinary fusion keeps byte-identical shapes),
canonical base+const folding under the same gate, a wider
shift-amount conversion unwrap, and start-section placement
recovery for passive data segments so the table verification works
on shared-memory (threads) links, whose segments are all passive.

Method per the asm-first rule: the ldr d + fcvtl form was first
patched by hand into a built bundle and measured (~3.5% on the
routing bench, NLL parity green) before this generation landed;
the generated form measures 9-14% faster than its baseline —
beyond the hand patch, because retiring the gather also retires
the per-element scalar load glue the hand patch left running.
Full suite, lint, and the go-llama gates (native NLL parity
included) are green; three of the four hot attention windows
rewrite, the fourth keeps the literal gather pending a lane-order
generalization.

Co-Authored-By: Claude Fable 5 <[email protected]>

* codegen: recognize word-unpack and offset-node f16 gather variants

The f16 gather rewrite only matched groups whose four table indices
chase to separate load16_u reads at literal base+0/+2/+4/+6. The
attention kernel's hottest gather takes two other shapes:

- Word-unpack: the guest loads four f16 values as one 64-bit word
  and derives each table index by shift and mask - lane 0 as
  (w & 0xffff) << 2 and lanes 1..3 as (w >>u 16k-2) & 0x3fffc plus
  the table base, the strength-reduced form clang emits. The mask
  after the pre-scaled shift selects exactly halfword k times 4, so
  the group equals the same f16x4_cvt(v128_load16x4_u(base)).
- Offset nodes: lane addresses that chase to base+const add NODES
  rather than scalar/sum forms; argOffsetBy now peels constant add
  arms on either side, carrying the delta arithmetically.

Chase additions, all gated on the gather pass so ordinary fusion
keeps byte-identical shapes: identity-mask elimination (load16_u
masked with low-16-ones), and bare *(*uint16) derefs assigned
straight to wider locals.

Measured on the routing benchmark (single thread, interleaved A/B,
two rounds): decode 14.4/14.6 -> 16.6/16.8 tok/s (+15%). Full
transpiler suite, golangci-lint, and the dependent go-llama suite
(token-parity and NLL gates included) all pass; rewritten-group
count rises from 55 to 97 with no rejected groups left in the
attention kernel.

Co-Authored-By: Claude Fable 5 <[email protected]>

* gcasm: carry host pointers in fused-loop splices (arm64)

Fused loops addressed every memory operand as mBase+scalar+const,
re-materializing the effective address in scratch per access. The
q8_0 SDOT decode kernel spends its time in exactly these loads, so
the splicer now carries a host pointer (mBase + bump-carried address
scalar) in a free argument register, recomputed at the top of each
unroll copy, and the load emitters use immediate-offset forms off it
with a running bias that steps the pointer when an offset leaves the
signed-9-bit window. The coalesced range checks fold to one compare
off the guest scalar with the same reassociated bound as before. The
f16-scale peephole reads its half-word through the carried pointer
the same way. Straight-line window splices are untouched, as is
amd64; WASM2GO_NO_HOSTPTR disables the fold for A/B runs.

Proven by hand-editing the generated kernel to the native shape
first (per the asm-first rule): the hand patch measured +5.5% on
single-thread decode, and this deterministic form reproduces it —
routing benchmark decode 16.8 -> 17.8 tok/s (+5-6%, two interleaved
rounds, all routes exact). A prefetch variant measured negative and
a bare address-glue fold measured neutral; neither is included.
Full transpiler suite, golangci-lint, and the dependent go-llama
suite (token-parity and NLL gates) all pass.

Co-Authored-By: Claude Fable 5 <[email protected]>

* codegen: batch vec_dot caller row loops through a companion (opt-in)

The mul_mat drivers call the verified vec_dot once per output row
through the indirect table; per-call machinery (frame, module-state
reloads, dispatch, reduction epilogue) measured ~20% of single-
thread decode on the reference workload. With the new -vec-dot-rows
flag (requires -vec-dot-pair-entry) the translator now:

- emits a row-looped companion of the verified vec_dot: its emitted
  body is cloned (print/reparse round-trip), returns rewrite to a
  continue label, parameters become fresh per-row copies so mutable
  locals and wasm zero-init semantics hold, and the whole loop runs
  in one call frame. The boundary is packed through the per-module
  outline scratch (m + nine scalars exceeds the amd64 register
  ABI), and the companion rides the synthetic-signature channel so
  the asm bundle transforms it like an outlined function — the
  fused-loop splices apply inside the row loop.
- rewrites matching driver row loops (indirect call + mBase reload
  + counter step + guarded backedge with pointer bumps) into a
  GUARDED one-call fast path; the original loop remains as the
  guard-miss branch, so semantics are preserved for every runtime
  table target, and modules without the flag are untouched.

The traits scan learns passive data segments (threads links emit
only those) via the start-section placements, gated on the same
flag; the nrows patch that switches the guest to paired two-row
calls is suppressed under it, since batching needs the drivers to
keep stepping one row per call.

Measured on the routing benchmark (single thread, interleaved A/B,
two rounds, all routes exact): decode 17.2 -> 18.6 tok/s (+8%).
Full transpiler suite, golangci-lint, and the dependent go-llama
suite (token-parity and NLL gates) all pass; unit tests cover the
loop matcher, the return rewriter and the shift-count reduction.

Co-Authored-By: Claude Fable 5 <[email protected]>

* gcasm: feature-gate synthetic bodies like wasm functions

The feature-gated twin split (SDOT/AVX2 body + baseline twin +
dispatch stub) excluded synthetic-signature functions, which was a
no-op while outlined extraction bodies never contained gated
instructions. The batched-rows companion clones a kernel whose body
selects those forms, so its plain symbol would fault on a baseline
CPU. Synthetic bodies now take the same split; the twin
declarations reuse the original name's (packed) signature.

Verified: the companion emits dotprod/generic twins plus the
dispatch stub; full transpiler suite, golangci-lint, the go-llama
suite and the routing benchmark (routes exact, throughput unchanged
within noise) all pass.

Co-Authored-By: Claude Fable 5 <[email protected]>

* codegen/gcasm: fuse goto-form countdown loops with per-iteration scale chains

Decode-critical mul_mat hosts inline their kernel loops as goto-form
countdowns (label + backedge/exit jumps), which the loop upgrade
previously skipped; and their per-iteration f16 scale loads reached the
fused window as opaque scalar arguments, freezing the entry-time value
for the whole loop. This lands the fixes that together let those loops
fuse correctly:

- Upgrade goto-form loops by synthesizing a ForStmt (label preserved as
  the live entry target, exit jump kept); on by default with
  WASM2GO_GOTO_FUSE_MAX as a bisection cap.
- Chase computed (non base+const) memory addresses into scalar node
  chains under the loop-trial wide chase, so LUT indexes like idx<<2
  with the table constant in the memarg offset are recomputed per
  iteration instead of frozen.
- Narrow (wideload >> C) & 0xffff into a 16-bit load at addr + C/8
  (little-endian), matching clang packing adjacent f16 scale reads
  into one wide load.
- Recurse identWrites through nested statements so loop-variant writes
  inside the backedge branch are seen by the intervener consumption
  check; unabsorbed chains now refuse the upgrade instead of
  miscomputing.
- Countdown-to-threshold exits (SUBS/CSEL floor form), reassociation of
  exactly-associative vector chains, cross-chain raw SDOT rewrite with
  dual alternating accumulators, whole-loop bounds-check hoisting for
  store-free loops, and const-pattern shuffles without ABI slots.
- Deterministic carried-pointer register assignment (map order leaked
  into emitted bytes).

Co-Authored-By: Claude Fable 5 <[email protected]>

* codegen: unit-test the halfword-narrowing chase; fix its shift semantics

The extract rewrite resolved shift amounts through matchShiftConst,
whose default mod-32 reduction encodes i32 shift semantics — a bare
64-bit lane offset (>> 32, >> 48) would wrap to the wrong byte
offset, and an out-of-range extract could pass the width check.
Resolve the amount locally with no implicit reduction (the % 64
spelling of memory64 emission reduces identically), and unwrap
ParenExpr so the match does not depend on the AST provenance.

Adds unit coverage for the narrowing chase (lane offsets, wrapped
masks, operand order, rejections), the recursive identWrites, and
the carried-pointer register assignment order.

Co-Authored-By: Claude Fable 5 <[email protected]>

* codegen: bound the shuffle-pattern word split explicitly

The four u32 lane words of a const shuffle pattern derive from the
parsed 64-bit halves; mask each split explicitly so the truncation
is a stated bound rather than an implicit narrowing (flagged by
code scanning as an unchecked 64-to-32 conversion).

Co-Authored-By: Claude Fable 5 <[email protected]>

* codegen: gofmt the shuffle-pattern split

Co-Authored-By: Claude Fable 5 <[email protected]>

* Remove diagnostic tracing and environment kill switches

The fused-loop work carried per-site trace logging (chase failures,
candidate dumps, plan-reject reasons) and WASM2GO_* environment
switches used while the upgrade was being isolated. The features are
validated and unconditional now, so the switches and the tracing are
dead weight: drop them, along with the now-unused counter, reject
reason field and statement renderer. Transpiled output is unchanged
(byte-identical bundle before and after).

Co-Authored-By: Claude Fable 5 <[email protected]>

---------

Co-authored-by: Claude Fable 5 <[email protected]>
2026-08-24 12:20:49 +09:00

67 lines
2.1 KiB
Go

package gcasm
import (
"strings"
"testing"
"github.com/goccy/wasm2go/internal/simdfuse"
)
// floorLoop is carryLoop rewritten as a do-while with a signed floor
// exit: repeat while counter > 1 (the rotated count-down-to-K shape).
func floorLoop() *simdfuse.Loop {
l := carryLoop()
l.PreTest = false
l.ExitGT = true
l.ExitThresh = 1
return l
}
// The floor form must exit on a SIGNED compare against the threshold,
// not the exact-zero test: a counter that skips past the floor (or
// starts at/below it) still terminates after the mandatory first
// iteration.
func TestX64LoopFloorBackedge(t *testing.T) {
var b strings.Builder
spliced, _, err := x64SpliceLoop(&b, floorLoop(), &ConstPool{}, fuseTestOffs, "0", false, 0)
if err != nil || !spliced {
t.Fatalf("spliced=%v err=%v", spliced, err)
}
body := b.String()
if !strings.Contains(body, "$1") || !strings.Contains(body, "JGT gcasmfxl0") {
t.Errorf("no signed floor backedge:\n%s", body)
}
if strings.Contains(body, "JNE gcasmfxl0") {
t.Errorf("exact-zero backedge left in floor loop:\n%s", body)
}
}
func TestA64LoopFloorBackedge(t *testing.T) {
var b strings.Builder
spliced, _, err := a64SpliceLoop(&b, floorLoop(), &ConstPool{}, fuseTestOffs, "0", false)
if err != nil || !spliced {
t.Fatalf("spliced=%v err=%v", spliced, err)
}
body := b.String()
if !strings.Contains(body, "CMPW $1,") || !strings.Contains(body, "BGT gcasmfxl0") {
t.Errorf("no signed floor backedge:\n%s", body)
}
if strings.Contains(body, "CBNZW") {
t.Errorf("exact-zero backedge left in floor loop:\n%s", body)
}
}
// A floor on the pre-tested while form has no emitted meaning; the
// splicers must refuse it rather than silently emit the wrong exit.
func TestLoopFloorRejectsPreTest(t *testing.T) {
l := floorLoop()
l.PreTest = true
var b strings.Builder
if _, _, err := x64SpliceLoop(&b, l, &ConstPool{}, fuseTestOffs, "0", false, 0); err == nil {
t.Error("x64: pre-tested floor loop not refused")
}
if _, _, err := a64SpliceLoop(&b, l, &ConstPool{}, fuseTestOffs, "0", false); err == nil {
t.Error("a64: pre-tested floor loop not refused")
}
}