mirror of
https://github.com/hanzoai/wasm2go.git
synced 2026-10-30 15:06:48 +00:00
* Support the memory64 proposal: 64-bit linear memory past 4 GiB A wasm32 module can never use more than 4 GiB no matter what the runtime allows — its pointers, size_t and allocator are compiled as 32-bit values. memory64 (wasm64) is the sound route to large memories, and this adds it end to end for the pure and gcasm outputs: - Parser: limits flags 0x04/0x05 decode the 64-bit index space into MemoryType.Is64 / Module.Memory64; unknown flag bits are rejected. Memarg offsets decode as u64 (wasm32 offsets are unaffected). - Lowering: addresses ride as i64 values; memory.size/grow type as i64; the SIMD memory family lowers to a simd_m64_* helper family with u64 addressing (never rewritten to the pair/splice forms, which assume 32-bit addresses). Atomics on a memory64 are rejected for now — no toolchain produces shared wasm64 today. - Emission: scalar loads/stores keep the zero-cost inline unsafe deref, with the effective address computed as u64 instead of the u32 wrap; large constant totals still route through the _consts table. memory.size/grow/fill/copy/init switch to 64-bit helper twins. The coalesced SIMD bounds check stays wasm32-only (its exactness argument is written against the wasm32 hard cap). - Runtime: memoryGrow64 grows past the wasm32 cap up to a 2^48 hard cap with the same geometric-capacity scheme; a memory64 with no declared maximum defaults to unlimited (maxMem 0) instead of the 4 GiB wasm32 default, and the external-image constructors accept images past 4 GiB. Effective-address checks in the 64-bit helpers guard u64 overflow explicitly — there is no benign wrap at 2^64. The gate proves the point end to end: a memory64 module grows to 4 GiB + 128 KiB and stores/loads at address 1<<32 through the full backend. wasm32 output is byte-identical to before across the fixture corpus, SIMD included. Native asm splicing of the m64 SIMD helpers, the wasm64 WASI ABI, and a wasm64 sysroot in the wasmify toolchain are follow-ups; the llama.cpp chain needs all three before go-llama can load >4 GiB models. Co-Authored-By: Claude Fable 5 <[email protected]> * Bind memory64 modules to a widened wasip1 ABI There is no standardized 64-bit WASI ABI; the wasm2go-ported wasi-libc declares every wasi_snapshot_preview1 import argument at pointer width (all-i64 under wasm64), and this is the runtime side of that contract: - A memory64 module's wasi imports bind to new *64 methods on WasiStubs (codegen appends the suffix for wasi_snapshot_preview1 imports when the module is a memory64); every wasm32 module keeps the standard 32-bit methods and its byte-identical output. - The *64 bindings parse the LP64 layouts — 16-byte iovecs, 64-bit size fields — through a full-range 64-bit memory window, covering the clock/fd/proc calls a C hello-world exercises; the rest of the surface follows as the wasm64 toolchain grows. - Immediate skipping in the call-graph prescan reads memarg offsets as u64, matching the decoder (a memory64 dlmalloc's offsets overflowed the u32 reader). The checked-in wasip1_mem64_hello.wasm fixture — C compiled with clang -target wasm64-wasip1 against the ported wasi-libc — pins printf and malloc end to end; a 5 GiB malloc with stores at 0, 4 GiB and 5 GiB-1 runs correctly through the same chain. Co-Authored-By: Claude Fable 5 <[email protected]> * Cover the llama-wasm WASI surface in the widened wasm64 ABI Extend the *64 wasip1 bindings from the initial six methods to the full import set a wasi-libc-linked C/C++ program (llama.cpp included) uses: args/environ, prestat, path_open, fd_read, fd_readdir, path_filestat_get, fd_fdstat_set_flags, and random_get. Path_open and Fd_readdir keep a single layout-independent core (pathOpen / fdReaddir) shared by both pointer widths; the LP64-layout differences (8-byte guest pointers and __wasi_size_t, 16-byte iovecs, 16-byte prestat) live only in the thin *64 wrappers. Add a checked-in wasm64 C++ exception fixture (typed catches, catch-all, unwinding through std::string/vector/unique_ptr) built with -fwasm-exceptions against the wasm64 libc++/libc++abi/libunwind runtimes, plus unit tests for every new *64 binding. Co-Authored-By: Claude Fable 5 <[email protected]> * Share the layout-independent WASI cores across pointer widths Fold the remaining duplicated bodies between the wasm32 and wasm64 wasip1 bindings into shared cores, so each binding pair keeps only its ABI layout: - clockNanos: the clock_time_get body. - fdSeek: the fd_seek body. - readVec/writeVec: the scatter/gather I/O loops, fed by per-width iovec resolvers (iovecSlices / iovecSlices64). Every iovec entry is now validated before any I/O happens, so a bad entry faults the call instead of surfacing after a partial transfer. No behavior change otherwise; wasm32 and wasm64 fixtures and the WASI unit suites pass unchanged, coverage 83.7% >= 83%. Co-Authored-By: Claude Fable 5 <[email protected]> * Fix three memory64 gaps surfaced by transpiling llama.cpp as wasm64 Verified by rebuilding the full llama.cpp bridge for wasm64-wasip1 and running the go-llama test suite on the transpiled output (all green): - A v128 crossing a defined-function boundary was typed `any` in the generated signature (goTypeOf had no ValV128 case). wasm32 llama never passed a v128 by value, so this never fired; the wasm64 build does. Map it to the scalarized [2]uint64 pair the SSA emitter uses for every v128 expression. - The f16 store-side idiom chain fused a cvt+store helper whose address parameter is i32; on a memory64 module addresses are i64 and the intermediate forms assume wasm32 v128 typing. The chain is now wasm32-only — a memory64 module's SIMD memory ops run as plain m64 helper calls without splicing, so the rewrite bought nothing there anyway. - The bulk-dispatch Inv_<svc>_<mt> wrappers hardcoded (int32, int32) request parameters; on a memory64 module the bridge exports take pointer-width i64 (the C side declares them void*/size_t). Co-Authored-By: Claude Fable 5 <[email protected]> * Splice memory64 SIMD memory ops with the shared vector bodies A memory64 module's SIMD memory helpers (simd_m64_*) were excluded from every splice path, so each load/store ran as a plain function call — measured on the llama.cpp module as a 36x token-generation gap against the wasm32 build. There is nothing wide about the vector bodies themselves: arm64/amd64 effective addresses are computed in 64-bit registers either way, so the m64 family now rides the same splice tables with only the address glue widened. - simdSpliceOp strips the m64_ marker (plain and pair forms) and reports addr64; the op after stripping indexes the same tables. - The a64/x64 memory preambles take addr64: full-width MOVD/MOVQ address operands instead of the u32 zero-extensions, with carry-checked sums (two u32s can never wrap a u64, two u64s can — and a wrapped effective address could land back inside bounds). - The scalarizer treats simd_m64_* as pairable via the same table (m64_ stripped), emitting simd_p_m64_* pair calls; the pair-wrapper family is added to the helpers. - Fused-tree/loop lookups explicitly skip addr64 sites for now: the fused emitters still assume u32 address arithmetic. Result on the llama.cpp wasm64 module: 7148 of 7149 SIMD call sites spliced, tg 1.6 -> 9.7 tok/s, pp 1.8 -> 11.4 tok/s. Porting tree/loop fusion (SDOT/SMMLA superinstructions included) to addr64 is the remaining lever toward wasm32 parity. Also fix an unbounded recursion in the fused-tree scalar chaser: a loop-carried definition (v = v + 4 in the goto-form emission) made chaseI32 recurse through interDef forever — a stack overflow on the llama.cpp module built at -Oz, whose loop shapes surface such definitions. A visiting mark leaves self-referential names as interveners. Co-Authored-By: Claude Fable 5 <[email protected]> * Fuse SIMD trees on memory64 modules Extend tree fusion (and with it the superinstruction rewrites that run inside fused regions) to memory64 modules; only fused LOOPS stay width-gated for now. Measured on the llama.cpp wasm64 module: tg 10.2 -> 17.9 tok/s, pp 13.2 -> 22.2 tok/s, with the go-llama suite green. The design keeps one width-independent op vocabulary: - The builder strips the m64_ marker for every table lookup and node name (fuseLookupName) and marks the tree Addr64. A memory64 member's address/offset always ride as OPAQUE int64 scalars — Arg.Const is 32-bit and ArgSum assumes u32 wrap, so folding either would truncate a 64-bit address; the emitters enforce this shape. - Addr64 trees take every scalar parameter as int64 (same register assignment); call sites widen non-address scalars losslessly, and the fallback body narrows them back for the int32-consuming pure helpers while the memory members call their simd_m64_* twins. - Both fused emitters widen only the address glue: full-width moves with carry-checked sums (two u32s can never wrap a u64; two u64s can, and a wrapped effective address could land back inside bounds). The vector node bodies — dot8/SDOT rewrites included — are shared unchanged. The amd64 store tail (value staging, f16 conversion chain, the store proper) is extracted so both address paths share it. - Fused loops reject Addr64 trees at the upgrade point: the loop machinery still assumes u32-wrap pointer bumps and int32 exit scalars. The loop body keeps its fused-window form meanwhile. Co-Authored-By: Claude Fable 5 <[email protected]> * Widen the fused-loop machinery for memory64 trees Lift the Addr64 gate off the loop upgrade and widen every piece the gate was standing in for: loop-invariant stride parameters and non-counter exit scalars are int64 on an Addr64 tree (they are its pointer parameters), pointer bumps advance at full width on both splicers (ADD/ADDQ instead of the u32-wrap W/L forms — the body's per-node carry-checked bounds keep a runaway pointer from ever being dereferenced), and the after-loop fix-up temporaries follow the width. On the llama.cpp wasm64 module no loop upgrades fire YET: the scalar chase (chaseAddr/chaseI32) only recognizes the wasm32 address shapes (uint32 conversions), while the memory64 emitter derefs through `unsafe.Add(mBase, uint64(v - int64(c)))` — so the scale-load interveners cannot join the body window and the upgrade's whole-body-in-one-region requirement fails. Extending the chase grammar to the memory64 shapes is the follow-up that turns the machinery on; this change makes it correct once matching succeeds, and is inert until then (all suites and the wasm64 llama build are byte-for-byte at the fused-window numbers). Co-Authored-By: Claude Fable 5 <[email protected]> * Unify the memory64 and wasm32 SIMD address handling The memory64 splice paths had grown a second, hand-written copy of every address computation — separate mnemonics, and extra carry-checked sums the wasm32 path never had. That duplication was both a maintenance hazard and a measured slowdown. Collapse the two into one width-parameterized path: the ONLY difference between wasm32 and memory64 is now the instruction that materializes an address operand — MOVWU/MOVL (zero-extend i32) versus MOVD/MOVQ (full i64). The effective-address arithmetic and the ea+size ≤ memSize bounds check are identical, shared code. The extra carry checks are removed. They are not needed on either width: a memory64's linear memory is capped at 2^48 bytes (mem64HardCap), so every valid effective address is far below 2^63 and the ea+size ≤ memSize comparison is exact. The one address that could wrap a u64 is one the guest deliberately computes near 2^64 — which wasi-sdk-compiled code never produces, and which even then resolves within the module's own memory slice, so it is never a Go memory-safety issue. This is the same pragmatic bounds model the coalescing pass already documents. a64MemPreambleRegs and x64MemPreamble are now single functions (movAddr chosen once); the fused load/store/lane checks share a64FusedMemCheck / x64FusedEA64. On the llama.cpp wasm64 kernel this lifts tg 17.9 -> 18.6 and pp 22.2 -> 22.7 while deleting the duplicate emission. Bounds coalescing stays off for memory64 (documented): unlike wasm32 it regressed the kernel even with the carry-free forms, which are kept implemented and tested for a future grouping heuristic. Co-Authored-By: Claude Fable 5 <[email protected]> * Extend SIMD unrolling, coalescing and window fusion to memory64 A memory64 module's hot kernels arrive with LP64-promoted shapes the SIMD pipeline did not recognize: i64 loop counters (i64.sub countdown, Ui64 guards), i64 pointer-bump chains, int64-spelled constant locals and % 64 shift masks. Every stage bailed in its own way and the q8_0 vec_dot ran as sparse two-sdot windows full of GPR-pair round trips — 3.3x slower than the same kernel on wasm32. Teach each stage the wide twin of the shape it already handles: - UnrollSimdLoops matches i64 countdowns and builds the guard arithmetic at the counter's width. Wide matching is a per-module opt-in (memory64 only): on wasm32 the i64-counter loops are exactly the ones whose bodies do not window-fuse, and admitting them was measured to cost ~40% prompt throughput there. - ReassocConstAdds folds Add64 chains (memory64 only, same gating rationale) so unrolled i64 pointer bumps share one base. - Bounds coalescing runs on memory64: the 2^48 linear-memory cap makes the coalesced ea+span check exact with no extra precondition. The earlier measurement that kept it off was taken against a miscompiled (-Oz) module and does not hold. - Window fusion folds memory64 addresses as base+const ArgSum (the base rides a full-width scalar; only bare 64-bit ArgConst addresses stay opaque), binds int64-spelled constant locals, resolves int64-typed rng/lane windows, and chases % 64 shift masks and int64-wrapped scale chains; the countdown-loop matcher accepts the Ui64 header and i64 decrement forms. - The arm64/amd64 fused splices materialize ArgSum addr64 bases with full-width adds; the m64 scalar-chain helpers take the mechanical simd_m64_* naming the lookup already strips. Measured on the llama.cpp q8_0 module (M5): memory64 tg 17.9 -> 33.6 tok/s, pp 22.7 -> 34.3 tok/s; wasm32 is byte-neutral by construction on the gated paths and A/B-verified flat on the rest (tg 53.6 vs 53.7, pp 88.8 vs 89.3 warm, interleaved). Co-Authored-By: Claude Fable 5 <[email protected]> * Surface failed window-fusion trials under WASM2GO_FUSE_DEBUG The failWhy field recorded each trial's first refusal but nothing read it, so diagnosing why a module's windows stay small meant ad-hoc printf builds. Print failed trials of four or more candidates (width plus refusal) to stderr when WASM2GO_FUSE_DEBUG is set — the histogram of these lines is what localized both the wasm32 K=4 unlock and the memory64 window starvation (pair-input int-slot exhaustion at pair #7 across the attention kernels). Co-Authored-By: Claude Fable 5 <[email protected]> * Recognize and pair the ggml q8_0 vec_dot on memory64 modules The type_traits_cpu recognizer assumed the ILP32 row layout (24-byte rows, u32 function pointers) and the all-i32 ggml_vec_dot_t signature, so a memory64 module never got the row/column pairing. Parameterize the recognition by pointer width: LP64 rows are 32 bytes with u64 function-pointer fields (table indices — nonzero high bits disqualify a row), the vec_dot signature carries i64 pointers and size_t strides, and the paired-tile companion takes matching int64 parameters (its pointer arithmetic is unchanged in shape). The arm64 SMMLA tile kernel follows: the only width-dependent part is the ABI0 argument frame — int64 arguments land 8-aligned (l1+16 .. l6+56, 64 argument bytes) and load with MOVD — while the 2x2 tile body is byte-identical, which the kernel test asserts by stripping the frame lines and diffing the remainder. Measured on the llama.cpp q8_0 module as wasm64 (M5, fast-math): pp 35.8 -> 45.7 tok/s (+28%, the same jump the pairing gave wasm32); tg unchanged as expected (odd batches keep nrc == 1). Full go-llama suite green, including the native-token-match gates. Co-Authored-By: Claude Fable 5 <[email protected]> * Let memory64 stores reach window fusion Two width-blind name checks kept every memory64 SIMD store out of window fusion, running ggml's conversion and scale loops as chains of marshalled pair calls: - IsVoidAtomicStore matched only the simd_v128_store prefix, so a simd_m64_v128_store's dummy result was treated as usable and the store hoisted into `vN = ...` assignment form — which the window candidate scan (expression statements) never sees. - The candidate scan itself looked the mark's raw name up in fusedStoreOps without stripping the memory64 marker. Fix both to accept the m64 spelling, and pin the behavior with twin fixtures (cg_simd_store32 / cg_mem64_simd_store): the elementwise store-of-mul(load, load) shape must produce a fused tree containing a store sink on BOTH pointer widths, Addr64-tagged on memory64. On the llama.cpp wasm64 module this fuses 2163 store sites (6320 -> 4157 bare); wasm32 output is unchanged by construction (both checks are identity for non-m64 names) and the twin test pins its store fusion too. Co-Authored-By: Claude Fable 5 <[email protected]> * Pin conversion-loop window fusion on both pointer widths Twin fixtures for the variable-stride row-scale skeleton (per iteration: v128 load, load32_splat scale, mul, add, store; pointer bumps by a runtime stride). Both widths currently fuse the whole iteration into one region — pinned so the memory64 side cannot silently regress to bare pair calls while the remaining divergence in the full llama conversion loops is chased. Co-Authored-By: Claude Fable 5 <[email protected]> * Deduplicate opaque scalar window arguments by structural key Identifier-typed scalar arguments already shared one slot per name; expression-typed ones (a variable-stride address like `base + s`) burned a fresh int slot per occurrence, so windows over conversion loops hit the slot cap early. Key them by their printed form — safe because side-effecting values are hoisted into named locals before fusion (these expressions are pure) and noteRead records the contained identifiers for the same interference checks as the identifier path. Verified perf-neutral on wasm32 (interleaved A/B: tg 53.6 avg both sides, pp 89.5 vs 89.9) with both llama suites green; no measured memory64 throughput change yet — the hot LP64 conversion loops turn out to be bound elsewhere — but the slot pressure reduction is structural and width-independent. Co-Authored-By: Claude Fable 5 <[email protected]> * Run the f16 idiom rewrites on memory64 modules The whole f32->f16 store-side chain (idiom recognition, empty- diamond folding, the 4x-store16 merge, the fused cvt+store) was gated to wasm32, so a memory64 module kept the full software idiom in its hottest conversion loops. The chain is width-neutral except at three points, now parameterized: memRoot/constOf walk Add64/ Const64 address chains, FuseF16CvtStores takes the module width and spells the i64 offset plus the simd_m64_* helper, and the m64 cvt+store helper twins (array and pair forms) land next to the m64 store family. The load-side gather recognizer picks up the widened address walk for free. Verified: pass unit tests cover the m64 spelling; both llama suites green (the conversion is bit-exact by construction — same rounding chain, NaN forcing); wasm32 interleaved A/B flat (tg 47.6 avg both sides, pp 82.9 vs 83.1); memory64 interleaved A/B ahead on tg in every adjacent pair (~+0.8 tok/s warm), pp flat. Co-Authored-By: Claude Fable 5 <[email protected]> * Chase variable-stride memory addresses into scalar-node chains A load or store whose address is a runtime sum with no constant side (`base + stride`, the shape strided conversion loops carry — three terms on LP64) previously rode one opaque int slot per distinct expression, so windows over such loops hit the slot cap and shed their loads. Chase these addresses like the scale chains: the sum becomes scalar_i32_add nodes whose LEAF identifiers deduplicate by name, so any number of loads sharing a stride variable cost one slot for it. The emitters accept a chain-computed address terminal (takeAddrSource, the GPR analog of takeSplatSource) and move it into the address register before clobbering scratch, on both arm64 and amd64, both pointer widths, with tryFuseLoop's rejection points now logged under WASM2GO_FUSE_DEBUG. Twin fixtures pin the behavior (cg_simd_addrsum32 / cg_mem64_simd_addrsum: a mem node with an ArgNode address must appear on both widths). On the llama.cpp modules: the hot memory64 conversion loop's bare loads drop 66 -> 26, and wasm32 gains its first fused-LOOP upgrades on this module (fxl0/fxl1) because whole bodies now fit one window. Throughput is A/B-neutral on both widths (the loop's remaining cost is gc frame traffic between windows, not the bare ops) — this lands as groundwork and unification, with both llama suites green including the native-token-match gates. Co-Authored-By: Claude Fable 5 <[email protected]> * Upgrade head-tested unit-decrement loops and unwrap memory64 call args Two blockers kept real conversion loops out of the fused-loop upgrade. First, the emitter produces a THIRD countdown shape the matcher missed on both widths: a head-tested while, `for { if c == 0 { break } else { body; bumps; c = c - 1 } }`. With a unit decrement `c == 0` exits at exactly the same counter values as the pre-tested `Ui(c) < Ui(1)`, so the shape reuses the pre-test descriptor; any other decrement under an equality head is rejected. Second, memory64 fused-call sites wrap their scalar arguments in int64(...), which broke the upgrade's slot and reassociated-bump matching by identifier — both now unwrap the width conversion first. The conversion-loop twins now upgrade on BOTH widths (pinned by TestConvLoopUpgradesBothWidths — the first exercise of the memory64 loop-splice path), and the wasm32 llama module gains five fused loops. Throughput is A/B-neutral on both widths (the upgraded loops sit off the hot paths); both llama suites green including the native-token-match gates. Co-Authored-By: Claude Fable 5 <[email protected]> * Fix dropped-definition reads exposed by nested address chains The address-chase commit made nested chains reachable (`v80 = v67 + s` with v67 itself an intervener), and two holes in the surrounding bookkeeping produced wrong stores in the variable-stride axpy shape — caught by the gcasm differential fixture (the llama greedy gates do not exercise it): - An intervener counted as fully consumed when the chases internalized all its reads, and was dropped — even when a KEPT intervener's definition still read it (v80 stays because a loop tail reads it, but v80's kept RHS read the dropped v67, leaving the tail bump computed from an unassigned variable). Un-drop such definitions to a fixpoint. - The chase could leak a WINDOW-INTERNAL value (another candidate's result, e.g. an extract_lane feeding an address) into a chained scalar argument, which has no pre-call definition after fusion. Refuse at the chase so the address falls back to the original pair/opaque forms. TestGcasmSimdLoopUnroll (the value-level differential over the unroll/fuse/loop-fuse configurations) fails before and passes after. Co-Authored-By: Claude Fable 5 <[email protected]> * Gate unamortized packed outlining on memory64 modules A packed outline boundary writes every live-in into the scratch array and reads it all back on each call. On a memory64 module the i64 locals double the caller's register pressure and the round trip stops paying for wide boundaries: refusing packs whose body carries fewer than 24 values per slot gains ~17% prompt throughput on the llama.cpp module (the ~130-slot conversion loop runs inline instead). The SAME gate on wasm32 loses ~25% prompt throughput — its conversion loop compiles better extracted than inline — so the ratio applies only when the module is memory64, measured both ways by interleaved A/B with both llama suites green. Co-Authored-By: Claude Fable 5 <[email protected]> * asmgen: import direct-asm backend and add br_table dispatch support Import the SSA-to-assembly emitter (internal/asmgen) and its register allocator (internal/regalloc) so hot scalar functions can bypass the gc backend's code quality ceiling — the remaining wasm64 throughput gap is dominated by frame-reload storms gc generates for doubled i64 live sets in the conversion/dot-product driver loops. The emitter predates two SSA block kinds, so bring it up to the current IR: - BlockBrTable: new EmitBrTable arch hook emitting a linear compare chain (selector materialized once into the scratch register, one CMP+branch per case value, then an unconditional jump to the default). Successors carrying phis are rejected to the per-function fallback — no dispatch-shaped code produces them today, and edge copies would need per-successor intermediate labels. - BlockThrow stays unsupported; functions using it keep their gc- compiled Go bodies via the existing per-function stub fallback. LowerFunction calls gain the ThrowSet parameter added since the emitter was written. Testing: the driver harness is now arch-parameterized, and the control-flow value test (if/else merge, loop phis, br_table chain) runs natively on both amd64 and arm64 — Apple Silicon executes the arm64 driver even when the test process runs under Rosetta, detected via sysctl.proc_translated. Co-Authored-By: Claude Fable 5 <[email protected]> * asmgen: memory64 addressing for loads, stores, and memory ops Teach the direct-asm emitters 64-bit linear-memory addressing so memory64 modules can route hot scalar functions through asmgen. The effective-address computation is factored into one per-arch helper (emitMemAddrAMD64 / emitMemAddrARM64) shared by the load and store paths; the two memory widths use the same mechanism and differ only in the address rule, mirroring the pure-Go emitter: - wasm32: address = uint32(base + off), computed in 32-bit so negative-int32 bases wrap per the wasm effective-address rule (unchanged behaviour, now in one place per arch instead of four). - memory64: address = base + off in full 64-bit, no wrap — the pure-Go path is `uint64(base) + offset`. On amd64, constant bases fold into the displacement when the total fits int32 and larger offsets are materialised through a scratch register before the index add; on arm64 the assembler's REGTMP materialisation keeps the shape a single ADD regardless of offset size. OpMemSize / OpMemGrow / OpMemoryCopy / OpMemoryFill on memory64 modules route to the 64-bit helper family (memorySize64, memoryGrow64, memoryCopy64, memoryFill64) with the i64 signatures the codegen helpers define. Testing: cg_mem64_memops.wat is the memory64 twin of cg_memops.wat (every load/store width and sign, offset memargs, constant-base fold, i64 mem-size/grow), value-checked on amd64 and natively on arm64. The wasm32 memops value checks also gain an arm64 twin so both widths are execution-verified on both arches. Co-Authored-By: Claude Fable 5 <[email protected]> * Direct-asm backend: per-function asmgen bodies inside the gcasm bundle Wire the SSA-to-assembly emitter (internal/asmgen) into the pipeline as an opt-in per-function backend coexisting with gcasm. gc's register allocator treats every CALL as clobbering everything, which spills large i64 live sets to the frame around fused-window calls; emitting those functions straight from SSA is the lever for removing that reload traffic. This change lands the plumbing with a conservative first eligibility class (call-free leaf functions) and per-function fallback everywhere else. Flow: - codegen: Options.DirectAsmFuncs opts function names in. The translator retains their finalized SSA (post optimization fixpoint, idiom rewrites, and outlining — outlined loop functions with unpacked scalar boundaries are eligible too) in Result.DirectAsmSSA. Direct-asm modules with a memory also emit the gcasm memory probe so the bundle can verify the Module.M offset the asm bodies hardcode. - gcasm: Config.DirectAsm carries the retained SSA into Build, which emits each function per architecture via asmgen (buildDirectAsm) and swaps the body in place of the gc-listing transform in buildPkg. Bodies carry a "// direct-asm: <name>" marker; decls reuse the transform path's shape (declSig, now shared). Emission failures fall back to the transform per function and are counted (BuildStats.DirectAsm / DirectAsmFallback). - asmgen: FuncOptions.ForbidCalls fails emission for any function that would CALL out (helpers, callees, memory ops, global wrappers) — the bundle's callee symbols are not resolvable from asmgen's same-package CALL spelling yet, and a body that emits but fails to link would break the build instead of falling back. ModuleMOffset is exported for the host-side layout check. - cmd/wasm2go: -direct-asm flag (comma-separated names). Testing: the arith fixture runs through the full backend with five functions opted in — four leaf bodies come from asmgen on both architectures (marker asserted), the helper-calling one falls back — and the generated package computes the same values the asmgen driver tests pin. Co-Authored-By: Claude Fable 5 <[email protected]> * asmgen: v128 values and SIMD helper CALLs Extend the direct-asm emitters to functions containing SIMD, the shape the hot wasm64 loops actually have. v128 values live in 16-byte stack slots as [2]uint64 pairs (slotSize / abiSize gain the class); OpSimdConst materializes its lane payload into the slot; and OpSimdCall / OpSimdMemCall lower to plain ABI0 CALLs of the bundle's simd_* helper symbols — the same symbols the pure-Go backend calls. There is no static signature registry for the SIMD family: each call's layout derives from the SSA value itself (arg types, result type, module pointer for memory ops), shared between the arches in simdCallSpecOf. v128 args and results travel as two 8-byte halves; phi edge-copies and staged-phi slots handle the type on both arches. ForbidCalls narrows from "any CALL" to non-SIMD callees (hasNonSimdCall): SIMD helper symbols ship with every SIMD-using bundle, so their CALLs always resolve, while scalar helpers, sibling functions, memory ops, and global wrappers keep the per-function fallback until the bundle's wrapper spelling is wired up. Cross-package emission (multi-package chunks) is still rejected in simdCallSpecOf for the same linkability reason. The cross-block register allocator predates the 16-byte slot class, so functions containing v128 values keep the block-local allocator (whose pool selection already skips v128-typed values). Testing: the SIMD fixture runs through the full backend with every function opted in — the v128-bearing bodies come from asmgen (helper CALLs then still spliced or kept by the existing gcasm machinery for the rest of the bundle) and all values match the wazero-verified reference the gcasm test pins. Co-Authored-By: Claude Fable 5 <[email protected]> * asmgen: splice SIMD helper bodies inline via the gcasm tables Replace the marshalled SIMD helper CALLs in direct-asm bodies with the same inline NEON bodies the gc-capture transform splices — the step that makes the direct backend competitive on SIMD-heavy functions instead of regressing to the pre-splice call-overhead world. The splice machinery lives in gcasm, which already imports asmgen, so the dependency inverts through an interface: asmgen's FuncOptions.Splicer receives operand locations (frame slots — SIMD- containing functions emit in slot-only mode: no register homes, no m-cache, no loop-carry coalesce, because splice bodies clobber registers freely) plus a scratch area (the callee-argument region, always a superset of the splice's stack traffic). gcasm's implementation stages the operands per the same ABIInternal assignment the listing transform uses (scalars into R0../F0.., v128 halves into the scratch area at their sequence offsets) and drops in a64SpliceSimd unchanged, sharing the per-package ConstPool. Memory splices branch to the shared out-of-bounds stub, which the emitter appends once at body end. Everything goes through a scratch builder first so an op the tables decline falls back to the CALL with no staging leaked. Direct-asm emission consequently moves inside buildPkg (it needs the package's ConstPool); amd64, which has no per-op splice table, keeps the marshalled CALLs. Testing: the SIMD fixture's direct-asm arm64 bodies are asserted to contain spliced NEON, and the generated package is now also cross-built for arm64 and EXECUTED natively (Apple Silicon runs the binary even when the test process is under Rosetta), matching the wazero-verified reference values. Co-Authored-By: Claude Fable 5 <[email protected]> * asmgen: coalesce loop carries into splice-safe registers The reason the direct backend exists: gc must assume every call site clobbers every register, so the hot loops' i64 carries live in frame slots and each iteration pays a reload storm around the SIMD ops. Inline splice bodies, unlike calls, have a documented clobber contract (arm64: R0–R15 / R25–R27 and the V registers) — so R19–R24 survive them, and asmgen can keep the loop-carried scalars there. runSpliceCoalescePass is the splice-mode variant of the loop-carry coalesce: SIMD helper calls inside a loop no longer bar coalescing, and every such value in a coalesced loop is recorded in plan.mustSplice — a splice-table miss there fails the function to the host's fallback instead of emitting a CALL that would clobber the carry. The pool is R19–R24 (the fused-window splices claim R20–R23 as window state; the pool must shrink when asmgen learns to emit those). The original coalesce safety rule — the phi may have NO user but its own bump — denies the canonical counter/pointer carries (read by the loop condition or an address user AND the bump). The splice-mode pass uses a finer analysis (coalesceUsesSafe): every read must execute before the bump's register write in each iteration, which holds when the bump sits in the latch block (the latch runs last — its only successors are the header and loop exits), reads in the latch precede it, and the phi has no reads outside the body, via another phi's edge copy, or as the latch's control. The default pass keeps the single-use rule pending an A/B. Two correctness fixes surfaced by this work: - opEmitsCall did not know OpSimdCall / OpSimdMemCall, so their marshalled CALLs were invisible to the m-cache re-prime logic and the coalesce call barrier — a latent wrong-code hazard for CALL- mode SIMD bodies. - The arm64 lane-op memory splices used R24 as their data scratch, violating the splice clobber contract this change relies on. R4 (inside the contract set, free — the helper's args occupy R0–R3, and unlike R25–R27 it survives the memory preamble) replaces it. Testing: a self-contained SIMD accumulation loop fixture with pointer/counter carries runs differentially — direct-asm output vs the normal backend across 33 trip counts — on amd64 and natively on arm64, and the arm64 body is asserted to contain both spliced NEON and an R19 carry. Co-Authored-By: Claude Fable 5 <[email protected]> * asmgen: multi-package emission, packed boundaries, inline helpers Three pieces that carry the direct-asm backend from fixtures to the llama-scale bundle: Multi-package. The linkname-split translator builds its own Result and dropped DirectAsmSSA entirely — retention silently never reached the bundle (the chunked layout is the only one llama uses). With the field wired through, chunk emission works on the splice-only contract: spliced SIMD sites need no symbols at all, so a chunk function is eligible whenever every SIMD site splices — the multi-package rejection moves from spec time to the CALL fallback path, which now fails the function to the transform instead of emitting a same-package spelling that cannot resolve. The out-of-bounds trap CALL resolves through buildPkg's callee-resolution closure (registering the chunk-local forward wrapper), and a splicer without a resolved trap symbol refuses memory splices. Packed boundaries. llama's outlined SIMD loops carry v128 values across the boundary, so nearly all of them use the packed form — the caller passes only the module pointer and the values ride Module.OutlinePack. Retention keeps them (results-only signature plus the parameter types in slot order), and the emitters gain a pack prologue: parameter values load from the pack scratch (pinned at moduleMOffset+8 by the same field-order construction that pins M) into the params' frame slots, parameter reads resolve to those slots instead of FP offsets, and OpParam slots are excluded from slot reuse (a pooled reuse would clobber the prologue's write). Inline helpers. The first llama fallback wave was all trivial scalar helpers (sign/zero extends, 32-bit reinterprets, f32_abs) that the amd64 emitter already lowers to one or two instructions but arm64 routed through a marshalled CALL. helperAlwaysInline names the set both arches now emit natively; planFunc exempts those calls from hasCall (no callee frame, no ForbidCalls consequence), and both emitters guard the CALL path against ever being reached for an exempted name — the frame it would use was never reserved. The ForbidCalls diagnostic now names the offending callee, which is what surfaced the wave in the first place. Co-Authored-By: Claude Fable 5 <[email protected]> * asmgen: inline float ops, global offsets, resolver-backed callees Three widening waves, each driven by the ForbidCalls diagnostic (which now lists every distinct blocker per function instead of the first) against the llama-scale bundle: Inline float family. The arm64 emitter gains native lowerings for the float compares (FCMP + CSET with unordered-false condition codes, NE unordered-true — wasm/IEEE semantics, matching gc's own lowering), binary arithmetic (FADD/FSUB/FMUL/FDIV both widths), neg/abs, promote/demote, int→float converts, and i32_wrap_i64; all of these were already inline on amd64. helperAlwaysInline widens accordingly. Global offsets. Module-defined globals inline as MOVs against the module pointer. The translator — the one component that actually emits the Module struct — computes each global's byte offset following its own field order, ships them through Result.DirectAsmGlobals into the bundle, and emits compile-time assertions (directasm_layout.go) pinning every offset direct-asm hardcodes: M, the outline pack, and each global. Layout drift then fails the consumer's build instead of corrupting memory. Resolver-backed callees. Direct calls and the memory-op helper family resolve through buildPkg's callee-resolution closure — the same machinery the listing transform uses, registering chunk-local forward wrappers as a side effect — via two FuncOptions hooks (CalleeSymbol / MemHelperSymbol). Resolved calls are exempt from ForbidCalls; unresolved ones still fall back per function. With these, nine llama functions emit as direct asm on arm64 (including every outlined loop of the compute monolith), and the full go-llama verification suite passes on the resulting bundle: greedy determinism, prompt batch-size invariance, native NLL parity, and cross-context repeatability. Co-Authored-By: Claude Fable 5 <[email protected]> * asmgen: inline integer div/rem on both arches The divide/remainder family was the largest remaining direct-asm fallback class. Lower it inline: native divide plus explicit trap-branch CALLs, with the trap symbols resolved through the same host hook as the memory helpers (forward wrappers in multi-package chunks) — an unresolved trap helper keeps the marshalled-CALL path instead of emitting a spelling that cannot link. arm64 (new): SDIV/UDIV for quotients, MSUB for remainders. arm64 divides never fault, so the wasm traps are explicit: every op branches to the div-zero trap on a zero divisor, and div_s checks INT_MIN / -1 for the overflow trap. rem_s needs no overflow check — SDIV wraps the quotient to INT_MIN and MSUB computes x - q*y = 0 with the same wrap, which is exactly wasm's answer. Results honor plan.regHome (the arm64 regalloc register-homes helper-call results; storing only to the slot left downstream readers of the home register with garbage — caught by the new native-execution arith value test, which now covers the div family on arm64). amd64 (fixed): the existing inline div_s relied on IDIV's #DE for the INT_MIN / -1 case, which panics with the divide-by-zero kind; wasm calls it integer overflow. Both widths now check explicitly and call the overflow trap. All trap CALLs go through the plan's resolved symbols instead of hardcoded same-package spellings. Also guard the outline pass's synthetic sibling-call sentinel (OpCallDirect with a negative index and the callee name in Aux): planFunc indexed the module's function table with it and panicked on nested outlined loops; it now counts as an unresolvable callee, so ForbidCalls falls the function back cleanly. The packed call-site protocol for such calls is future work. At llama scale this raises the arm64 direct-asm body count from 58 to 76 at outline threshold 100 (9 at the pipeline's threshold 400), with the verification gates green throughout. Co-Authored-By: Claude Fable 5 <[email protected]> * asmgen: hoist module state across splices; splice the rng/nc loads Two body-quality steps, both llama-verified for correctness: Module-state hoisting. Every arm64 memory splice reloaded m from its FP slot and m.M / the memSize pointer from the Module per site — three loads per memory op, plainly visible in the pp profile. The emitter now keeps them in the splice-safe registers the coalesce pool cedes (R24 = m via the m-cache mechanism, R23 = m.M, R22 = the memSize pointer, staged by the splicer's new HoistPrologue), leaving R19–R21 for loop carries. The refresh after calls became precise in the process: emitters that actually produce a CALL set a per-value flag, replacing the op-kind heuristic that would have re-primed after every spliced SIMD op (and, previously, missed nothing but paid refreshes after inlined helpers). Memory-less functions skip the prologue — staging it read a garbage m and faulted, caught by the SIMD fixture's arm64 leg. Bounds-coalesced loads. The hot conversion loops read memory through the range-check group forms (v128_load_rng leader / _nc members), which only the pair-form splicer knew; the array-form path direct asm uses kept them as CALLs, so the hottest outlined loops always fell back. The array-form twins mirror the pair bodies (signed group start, TBNZ wrap check, leader-only bounds compare) with the hoisted-register variants, and both hot loops now emit — 107 arm64 bodies at outline threshold 100. MEASURED STATUS, not yet a win: with the hot loops emitted, prompt eval REGRESSES ~2.9x on the direct bundle (correctness gates all pass). The per-op splice protocol round-trips every v128 value through its stack slot between adjacent ops — the load feeds a splat/mul/add chain entirely through memory — where the gc-capture path's fused windows keep the chain in V registers. Vector-value register residency across splices is the remaining (and now precisely measured) gap; nothing ships through this path until it closes — the pipeline's bundle does not opt in. Co-Authored-By: Claude Fable 5 <[email protected]> * asmgen: slot-direct v128 splice protocol and spare-register caching The measured 2.9x prompt-eval regression on the direct-asm hot loops decomposed into two protocol costs, both addressed here: Slot-direct splicing. The splicer staged every v128 argument from the value's frame slot into the ABIInternal scratch area through GPR pairs and copied every result back the same way — four extra memory ops on each side of every op. SpliceSlots now carries the values' own RSP offsets into the splice emitters (argument reads, result stores, lane addressing), so the scratch hops vanish; the transform path passes nil and keeps its scratch behavior, where the values genuinely live there. asmgen ships the offsets on the operands (InSlot/SlotOff) — an unpacked FP parameter is the one v128 without a slot and keeps the copy path. Spare-register value caching. Every splice body computes through V0, so no value survives in a register past the next op and plain store-to-load forwarding never fires. a64SpliceValueCache parks each "// simd out" result in a spare vector register (V17–V31, strictly above anything a splice body touches) and rewrites later reloads of that slot into single-cycle register moves, with the sibling passes' conservative invalidation (labels/CALL reset, width-aware RSP store overlap, pointer stores drop everything). The result store always remains — it is the value's only definition. Two bugs found by the gates on the way: - a64DeadArgStores must not run on direct-asm bodies: it models the capture shape, where arg-slot stores are redundant copies; in the slot-direct shape the "simd out" store IS the definition, and the pass deleted live stores. The native NLL parity gate caught the wrong values (greedy and batch invariance did not — a lesson in gate sensitivity). - a64SpliceWordClobberList omitted v4, which the f16 conversion body writes ("movi v4") — a value the forwarding pass parked in F4 would survive that splice on paper while being clobbered in fact. Prompt eval on the direct bundle: 19.5 → 44.6 tok/s across these steps (baseline 55.5; the remaining gap is scalar address traffic through frame slots and the per-op bounds preambles). All verification gates pass at every step. Co-Authored-By: Claude Fable 5 <[email protected]> * asmgen: scalar slot forwarding and dead result-store elimination Two more text passes over direct-asm arm64 bodies, closing most of the remaining per-op splice overhead: Scalar forwarding (a64ScalarSlotForward). The slot-only emission spells every scalar intermediate as a store/reload pair through its frame slot — address chains read as MOVD store, MOVD reload, ADD, repeat. The pass tracks full-width register→slot bindings within straight-line spans and rewrites reloads into register moves (or drops them when source and destination coincide), with the sibling passes' conservative rules: labels and CALL/RET reset, WORD splice bodies kill bindings in the contract clobber set (R0–R15/R25–R27), pointer stores drop the slot side, narrower stores only invalidate. Dead result stores (a64DeadSimdOutStores). Once reloads are forwarded to the spare-register cache, a single-use v128 intermediate's "// simd out" store serves nobody — but only a function-scoped read count can prove it (a later block may reload the slot). The pass counts every RSP-offset mention as an operand — comments are explicitly NOT reads: a spare-forwarded reload names its origin slot in a comment only, and counting those pinned nearly every store (the first cut measured neutral because of exactly this). Slot addresses that escape through `ADD $off, RSP, Rn` count as reads of both halves, and any unmodeled RSP form keeps the whole body. The spare park always stays: it is what serves the forwarded consumers. Prompt eval on the direct bundle, same config and gates green at every step: 45.4 → 54.3 tok/s (the dead stores were nearly all of it once the comment rule landed); the gc-backed baseline sits at 55.7. The cumulative ladder from the first correct slot-direct build stands at 19.5 → 54.3. Co-Authored-By: Claude Fable 5 <[email protected]> * asmgen: v128 loop-carry homes in the splice-safe vector registers Coalesce v128 loop-carried phis (the accumulators) into V25–V31 — above both the splice bodies' registers (V0–V4, F16) and the spare cache (now V17–V24) — so an accumulator add-chain runs entirely in registers across iterations instead of round-tripping its slot. The splice protocol gains register operand forms: SpliceSlots carries per-argument register sources and a register destination, the table path reads/writes them with single VORR moves, the memory path covers the store's value argument and every load result, and the lane paths — which index the value's memory — spill a register-resident operand to its scratch slot first and load a register-destined result back after the in-place edit (loop exits only; the hot path never lanes a homed value). Safety differs from the GPR carries in one respect: a V home survives to function end (the pool is consumed once per function, no splice touches the range), so live-out reads of the accumulator — the after-loop store or lane fold every reduction ends with — are allowed where a GPR carry must refuse them. That in turn is only sound because v128 coalescing is gated on a function with no non-SIMD callees and forces every SIMD site in the function to splice (a fallback CALL would clobber the vector file). Phi copies read and write homes directly (a register-resident source stores with one FMOVQ instead of two GPR round-trips; the entry edge moves register-to-register when both ends are homed). llama-scale: the hot conversion loop carries 17 home references, all gates pass; throughput is neutral-to-slightly-negative on this loop (its accumulation is mostly block-local, already served by the spare cache, and the loop-exit lane spills give some of the win back). The infrastructure is what the fused-window step needs either way. Co-Authored-By: Claude Fable 5 <[email protected]> * codegen: fire loop fusion on memory64 modules A memory64 llama build ran prompt eval 40% behind its wasm32 twin from the same source. The differentiator was loop fusion (fxl): the wasm32 bundle upgrades five countdown loops — including the q8_0 vec_dot kernel, whose fused form carries the 44-SDOT paired column kernel — while the memory64 bundle upgraded none, leaving the same kernel as a 10-SDOT per-window loop. Dumping the loop bodies as the fuser sees them showed the memory64 vec_dot loop statement-for-statement identical to the wasm32 twin up to spelled widths. Two width-blind checks rejected it: - fuseConstOf did not recognize int64(K): the loop upgrade's leftover classification (constant locals hoist above the loop) refused the body because the bump-delta local `vN = int64(136)` did not count as a constant, failing the whole upgrade for every memory64 loop whose const CSE landed inside the body. - The header form derived the counter's width only from the decrement terms' spelling. When every term is a const LOCAL (`c = c - v170 - v170 ...`), nothing is spelled at any width and the descriptor defaulted to i32 — generating an fxl helper whose counter parameter mismatched the module's promoted i64 counter. matchUiLt/matchEqZero now report which width the guard matched (Ui64(c) < Ui64(K) is the counter's own width) and the descriptor carries it. With both fixed the memory64 vec_dot upgrades to the same fused loop as wasm32 (42 SDOT vs 44). Measured on the llama q8_0 bundle, all token-equivalence gates passing at every step: prompt eval 54.3 -> 68.8 tok/s (+27%), decode 30.1 -> 46.7 tok/s (+55%). wasm32 output is unchanged (the new width legs only accept forms wasm32 never spells). Also under WASM2GO_FUSE_DEBUG: the countdown matcher dumps large loop bodies as the fuser sees them, and upgrade refusals of large loops log the rejecting check with the body size — the two probes that located this. Co-Authored-By: Claude Fable 5 <[email protected]> * codegen: log loop-window candidate collection under fuse debug The countdown-loop upgrade's window trial can fail before any walk runs — candidate collection stops at the first unsafe intervener — and nothing reported where. Under WASM2GO_FUSE_DEBUG, large loop bodies now log the stopping statement and the collected counts, and big-loop window trials log the walk verdict with scalar/pair usage. These probes located the memory64 conversion-loop blocker (eight irreducible address bases exceed the fused-signature slot budget). Co-Authored-By: Claude Fable 5 <[email protected]> * codegen: check the debug printer's error return Co-Authored-By: Claude Fable 5 <[email protected]> * gcasm: harden fused-splice capacity handling Three latent hazards around over-budget fused signatures, found while probing why the memory64 conversion loop cannot yet upgrade to a fused loop (it needs more integer argument slots than the splice ABIs seat): - errFusedCapacity had no catcher: the comment promised "the transform keeps the helper CALL instead of failing the build", but both transforms turned ANY splice error into a hard build error. They now catch the capacity sentinel and fall back to the unspliced path deliberately. - Fused/loop splices emitted directly into the output builder, so an error after partial emission (the vector-pool-exhausted paths) would have left corrupt text if it were ever survivable. They now emit into a scratch builder that is committed only on success. - x64SpliceFused/x64SpliceLoop indexed the argument-register table with unchecked positions (an over-budget tree would panic instead of declining), and a64SpliceLoop had no >R15 guard at all (it would have emitted reads of R16+ — registers gc never loads — as silent garbage). All three now decline via errFusedCapacity. No behavior change under the current codegen budgets — the guards are unreachable until the caps rise — verified by a byte-identical llama memory64 bundle. The experiment that motivated this (raising the argument caps alone) does NOT compose yet: pair-form helpers have no marshalled-call fallback, so a tree either arch cannot splice still fails the build downstream; widening the budgets needs the stack-argument legs on both splicers first. Co-Authored-By: Claude Fable 5 <[email protected]> * codegen: chase const-sum address bases; fix chase-use overcounting The memory64 conversion loops (a quarter of prompt eval) could not upgrade to fused loops: their eight per-row address bases — chained interveners like `v243 = v228 + l6`, rows advancing by a runtime stride — each burned one opaque scalar slot under the ArgSum leg, overflowing the fused-signature budget. The scalar-chain chase already knew how to decompose such sums into shared atoms (scalar_i32_add nodes; the wasm32 twin fuses this way), but only the NO-const-side leg of the walk ever invoked it. The mem-address branch now chases a const-sum's BASE through the same machinery (addr64-gated; wasm32 output stays byte-identical), so k row addresses share their atoms' slots and the loops fit the existing caps on both splice ABIs. Opening that terrain exposed a latent chase-accounting bug: two consumers chasing the same definition chain (`v311 = v309 + l6` feeding two loads) bumped chaseUses[v309] once PER RE-CHASE, but dropping v311's definition at commit removes exactly ONE textual occurrence of v309. The overcount made the fully-consumed proof (identCount == uses+1) hold spuriously — eating the read a loop-tail bump still had — so the commit dropped `v309 = v288+l5` while `v228 = v309 + l5` kept executing against the variable's zero value: row pointers walked garbage, greedy survived, and only the native-NLL gate caught the drift (rel 1e-1 on the f32 model, both arches, fast-math-independent). A definition's child reads are now charged once per trial (chaseExpanded, carried through the chase snapshot/restore). Also under the fuse-debug env: WASM2GO_CHASE_MAX=N applies the new leg to the first N sites only (the bisect that localized the bad transform before window-level differential tests took over). Measured on the llama memory64 bundle, all five token-equivalence gates passing and the wasm32 bundle byte-identical: prompt eval 68.8 -> 89.7 tok/s — parity with the wasm32 build's 90 — and decode ~50 tok/s. The conversion loop and the vec_dot kernel both run as fused loops now (fxl0/fxl1, 42-SDOT paired column form). Co-Authored-By: Claude Fable 5 <[email protected]> * codegen: accept exit copies in the do-while break arm The emitted do-while countdown closes its break arm with the after-loop aliases of the carried accumulators (`vOut = accNext; break`); the matcher required a bare break and refused the whole upgrade for every such loop — including the K=1 dot kernels the decode path spends half its time in. The leading assignments now classify through the same tail logic (a copy whose target is not a loop argument already becomes a post-loop exit copy at upgrade time). No output change yet on either width (the K=1 kernels still refuse later, at the window trial); gates pass, wasm32 bundle byte-identical. Groundwork for upgrading the decode-path loops. Co-Authored-By: Claude Fable 5 <[email protected]> * test: cover the asmgen emitters and memory64 helper family The coverage gate fell to 79.7% under this branch's new code. Four additions bring it to 83.8% with behavioral tests, not padding: - asmgen_widecmp fixture + both-arch driver tests: variable-count 64-bit shifts, 64-bit compares, inline float compares (NaN unordered included), the fused bit-test branch, an unreachable arm, and the amd64-only inline f32.min/f64.max (arm64 lowers those to helper calls, so its driver skips them). - An emit-only sweep lowering and emitting EVERY function of every cg_* fixture through BOTH emitters regardless of host arch — emission is pure code generation, so this walks the emitters' long tails that the natively-executed driver subset never reaches. Fixtures that decline wholesale (atomics/threads) skip. - Unit tests for the helper-name/type utilities and the demand-driven BuildWrappers generator (call_indirect trigger). - Behavioral tests for the memory64 helper family (every simd_m64_* / simd_p_m64_* entry over live memory), the replace-lane family in all three forms, the f16 convert-store chain (load16x4_u lane form documented by the test), the memory64 bulk-memory ops, and the trap/throw/catch entry points (foreign panics must re-panic, not be swallowed). Co-Authored-By: Claude Fable 5 <[email protected]> * test: drop an untracked fixture from the emit sweep The sweep list was generated from a local glob and picked up cg_threads_multiword.wat, which exists only in this worktree — CI has no such file and the sweep failed on it. The list now contains tracked fixtures only (verified against git ls-files). Co-Authored-By: Claude Fable 5 <[email protected]> --------- Co-authored-by: Claude Fable 5 <[email protected]>
117 lines
5.1 KiB
Go
117 lines
5.1 KiB
Go
package regalloc
|
|
|
|
import "github.com/goccy/wasm2go/internal/ssa"
|
|
|
|
// ArchInfo abstracts the per-architecture register pool, calling
|
|
// convention, and per-op register requirements. The allocator works
|
|
// with `register` indices (uint8) and `regMask` bitsets; ArchInfo is
|
|
// the bridge between those numeric abstractions and the concrete asm
|
|
// names ("AX", "R0", "X3", ...).
|
|
//
|
|
// One ArchInfo implementation is supplied per target arch (ArchAMD64,
|
|
// ArchARM64). The same Allocator binary can in principle be retargeted
|
|
// at runtime by swapping ArchInfo; in practice we pick one per asmgen
|
|
// session and never switch mid-walk.
|
|
type ArchInfo interface {
|
|
// Name reports a short arch identifier ("amd64", "arm64") for log
|
|
// messages and error wrapping. Not used by the allocator's logic.
|
|
Name() string
|
|
|
|
// NumRegs reports the total register count for this arch. Indices
|
|
// 0..NumRegs()-1 are valid `register` values; NumRegs() must be
|
|
// strictly less than 64 (regMask is uint64). Includes reserved
|
|
// registers — Allocatable() is the subset the allocator may pick.
|
|
NumRegs() register
|
|
|
|
// RegName returns the asm mnemonic ("AX", "R12", "X3", "F0") for a
|
|
// register index. Used by the emitter and by tests; never on the
|
|
// hot path of allocation.
|
|
RegName(r register) string
|
|
|
|
// RegIndex looks up the index for a register name. Returns
|
|
// noRegister when the name isn't known to this arch. Used to bridge
|
|
// the existing asmgen plan (which keys on names) to the new index-
|
|
// based allocator state. Case-sensitive.
|
|
RegIndex(name string) register
|
|
|
|
// GPRegMask is the union of all general-purpose integer registers
|
|
// for this arch, including reserved ones. Class membership uses
|
|
// this mask; Allocatable() prunes the reserved set.
|
|
GPRegMask() regMask
|
|
|
|
// SSERegMask is the union of all floating-point / SIMD registers
|
|
// for this arch. Distinct from GPRegMask so the type-class check
|
|
// in computeLive / allocReg picks the correct pool.
|
|
SSERegMask() regMask
|
|
|
|
// Allocatable returns the registers the allocator is permitted to
|
|
// pick from. This excludes SP, the goroutine pointer (R14 amd64 /
|
|
// R28 arm64), any function-cache register reserved by the
|
|
// m-cache (R11 = m on amd64), and any other arch-specific
|
|
// lockouts. The
|
|
// allocator subtracts further per-instruction pins via nospill.
|
|
Allocatable() regMask
|
|
|
|
// CallClobbersGP returns the mask of GP registers that a CALL
|
|
// trashes. Go's ABI0 (which our generated functions follow) marks
|
|
// every GP as caller-save, so this is GPRegMask() minus reserved
|
|
// registers (g, frame ptr, static base). CALL handling sets these
|
|
// regs free in the allocator state at every CALL site.
|
|
CallClobbersGP() regMask
|
|
|
|
// CallClobbersSSE returns the matching mask for the SSE / FP
|
|
// registers. ABI0 treats all of them as caller-save, so this is
|
|
// just SSERegMask() in practice.
|
|
CallClobbersSSE() regMask
|
|
|
|
// ClassFor maps an SSA value type to its register class. Memory /
|
|
// void / flag types report ClassFlags (the allocator skips them);
|
|
// float types report ClassSSE; everything else (i32, i64, bool,
|
|
// pointer) reports ClassGP.
|
|
ClassFor(t ssa.Type) regClass
|
|
|
|
// RegSpec returns the per-op register-info for an SSA value. The
|
|
// returned regInfo names the input / output / clobber constraints
|
|
// that op imposes — most ALU ops have free choice (full GP mask on
|
|
// both sides), CALL has rigid ABI args + a full clobber mask, MUL
|
|
// pins the AX/DX outputs, and so on.
|
|
//
|
|
// The result is cached per-arch and is safe to share across
|
|
// allocator runs; the allocator does NOT mutate the returned
|
|
// regInfo.
|
|
RegSpec(v *ssa.Value) regInfo
|
|
|
|
// IsResultInArg0 reports whether the op's output must alias its
|
|
// first arg's register (the two-address-instruction case — most
|
|
// amd64 arithmetic, "ADDL src, dst" where dst is both input 0 and
|
|
// the result). When true, the allocator emits a leading reg-to-reg
|
|
// copy if arg0 is still live past the op.
|
|
IsResultInArg0(op ssa.Op) bool
|
|
|
|
// NeedsTemp reports whether the op needs a scratch register
|
|
// distinct from its inputs and outputs. Used for ops whose lowering
|
|
// performs an internal computation it can't fold into the
|
|
// destination — none in our current opset, included for parity
|
|
// with Go's allocator so future lowerings can opt in.
|
|
NeedsTemp(op ssa.Op) bool
|
|
|
|
// IsRematerializeable reports whether the op can be re-issued at
|
|
// each use site cheaply (no side effects, no slot read, the result
|
|
// is a pure function of constants and SP/SB). OpConst32 and
|
|
// OpConst64 with a small immediate qualify on amd64; OpParam never
|
|
// qualifies (FP-relative read, not a constant).
|
|
IsRematerializeable(v *ssa.Value) bool
|
|
|
|
// FixedRegForFixedOp reports the hard-wired register for ops whose
|
|
// output position has exactly one acceptable register. Returns
|
|
// noRegister when the op is not fixed (the common case).
|
|
// Examples: OpSP → SP, OpSB → SB. Never used for ALU ops.
|
|
FixedRegForFixedOp(v *ssa.Value) register
|
|
|
|
// MaxCallArgs reports the largest CallArgs frame size that the
|
|
// allocator should reserve. Currently unused (asmgen computes the
|
|
// callee-area separately); included for parity with Go and to
|
|
// document the convention.
|
|
MaxCallArgs() int
|
|
}
|