mirror of
https://github.com/hanzoai/wasm2go.git
synced 2026-10-30 15:06:48 +00:00
743b14bbfa
* asmgen: emit the branch-based EH ops in direct-asm bodies Functions containing exception handling fell back to the listing transform because the direct emitters rejected every exc op. The branch-based EH state is plain Module fields, so the whole family lowers to inline MOVs against the module pointer, mirroring the inlined-global path on both architectures: - OpExcPending / OpExcTag / OpExcVal: loads into the value's slot (OpExcVal narrows the uint64 operand slot by loading the low bytes, or the raw bits for floats, at the slot's base address. - OpExcRaise: store the tag, widen each operand into its slot (zero-extend for 32-bit values, raw bits for floats), set the pending flag. - OpExcRearm / OpExcClear: flag stores. - OpCatchArg: narrow the dispatch's int64-typed snapshot. The exception-state fields move into the Module struct's fixed-offset leading region (right after the outline pack) so the offsets stay trivially modeled instead of requiring the whole struct tail: codegen computes them (moduleExcOffsets), they ride Result.DirectAsmExc -> gcasm.Config -> asmgen FuncOptions.Exc, and the generated compile-time layout pins assert them — including, now, for memory-less modules. The bundle passes Config to emitDirectAsmBody directly: offsets previously rode the SIMD memory probe, which does not exist for modules without SIMD memory ops. Two adjacent defects fixed along the way: - The pure-Go emitter produced math.Float64frombits(int64) — a compile error — when a catch handler read an f64 operand, because narrowExcSlot assumed a uint64-typed slot while OpCatchArg feeds it the dispatch's int64 snapshot. - i64_reinterpret_f64 / f64_reinterpret_i64 were missing from the always-inline helper set, forcing a per-function fallback under ForbidCalls; amd64 already carried (dead) inline coverage and arm64 gains the two-instruction FMOV lowering. New gates: an EH fixture whose seven functions all emit direct-asm on both architectures, verified differentially against the normal backend plus pinned values across all four operand widths, executed on amd64 and natively on arm64; a unit test pins the offset model. Co-Authored-By: Claude Fable 5 <[email protected]> EOF ) * helpers: fix memoryInit64 disabling shared-image CoW for threads builds memoryInit64 saturated dataEnd to MaxUint32 on EVERY install - a leftover from before memory64 images were shareable. A threads build makes every data segment passive (__wasm_init_memory installs them via memory.init at start-up), so the very first install saturated the extent and the sharing layer refused the image: every wasm64-threads engine silently lost copy-on-write model sharing and fell back to private loads. Raise dataEnd to the real extent like the 32-bit memoryInit, and saturate only when a segment genuinely lands beyond the uint32 bookkeeping range. Also adopt the 32-bit variant's compare-then-write path so an image-backed instance re-running the start section over a copy-on-write map does not fault identical pages private. Co-Authored-By: Claude Fable 5 <[email protected]> * codegen: carry the source SSA value on SIMD call marks The fusion pass will use it to report each fused window's member and root values back to direct-asm retention, so the asm emitter can splice the same windows straight from the retained SSA instead of re-deriving the match. Co-Authored-By: Claude Fable 5 <[email protected]> * codegen: record SSA provenance for fused-signature parameters Each fused-tree parameter (scalar/float/pair) now carries where it came from: a compile-time constant, or a member SSA value's argument position. Chased compound expressions and named-local sum bases get no provenance — a window carrying one will keep per-op splices in direct-asm bodies rather than guess. The chase-rollback path truncates the provenance array in lockstep with the parameters. Pure bookkeeping: fusion decisions and emitted output are unchanged; the direct-asm window retention that consumes this lands next. Co-Authored-By: Claude Fable 5 <[email protected]> * codegen: track member SSA values per fused-tree node nodeVals parallels the builder's node list with the SSA value each member call was built from (nil for synthesized scalar-chain nodes), remapped through scheduleNodes and rolled back by restoreChase in lockstep. Together with the parameter provenance this gives window retention everything it needs to name a window's members, roots and arguments in the retained SSA. Co-Authored-By: Claude Fable 5 <[email protected]> * codegen: report committed fused windows to direct-asm retention At both fusion commit points (single-tree and window), a retained function now gets a DirectAsmWindow descriptor: the interned tree, the member and root SSA values, and the fused signature's parameter sources. Recording is strict — any parameter without provenance or root without a member value skips the window (FuseDebug logs why), leaving the direct body on per-op splices. Verified against the llama bundle: the mechanism records windows with full provenance end to end (23 windows in one retained function); the asm-bundle consumer lands next. Co-Authored-By: Claude Fable 5 <[email protected]> * asmgen: plumb fused-window descriptors to the direct-asm emitters Canonical FusedWindow/FusedParamSrc types live in asmgen; transpile converts the codegen retention's descriptors, gcasm carries them on DirectAsmFn, and emitDirectAsmBody hands them to FuncOptions. No consumer yet — emission still takes the per-op path for every window member. Co-Authored-By: Claude Fable 5 <[email protected]> * asmgen: define the optional FusedSplicer interface Per-window splice entry point the emitters will call: operands in fused-signature order as immediates or frame slots, root result slots for the epilogue, false to keep per-op emission. Type-asserted from the existing Splicer so per-op-only implementations are unaffected. Co-Authored-By: Claude Fable 5 <[email protected]> * asmgen: plan fused windows against the retained SSA planFusedWindows validates each codegen-claimed region — members in one block and unclaimed, non-root members consumed only inside the window — and resolves the fused signature's parameter sources to frame-slot/immediate operands plus root slot targets. Any mismatch is a quiet per-op fallback. Emission consumption lands next. Co-Authored-By: Claude Fable 5 <[email protected]> * asmgen: emit validated fused windows at their last member The whole window body emits at the LAST member — every interleaved scalar input is materialized by then — and earlier members emit nothing. Plan validation makes the deferral observation-free: values between the members must be pure scalar glue (no memory ops, no effects, no reads of a root), non-root members must have no consumers outside the window, and planning only runs when the splicer implements FusedSplicer. A splicer decline at emission is a per-function fallback to the listing transform, never a half-emitted body. Co-Authored-By: Claude Fable 5 <[email protected]> * gcasm: emit fused-window bodies inside direct-asm functions SpliceFused stages the fused ABIInternal signature exactly as gc's call site would — m in R0 from the emitter's R24 cache, scalars and pair halves in R1.., float32 leaves in F0.. — drops in the same prolog+core+epilogue the capture path splices, lands each root's result register pair in the root value's slot, and re-primes the m cache and hoisted module state the fused body clobbers. Two defects the SIMD differential gate caught on the way in: - The fused pool registers (v16..v30) overlap the direct-asm text passes' cache and home ranges, and the passes' line model does not understand window interiors: window-bearing bodies skip the four passes (a marker comment keys it; integrating the passes around windows is tracked separately). - Unpacked parameters are FP-resident and their RSP slots are never written: operand resolution now emits FP-relative references for them, exactly like the per-op operand resolvers, instead of reading an unwritten slot. WASM2GO_FUSEDWIN_MAX (env) caps windows per function for bisection. Co-Authored-By: Claude Fable 5 <[email protected]> * asmgen: allow interleaved plain loads across load-only fused windows The codegen fusion proved exactly this hoist when it claimed a region past scalar loads (f16 scale reads sit between the gather members in every hot attention window); a window that writes no memory can be deferred past them without changing any observation. Storing windows still reject interleaved loads. Plan rejects now report their reason under WASM2GO_FUSEDWIN_DEBUG. Co-Authored-By: Claude Fable 5 <[email protected]> * asmgen: pin window operands and disable splice coalesce around windows Two llama-scale correctness fixes the NLL parity gate demanded: window operand sources keep dedicated slots (their reads happen at the window's deferred emission point, past the reuse model's lifetimes), and the splice coalesce pass is off for window-bearing bodies — a coalesced v128 loop carry lives in a home register and never materializes the slot the window stages from, and the fused pool clobbers the home ranges regardless. With these the full go-llama suite (native NLL parity included) passes on a bundle emitting the hottest attention function through five fused windows. The configuration is slower than the gc path for now: skipping the text passes and the coalesce costs the function's per-op remainder more than the windows gain, so the feature stays opt-in while pass integration around windows lands. Co-Authored-By: Claude Fable 5 <[email protected]> * gcasm: run the text passes around fused windows instead of skipping The register-rewriting passes (slot forwarding, spare-register cache) operate per outside-window segment with state flushed at each window boundary; the read-counting dead-store pass runs whole-body so a store whose only reader sits inside a window survives; windows are bracketed by begin/end marker comments and passed through verbatim. Full go-llama gates (NLL parity included) pass on the hottest attention function emitted with five fused windows. Perf remains behind the gc path for the whole function (the direct body's per-op remainder still lacks gc-grade register allocation), so the feature stays opt-in. Co-Authored-By: Claude Fable 5 <[email protected]> * gcasm: convert f16 scales with FCVT instead of the f32 table lookup The guest code converts f16 scale factors through ggml's f32 table (load16_u -> <<2 -> + table base -> f32 load); the arm64 host has hardware half-to-single conversion, so the fused scalar-chain splicer now emits one H load plus FCVT where it recognized the lookup, dropping the dependent second load. The conversion is bit-exact (IEEE f16->f32 is exact, which is precisely what the table encodes). Method per the asm-first rule: the sequence was first patched by hand into a built bundle and measured (~6% faster tg on the q8_0 routing bench, NLL parity green) before being wired into generation; the generated form reproduces the measurement (80 sites across the dot kernels and the attention windows vs 40 in the hand patch). A companion hand experiment (eliding the mov feeding scvtf) measured no additional win and is intentionally not implemented. Co-Authored-By: Claude Fable 5 <[email protected]> * codegen: rewrite f16 table gathers to vector loads in fused windows The guest converts f16 values (attention KV, activations) through ggml's f32 lookup table one element at a time: scalar load16_u reads feed shifted indices into a load32_zero + 3x load32_lane chain over the table — eight dependent loads per four elements. The sources are consecutive halfwords, so the whole group equals f16x4_cvt(v128_load16x4_u(base)): one D-register load plus FCVTL, the exact shape the native kernel compiles to. The SSA-level gather pass cannot claim this form (the shift amount rides a single-assignment local, constant only through definition chasing), so the normalization runs on the built fused tree, where the chase machinery lives: each lane's index expression is chased to a scalar load16_u, lane adjacency (+0,+2,+4,+6) is proven structurally with the delta carried by one constant arm (argOffsetBy), the table is verified byte-for-byte as the IEEE conversion, and the chain is rewritten with dead nodes and retired parameters compacted. Supporting pieces: a function-wide single-assignment definition map consulted by the chase (gated to this rewrite so ordinary fusion keeps byte-identical shapes), canonical base+const folding under the same gate, a wider shift-amount conversion unwrap, and start-section placement recovery for passive data segments so the table verification works on shared-memory (threads) links, whose segments are all passive. Method per the asm-first rule: the ldr d + fcvtl form was first patched by hand into a built bundle and measured (~3.5% on the routing bench, NLL parity green) before this generation landed; the generated form measures 9-14% faster than its baseline — beyond the hand patch, because retiring the gather also retires the per-element scalar load glue the hand patch left running. Full suite, lint, and the go-llama gates (native NLL parity included) are green; three of the four hot attention windows rewrite, the fourth keeps the literal gather pending a lane-order generalization. Co-Authored-By: Claude Fable 5 <[email protected]> * codegen: recognize word-unpack and offset-node f16 gather variants The f16 gather rewrite only matched groups whose four table indices chase to separate load16_u reads at literal base+0/+2/+4/+6. The attention kernel's hottest gather takes two other shapes: - Word-unpack: the guest loads four f16 values as one 64-bit word and derives each table index by shift and mask - lane 0 as (w & 0xffff) << 2 and lanes 1..3 as (w >>u 16k-2) & 0x3fffc plus the table base, the strength-reduced form clang emits. The mask after the pre-scaled shift selects exactly halfword k times 4, so the group equals the same f16x4_cvt(v128_load16x4_u(base)). - Offset nodes: lane addresses that chase to base+const add NODES rather than scalar/sum forms; argOffsetBy now peels constant add arms on either side, carrying the delta arithmetically. Chase additions, all gated on the gather pass so ordinary fusion keeps byte-identical shapes: identity-mask elimination (load16_u masked with low-16-ones), and bare *(*uint16) derefs assigned straight to wider locals. Measured on the routing benchmark (single thread, interleaved A/B, two rounds): decode 14.4/14.6 -> 16.6/16.8 tok/s (+15%). Full transpiler suite, golangci-lint, and the dependent go-llama suite (token-parity and NLL gates included) all pass; rewritten-group count rises from 55 to 97 with no rejected groups left in the attention kernel. Co-Authored-By: Claude Fable 5 <[email protected]> * gcasm: carry host pointers in fused-loop splices (arm64) Fused loops addressed every memory operand as mBase+scalar+const, re-materializing the effective address in scratch per access. The q8_0 SDOT decode kernel spends its time in exactly these loads, so the splicer now carries a host pointer (mBase + bump-carried address scalar) in a free argument register, recomputed at the top of each unroll copy, and the load emitters use immediate-offset forms off it with a running bias that steps the pointer when an offset leaves the signed-9-bit window. The coalesced range checks fold to one compare off the guest scalar with the same reassociated bound as before. The f16-scale peephole reads its half-word through the carried pointer the same way. Straight-line window splices are untouched, as is amd64; WASM2GO_NO_HOSTPTR disables the fold for A/B runs. Proven by hand-editing the generated kernel to the native shape first (per the asm-first rule): the hand patch measured +5.5% on single-thread decode, and this deterministic form reproduces it — routing benchmark decode 16.8 -> 17.8 tok/s (+5-6%, two interleaved rounds, all routes exact). A prefetch variant measured negative and a bare address-glue fold measured neutral; neither is included. Full transpiler suite, golangci-lint, and the dependent go-llama suite (token-parity and NLL gates) all pass. Co-Authored-By: Claude Fable 5 <[email protected]> * codegen: batch vec_dot caller row loops through a companion (opt-in) The mul_mat drivers call the verified vec_dot once per output row through the indirect table; per-call machinery (frame, module-state reloads, dispatch, reduction epilogue) measured ~20% of single- thread decode on the reference workload. With the new -vec-dot-rows flag (requires -vec-dot-pair-entry) the translator now: - emits a row-looped companion of the verified vec_dot: its emitted body is cloned (print/reparse round-trip), returns rewrite to a continue label, parameters become fresh per-row copies so mutable locals and wasm zero-init semantics hold, and the whole loop runs in one call frame. The boundary is packed through the per-module outline scratch (m + nine scalars exceeds the amd64 register ABI), and the companion rides the synthetic-signature channel so the asm bundle transforms it like an outlined function — the fused-loop splices apply inside the row loop. - rewrites matching driver row loops (indirect call + mBase reload + counter step + guarded backedge with pointer bumps) into a GUARDED one-call fast path; the original loop remains as the guard-miss branch, so semantics are preserved for every runtime table target, and modules without the flag are untouched. The traits scan learns passive data segments (threads links emit only those) via the start-section placements, gated on the same flag; the nrows patch that switches the guest to paired two-row calls is suppressed under it, since batching needs the drivers to keep stepping one row per call. Measured on the routing benchmark (single thread, interleaved A/B, two rounds, all routes exact): decode 17.2 -> 18.6 tok/s (+8%). Full transpiler suite, golangci-lint, and the dependent go-llama suite (token-parity and NLL gates) all pass; unit tests cover the loop matcher, the return rewriter and the shift-count reduction. Co-Authored-By: Claude Fable 5 <[email protected]> * gcasm: feature-gate synthetic bodies like wasm functions The feature-gated twin split (SDOT/AVX2 body + baseline twin + dispatch stub) excluded synthetic-signature functions, which was a no-op while outlined extraction bodies never contained gated instructions. The batched-rows companion clones a kernel whose body selects those forms, so its plain symbol would fault on a baseline CPU. Synthetic bodies now take the same split; the twin declarations reuse the original name's (packed) signature. Verified: the companion emits dotprod/generic twins plus the dispatch stub; full transpiler suite, golangci-lint, the go-llama suite and the routing benchmark (routes exact, throughput unchanged within noise) all pass. Co-Authored-By: Claude Fable 5 <[email protected]> * codegen/gcasm: fuse goto-form countdown loops with per-iteration scale chains Decode-critical mul_mat hosts inline their kernel loops as goto-form countdowns (label + backedge/exit jumps), which the loop upgrade previously skipped; and their per-iteration f16 scale loads reached the fused window as opaque scalar arguments, freezing the entry-time value for the whole loop. This lands the fixes that together let those loops fuse correctly: - Upgrade goto-form loops by synthesizing a ForStmt (label preserved as the live entry target, exit jump kept); on by default with WASM2GO_GOTO_FUSE_MAX as a bisection cap. - Chase computed (non base+const) memory addresses into scalar node chains under the loop-trial wide chase, so LUT indexes like idx<<2 with the table constant in the memarg offset are recomputed per iteration instead of frozen. - Narrow (wideload >> C) & 0xffff into a 16-bit load at addr + C/8 (little-endian), matching clang packing adjacent f16 scale reads into one wide load. - Recurse identWrites through nested statements so loop-variant writes inside the backedge branch are seen by the intervener consumption check; unabsorbed chains now refuse the upgrade instead of miscomputing. - Countdown-to-threshold exits (SUBS/CSEL floor form), reassociation of exactly-associative vector chains, cross-chain raw SDOT rewrite with dual alternating accumulators, whole-loop bounds-check hoisting for store-free loops, and const-pattern shuffles without ABI slots. - Deterministic carried-pointer register assignment (map order leaked into emitted bytes). Co-Authored-By: Claude Fable 5 <[email protected]> * codegen: unit-test the halfword-narrowing chase; fix its shift semantics The extract rewrite resolved shift amounts through matchShiftConst, whose default mod-32 reduction encodes i32 shift semantics — a bare 64-bit lane offset (>> 32, >> 48) would wrap to the wrong byte offset, and an out-of-range extract could pass the width check. Resolve the amount locally with no implicit reduction (the % 64 spelling of memory64 emission reduces identically), and unwrap ParenExpr so the match does not depend on the AST provenance. Adds unit coverage for the narrowing chase (lane offsets, wrapped masks, operand order, rejections), the recursive identWrites, and the carried-pointer register assignment order. Co-Authored-By: Claude Fable 5 <[email protected]> * codegen: bound the shuffle-pattern word split explicitly The four u32 lane words of a const shuffle pattern derive from the parsed 64-bit halves; mask each split explicitly so the truncation is a stated bound rather than an implicit narrowing (flagged by code scanning as an unchecked 64-to-32 conversion). Co-Authored-By: Claude Fable 5 <[email protected]> * codegen: gofmt the shuffle-pattern split Co-Authored-By: Claude Fable 5 <[email protected]> * Remove diagnostic tracing and environment kill switches The fused-loop work carried per-site trace logging (chase failures, candidate dumps, plan-reject reasons) and WASM2GO_* environment switches used while the upgrade was being isolated. The features are validated and unconditional now, so the switches and the tracing are dead weight: drop them, along with the now-unused counter, reject reason field and statement renderer. Transpiled output is unchanged (byte-identical bundle before and after). Co-Authored-By: Claude Fable 5 <[email protected]> --------- Co-authored-by: Claude Fable 5 <[email protected]>