Files
f57f53361e Replace the engine-specific body retargets with project-supplied assembly overrides (#67)
* codegen: keep fused-tree operands ahead of their consumers after the f16 gather rewrite

rewriteF16GatherGroup appends the collapsed v128_load16x4_u at the end
of the node list and points the chain's tail (now f16x4_cvt) at it, a
forward reference. The asm splicers walk the tree recursively and never
noticed, but the pure twin body emits nodes in list order, so any tree
that reached it through this rewrite declared the load after its use:
"undefined: n6 / declared and not used: n6" from the gcasm capture
build. It surfaced once ggml's wasm f16 vector loads started producing
the gather idiom everywhere (vec_dot_f16, vec_mad_f16, flash attention)
instead of only in the repack kernels.

compactDeadNodes, which every rewrite already runs, now places a node's
operands before the node while otherwise keeping list order — the
collapsed load lands where the chain's last lane load used to be.

* helpers: detect FEAT_I8MM on linux/arm64 (HWCAP2 bit 13)

The kernel-override dispatch and the SMMLA splices gate on it; darwin
already reported it through hw.optional.arm.FEAT_I8MM.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* Replace the engine-specific body retargets with project-supplied kernel overrides

wasm2go's job is to read a wasm module and emit Go and, where it can do
better than gc, assembly derived from that wasm. Two features had grown
past that line: the repack GEMM retarget swapped the body of an export
named dbg_gemm_q8_0_4x4 for a hand-written kernel that knows one
engine's quantized block layout, and the vec_dot row/column pairing
recognized one engine's trait-table layout and handed the paired
companion to another hand-written kernel of the same kind. Neither body
came from the wasm, and both tied the transpiler to that engine. On the
current engine build the pairing also no longer measures (prompt and
decode throughput within noise with it off), because the repack path
carries the hot matmuls.

Both are removed: the retarget and its -no-repack-gemm switch, the
pairing and its -vec-dot-pair-entry / -vec-dot-rows switches, the
kernels and the trait-table recognizer. The passive-segment placement
scan the f16-table detection shares moves to its own file.

In their place, a generic mechanism: -kernel-overrides <manifest>. The
project that produced the wasm may supply the assembly body of an
exported leaf function, per architecture and CPU feature level.
wasm2go validates the manifest against the module (export present,
signature and pointer width as declared), wraps each body in a fixed
ABI — the linear-memory base and size in named registers, arguments by
the ABI0 layout rule, a kov_oob label that reaches the module's
out-of-bounds trap — emits a feature-dispatch stub (i8mm > dotprod >
neon; avx512vnni > avx2 > sse4) with the lowered body as the portable
fallback, and pins the exports out of line. It never inspects what a
body computes; the body must stay inside the wasm execution model
(linear memory through the given base and size, bounds-checked, no
calls), and the manifest loader rejects TEXT, calls, system
instructions and foreign symbol references. docs/kernel-overrides.md
states the contract and the verification the project owes.

Tests: manifest validation, ABI text and dispatch shape, and an
end-to-end transpile of a module with and without an override body on
the host architecture — same sums, and the override traps past the end
of memory.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* helpers: detect FEAT_I8MM on darwin/arm64 too

The linux and fallback twins gained CPUI8MM; a darwin build of a bundle
whose kernel overrides declare an i8mm body needs the same symbol in
the base package.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* Rename kernel overrides to assembly overrides

"Kernel" is numeric-computing vocabulary; the mechanism itself knows
nothing about what the function computes — it replaces the body of an
exported function with project-supplied assembly. The name now says
that: -asm-overrides / Options.AsmOverrides, docs/asm-overrides.md,
the manifest's "functions" array, the ovr_oob label and ovr_ data
prefix, and Fn<idx>ovr<feature> body symbols.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

---------

Co-authored-by: Claude Fable 5.1 <[email protected]>
2026-09-04 00:10:00 +09:00

326 lines
11 KiB
Go
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
package gcasm
import (
"bytes"
"strings"
"testing"
"github.com/goccy/wasm2go/internal/codegen"
"github.com/goccy/wasm2go/internal/testfixture"
"github.com/goccy/wasm2go/internal/wasm"
)
func TestAsmSpellPath(t *testing.T) {
got := asmSpellPath("github.com/goccy/llamawasm2go/p1")
want := "github·com∕goccy∕llamawasm2go∕p1"
if got != want {
t.Fatalf("asmSpellPath = %q, want %q", got, want)
}
}
// buildMultiChunk translates a fixture in linkname-split
// multi-package mode under importPath and hands it to Build. The
// tiny 64-byte chunk budget forces the fixture across several chunk
// packages, so transformed bodies make cross-chunk calls.
func buildMultiChunk(t *testing.T, fixture, importPath string) map[string][]byte {
t.Helper()
defer codegen.SetMultiPackageThreshold(64)()
bin := testfixture.Wasm(t, fixture)
mod, err := wasm.Parse(bytes.NewReader(bin))
if err != nil {
t.Fatal(err)
}
var buf bytes.Buffer
res, err := codegen.Translate(&buf, mod, codegen.Options{Package: "pkg", OutputImportPath: importPath})
if err != nil {
t.Fatal(err)
}
treeIn := map[string][]byte{}
for name, data := range res.Sidecars {
treeIn[name] = data
}
for name, data := range res.Files {
treeIn[name] = data
}
files, _, err := Build(mod, buf.Bytes(), treeIn, importPath, nil, nil, nil, nil, Config{})
if err != nil {
t.Fatal(err)
}
return files
}
// TestCrossChunkDirectCalls: a plan-9-spellable import path (dots and
// slashes only, like the production bundles) gets direct cross-chunk
// asm CALLs — the spelled remote symbol appears in the emitted asm and
// no gcasmFwdFn forwarding wrapper is generated.
func TestCrossChunkDirectCalls(t *testing.T) {
files := buildMultiChunk(t, "cg_crosscall", "github.com/gentest/pkg")
var direct, wrappers int
for name, data := range files {
if strings.HasSuffix(name, ".s") {
direct += strings.Count(string(data), "CALL github·com∕gentest∕pkg")
}
if strings.HasSuffix(name, ".go") {
wrappers += strings.Count(string(data), "func gcasmFwdFn")
}
}
if direct == 0 {
t.Error("no spelled direct cross-chunk CALL emitted")
}
if wrappers != 0 {
t.Errorf("%d gcasmFwdFn forwarding wrappers emitted; want 0 (direct calls should replace them)", wrappers)
}
// The fixture's wide-signature fn falls back to its pure Go body,
// so its package must anchor the ABI0 wrapper the remote direct
// CALL links against.
var anchors int
for name, data := range files {
if strings.HasSuffix(name, ".s") {
anchors += strings.Count(string(data), "TEXT ·gcasmABI0Keep(SB)")
}
}
if anchors == 0 {
t.Error("no gcasmABI0Keep anchor emitted for the fallback fn's package")
}
}
// TestCrossChunkWrapperFallback: an import path plan 9 asm cannot
// spell keeps the historical Go-wrapper forwarding shape. Both
// unspellable classes are covered: a hyphenated host path, and a
// digit-leading host path (the asm lexer accepts digits only after
// the first identifier rune, so "4d63·com∕..." would lex as a
// number, not a symbol).
func TestCrossChunkWrapperFallback(t *testing.T) {
for _, importPath := range []string{"github.com/gen-test/pkg", "4d63.com/gentest/pkg"} {
t.Run(importPath, func(t *testing.T) { assertWrapperFallback(t, importPath) })
}
}
func assertWrapperFallback(t *testing.T, importPath string) {
files := buildMultiChunk(t, "cg_crosscall", importPath)
var spelled, wrappers, anchors int
for name, data := range files {
if strings.HasSuffix(name, ".s") {
spelled += strings.Count(string(data), "∕")
anchors += strings.Count(string(data), "TEXT ·gcasmABI0Keep(SB)")
}
if strings.HasSuffix(name, ".go") {
wrappers += strings.Count(string(data), "func gcasmFwdFn")
}
}
if spelled != 0 {
t.Errorf("%d spelled path references emitted for an unspellable import path", spelled)
}
if wrappers == 0 {
t.Error("no gcasmFwdFn wrappers emitted; the unspellable path must keep the forwarding shape")
}
// No direct CALL is ever made on this path, so the ABI0 keep
// anchor would be pure dead weight — it must not be emitted.
if anchors != 0 {
t.Errorf("%d gcasmABI0Keep anchors emitted for an unspellable path; none should be (no direct calls)", anchors)
}
}
// TestCrossChunkDirectAsmCallSymbols: a function retained for
// direct-asm emission whose body makes cross-chunk calls must emit
// the same CALL operand shapes the listing transform does — a
// spelled remote symbol on a spellable path, never a
// current-package-qualified mangling of it ("CALL ·github·com∕...")
// that could not link.
func TestCrossChunkDirectAsmCallSymbols(t *testing.T) {
defer codegen.SetMultiPackageThreshold(64)()
bin := testfixture.Wasm(t, "cg_crosscall_directasm")
mod, err := wasm.Parse(bytes.NewReader(bin))
if err != nil {
t.Fatal(err)
}
const importPath = "github.com/gentest/pkg"
var buf bytes.Buffer
// Fn2 ($mid) calls Fn0/Fn1 across chunks; Fn4 (the export) also
// calls the fallback fn Fn3 ($wide), covering the anchor-resolved
// shape from a direct-asm body too.
res, err := codegen.Translate(&buf, mod, codegen.Options{
Package: "pkg", OutputImportPath: importPath,
DirectAsmFuncs: []string{"Fn2", "Fn4"},
})
if err != nil {
t.Fatal(err)
}
if len(res.DirectAsmSSA) == 0 {
t.Fatal("translator retained no direct-asm SSA; the direct-asm path is not exercised")
}
directAsm := map[string]DirectAsmFn{}
for name, df := range res.DirectAsmSSA {
directAsm[name] = DirectAsmFn{Fn: df.Fn, Sig: df.Sig, Packed: df.Packed, PackedParams: df.PackedParams}
}
treeIn := map[string][]byte{}
for name, data := range res.Sidecars {
treeIn[name] = data
}
for name, data := range res.Files {
treeIn[name] = data
}
files, stats, err := Build(mod, buf.Bytes(), treeIn, importPath, nil, nil, nil, nil,
Config{DirectAsm: directAsm, DirectAsmGlobals: res.DirectAsmGlobals})
if err != nil {
t.Fatal(err)
}
if stats.DirectAsm == 0 {
t.Fatal("no direct-asm body emitted; the direct-asm path is not exercised")
}
var directBodies, spelledInDirect int
for name, data := range files {
if !strings.HasSuffix(name, ".s") {
continue
}
s := string(data)
// A spelled remote symbol re-wrapped as current-package-local
// can never link; it must not appear anywhere in the bundle.
if strings.Contains(s, "·github·com") {
t.Errorf("%s: current-package-mangled spelled symbol (CALL ·github·com...) emitted", name)
}
if !strings.Contains(s, "// direct-asm: ") {
continue
}
directBodies++
spelledInDirect += strings.Count(s, "CALL github·com∕gentest∕pkg")
}
if directBodies == 0 {
t.Fatal("no .s file carries a direct-asm body marker")
}
if spelledInDirect == 0 {
t.Error("no spelled cross-chunk CALL emitted from a direct-asm body")
}
}
// finalBundleTree returns the file set an amd64/arm64 consumer would
// actually compile: the codegen multi-package output with the gcasm
// backend's delta merged in (a nil gcasm entry deletes the file).
func finalBundleTree(t *testing.T, fixture, importPath string) map[string][]byte {
t.Helper()
return finalBundleTreeAt(t, fixture, importPath, 64)
}
// finalBundleTreeAt is finalBundleTree under an explicit chunk byte
// budget, for fixtures whose interesting shape needs two functions in
// ONE chunk package (the planner first-fit packs SCCs by size).
func finalBundleTreeAt(t *testing.T, fixture, importPath string, chunkBytes int) map[string][]byte {
t.Helper()
defer codegen.SetMultiPackageThreshold(chunkBytes)()
bin := testfixture.Wasm(t, fixture)
mod, err := wasm.Parse(bytes.NewReader(bin))
if err != nil {
t.Fatal(err)
}
var buf bytes.Buffer
res, err := codegen.Translate(&buf, mod, codegen.Options{Package: "pkg", OutputImportPath: importPath})
if err != nil {
t.Fatal(err)
}
tree := map[string][]byte{}
for name, data := range res.Sidecars {
tree[name] = data
}
for name, data := range res.Files {
tree[name] = data
}
delta, _, err := Build(mod, buf.Bytes(), tree, importPath, nil, nil, nil, nil, Config{})
if err != nil {
t.Fatal(err)
}
for name, data := range delta {
if data == nil {
delete(tree, name)
continue
}
tree[name] = data
}
return tree
}
// buildDirectiveOf returns the first //go:build expression in a Go
// source file, or "" for an untagged (all-arch) file.
func buildDirectiveOf(src []byte) string {
for _, ln := range strings.Split(string(src), "\n") {
ln = strings.TrimSpace(ln)
if strings.HasPrefix(ln, "//go:build ") {
return strings.TrimSpace(strings.TrimPrefix(ln, "//go:build "))
}
if strings.HasPrefix(ln, "package ") {
break
}
}
return ""
}
// TestPlan9SafeBundleFileStructure asserts the final bundle a
// plan-9-spellable consumer compiles has no amd64/arm64 linkname
// alias file: the codegen amd64||arm64 alias.go is deleted by the
// gcasm backend, its cross-chunk declarations fold into the
// decls_<arch>.go the asm bodies need anyway, and every remaining
// linkname-only alias file is guarded off the asm arches. This is the
// file-set contract the direct-call path relies on for parse economy.
func TestPlan9SafeBundleFileStructure(t *testing.T) {
// The file-set contract (no separate amd64/arm64 linkname alias
// file; cross-chunk decls in decls_<arch>.go) holds for BOTH the
// spellable and the hyphenated path — only the CONTENT of
// decls_<arch>.go differs (direct symbols vs gcasmFwd wrappers).
for _, importPath := range []string{"github.com/gentest/pkg", "github.com/gen-test/pkg"} {
t.Run(importPath, func(t *testing.T) { assertBundleFileStructure(t, importPath) })
}
}
func assertBundleFileStructure(t *testing.T, importPath string) {
tree := finalBundleTree(t, "cg_indirect", importPath)
// Collect the chunk packages (pN directories).
chunkDirs := map[string]bool{}
for name := range tree {
if i := strings.IndexByte(name, '/'); i > 0 && strings.HasPrefix(name[:i], "p") {
chunkDirs[name[:i]] = true
}
}
if len(chunkDirs) < 2 {
t.Fatalf("fixture did not split into multiple chunks: %v", chunkDirs)
}
for dir := range chunkDirs {
// No amd64||arm64 alias.go survives — it was deleted, its
// decls folded into decls_<arch>.go.
if _, ok := tree[dir+"/alias.go"]; ok {
t.Errorf("%s/alias.go survived; the gcasm backend must delete the amd64||arm64 alias", dir)
}
hasArchDecls := false
for _, arch := range []string{"amd64", "arm64"} {
if data, ok := tree[dir+"/decls_"+arch+".go"]; ok {
hasArchDecls = true
if d := buildDirectiveOf(data); !strings.Contains(d, arch) {
t.Errorf("%s/decls_%s.go build guard %q does not restrict to %s", dir, arch, d, arch)
}
}
}
if !hasArchDecls {
t.Errorf("%s: no decls_amd64.go/decls_arm64.go to carry the asm-body and cross-chunk decls", dir)
}
// Every linkname-only alias file left in the chunk must be
// guarded OFF the asm arches (the pure fallback path only).
for name, data := range tree {
if !strings.HasPrefix(name, dir+"/") || !strings.HasSuffix(name, ".go") {
continue
}
if !strings.Contains(string(data), "//go:linkname") {
continue
}
d := buildDirectiveOf(data)
// The arch decls legitimately carry linkname forwards; the
// concern is a SEPARATE alias file compiled on the asm arch.
if strings.HasSuffix(name, "/decls_amd64.go") || strings.HasSuffix(name, "/decls_arm64.go") {
continue
}
if !strings.Contains(d, "!amd64") && !strings.Contains(d, "!arm64") {
t.Errorf("%s carries //go:linkname but is not guarded off the asm arches (build %q)", name, d)
}
}
}
}