mirror of
https://github.com/hanzoai/wasm2go.git
synced 2026-10-30 15:06:48 +00:00
* gcasm: reach remote fns from fallback bodies via trampolines, never linkname pulls; link gates
Root cause of the unlinkable llamawasm2go v0.3.1 bundle ("nosplit stack
over 792 byte limit ... infinite cycle" on linux/arm64 and linux/amd64
GOAMD64=v2), latent since the direct cross-chunk calls (#61):
A chunk package whose pure-Go fallback bodies call a remote FnN
declared that FnN through a bodyless //go:linkname pull, while the
same package's transformed asm CALLed the remote through its spelled
dotted symbol. symabis reports the asm reference, the pull supplies a
declaration, and the compiler emits a DUPOK ABI0 wrapper under the
REMOTE symbol's name. The linker keeps whichever of that wrapper and
the remote's real TEXT it loads first; when the wrapper wins, the
remote's ABIInternal wrapper and the DUPOK ABI0 wrapper call each other
forever and the link fails. Small fixtures never had a fallback body
calling a remote fn, so the shape went unexercised; the llama bundle
has hundreds of such overlaps, and which of them fail is decided by
package load order.
Fix (the direct asm-to-asm CALLs are untouched — they are the hop-count
win and are harmless on their own):
- Remote fns referenced from fallback bodies on a plan-9-spellable
owner path get a LOCAL bodyless decl implemented by a NOSPLIT $0
tail-JMP trampoline to the spelled remote symbol, so the compiler
binds the decl to this package's own asm and never declares the
remote symbol. Unspellable owner paths keep the linkname pull (no
asm reference can be spelled there). The trampoline design the
bundle.go comments described was never implemented; it is now.
- abi0ArgBytes derives the trampolines' argument frame from the wasm
signature (validated against real bundle TEXT frames).
- Build refuses to emit a package whose //go:linkname-pulled symbols
intersect its spelled asm references (crossChunkPullConflicts), so
the hazard is a generator error, never a downstream link failure.
Gates:
- cg_crosscall_fallbackcaller fixture (fallback caller + transformed
sibling packed into one chunk, calling a recursive remote); shape
tests for the trampoline / pull split; a consumer link test for
linux/arm64, linux/amd64 v2 and v1 with the chunk packages imported
in ascending AND descending order (the pre-fix generator fails the
descending order exactly like the shipped bundle); gate4 executes
the fixture's exports.
- make llama-bundle-link (+ CI job): transpile the pinned llama-wasm
release wasm with llama-wasm's own options and link a consumer that
reaches every export for the three targets. With this change the
v0.3.1 llama.wasm links on all of them.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
* ci: link the llama bundle consumer in both chunk import orders; serialize the capture builds
The consumer now imports the chunk packages explicitly in ascending
and descending order (see scripts/bundle-link-consumer.sh): the linker
resolves DUPOK symbols first-come, so a wrapper emitted under another
package's symbol breaks the link only when the emitting package loads
first, and a single import order passed on a defective bundle by
layout luck in a downstream CI run while the shipped v0.3.1 failed.
The llama-bundle-link job was terminated (exit 143) four minutes in:
the gcasm capture builds the chunk packages in parallel and exhausts
the runner. GOFLAGS=-p=1 and GOMEMLIMIT bound it.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
* ci: trace memory/disk around the llama bundle link job
The job is terminated with exit 143 about four minutes after the SSA
phase, during the gcasm capture, with no message from the runner and
regardless of GOFLAGS=-p=1 / GOMEMLIMIT. Sample free/df/top-RSS every
ten seconds and print the tail (and any OOM kernel lines) in an
always-run step so the next termination carries its cause.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
* ci: stream the llama bundle job's resource samples into the step log
The always() trace step never ran: after the exit-143 termination the
runner skipped every remaining step, post steps included, i.e. the
runner itself went away. Emit the free/df/top-RSS samples from a
background loop inside the transpile step so the streamed step log
keeps everything sampled before the runner dies, and print the
runner's cpu/RAM/disk at the start.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
* ci: name the package the top-RSS compile is building in the resource samples
The samples show one gc compile process at 15 GiB on the 16 GiB
runner before it dies; record its -p import path so the offending
chunk package is identified.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
* ci: build the llama bundle gate with the latest Go 1.25 patch, not go.mod's floor
The job was terminated (exit 143) on every run: go1.25.0's compiler
needs 12-15 GiB for the largest chunk package (p0) and takes the 16 GiB
runner down, with every later step skipped. The same package compiles
in 1.8 GiB with go1.26.2 (measured locally, cold cache, -p=1), and the
production bundle is built from golang:1.25 (latest patch) inside the
wasmify image, so pin the gate to the latest 1.25 patch. GOMEMLIMIT is
dropped (it did not help; the live heap is the problem); -p=1 stays.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
* gcasm: drop the llama-specific bundle gate from this repo
wasm2go is a general transpiler and carries no configuration tied to
one downstream project: the pinned goccy/llama-wasm release, its
transpile options, the Makefile target and the CI job that reproduced
its bundle do not belong here. The consumer-link check lives in the
repos that own that bundle (goccy/llama-wasm, goccy/llamawasm2go);
this repo keeps the project-neutral gates — the Build-time
pull/spelled-reference conflict error and the fixture link test in
both chunk import orders.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
---------
Co-authored-by: Claude Fable 5.1 <[email protected]>
180 lines
6.9 KiB
Go
180 lines
6.9 KiB
Go
package gcasm
|
||
|
||
import (
|
||
"os"
|
||
"os/exec"
|
||
"path/filepath"
|
||
"regexp"
|
||
"sort"
|
||
"strings"
|
||
"testing"
|
||
|
||
"github.com/goccy/wasm2go/internal/codegen"
|
||
)
|
||
|
||
// fallbackCallerChunkBytes packs the fixture's transformed sibling and
|
||
// its pure-Go fallback caller into ONE chunk package (the planner
|
||
// first-fit packs SCCs by size), so that package both references the
|
||
// remote callee from Go (the fallback body) and CALLs it from asm
|
||
// (the sibling's spelled direct call).
|
||
const fallbackCallerChunkBytes = 100
|
||
|
||
var (
|
||
plainPullRe = regexp.MustCompile(`(?m)^//go:linkname (Fn\d+) (\S+)$`)
|
||
trampRe = regexp.MustCompile(`(?m)^TEXT ·(Fn\d+)\(SB\), NOSPLIT, \$0-(\d+)\n\tJMP (\S+)·(Fn\d+)\(SB\)`)
|
||
)
|
||
|
||
// chunkDirsOf lists the pN chunk packages of a materialised tree.
|
||
func chunkDirsOf(tree map[string][]byte) []string {
|
||
seen := map[string]bool{}
|
||
for name := range tree {
|
||
if i := strings.IndexByte(name, '/'); i > 0 && strings.HasPrefix(name, "p") {
|
||
seen[name[:i]] = true
|
||
}
|
||
}
|
||
var dirs []string
|
||
for d := range seen {
|
||
dirs = append(dirs, d)
|
||
}
|
||
sort.Strings(dirs)
|
||
return dirs
|
||
}
|
||
|
||
// TestCrossChunkFallbackBodyCallsUseTrampolines: on a plan-9-spellable
|
||
// import path, a fallback body's Go-level reference to a remote FnN
|
||
// reaches it through a local NOSPLIT tail-JMP trampoline — never a
|
||
// bodyless //go:linkname pull of the remote symbol. A pull next to the
|
||
// same package's spelled asm CALLs makes the compiler emit a DUPOK
|
||
// ABI0 wrapper under the REMOTE name; whichever of that wrapper and
|
||
// the remote's real TEXT the linker loads first wins, and when the
|
||
// wrapper wins the two ABI wrappers call each other forever (the
|
||
// "nosplit stack over 792 byte limit ... infinite cycle" link error).
|
||
// The direct asm CALLs themselves stay: they are the hop-count win of
|
||
// the direct-call path and are harmless on their own.
|
||
func TestCrossChunkFallbackBodyCallsUseTrampolines(t *testing.T) {
|
||
const importPath = "github.com/gentest/pkg"
|
||
tree := finalBundleTreeAt(t, "cg_crosscall_fallbackcaller", importPath, fallbackCallerChunkBytes)
|
||
spelledRoot := asmSpellPath(importPath)
|
||
var directCalls, tramps int
|
||
for _, dir := range chunkDirsOf(tree) {
|
||
for _, arch := range []string{"amd64", "arm64"} {
|
||
decls := string(tree[dir+"/decls_"+arch+".go"])
|
||
asm := string(tree[dir+"/"+arch+".s"])
|
||
for _, m := range plainPullRe.FindAllStringSubmatch(decls, -1) {
|
||
t.Errorf("%s/decls_%s.go: bodyless //go:linkname pull of %s (%s) — asm-referenced remote fns must use a trampoline", dir, arch, m[2], m[1])
|
||
}
|
||
directCalls += strings.Count(asm, "CALL "+spelledRoot)
|
||
for _, m := range trampRe.FindAllStringSubmatch(asm, -1) {
|
||
tramps++
|
||
if m[1] != m[4] {
|
||
t.Errorf("%s/%s.s: trampoline %s jumps to %s", dir, arch, m[1], m[4])
|
||
}
|
||
if !strings.HasPrefix(m[3], spelledRoot+"∕") {
|
||
t.Errorf("%s/%s.s: trampoline %s target %q is not a spelled remote symbol", dir, arch, m[1], m[3])
|
||
}
|
||
// The local decl the trampoline implements must be a bare
|
||
// asm decl (no linkname) so the compiler binds it locally.
|
||
if !strings.Contains(decls, "\nfunc "+m[1]+"(") {
|
||
t.Errorf("%s/decls_%s.go: no local decl for trampoline %s", dir, arch, m[1])
|
||
}
|
||
}
|
||
}
|
||
}
|
||
if tramps == 0 {
|
||
t.Error("no cross-chunk trampoline emitted for the fallback body's remote call")
|
||
}
|
||
if directCalls == 0 {
|
||
t.Error("no spelled direct cross-chunk CALL emitted; the direct-call path must survive the trampoline change")
|
||
}
|
||
}
|
||
|
||
// An unspellable import path never spells asm references, so the
|
||
// bodyless linkname pull is safe there and stays (no trampoline can
|
||
// name the remote either).
|
||
func TestCrossChunkFallbackBodyCallsKeepPullOnUnspellablePath(t *testing.T) {
|
||
const importPath = "github.com/gen-test/pkg"
|
||
tree := finalBundleTreeAt(t, "cg_crosscall_fallbackcaller", importPath, fallbackCallerChunkBytes)
|
||
var pulls, tramps, spelled int
|
||
for _, dir := range chunkDirsOf(tree) {
|
||
for _, arch := range []string{"amd64", "arm64"} {
|
||
pulls += len(plainPullRe.FindAllString(string(tree[dir+"/decls_"+arch+".go"]), -1))
|
||
asm := string(tree[dir+"/"+arch+".s"])
|
||
tramps += len(trampRe.FindAllString(asm, -1))
|
||
spelled += strings.Count(asm, "∕")
|
||
}
|
||
}
|
||
if pulls == 0 {
|
||
t.Error("unspellable path emitted no //go:linkname pull for the fallback body's remote call")
|
||
}
|
||
if tramps != 0 || spelled != 0 {
|
||
t.Errorf("unspellable path emitted %d trampolines / %d spelled references; must be 0", tramps, spelled)
|
||
}
|
||
}
|
||
|
||
// TestCrossChunkBundleLinksInEveryPackageOrder links a consumer of the
|
||
// fixture bundle for both asm targets with the chunk packages imported
|
||
// in ascending AND descending order. The linker resolves DUPOK symbols
|
||
// first-come, so a bundle that is only correct for one package load
|
||
// order is not correct: the production consumer's import graph decides
|
||
// the order, not the bundle.
|
||
func TestCrossChunkBundleLinksInEveryPackageOrder(t *testing.T) {
|
||
const modPath = "github.com/gentest"
|
||
tree := finalBundleTreeAt(t, "cg_crosscall_fallbackcaller", modPath+"/pkg", fallbackCallerChunkBytes)
|
||
dirs := chunkDirsOf(tree)
|
||
if len(dirs) < 2 {
|
||
t.Fatalf("fixture did not split into multiple chunks: %v", dirs)
|
||
}
|
||
root := t.TempDir()
|
||
write := func(name string, data []byte) {
|
||
t.Helper()
|
||
p := filepath.Join(root, name)
|
||
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
|
||
t.Fatal(err)
|
||
}
|
||
if err := os.WriteFile(p, data, 0o644); err != nil {
|
||
t.Fatal(err)
|
||
}
|
||
}
|
||
write("go.mod", []byte("module "+modPath+"\n\ngo 1.25.0\n"))
|
||
for name, data := range tree {
|
||
write("pkg/"+name, data)
|
||
}
|
||
orders := map[string][]string{"ascending": dirs}
|
||
rev := make([]string, len(dirs))
|
||
for i, d := range dirs {
|
||
rev[len(dirs)-1-i] = d
|
||
}
|
||
orders["descending"] = rev
|
||
targets := []struct {
|
||
name string
|
||
env []string
|
||
}{
|
||
{"linux/arm64", []string{"GOOS=linux", "GOARCH=arm64"}},
|
||
{"linux/amd64-v2", []string{"GOOS=linux", "GOARCH=amd64", "GOAMD64=v2"}},
|
||
{"linux/amd64-v1", []string{"GOOS=linux", "GOARCH=amd64", "GOAMD64=v1"}},
|
||
}
|
||
for orderName, order := range orders {
|
||
// The consumer must REACH the exports: the linker's dead-code
|
||
// pass drops unreferenced text before the nosplit walk, so a
|
||
// blank-import consumer would link even a poisoned bundle.
|
||
var main strings.Builder
|
||
main.WriteString("package main\n\nimport (\n\t\"fmt\"\n\n")
|
||
for _, d := range order {
|
||
main.WriteString("\t_ \"" + modPath + "/pkg/" + d + "\"\n")
|
||
}
|
||
main.WriteString("\t\"" + modPath + "/pkg\"\n)\n\nfunc main() {\n\tm := pkg.New()\n\tfmt.Println(pkg." +
|
||
codegen.ExportMethodName("fallbackcaller") + "(m, 1, 2))\n}\n")
|
||
write("cmd/"+orderName+"/main.go", []byte(main.String()))
|
||
for _, tg := range targets {
|
||
t.Run(orderName+"/"+tg.name, func(t *testing.T) {
|
||
cmd := exec.Command("go", "build", "-o", os.DevNull, "./cmd/"+orderName)
|
||
cmd.Dir = root
|
||
cmd.Env = append(append(os.Environ(), "CGO_ENABLED=0"), tg.env...)
|
||
if out, err := cmd.CombinedOutput(); err != nil {
|
||
t.Fatalf("consumer does not link with chunks imported in %s order on %s: %v\n%s", orderName, tg.name, err, out)
|
||
}
|
||
})
|
||
}
|
||
}
|
||
}
|