Files
base/plugins/v8vm/runtime_test.go
zeekayandGitHub 2fdcf61809 feat(base): pluggable extension runtimes — wazero + v8go + goja + native + bench (#3)
* feat(base): extension runtime interface + native impl

Foundation for pluggable extension runtimes. One Runtime interface
that wazero, v8go, goja, and native-Go all satisfy. Extensions
declare their preferred runtime in extension.json; the loader
dispatches.

This sits alongside plugins/jsvm — goja-via-the-existing-jsvm-path
remains the default for .base.js hooks. The new path is for users
who want hard-sandboxed WASM extensions or full V8 perf.

Files:
- plugins/extruntime/extruntime.go — Runtime + Module interfaces, Manifest
- plugins/extruntime/manifest.go — JSON manifest loader (stdlib only)
- plugins/extruntime/native.go — reference impl, registers Go funcs

Concrete runtimes land in sibling commits:
- plugins/wasmvm — wazero (pure Go, hard sandbox)
- plugins/v8vm — v8go (cgo, -tags v8vm)
- plugins/gojavm — goja adapter (no new deps)
- plugins/extbench — fixtures + benchmarks across all 5

* feat(base): goja runtime adapter + extension loader

* feat(base): v8go extension runtime (cgo, -tags v8vm)

V8-via-cgo runtime satisfying plugins/extruntime.Runtime. Build-tag
gated so the default base binary never pulls v8go's ~50MB of cgo
deps — a stub at !v8vm reports the same Name/Capabilities and
errors at Load with ErrUnsupported.

JS calling convention: payload bytes are JSON.parse'd into a real
JS value and passed as the single argument to globalThis[fn]; the
return value is JSON.stringify'd back to bytes. Much simpler than
the wasm ptr/len dance — V8 has actual objects.

Single shared Isolate across all modules (creation is expensive,
isolates are not thread-safe so a global mutex serializes anyway).
Per-module Context pool (default 8, BASE_V8VM_POOL_SIZE) for state
isolation across back-to-back invocations. Heap cap intent via
BASE_V8VM_HEAP_MB_MAX; v8go v0.9.0 has no public per-isolate heap
API, so the value is advisory and enforcement is the V8 default
plus ctx-cancel TerminateExecution() watchdog.

Tests cover the cancel path (200ms timeout on `while(true)`), heap
bomb (2^30-char string), pool reuse cap, wrong-runtime manifest,
and closed-module rejection.

* feat(base): wazero extension runtime

Implements plugins/wasmvm — the wazero-backed extruntime.Runtime.

- Shared wazero.Runtime with compilation cache + WASI snapshot preview1
  instantiated once so guest modules can do stdio/clock/random.
- Per-Module instance pool (default 8, BASE_WASMVM_POOL_SIZE override)
  to amortize InstantiateModule across invocations.
- Host-guest ABI: guest exports fn(i32 ptr, i32 len) -> i64 (packed
  ptr<<32 | len), plus __base_alloc / __base_free for the buffer
  handshake. Missing alloc/free returns ErrUnsupported.
- Hard abort on ctx cancel via api.Module.Close(); cancelled instances
  are replaced from a background ctx so pool size stays stable.
- Tests cover identity, load+invoke, ctx cancel, concurrent pool
  reuse (16 invokes over pool size 4), invoke-after-close, and
  wrong-runtime manifest rejection. Fixture-dependent tests skip
  with TODO until extbench fixtures land.

* feat(base): extension benchmark fixtures across 5 runtimes

* feat(base): extension runtime benchmarks + writeup

plugins/extbench/ benches all five extension runtimes (native, goja,
wazero AS, wazero Javy, v8go) against an identical validate(email,age)
workload. Per-runtime Serial + Parallel + Coldstart + Memory tests.
run.sh runs the suite at -benchtime=2s -count=3 (~3 min total).

Headline numbers (Apple M1 Max, Go 1.26.3, median of 3):
  native     1,197 ns/op serial  /     694 ns/op parallel
  goja       4,513 ns/op serial  /   1,652 ns/op parallel
  wazero AS  9,956 ns/op serial  /   3,271 ns/op parallel
  v8go      11,895 ns/op serial  /  12,667 ns/op parallel (no parallelism)

Cold start ranges from native 17us to wazero AS 3.9ms. Binary size
delta from +89 KB (native) to +47.8 MB (all four with -tags v8vm).
Javy is wired but skipped pending a stdio-shim or custom Javy plugin
(its WASI command ABI doesn't match wasmvm's pointer-based contract).

Full writeup in docs/EXTENSIONS_BENCHMARK.md, raw runs in
plugins/extbench/bench-results.txt.

* feat(base): scale study — concurrent module / tenant / goroutine fanout

Five-section fan-out study answering:
1) per-loaded-module marginal memory (N up to 10000)
2) per-tenant concurrent invocation latency (T up to 1000) + pool-size elbow
3) sustained concurrency on ONE module (M up to 10000)
4) goroutine cost (1000 parked goroutines, HeapInuse delta per runtime)
5) cold pool startup vs steady-state (N=100 sequential Load)

Headlines: goja is the multi-tenant winner (~9KB/tenant, 188k ops/s at
M=1000); wazero's per-module cost is the 8-instance pool — elbow at
pool=4; v8go fatal-aborts above ~10 concurrent invokes (darwin/arm64
SIGSEGV inside libv8); native pays nothing per goroutine. Wazero
pool=256 explodes RSS to 10.8 GB at 1000 modules without latency win.

Implementation: cloneTenants() materializes a temp dir per tenant with
a unique extension name, symlinked module artifact, and native-handler
re-registration. v8go capped at concurrency=10 and skipped in
goroutine-cost — the cap itself is the data, recorded in the writeup.

Exports Validate from fixtures/native-go so scale_test can register
the same handler under N unique extension names without re-implementing.

Wall clock: <1 minute for the full study on Apple M1 Max.

* feat(base): pyvm runtime — cgo CPython 3.13 (-tags pyvm)

Fifth extension runtime alongside native/goja/wazero/v8go. Default
build links the stub; real impl behind `-tags pyvm` requires
pkg-config python-3.13-embed.

Architecture: one process-wide CPython init, per-module pool of PEP
684 OWN_GIL sub-interpreters (default 4 / BASE_PYVM_POOL_SIZE).
Each Invoke pins to OS thread, attaches a sub-interpreter, runs
__main__.fn(json.loads(payload)), returns json.dumps(result).
Free-threading (PEP 703) detected at runtime via Py_GIL_DISABLED.

Honest limitations documented in README + capability flags:
- HardSandbox=false (C extension segfault kills the host)
- SupportsAbort=false (no primitive to abort a running sub-interp
  from another OS thread without holding its GIL)

Direct cgo bridge (~150 lines), no DataDog/go-python3 or cpy3
dependency — both fail to compile against CPython 3.12+ because
they reference removed PyFloat_ClearFreeList /
PyDict_ClearFreeList. JSON marshaling stays inside Python's
stdlib json module so the cgo surface stays narrow.

Tests: 7 table tests pass on darwin/arm64 + Python 3.13.13.

* feat(base): pyvm fixture + bench integration

Adds plugins/extbench/fixtures/pyvm-py/ (validate.py + manifest)
implementing the same validate(email, age) spec as the other four
fixtures — pure stdlib, no third-party Python deps.

bench_test.go: BenchmarkPyvm_Serial + BenchmarkPyvm_Parallel skip
gracefully when built without -tags pyvm (same shape as v8go).

scale_test.go: pyvm registered alongside the four existing runtimes
with a 100-module cap (each module holds up to 4 sub-interpreters
at ~4.6 MB each — 1000 modules would be ~18 GB and dominate the
study). BASE_PYVM_POOL_SIZE plumbed through setPoolEnv /
restorePoolEnv for the pool-elbow subtest.

* docs: pyvm benchmark numbers vs existing 4 runtimes

Five-way throughput table on darwin/arm64 + Apple M1 Max + Python
3.13.13 (GIL-enabled). Headline:

- pyvm Serial 4,089 ns/op — between native (1,229) and goja (4,646)
- pyvm Parallel 2,765 ns/op — faster than wazero (4,530) and v8go
  (12,311), close to goja (1,982)
- pyvm B/op 496, allocs/op 10 — smallest footprint of any runtime
- pyvm cold start 14.3 ms/module — worst in the field; warm-load
  at boot, never lazy-load
- pyvm per-module RSS asymptote 4.6 MB — highest in the field

HIP-0105 verdict: experimental-by-default, production-recommended
for single-tenant runtime:pyvm deployments only. Multi-tenant must
add crash-isolation:none + hard-abort:none labels so operators
can policy-reject.

Free-threading (PEP 703) NOT measured — homebrew does not ship
[email protected] on darwin/arm64 as of 2026-05-18. Runtime detection
path is wired; verification needs a Linux host.

Bench crash mode found + fixed: defer-order LIFO race in
pyvm_leave/release on Parallel — restructured to straight-line
ordering. No further crash modes observed across 90+ s of load.

* feat(base): starlark extension runtime (plugins/starkvm)

Adds Google's go.starlark.net interpreter as the sixth extruntime
Runtime. Pure Go, no cgo, no GIL, per-thread Cancel() for cooperative
abort. Same Module shape as gojavm/wasmvm: pre-warmed pool, JSON payload
in / JSON result out, ctx-cancel honored.

Manifest: "runtime": "starlark", "module": "validate.star".

Fixture plugins/extbench/fixtures/starkvm-star/ implements the shared
validate(email, age) workload. Uses the permissive Starlark dialect
(while + recursion enabled via FileOptions); strict Bazel Starlark
would forbid while, doc note in the .star file.

Wired into bench_test.go (Serial/Parallel), coldstart_test.go,
memory_test.go (TestMemory_Starkvm), and scale_test.go (spec entry +
BASE_STARKVM_POOL_SIZE pool-elbow sweep). Tests cover capabilities,
roundtrip, unknown fn, post-close, ctx cancel mid-loop, pool reuse,
and the full validate fixture against 5 input cases (8 tests total).

First measurement on Apple M1 Max: 3701 B/invoke, 1 GC per 1000 calls
-- comparable to goja (3433 B/invoke).

* feat(extbench): wasm-Rust fixture + bench

Pure-Rust validate(email, age) compiled to wasm32-unknown-unknown,
loaded by the existing wasmvm runtime. Implements HIP-0105 pointer/
length ABI directly (no zip-rs dep, no Hanzo-internal deps). Compares
apples-to-apples with the AssemblyScript fixture: same ABI, same
payload, different source language.

- Cargo.toml: standalone crate (workspace=[] to escape hanzo workspace)
- src/lib.rs: no_std + bump allocator + tolerant JSON scrape
- build.sh: cargo build --release --target wasm32-unknown-unknown
- validate.wasm: 9.1 KB committed artifact (no toolchain dep at bench run)
- bench_test.go: BenchmarkWazero_Rust_{Serial,Parallel}

Run: ~4.5us serial / 3.9us parallel on M1 Max.

* docs: deferred CPython-WASI + RustPython ABI shim

Both runtimes emit WASI command modules (stdin/stdout via _start) which
don't match Base's wasmvm pointer ABI (validate(ptr,len) -> i64).

CPython-WASI deferred: requires either a 30 MB python.wasm artifact +
a stdio-piping Rust/AS shim, or building CPython from source against
the WASI SDK (~15-20 min) and forking _start. Out of 60-min budget.

RustPython deferred: tried embedding rustpython-vm 0.5.0 in a Rust
shim that exposes the pointer ABI directly. The {freeze-stdlib +
rustpython-compiler} feature combo fails to compile on wasm32-wasip1
because of cfg-gated paths in define_exception_fn!. Resolving requires
either patching the crate or waiting on upstream 0.6.x. Documented
the exact attempt + failure in fixtures/wazero-rustpython/README.md so
the next agent doesn't repeat the work.

Both fixtures stay in tree with extension.json + validate.py + README;
the wasmvm runtime reports "module not built" and the bench harness
Skip()s them with a clear message.

* test(extbench): add pyvm + wasm-Rust memory + coldstart tests

memory_test.go gains TestMemory_Pyvm + TestMemory_Wazero_Rust +
deferred-fixture skips (TestMemory_Wazero_CPythonWASI/RustPython).
runMemory helper now Skip()s when manifest.module artifact is
missing — same loadFixture path the bench harness uses, so
deferred fixtures (extension.json present, .wasm not built) skip
cleanly instead of failing.

coldstart_test.go gains BenchmarkColdstart_Pyvm + ..._Wazero_Rust +
the two deferred-fixture skips. benchColdstart already probes Load
and skips on failure, so the new ones inherit clean Skip behavior.

All tests pass under default + v8vm + pyvm tag combinations.

* bench: full cross-runtime comparison results

Captured 2026-05-19 on Apple M1 Max, default + -tags 'v8vm pyvm'.

8 runtimes measured (5 built, 3 deferred-by-ABI):
- native-go, goja, starkvm, wazero-as, wazero-rust, v8go, pyvm: full
  throughput + memory + cold-start numbers
- wazero-cpython-wasi, wazero-rustpython: Skip with "module not built"
  message (deferred per fixture README)

Headline:
- pyvm 6.9us serial (only native-go is faster) but 34ms cold-start
- starkvm 21us serial — comparable to goja, pure-Go, no JIT
- wazero-rust 14.6us serial — slightly better than wazero-as
- v8go is the slowest non-deferred runtime (cgo marshalling cost)

* docs: consolidated runtime comparison in EXTENSIONS_BENCHMARK.md

New section "Full Cross-Runtime Comparison (2026-05-19)" appended.

Covers 8 runtimes side-by-side:
- 5 originally measured (native, goja, wazero-as, v8go, pyvm)
- 3 added in this sprint (starkvm, wazero-rust, plus two
  documented deferrals: wazero-cpython-wasi + wazero-rustpython)

Sections:
- Throughput table (serial / parallel / B/op / allocs/op / coldstart)
- Per-invocation memory (TestMemory)
- Cold-start (BenchmarkColdstart)
- Scale + crash characteristics (qualitative)
- Recommendation grid (use case -> best/second/avoid)
- Headline findings (5 bullets, honest, including the deferral wall)

Numbers come from plugins/extbench/bench-results-all.txt; reproduce
section at the bottom.

* docs(pyvm): add measured 3.13t free-threading numbers + cpy3 fork note

- EXTENSIONS_BENCHMARK.md "Free-threading vs sub-interpreter
  parallelism" section now has the measured numbers (3.13 vs 3.13t,
  Apple M1 Max, default-bridge pyvm): free-threading is a non-event
  for the embedding workload (4,073 / 5,733 ns/op default-GIL vs
  4,478 / 5,805 ns/op free-threaded). PEP 684 OWN_GIL pool already
  delivers per-OS-thread parallelism; PEP 703 doesn't add headroom
  in this contract.
- Library-choice paragraph now references hanzoai/cpy3 (the
  canonical Hanzo Go binding for embedded CPython — fork of
  go-python/cpy3 with 3.12+ compat + sub-interp wrapper +
  free-threading detection + JSON hot-path helper + 3.13t build
  script) and explains why pyvm itself keeps the consolidated
  direct cgo bridge (one cgo call per Invoke, ~30% faster than
  routing through cpy3 wrappers for this workload).
- go.mod tidied — cpy3 no longer in pyvm's import graph; cpy3 is
  documented but not directly used by base.
2026-05-18 23:38:22 -07:00

234 lines
5.8 KiB
Go

//go:build v8vm
// +build v8vm
package v8vm
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/hanzoai/base/plugins/extruntime"
)
// writeExt builds a minimal extension dir on disk and returns its path.
func writeExt(t *testing.T, name, src string, exports ...string) string {
t.Helper()
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "extension.json"), []byte(
`{"name":"`+name+`","version":"0.1.0","runtime":"v8go","module":"mod.js","exports":`+jsArr(exports)+`}`,
), 0o644); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "mod.js"), []byte(src), 0o644); err != nil {
t.Fatal(err)
}
return dir
}
func jsArr(s []string) string {
if len(s) == 0 {
return "[]"
}
var b strings.Builder
b.WriteByte('[')
for i, v := range s {
if i > 0 {
b.WriteByte(',')
}
b.WriteByte('"')
b.WriteString(v)
b.WriteByte('"')
}
b.WriteByte(']')
return b.String()
}
func TestRuntime_Capabilities(t *testing.T) {
rt := NewRuntime()
defer rt.Close()
if rt.Name() != "v8go" {
t.Fatalf("Name = %q, want v8go", rt.Name())
}
caps := rt.Capabilities()
if !caps.Cgo || !caps.HardSandbox || !caps.SupportsAbort {
t.Fatalf("unexpected caps: %+v", caps)
}
if len(caps.AcceptsLanguages) != 1 || caps.AcceptsLanguages[0] != "js" {
t.Fatalf("languages = %v, want [js]", caps.AcceptsLanguages)
}
}
func TestModule_Invoke(t *testing.T) {
rt := NewRuntime()
defer rt.Close()
dir := writeExt(t, "echo", `
globalThis.echo = function(p) { return { in: p, ok: true }; };
globalThis.upper = function(p) { return p.s.toUpperCase(); };
globalThis.sum = function(p) { return p.a + p.b; };
globalThis.nothing = function() { /* returns undefined */ };
`, "echo", "upper", "sum", "nothing")
mod, err := rt.Load(context.Background(), dir)
if err != nil {
t.Fatal(err)
}
defer mod.Close()
cases := []struct {
name string
fn string
in string
want string
}{
{"echo-object", "echo", `{"x":1}`, `{"in":{"x":1},"ok":true}`},
{"upper-string", "upper", `{"s":"hello"}`, `"HELLO"`},
{"sum-numbers", "sum", `{"a":2,"b":3}`, `5`},
{"undefined-result", "nothing", `null`, `null`},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got, err := mod.Invoke(context.Background(), tc.fn, []byte(tc.in))
if err != nil {
t.Fatal(err)
}
if string(got) != tc.want {
t.Fatalf("got %s, want %s", got, tc.want)
}
})
}
}
func TestModule_UnknownFunction(t *testing.T) {
rt := NewRuntime()
defer rt.Close()
dir := writeExt(t, "ext", `globalThis.exists = function() { return 1; };`, "exists")
mod, err := rt.Load(context.Background(), dir)
if err != nil {
t.Fatal(err)
}
defer mod.Close()
if _, err := mod.Invoke(context.Background(), "missing", []byte("null")); !errors.Is(err, extruntime.ErrUnknownFn) {
t.Fatalf("err = %v, want ErrUnknownFn", err)
}
}
func TestModule_CancelTerminates(t *testing.T) {
rt := NewRuntime()
defer rt.Close()
dir := writeExt(t, "spin", `globalThis.spin = function() { while (true) { /* tight loop */ } };`, "spin")
mod, err := rt.Load(context.Background(), dir)
if err != nil {
t.Fatal(err)
}
defer mod.Close()
ctx, cancel := context.WithTimeout(context.Background(), 200*time.Millisecond)
defer cancel()
start := time.Now()
_, err = mod.Invoke(ctx, "spin", []byte("null"))
dur := time.Since(start)
if err == nil {
t.Fatal("expected error from cancelled infinite loop")
}
if dur > 5*time.Second {
t.Fatalf("TerminateExecution did not fire in time: %v", dur)
}
}
func TestModule_HeapBomb(t *testing.T) {
rt := NewRuntime()
defer rt.Close()
// Allocate ~200MB string to blow past default heap; V8 throws
// RangeError. We just need the call to error, not panic.
dir := writeExt(t, "bomb", `
globalThis.bomb = function() {
var s = "x";
for (var i = 0; i < 30; i++) { s = s + s; } // 2^30 chars
return s.length;
};`, "bomb")
mod, err := rt.Load(context.Background(), dir)
if err != nil {
t.Fatal(err)
}
defer mod.Close()
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
_, err = mod.Invoke(ctx, "bomb", []byte("null"))
if err == nil {
t.Fatal("expected OOM / range error from heap bomb")
}
}
func TestModule_PoolReuse(t *testing.T) {
// Pool size of 2; 10 invocations should reuse contexts (we check
// the pool never grows past poolSize).
t.Setenv(envPoolSize, "2")
rt := NewRuntime()
defer rt.Close()
dir := writeExt(t, "p", `globalThis.id = function(p) { return p; };`, "id")
mod, err := rt.Load(context.Background(), dir)
if err != nil {
t.Fatal(err)
}
defer mod.Close()
for i := 0; i < 10; i++ {
out, err := mod.Invoke(context.Background(), "id", []byte(`{"i":1}`))
if err != nil {
t.Fatal(err)
}
if string(out) != `{"i":1}` {
t.Fatalf("got %s", out)
}
}
mImpl, ok := mod.(*module)
if !ok {
t.Fatal("module is not *module")
}
mImpl.mu.Lock()
n := len(mImpl.pool)
mImpl.mu.Unlock()
if n > 2 {
t.Fatalf("pool grew past cap: %d", n)
}
}
func TestRuntime_LoadRejectsWrongRuntime(t *testing.T) {
rt := NewRuntime()
defer rt.Close()
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "extension.json"),
[]byte(`{"name":"x","runtime":"wazero","module":"x.wasm"}`), 0o644); err != nil {
t.Fatal(err)
}
_, err := rt.Load(context.Background(), dir)
if !errors.Is(err, extruntime.ErrUnsupported) {
t.Fatalf("err = %v, want ErrUnsupported", err)
}
}
func TestModule_ClosedRejects(t *testing.T) {
rt := NewRuntime()
defer rt.Close()
dir := writeExt(t, "c", `globalThis.f = function() { return 1; };`, "f")
mod, err := rt.Load(context.Background(), dir)
if err != nil {
t.Fatal(err)
}
if err := mod.Close(); err != nil {
t.Fatal(err)
}
if _, err := mod.Invoke(context.Background(), "f", []byte("null")); !errors.Is(err, extruntime.ErrClosed) {
t.Fatalf("err = %v, want ErrClosed", err)
}
}