Files
base/plugins/gojavm/runtime_test.go
zeekayandGitHub 65ef705610 feat(base): pluggable extension runtimes — wazero + v8go + goja + native + bench (#3)
* feat(base): extension runtime interface + native impl

Foundation for pluggable extension runtimes. One Runtime interface
that wazero, v8go, goja, and native-Go all satisfy. Extensions
declare their preferred runtime in extension.json; the loader
dispatches.

This sits alongside plugins/jsvm — goja-via-the-existing-jsvm-path
remains the default for .base.js hooks. The new path is for users
who want hard-sandboxed WASM extensions or full V8 perf.

Files:
- plugins/extruntime/extruntime.go — Runtime + Module interfaces, Manifest
- plugins/extruntime/manifest.go — JSON manifest loader (stdlib only)
- plugins/extruntime/native.go — reference impl, registers Go funcs

Concrete runtimes land in sibling commits:
- plugins/wasmvm — wazero (pure Go, hard sandbox)
- plugins/v8vm — v8go (cgo, -tags v8vm)
- plugins/gojavm — goja adapter (no new deps)
- plugins/extbench — fixtures + benchmarks across all 5

* feat(base): goja runtime adapter + extension loader

* feat(base): v8go extension runtime (cgo, -tags v8vm)

V8-via-cgo runtime satisfying plugins/extruntime.Runtime. Build-tag
gated so the default base binary never pulls v8go's ~50MB of cgo
deps — a stub at !v8vm reports the same Name/Capabilities and
errors at Load with ErrUnsupported.

JS calling convention: payload bytes are JSON.parse'd into a real
JS value and passed as the single argument to globalThis[fn]; the
return value is JSON.stringify'd back to bytes. Much simpler than
the wasm ptr/len dance — V8 has actual objects.

Single shared Isolate across all modules (creation is expensive,
isolates are not thread-safe so a global mutex serializes anyway).
Per-module Context pool (default 8, BASE_V8VM_POOL_SIZE) for state
isolation across back-to-back invocations. Heap cap intent via
BASE_V8VM_HEAP_MB_MAX; v8go v0.9.0 has no public per-isolate heap
API, so the value is advisory and enforcement is the V8 default
plus ctx-cancel TerminateExecution() watchdog.

Tests cover the cancel path (200ms timeout on `while(true)`), heap
bomb (2^30-char string), pool reuse cap, wrong-runtime manifest,
and closed-module rejection.

* feat(base): wazero extension runtime

Implements plugins/wasmvm — the wazero-backed extruntime.Runtime.

- Shared wazero.Runtime with compilation cache + WASI snapshot preview1
  instantiated once so guest modules can do stdio/clock/random.
- Per-Module instance pool (default 8, BASE_WASMVM_POOL_SIZE override)
  to amortize InstantiateModule across invocations.
- Host-guest ABI: guest exports fn(i32 ptr, i32 len) -> i64 (packed
  ptr<<32 | len), plus __base_alloc / __base_free for the buffer
  handshake. Missing alloc/free returns ErrUnsupported.
- Hard abort on ctx cancel via api.Module.Close(); cancelled instances
  are replaced from a background ctx so pool size stays stable.
- Tests cover identity, load+invoke, ctx cancel, concurrent pool
  reuse (16 invokes over pool size 4), invoke-after-close, and
  wrong-runtime manifest rejection. Fixture-dependent tests skip
  with TODO until extbench fixtures land.

* feat(base): extension benchmark fixtures across 5 runtimes

* feat(base): extension runtime benchmarks + writeup

plugins/extbench/ benches all five extension runtimes (native, goja,
wazero AS, wazero Javy, v8go) against an identical validate(email,age)
workload. Per-runtime Serial + Parallel + Coldstart + Memory tests.
run.sh runs the suite at -benchtime=2s -count=3 (~3 min total).

Headline numbers (Apple M1 Max, Go 1.26.3, median of 3):
  native     1,197 ns/op serial  /     694 ns/op parallel
  goja       4,513 ns/op serial  /   1,652 ns/op parallel
  wazero AS  9,956 ns/op serial  /   3,271 ns/op parallel
  v8go      11,895 ns/op serial  /  12,667 ns/op parallel (no parallelism)

Cold start ranges from native 17us to wazero AS 3.9ms. Binary size
delta from +89 KB (native) to +47.8 MB (all four with -tags v8vm).
Javy is wired but skipped pending a stdio-shim or custom Javy plugin
(its WASI command ABI doesn't match wasmvm's pointer-based contract).

Full writeup in docs/EXTENSIONS_BENCHMARK.md, raw runs in
plugins/extbench/bench-results.txt.

* feat(base): scale study — concurrent module / tenant / goroutine fanout

Five-section fan-out study answering:
1) per-loaded-module marginal memory (N up to 10000)
2) per-tenant concurrent invocation latency (T up to 1000) + pool-size elbow
3) sustained concurrency on ONE module (M up to 10000)
4) goroutine cost (1000 parked goroutines, HeapInuse delta per runtime)
5) cold pool startup vs steady-state (N=100 sequential Load)

Headlines: goja is the multi-tenant winner (~9KB/tenant, 188k ops/s at
M=1000); wazero's per-module cost is the 8-instance pool — elbow at
pool=4; v8go fatal-aborts above ~10 concurrent invokes (darwin/arm64
SIGSEGV inside libv8); native pays nothing per goroutine. Wazero
pool=256 explodes RSS to 10.8 GB at 1000 modules without latency win.

Implementation: cloneTenants() materializes a temp dir per tenant with
a unique extension name, symlinked module artifact, and native-handler
re-registration. v8go capped at concurrency=10 and skipped in
goroutine-cost — the cap itself is the data, recorded in the writeup.

Exports Validate from fixtures/native-go so scale_test can register
the same handler under N unique extension names without re-implementing.

Wall clock: <1 minute for the full study on Apple M1 Max.

* feat(base): pyvm runtime — cgo CPython 3.13 (-tags pyvm)

Fifth extension runtime alongside native/goja/wazero/v8go. Default
build links the stub; real impl behind `-tags pyvm` requires
pkg-config python-3.13-embed.

Architecture: one process-wide CPython init, per-module pool of PEP
684 OWN_GIL sub-interpreters (default 4 / BASE_PYVM_POOL_SIZE).
Each Invoke pins to OS thread, attaches a sub-interpreter, runs
__main__.fn(json.loads(payload)), returns json.dumps(result).
Free-threading (PEP 703) detected at runtime via Py_GIL_DISABLED.

Honest limitations documented in README + capability flags:
- HardSandbox=false (C extension segfault kills the host)
- SupportsAbort=false (no primitive to abort a running sub-interp
  from another OS thread without holding its GIL)

Direct cgo bridge (~150 lines), no DataDog/go-python3 or cpy3
dependency — both fail to compile against CPython 3.12+ because
they reference removed PyFloat_ClearFreeList /
PyDict_ClearFreeList. JSON marshaling stays inside Python's
stdlib json module so the cgo surface stays narrow.

Tests: 7 table tests pass on darwin/arm64 + Python 3.13.13.

* feat(base): pyvm fixture + bench integration

Adds plugins/extbench/fixtures/pyvm-py/ (validate.py + manifest)
implementing the same validate(email, age) spec as the other four
fixtures — pure stdlib, no third-party Python deps.

bench_test.go: BenchmarkPyvm_Serial + BenchmarkPyvm_Parallel skip
gracefully when built without -tags pyvm (same shape as v8go).

scale_test.go: pyvm registered alongside the four existing runtimes
with a 100-module cap (each module holds up to 4 sub-interpreters
at ~4.6 MB each — 1000 modules would be ~18 GB and dominate the
study). BASE_PYVM_POOL_SIZE plumbed through setPoolEnv /
restorePoolEnv for the pool-elbow subtest.

* docs: pyvm benchmark numbers vs existing 4 runtimes

Five-way throughput table on darwin/arm64 + Apple M1 Max + Python
3.13.13 (GIL-enabled). Headline:

- pyvm Serial 4,089 ns/op — between native (1,229) and goja (4,646)
- pyvm Parallel 2,765 ns/op — faster than wazero (4,530) and v8go
  (12,311), close to goja (1,982)
- pyvm B/op 496, allocs/op 10 — smallest footprint of any runtime
- pyvm cold start 14.3 ms/module — worst in the field; warm-load
  at boot, never lazy-load
- pyvm per-module RSS asymptote 4.6 MB — highest in the field

HIP-0105 verdict: experimental-by-default, production-recommended
for single-tenant runtime:pyvm deployments only. Multi-tenant must
add crash-isolation:none + hard-abort:none labels so operators
can policy-reject.

Free-threading (PEP 703) NOT measured — homebrew does not ship
[email protected] on darwin/arm64 as of 2026-05-18. Runtime detection
path is wired; verification needs a Linux host.

Bench crash mode found + fixed: defer-order LIFO race in
pyvm_leave/release on Parallel — restructured to straight-line
ordering. No further crash modes observed across 90+ s of load.

* feat(base): starlark extension runtime (plugins/starkvm)

Adds Google's go.starlark.net interpreter as the sixth extruntime
Runtime. Pure Go, no cgo, no GIL, per-thread Cancel() for cooperative
abort. Same Module shape as gojavm/wasmvm: pre-warmed pool, JSON payload
in / JSON result out, ctx-cancel honored.

Manifest: "runtime": "starlark", "module": "validate.star".

Fixture plugins/extbench/fixtures/starkvm-star/ implements the shared
validate(email, age) workload. Uses the permissive Starlark dialect
(while + recursion enabled via FileOptions); strict Bazel Starlark
would forbid while, doc note in the .star file.

Wired into bench_test.go (Serial/Parallel), coldstart_test.go,
memory_test.go (TestMemory_Starkvm), and scale_test.go (spec entry +
BASE_STARKVM_POOL_SIZE pool-elbow sweep). Tests cover capabilities,
roundtrip, unknown fn, post-close, ctx cancel mid-loop, pool reuse,
and the full validate fixture against 5 input cases (8 tests total).

First measurement on Apple M1 Max: 3701 B/invoke, 1 GC per 1000 calls
-- comparable to goja (3433 B/invoke).

* feat(extbench): wasm-Rust fixture + bench

Pure-Rust validate(email, age) compiled to wasm32-unknown-unknown,
loaded by the existing wasmvm runtime. Implements HIP-0105 pointer/
length ABI directly (no zip-rs dep, no Hanzo-internal deps). Compares
apples-to-apples with the AssemblyScript fixture: same ABI, same
payload, different source language.

- Cargo.toml: standalone crate (workspace=[] to escape hanzo workspace)
- src/lib.rs: no_std + bump allocator + tolerant JSON scrape
- build.sh: cargo build --release --target wasm32-unknown-unknown
- validate.wasm: 9.1 KB committed artifact (no toolchain dep at bench run)
- bench_test.go: BenchmarkWazero_Rust_{Serial,Parallel}

Run: ~4.5us serial / 3.9us parallel on M1 Max.

* docs: deferred CPython-WASI + RustPython ABI shim

Both runtimes emit WASI command modules (stdin/stdout via _start) which
don't match Base's wasmvm pointer ABI (validate(ptr,len) -> i64).

CPython-WASI deferred: requires either a 30 MB python.wasm artifact +
a stdio-piping Rust/AS shim, or building CPython from source against
the WASI SDK (~15-20 min) and forking _start. Out of 60-min budget.

RustPython deferred: tried embedding rustpython-vm 0.5.0 in a Rust
shim that exposes the pointer ABI directly. The {freeze-stdlib +
rustpython-compiler} feature combo fails to compile on wasm32-wasip1
because of cfg-gated paths in define_exception_fn!. Resolving requires
either patching the crate or waiting on upstream 0.6.x. Documented
the exact attempt + failure in fixtures/wazero-rustpython/README.md so
the next agent doesn't repeat the work.

Both fixtures stay in tree with extension.json + validate.py + README;
the wasmvm runtime reports "module not built" and the bench harness
Skip()s them with a clear message.

* test(extbench): add pyvm + wasm-Rust memory + coldstart tests

memory_test.go gains TestMemory_Pyvm + TestMemory_Wazero_Rust +
deferred-fixture skips (TestMemory_Wazero_CPythonWASI/RustPython).
runMemory helper now Skip()s when manifest.module artifact is
missing — same loadFixture path the bench harness uses, so
deferred fixtures (extension.json present, .wasm not built) skip
cleanly instead of failing.

coldstart_test.go gains BenchmarkColdstart_Pyvm + ..._Wazero_Rust +
the two deferred-fixture skips. benchColdstart already probes Load
and skips on failure, so the new ones inherit clean Skip behavior.

All tests pass under default + v8vm + pyvm tag combinations.

* bench: full cross-runtime comparison results

Captured 2026-05-19 on Apple M1 Max, default + -tags 'v8vm pyvm'.

8 runtimes measured (5 built, 3 deferred-by-ABI):
- native-go, goja, starkvm, wazero-as, wazero-rust, v8go, pyvm: full
  throughput + memory + cold-start numbers
- wazero-cpython-wasi, wazero-rustpython: Skip with "module not built"
  message (deferred per fixture README)

Headline:
- pyvm 6.9us serial (only native-go is faster) but 34ms cold-start
- starkvm 21us serial — comparable to goja, pure-Go, no JIT
- wazero-rust 14.6us serial — slightly better than wazero-as
- v8go is the slowest non-deferred runtime (cgo marshalling cost)

* docs: consolidated runtime comparison in EXTENSIONS_BENCHMARK.md

New section "Full Cross-Runtime Comparison (2026-05-19)" appended.

Covers 8 runtimes side-by-side:
- 5 originally measured (native, goja, wazero-as, v8go, pyvm)
- 3 added in this sprint (starkvm, wazero-rust, plus two
  documented deferrals: wazero-cpython-wasi + wazero-rustpython)

Sections:
- Throughput table (serial / parallel / B/op / allocs/op / coldstart)
- Per-invocation memory (TestMemory)
- Cold-start (BenchmarkColdstart)
- Scale + crash characteristics (qualitative)
- Recommendation grid (use case -> best/second/avoid)
- Headline findings (5 bullets, honest, including the deferral wall)

Numbers come from plugins/extbench/bench-results-all.txt; reproduce
section at the bottom.

* docs(pyvm): add measured 3.13t free-threading numbers + cpy3 fork note

- EXTENSIONS_BENCHMARK.md "Free-threading vs sub-interpreter
  parallelism" section now has the measured numbers (3.13 vs 3.13t,
  Apple M1 Max, default-bridge pyvm): free-threading is a non-event
  for the embedding workload (4,073 / 5,733 ns/op default-GIL vs
  4,478 / 5,805 ns/op free-threaded). PEP 684 OWN_GIL pool already
  delivers per-OS-thread parallelism; PEP 703 doesn't add headroom
  in this contract.
- Library-choice paragraph now references hanzoai/cpy3 (the
  canonical Hanzo Go binding for embedded CPython — fork of
  go-python/cpy3 with 3.12+ compat + sub-interp wrapper +
  free-threading detection + JSON hot-path helper + 3.13t build
  script) and explains why pyvm itself keeps the consolidated
  direct cgo bridge (one cgo call per Invoke, ~30% faster than
  routing through cpy3 wrappers for this workload).
- go.mod tidied — cpy3 no longer in pyvm's import graph; cpy3 is
  documented but not directly used by base.
2026-05-18 23:38:22 -07:00

218 lines
5.7 KiB
Go

package gojavm
import (
"context"
"encoding/json"
"os"
"path/filepath"
"testing"
"time"
"github.com/hanzoai/base/plugins/extruntime"
)
// writeExt writes a tiny manifest+source pair into a temp dir and
// returns the directory path. Helpers like this make every test below
// a one-liner — write the JS, point at the dir, invoke.
func writeExt(t *testing.T, name, src string) string {
t.Helper()
dir := t.TempDir()
manifest := extruntime.Manifest{
Name: name,
Version: "0.0.1",
Runtime: "goja",
Module: "index.js",
Exports: []string{"fn"},
}
mb, err := json.Marshal(manifest)
if err != nil {
t.Fatalf("marshal manifest: %v", err)
}
if err := os.WriteFile(filepath.Join(dir, "extension.json"), mb, 0644); err != nil {
t.Fatalf("write manifest: %v", err)
}
if err := os.WriteFile(filepath.Join(dir, "index.js"), []byte(src), 0644); err != nil {
t.Fatalf("write src: %v", err)
}
return dir
}
func TestRuntime_NameAndCaps(t *testing.T) {
r := NewRuntime()
defer r.Close()
if r.Name() != "goja" {
t.Fatalf("Name = %q, want goja", r.Name())
}
caps := r.Capabilities()
if len(caps.AcceptsLanguages) != 1 || caps.AcceptsLanguages[0] != "js" {
t.Fatalf("AcceptsLanguages = %v, want [js]", caps.AcceptsLanguages)
}
if caps.HardSandbox {
t.Fatal("HardSandbox should be false for goja")
}
if caps.Cgo {
t.Fatal("Cgo should be false for goja")
}
if !caps.SupportsAbort {
t.Fatal("SupportsAbort should be true (Interrupt is cooperative but present)")
}
}
func TestRuntime_LoadRejectsWrongRuntime(t *testing.T) {
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "extension.json"),
[]byte(`{"name":"x","runtime":"wazero","module":"x.wasm"}`), 0644); err != nil {
t.Fatal(err)
}
r := NewRuntime()
defer r.Close()
if _, err := r.Load(context.Background(), dir); err == nil {
t.Fatal("expected error loading wazero manifest in goja runtime")
}
}
func TestModule_InvokeRoundtrip(t *testing.T) {
dir := writeExt(t, "echo", `globalThis.fn = function(p) { return { got: p, doubled: p.n * 2 }; };`)
r := NewRuntime()
defer r.Close()
m, err := r.Load(context.Background(), dir)
if err != nil {
t.Fatalf("Load: %v", err)
}
defer m.Close()
if m.Name() != "echo" || m.Runtime() != "goja" {
t.Fatalf("Name/Runtime: %s/%s", m.Name(), m.Runtime())
}
out, err := m.Invoke(context.Background(), "fn", []byte(`{"n":21}`))
if err != nil {
t.Fatalf("Invoke: %v", err)
}
var got struct {
Got map[string]any `json:"got"`
Doubled int `json:"doubled"`
}
if err := json.Unmarshal(out, &got); err != nil {
t.Fatalf("unmarshal result %q: %v", out, err)
}
if got.Doubled != 42 {
t.Fatalf("doubled = %d, want 42", got.Doubled)
}
if got.Got["n"].(float64) != 21 {
t.Fatalf("got.n = %v, want 21", got.Got["n"])
}
}
func TestModule_InvokeUnknownFn(t *testing.T) {
dir := writeExt(t, "u", `globalThis.fn = function() { return 1; };`)
r := NewRuntime()
defer r.Close()
m, err := r.Load(context.Background(), dir)
if err != nil {
t.Fatalf("Load: %v", err)
}
defer m.Close()
_, err = m.Invoke(context.Background(), "missing", nil)
if err == nil {
t.Fatal("expected ErrUnknownFn")
}
}
func TestModule_InvokeAfterClose(t *testing.T) {
dir := writeExt(t, "c", `globalThis.fn = function() { return 1; };`)
r := NewRuntime()
defer r.Close()
m, err := r.Load(context.Background(), dir)
if err != nil {
t.Fatalf("Load: %v", err)
}
if err := m.Close(); err != nil {
t.Fatalf("Close: %v", err)
}
if _, err := m.Invoke(context.Background(), "fn", nil); err == nil {
t.Fatal("Invoke after Close should fail")
}
}
func TestModule_InvokeContextCancel(t *testing.T) {
// A while(true) loop that calls a host-visible function each
// iteration — goja's interrupt only fires at function-call
// opcodes so we keep one in the loop body.
src := `globalThis.fn = function() {
var x = 0;
while (true) { x = (x + 1) | 0; Math.sin(x); }
return x;
};`
dir := writeExt(t, "loop", src)
r := NewRuntime()
defer r.Close()
m, err := r.Load(context.Background(), dir)
if err != nil {
t.Fatalf("Load: %v", err)
}
defer m.Close()
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
defer cancel()
start := time.Now()
_, err = m.Invoke(ctx, "fn", []byte(`{}`))
elapsed := time.Since(start)
if err == nil {
t.Fatal("expected context error from infinite loop")
}
if elapsed > 2*time.Second {
t.Fatalf("interrupt took too long: %v", elapsed)
}
}
func TestModule_PrecompiledOnceAcrossInvocations(t *testing.T) {
// If we re-run the script on every Invoke, top-level state mutates
// each time. A counter on globalThis is the canary.
src := `
if (globalThis.__count === undefined) globalThis.__count = 0;
globalThis.__count++;
globalThis.fn = function() { return { c: globalThis.__count }; };
`
dir := writeExt(t, "once", src)
r := NewRuntime()
defer r.Close()
m, err := r.Load(context.Background(), dir)
if err != nil {
t.Fatalf("Load: %v", err)
}
defer m.Close()
// Force the same pool slot every time by serializing — pool runs
// sequentially so the first non-busy slot is reused. We expect the
// counter to stay at 1 across many invocations on the same slot.
for i := 0; i < 5; i++ {
out, err := m.Invoke(context.Background(), "fn", []byte(`{}`))
if err != nil {
t.Fatalf("Invoke[%d]: %v", i, err)
}
// All we care about is "the counter didn't grow with each
// invocation on the same runtime". We don't assert ==1 since
// the pool may hand us different slots, but it must never
// exceed the pool size.
var got struct {
C int `json:"c"`
}
_ = json.Unmarshal(out, &got)
if got.C > defaultPoolSize {
t.Fatalf("counter %d exceeds pool size — script ran more than once per slot", got.C)
}
}
}