mirror of
https://github.com/hanzoai/next.js.git
synced 2026-09-24 17:00:51 +00:00
[internal] Add a skill for benching changes in a sandbox (#95943)
Vibed. This should help with running React PRs on sandboxes to measure impact on `bench/`. ## Usage ``` use bench skill to to measure impact of https://github.com/react/react/pull/37066 ``` - Should be able to point at React or Next PRs - Compares against mergebase - Grabs React builds from CI when possible - Before benching, asserts both React builds are green, or that tests pass - Runs on many sandboxes with interleaved runs - Caches reusable stuff (builds) between sandboxes - Uses `bench/` examples with realistic payloads ## Test Plan Running a bench: <img width="1445" height="584" alt="Screenshot 2026-07-20 at 03 08 02" src="https://github.com/user-attachments/assets/ac8b6aac-1e62-4695-9a59-fd0577a8a441" /> Bench that shows good results: <img width="1078" height="725" alt="Screenshot 2026-07-20 at 03 46 20" src="https://github.com/user-attachments/assets/ce49a806-f1b1-497c-a3dd-e5f8d08ea25b" /> Neutral bench: <img width="1032" height="455" alt="Screenshot 2026-07-20 at 03 47 27" src="https://github.com/user-attachments/assets/de3b700c-4875-4c70-b1c1-55cdddd73a1a" /> --------- Co-authored-by: Claude Fable 5 <[email protected]>
This commit is contained in:
dan
@@ -0,0 +1,298 @@
|
||||
---
|
||||
name: sandbox-bench
|
||||
description: >
|
||||
Benchmark React or Next.js changes on Vercel Sandbox VMs with paired
|
||||
A/B statistics: react PR/commit vs base, or Next.js PR/commit vs base,
|
||||
measured end-to-end through the bench/render-pipeline app (rps,
|
||||
latency, p95; TTFB, RSS and document/Flight bytes when the Next
|
||||
side captures them). Use whenever the user asks to bench, perf
|
||||
test, or A/B a React PR, a react-server-dom / Flight / vendored React
|
||||
change, or a Next.js PR ("is this PR faster", "does this regress
|
||||
RSC?", "measure the perf impact of <commit>"), even if they don't say
|
||||
"benchmark" — any request to quantify a server-side performance
|
||||
difference between two revisions belongs here. Runs remotely
|
||||
(laptop-free), applies correctness gates before measuring, and
|
||||
reports boot-level confidence intervals.
|
||||
metadata:
|
||||
internal: true
|
||||
---
|
||||
|
||||
# Sandbox bench: paired A/B perf runs for React and Next.js changes
|
||||
|
||||
Measures what a change is actually worth, end to end: two revisions
|
||||
("arms") built into otherwise-identical Next.js apps, exercised by the
|
||||
`bench/render-pipeline` harness on Vercel Sandbox VMs, compared with
|
||||
paired statistics that treat the VM boot as the unit of replication.
|
||||
All heavy work happens on sandbox VMs; the laptop only orchestrates.
|
||||
|
||||
Scripts live in `scripts/` next to this file and run from anywhere.
|
||||
Arms are git refs, resolved in cached clones of react and next.js;
|
||||
the Next side defaults to canary. Everything is cached
|
||||
content-addressed: first use of a new pair builds caches (~45-60 min
|
||||
extra, once); later runs boot straight into measurement.
|
||||
|
||||
## One-time setup
|
||||
|
||||
1. `node scripts/config.mjs show` — if it reports NOT CONFIGURED, ask
|
||||
the user which Vercel **team** and **project** the sandbox VMs
|
||||
should run under (these are billed resources; never guess, never
|
||||
default), then `node scripts/config.mjs set team=<slug> project=<name>`.
|
||||
Config lives in `~/.config/sandbox-bench/config.json` — never commit
|
||||
team/project names into the repo.
|
||||
2. The Vercel CLI session must have access to that team. On a 403,
|
||||
stop launching (don't retry through it) and check whether access is
|
||||
already back: `vercel whoami --scope <team-slug>` plus one scoped
|
||||
read call (e.g. `vercel sandbox ls`) — grants drop and recover on
|
||||
their own, and a transient 403 needs no login at all. If
|
||||
verification still fails, run `vercel login <team-slug>` yourself
|
||||
as a background task (the token lives with the CLI session, not
|
||||
with the user). It opens a browser/device confirmation — relay the
|
||||
URL if one is printed — but keep re-running the verification pair
|
||||
every minute or two while it waits: access often returns before
|
||||
the login flow reports success, and once verification passes, kill
|
||||
the pending login and resume. After a 403 outage, expect in-flight runs to have died:
|
||||
run `node scripts/bench-status.mjs` and follow its recovery
|
||||
actions (measurement VMs will have hit their ~5h timeout if the
|
||||
outage was long — those cells need relaunching, not collecting).
|
||||
3. react and next.js clones land in the cache on first use (or point
|
||||
`reactRepo`/`nextRepo` in the config at existing checkouts).
|
||||
|
||||
Before the first real run with a new configuration, sanity-check the
|
||||
plan with `--dry-run` (prints what would happen, touches nothing).
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Resolve what's being compared
|
||||
|
||||
- **React PR**: `--pr <url|number>` — base is computed automatically
|
||||
(merge-base of the PR head with react main).
|
||||
- **React refs**: `--arms base=<ref>,cand=<ref>` — base FIRST. For a
|
||||
multi-commit branch, base is the merge-base with main, not `cand^`.
|
||||
- **Next.js PR**: `--next-pr <url|number>`. The React side defaults to
|
||||
whatever each Next ref vendors (that's what would ship); pass
|
||||
`--react-ref` only to pin both arms to one specific React build.
|
||||
- **Next refs**: `--next-arms base=<ref>,cand=<ref>`.
|
||||
|
||||
Exactly one side varies; the other is identical in both arms. That
|
||||
isolation is what makes the numbers attributable — never vary both.
|
||||
|
||||
### 2. Gate correctness before spending bench compute
|
||||
|
||||
A bench number from an arm that fails its own tests is meaningless.
|
||||
For any arm that is not already CI-green upstream (hand-assembled
|
||||
branches, cherry-picks with resolved conflicts, local commits):
|
||||
|
||||
```sh
|
||||
node scripts/sandbox-gate.mjs --arms cand=<ref>
|
||||
```
|
||||
|
||||
The bench itself enforces the primary gate: every react arm's commit
|
||||
must have green CI on the react repo, checked automatically before any
|
||||
build or VM is spent. PRs and main-history commits normally satisfy
|
||||
this with no extra work. For local or unpushed refs (no CI exists),
|
||||
gate on a VM with sandbox-gate.mjs and then pass --allow-ungated to
|
||||
the bench. The VM gate runs the full test suite in prod mode (the
|
||||
channel that gets benched). PASS requires seeing the actual test
|
||||
counts in the output. If a gate fails, report the failures and stop —
|
||||
do not bench a broken arm. Each arm is gated in its own
|
||||
lockfile's environment. Bench the exact sha the gate prints (a branch
|
||||
ref can move between gate and bench).
|
||||
|
||||
### 3. Launch the bench (background, non-blocking)
|
||||
|
||||
```sh
|
||||
bash -c 'node scripts/sandbox-e2e.mjs --pr <url> --label <slug> \
|
||||
2>&1 | grep --line-buffered -v "^live "; exit ${PIPESTATUS[0]}'
|
||||
```
|
||||
|
||||
- Run it as a background task and proceed on its completion
|
||||
notification. Never hold a foreground wait; never poll in a loop —
|
||||
the rule is about control flow, not status relay: reading the
|
||||
output tail to answer "how's it going" is always fine.
|
||||
- The harness handles the invariants internally: both arms in the same
|
||||
VM, interleaved ABBA, paired per (vm, run); detached remote
|
||||
execution (transport drops don't kill runs); build fingerprints
|
||||
recorded in every result row.
|
||||
- `live ...` lines are streaming estimates for progress display only.
|
||||
Never stop a run early because a live p-value looks good, and never
|
||||
report a live number — sequential peeking manufactures false
|
||||
positives. Only the final analysis counts.
|
||||
- Defaults (16 VMs × 2 paired runs) implement the methodology; don't
|
||||
reduce VM count to save time — boots are the unit of inference, and
|
||||
fewer boots means wider intervals, not faster answers.
|
||||
- Sandbox compute is internal capacity, not a budget: launch, relaunch,
|
||||
and confirm runs without asking about cost or shrinking them to
|
||||
save it.
|
||||
- The Next side's default, `canary`, is the latest published canary
|
||||
release (the launcher prints its version and sha), so repeat benches
|
||||
reuse the built snapshot until a new canary ships.
|
||||
- Useful flags: `--bench-env KEY=VALUE` (runtime-only env for the
|
||||
bench process — it does NOT affect the snapshot's app build), `--isolate-routes`
|
||||
(tail investigations), `--no-profile` (skip the CPU capture that
|
||||
runs by default after the timed runs), `--prepare` (build caches
|
||||
only — use when two cells will share an arm, to avoid duplicate
|
||||
builds racing).
|
||||
- CPU profiles are captured by default: one profile pass per arm runs
|
||||
strictly AFTER the timed runs (it cannot touch the numbers), costs
|
||||
~45-60 min extra VM wall-clock, and lands in `<runDir>/prof-vm<N>/`
|
||||
as standard V8 `.cpuprofile` files. Cross-VM profile diffs are
|
||||
highly stable (observed 16/16 sign agreement on real movers), so one
|
||||
profiled cell suffices to rank hot paths. Analysis caveats:
|
||||
aggregate by (functionName, line, column) — bare minified names
|
||||
collide across the bundle — and never diff arms by minified name
|
||||
(the minifier renames between builds); match positions or code
|
||||
snippets instead.
|
||||
- The bench exercises Next's node-streams path
|
||||
(`__NEXT_USE_NODE_STREAMS` is inlined as true for the node runtime
|
||||
at build time). React changes that only touch the EDGE stream
|
||||
configs are not exercised end-to-end and will (correctly) bench as
|
||||
no detected difference.
|
||||
|
||||
### 4. Read the result like a skeptical data scientist
|
||||
|
||||
The goal is the truth about the change, not making its author feel
|
||||
good. The final analysis prints, per route/phase/metric, the
|
||||
boot-level mean, ±95% CI, and p across boots. Apply the policy in
|
||||
[references/methodology.md](references/methodology.md):
|
||||
|
||||
- Claim only boot-level p < 0.01, with the CI, on an A/A-validated
|
||||
team/config (see methodology).
|
||||
- The PR is a hypothesis, not an explanation. Claims come from the
|
||||
analysis output alone. When the numbers agree with the PR's story,
|
||||
check whether the captured data actually discriminates that
|
||||
mechanism from alternatives — a latency win attributed to smaller
|
||||
payloads should come with a document-bytes delta; if the bytes
|
||||
didn't move, the story doesn't hold and the report says so.
|
||||
- Use every captured metric, and voice anything that does not add up:
|
||||
one metric family moving against the others, effects with no
|
||||
byte-level or RSS trace, throughput moving without latency,
|
||||
sign flips across boots. An inconsistency you cannot explain
|
||||
belongs in the report, not in the drawer.
|
||||
- The `within-run p` shown in brackets is a diagnostic, never a claim.
|
||||
- Check the fingerprint header first: two distinct fingerprints = valid
|
||||
A/B; "inconsistent fingerprints" = invalid, report no numbers. The
|
||||
fingerprint hashes both bundlers' compiled server files — arms
|
||||
touching only client files can still legitimately show identical
|
||||
fingerprints with different version strings.
|
||||
- Per-boot values are printed; if boots disagree in sign, say so.
|
||||
- Any claim that will drive a decision gets one independent
|
||||
confirmation run before it's stated as fact.
|
||||
|
||||
Re-analyze any past run without re-running it:
|
||||
`node scripts/bench-analyze.mjs <runDir>`.
|
||||
|
||||
### 5. Report
|
||||
|
||||
Name what was measured with links: the PR title (printed in the
|
||||
analysis header, stored in meta.json) linking to the PR; for ref
|
||||
arms, the commit title. Lead with a table of the significant cells,
|
||||
each row carrying the effect with its unit, the CI, and p:
|
||||
|
||||
```
|
||||
## [<PR title>](<PR url>) — e2e, Vercel Sandbox (x86 Xeon), <n> boots
|
||||
|
||||
Significant (boot-level p < 0.01, A/A-validated):
|
||||
| cell | effect | 95% CI | p |
|
||||
|---|---|---|---|
|
||||
| /dashboard under load | +14.4% throughput (req/s) | ±3.2% | <0.0001 |
|
||||
| /dashboard serial | −10.7% median latency (ms) | ±0.6% | <0.0001 |
|
||||
|
||||
No detected difference: <every cell not in the table, by name>.
|
||||
Flags: <cells at 0.01 ≤ p < 0.05, sign disagreements across boots,
|
||||
fingerprint caveats, anything that does not add up>
|
||||
```
|
||||
|
||||
One row per cell: rps and median restate each other, so report the
|
||||
throughput number (add a p95 row only when the tail moves differently
|
||||
from the median). Document metrics (raw/gzip/Flight KB) get their own
|
||||
rows when they differ — they are the mechanism evidence. When the
|
||||
Next side predates the document-metrics harness (vercel/next.js#95828)
|
||||
those cells are absent; say so instead of silently reporting less. State the platform next to the numbers. Magnitudes
|
||||
are platform-dependent (GC share differs by CPU); direction and
|
||||
mechanism transfer, percentages do not. Never present a
|
||||
noise-compatible delta as a small win or loss — it is "no detected
|
||||
difference".
|
||||
|
||||
## Results database
|
||||
|
||||
Every collected run lands in one SQLite file,
|
||||
`~/.cache/sandbox-bench/results.db` — raw measurements and artifacts
|
||||
(CPU profiles, logs) only, written exclusively by the importer, never
|
||||
by hand. The launcher imports and verifies automatically at
|
||||
collection; `bench-analyze` reads the db and nothing else, so every
|
||||
statistic is a pure function of it. Numbers in reports come from the
|
||||
analysis output verbatim — never retype, recompute, or aggregate them
|
||||
yourself.
|
||||
|
||||
- `node scripts/bench-db.mjs ls` — all runs with sample/artifact counts.
|
||||
- `node scripts/bench-db.mjs verify [runId]` — integrity checks:
|
||||
sqlite-level, referential, one fingerprint per arm, paired sample
|
||||
counts, artifact sha256. Run it before drawing on old data.
|
||||
- `node scripts/bench-db.mjs export out.db <runId...>` — cut a
|
||||
self-contained db of specific runs (with their profiles) to send to
|
||||
someone. It opens in any SQLite tool.
|
||||
- `node scripts/bench-analyze.mjs <runId>` — re-analyze anything in
|
||||
the db; a run-dir argument imports it first.
|
||||
|
||||
## Keeping the user informed
|
||||
|
||||
The launcher narrates itself on stdout: launch facts first (run dir,
|
||||
arms, CI verdicts), then a progress line every ~2 minutes with rows
|
||||
collected and interim per-route effects with confidence. Relay to the
|
||||
user: the run dir and expected duration right after launching,
|
||||
notable interim shifts if they ask how it's going, and the full
|
||||
verdict from the final analysis when the completion notification
|
||||
arrives. The analysis names metrics that were `not captured on this
|
||||
run` — repeat that in the verdict when it limits what the data can
|
||||
say (document metrics absent means the payload mechanism is
|
||||
unverified, not verified-identical).
|
||||
|
||||
While a run is active, open any reply with a one-line status per run:
|
||||
read the tail of the launcher's output and quote its latest progress
|
||||
line. If the session supports timed wakeups or reminders, schedule a
|
||||
check at each expected transition (arm builds -> experiment snapshot
|
||||
-> measuring, then every ~15 minutes of measurement) and post the
|
||||
progress line; if not, say when the next update will arrive so
|
||||
silence is never ambiguous. Interim effects in progress lines are
|
||||
streaming estimates — share them as progress, never as claims.
|
||||
|
||||
If a launcher process dies (session teardown, crash), the remote VMs
|
||||
keep executing their measurement loops — the data is not lost. `node
|
||||
scripts/bench-collect.mjs <runDir>` reconnects, waits for the loops,
|
||||
downloads the results, cleans up, and analyzes. Run it before the VMs
|
||||
hit their ~5h timeout.
|
||||
|
||||
## Failure recovery
|
||||
|
||||
- **First move, always: `node scripts/bench-status.mjs`.** Session
|
||||
restarts silently kill background launchers while their detached VMs
|
||||
keep measuring, and a dead launcher's log still ends with a
|
||||
healthy-looking progress line — never infer liveness from log tails
|
||||
or task output files. bench-status checks each run's recorded
|
||||
launcher pid and prints the per-run recovery action (running /
|
||||
collect now / relaunch). Run it at the start of any session that
|
||||
expects work in flight, after any crash, and before telling the user
|
||||
what is or isn't running. Launcher crashes are also recorded in the
|
||||
run's status.json (`phase: "failed"` plus the error).
|
||||
- **Interrupted local process**: remote VMs keep running detached.
|
||||
`vercel sandbox list` (with the configured team/project) to find
|
||||
them; poll each VM's `/vercel/sandbox/loop.done`, `cp` its
|
||||
`results.jsonl` down when done, then remove the VM and analyze with
|
||||
`bench-analyze.mjs`.
|
||||
- **Leaked VMs** after any crash: `node scripts/sandbox-sweep.mjs`
|
||||
lists this skill's VMs (matched by sbench-\* name AND the
|
||||
purpose=sandbox-bench tag, and only when older than --min-age-hours,
|
||||
default 3, so healthy in-flight runs are never touched); `--yes`
|
||||
removes them by exact listed name.
|
||||
- **Flaky uploads/transports**: the harnesses size-check artifacts and
|
||||
abort on truncation. A failed cell is safe to relaunch; caches make
|
||||
the retry cheap. Don't relaunch two cells that need the same uncached
|
||||
arm at the same moment — they'll race to build it; use `--prepare`
|
||||
first instead.
|
||||
|
||||
## Cost expectations (set these with the user before big runs)
|
||||
|
||||
Per cell at defaults: ~18 VMs (8 measurement + build/snapshot VMs),
|
||||
~1-2h wall-clock cold, ~30-60 min warm. A/A calibration and
|
||||
confirmation runs are extra cells. VMs are billed to the configured
|
||||
team — for anything beyond a single PR check, confirm scope first.
|
||||
@@ -0,0 +1,103 @@
|
||||
# Statistical methodology
|
||||
|
||||
The design decisions below are what make the numbers trustworthy.
|
||||
They follow standard rigorous-benchmarking practice (Kalibera & Jones,
|
||||
"Rigorous Benchmarking in Reasonable Time"; Georges et al.,
|
||||
"Statistically Rigorous Java Performance Evaluation"; JMH's forks
|
||||
model) — don't relax them to save wall-clock without understanding
|
||||
what they protect against.
|
||||
|
||||
## The VM boot is the unit of replication
|
||||
|
||||
JIT state, code layout, GC rhythm, host hardware, and phase order are
|
||||
random effects fixed at boot time. Iterations within a boot share all
|
||||
of them, so they are not independent samples — treating them as
|
||||
independent produces confident nonsense (a p<1e-4 effect in one run
|
||||
that vanishes in the next). The harness therefore:
|
||||
|
||||
- computes each metric's delta per boot (mean of that boot's paired
|
||||
within-boot deltas), and
|
||||
- runs t-based inference across boots (`bench-stats.mjs`); the CI and
|
||||
p-value you see are boot-level.
|
||||
|
||||
The `within-run p` printed in brackets is a diagnostic only. It tells
|
||||
you the pairing worked; it cannot support a claim.
|
||||
|
||||
Allocation follows from this: more boots × fewer runs beats fewer
|
||||
boots × more runs at fixed budget, and boots run in parallel — adding
|
||||
them costs money, not wall-clock, so don't be shy with them when
|
||||
sensitivity matters (CI shrinks with sqrt of boots). Defaults: e2e 16
|
||||
VMs × 1 block × 2
|
||||
runs. Both arms always run in the SAME VM, interleaved ABBA, paired
|
||||
per (vm, run) — hosts differ by up to ~20%, so cross-VM comparisons
|
||||
are meaningless; VMs exist to replicate, not to compare.
|
||||
|
||||
## Claims
|
||||
|
||||
- A claim requires boot-level p < 0.01, reported with its CI, on a
|
||||
team/config whose boot-level inference an A/A run has validated.
|
||||
- Anything that will drive a decision (merge, revert, ship) gets one
|
||||
independent confirmation cell first (standard perf-CI practice —
|
||||
Mozilla/Chrome retrigger, MongoDB change-point detection — nobody
|
||||
alerts off one detection).
|
||||
- Everything else is "noise-compatible", reported as no detected
|
||||
difference — not as a small effect.
|
||||
|
||||
## Percentiles
|
||||
|
||||
A per-run p95/p99 from a small load phase is a near-max order
|
||||
statistic, not a distribution estimate. Compare percentiles only at
|
||||
boot level; p99 needs ~1k+ samples per boot to mean anything at all.
|
||||
Tail effects are also where phase carryover lives (a route's serial
|
||||
tail depends on the previous route's load phase) — investigate tails
|
||||
with `--isolate-routes`, which restarts the server between routes.
|
||||
|
||||
## A/A validation
|
||||
|
||||
Before claiming anything on a new team/config, run an A/A cell (same
|
||||
ref as both arms) at default allocation:
|
||||
|
||||
```sh
|
||||
node scripts/sandbox-e2e.mjs --arms base=<ref>,cand=<ref> --label aa
|
||||
```
|
||||
|
||||
The A/A validates the inference itself: with identical arms, no
|
||||
route/phase/metric cell should reach p < 0.01 (beyond the nominal
|
||||
false-positive rate). Cells that do mean boots are sharing conditions
|
||||
and the CIs are too narrow — fix the allocation (more spread-out
|
||||
boots) before claiming; a magnitude gate is not a substitute. The A/A
|
||||
CIs also tell you what effect sizes this config can resolve. Re-run
|
||||
the A/A when the platform changes (VM hardware generation, node
|
||||
version, bench app changes).
|
||||
|
||||
## Interim progress display
|
||||
|
||||
The launcher's progress digest shows, per route/phase, the running rps
|
||||
effect and a directional confidence — P(candidate is actually
|
||||
faster/slower | boots so far), the Student-t posterior under a flat
|
||||
prior (not a p-value). It appears from 4 completed boots. Display
|
||||
only: runs complete their planned allocation, and claims come only
|
||||
from the final fixed-n analysis.
|
||||
|
||||
## Fingerprints
|
||||
|
||||
Every result row records a build fingerprint (dist-file hash + version
|
||||
string) read from the tree that actually served requests. The analysis
|
||||
header verifies them: inconsistent fingerprints within an arm
|
||||
invalidate the run (a VM measured the wrong build). Identical
|
||||
fingerprints across arms with different version strings can be
|
||||
legitimate — the hash covers the compiled server files, so client-only
|
||||
changes don't move it; check the version strings before declaring an
|
||||
accidental A/A.
|
||||
|
||||
## Honest reporting
|
||||
|
||||
- State the platform (e.g. "x86 Xeon sandbox VMs") next to any number.
|
||||
Magnitudes are platform-dependent — GC's share of request time
|
||||
scales with CPU speed — so direction and mechanism transfer between
|
||||
platforms, percentages do not.
|
||||
- Print per-boot values; if boots disagree in sign, the mean is not a
|
||||
finding, and say so.
|
||||
- Never conclude anything from the live streaming estimates. Peeking
|
||||
at sequential results and stopping when p looks good is a classic
|
||||
way to manufacture false positives.
|
||||
@@ -0,0 +1,83 @@
|
||||
#!/usr/bin/env node
|
||||
// Post-hoc (re-)analysis of runs at the boot level. Reads ONLY the
|
||||
// results db: a run-dir argument is imported (idempotently) and
|
||||
// verified first, a bare run id analyzes what the db already holds.
|
||||
// node bench-analyze.mjs <runDir|runId> [--base <arm>] [--cand <arm>]
|
||||
import fs from 'node:fs'
|
||||
import path from 'node:path'
|
||||
import { analyzeE2eRows } from './bench-stats.mjs'
|
||||
import { openDb, importRun, loadRows, runMeta, verify } from './bench-db.mjs'
|
||||
|
||||
const argv = process.argv.slice(2)
|
||||
const target = argv.find((a) => !a.startsWith('--'))
|
||||
const get = (name) => {
|
||||
const i = argv.indexOf(name)
|
||||
return i >= 0 ? argv[i + 1] : undefined
|
||||
}
|
||||
if (!target) {
|
||||
console.error(
|
||||
'usage: node bench-analyze.mjs <runDir|runId> [--base arm] [--cand arm]'
|
||||
)
|
||||
process.exit(1)
|
||||
}
|
||||
const db = openDb()
|
||||
let runId
|
||||
if (fs.existsSync(target)) {
|
||||
runId = importRun(db, target).runId
|
||||
} else {
|
||||
runId = path.basename(target)
|
||||
}
|
||||
const { failures, notes } = verify(db, runId)
|
||||
for (const n of notes) console.log(`note: ${n}`)
|
||||
if (failures.length) {
|
||||
console.error(`RESULTS DB VERIFY FAILED:\n ${failures.join('\n ')}`)
|
||||
process.exit(1)
|
||||
}
|
||||
const rows = loadRows(db, runId)
|
||||
const arms = [...new Set(rows.map((r) => r.arm))]
|
||||
if (arms.length !== 2) {
|
||||
console.error(`expected exactly 2 arms, found: ${arms.join(', ')}`)
|
||||
process.exit(1)
|
||||
}
|
||||
// Which arm is base? Row order cannot answer this (ABBA runs the
|
||||
// candidate first), so require an explicit source: the run's meta,
|
||||
// a conventional name, or the --base flag.
|
||||
const meta = runMeta(db, runId)?.meta
|
||||
const base =
|
||||
get('--base') ??
|
||||
(meta && arms.includes(meta.base) ? meta.base : undefined) ??
|
||||
['base', 'synced'].find((n) => arms.includes(n))
|
||||
if (!base) {
|
||||
console.error(
|
||||
`cannot determine the base arm from {${arms.join(', ')}}: ` +
|
||||
'no meta, no arm named "base"/"synced" — pass --base <arm>'
|
||||
)
|
||||
process.exit(1)
|
||||
}
|
||||
const cand = get('--cand') ?? arms.find((a) => a !== base)
|
||||
const boots = new Set(rows.map((r) => r.vm)).size
|
||||
console.log(`${runId} boots=${boots} arms: ${base} (base) vs ${cand}`)
|
||||
if (meta?.pr)
|
||||
console.log(
|
||||
`PR: ${meta.pr.title ? `"${meta.pr.title}" — ` : ''}${meta.pr.url}`
|
||||
)
|
||||
for (const a of meta?.arms ?? [])
|
||||
if (a.title) console.log(` ${a.name}: "${a.title}"`)
|
||||
if (rows[0].route === '') {
|
||||
// Micro (sandbox-ab) samples: phase holds the payload name.
|
||||
analyzeE2eRows(rows, base, cand, ['mean', 'p50', 'p95', 'p99', 'gcMs'])
|
||||
} else {
|
||||
// No p99 for e2e: the load phases are far too small for it.
|
||||
analyzeE2eRows(rows, base, cand, [
|
||||
'rps',
|
||||
'median',
|
||||
'mean',
|
||||
'p95',
|
||||
'ttfb',
|
||||
'docKb',
|
||||
'gzipKb',
|
||||
'flightKb',
|
||||
'rss',
|
||||
'rssHw',
|
||||
])
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
#!/usr/bin/env node
|
||||
// Recover a run whose local launcher died (session teardown, crash):
|
||||
// the remote VMs keep executing their detached loops regardless, so the
|
||||
// data is usually still there. Reads the run's status.json for the VM
|
||||
// names, waits for each VM's loop to finish, downloads its results,
|
||||
// removes the VM, and runs the boot-level analysis.
|
||||
//
|
||||
// node bench-collect.mjs <runDir> [--deadline-min 280]
|
||||
import { execFile } from 'node:child_process'
|
||||
import { promisify } from 'node:util'
|
||||
import fs from 'node:fs'
|
||||
import path from 'node:path'
|
||||
import { loadConfig, sandboxScope } from './config.mjs'
|
||||
|
||||
const execFileP = promisify(execFile)
|
||||
const CONFIG = loadConfig()
|
||||
const SCOPE = sandboxScope(CONFIG)
|
||||
const argv = process.argv.slice(2)
|
||||
const dir = argv.find((a) => !a.startsWith('--'))
|
||||
const deadlineMin = argv.includes('--deadline-min')
|
||||
? Number(argv[argv.indexOf('--deadline-min') + 1])
|
||||
: 280
|
||||
if (!dir || !fs.existsSync(path.join(dir, 'status.json'))) {
|
||||
console.error('usage: node bench-collect.mjs <runDir with status.json>')
|
||||
process.exit(1)
|
||||
}
|
||||
const statusPath = path.join(dir, 'status.json')
|
||||
const st = JSON.parse(fs.readFileSync(statusPath, 'utf8'))
|
||||
const vms = Object.keys(st.vms ?? {})
|
||||
function writeStatus(patch) {
|
||||
Object.assign(st, patch, { updatedAt: new Date().toISOString() })
|
||||
fs.writeFileSync(statusPath, JSON.stringify(st, null, 2))
|
||||
}
|
||||
writeStatus({ phase: 'collecting (recovery)', pid: process.pid })
|
||||
if (vms.length === 0) {
|
||||
console.error(
|
||||
'status.json lists no VMs — the run died before measurement; nothing to collect. ' +
|
||||
'Check for a leaked snapshot builder with sandbox-sweep.mjs and relaunch the run.'
|
||||
)
|
||||
process.exit(1)
|
||||
}
|
||||
|
||||
async function sb(args) {
|
||||
const scoped = ['sandbox', ...args]
|
||||
const sep = scoped.indexOf('--')
|
||||
scoped.splice(sep < 0 ? scoped.length : sep, 0, ...SCOPE)
|
||||
const { stdout } = await execFileP(CONFIG.vercelBin, scoped, {
|
||||
maxBuffer: 1 << 26,
|
||||
})
|
||||
return stdout
|
||||
}
|
||||
|
||||
const pending = new Map(vms.map((vm, i) => [vm, i]))
|
||||
const collected = []
|
||||
const deadline = Date.now() + deadlineMin * 60000
|
||||
while (pending.size > 0 && Date.now() < deadline) {
|
||||
for (const [vm, idx] of [...pending]) {
|
||||
let out
|
||||
try {
|
||||
out = await sb([
|
||||
'exec',
|
||||
vm,
|
||||
'--timeout',
|
||||
'2m',
|
||||
'--',
|
||||
'bash',
|
||||
'-c',
|
||||
'cat /vercel/sandbox/loop.done 2>/dev/null || echo RUNNING',
|
||||
])
|
||||
} catch (e) {
|
||||
// VM gone (timed out or removed): its data is lost; say so and move on.
|
||||
console.error(
|
||||
`${vm}: unreachable (${e.message.split('\n')[0].slice(0, 100)}) — boot lost`
|
||||
)
|
||||
st.vms[vm] = { ...st.vms[vm], state: 'lost' }
|
||||
writeStatus({})
|
||||
pending.delete(vm)
|
||||
continue
|
||||
}
|
||||
const state = out.trim().split('\n').pop()
|
||||
if (state === 'RUNNING') continue
|
||||
if (state === 'LOOPOK') {
|
||||
const local = path.join(dir, `results-vm${idx}.jsonl`)
|
||||
await sb(['cp', `${vm}:/vercel/sandbox/results.jsonl`, local])
|
||||
if (fs.existsSync(local) && fs.statSync(local).size > 500) {
|
||||
collected.push(local)
|
||||
st.vms[vm] = { ...st.vms[vm], state: 'done' }
|
||||
writeStatus({})
|
||||
console.error(`${vm}: collected -> ${local}`)
|
||||
} else {
|
||||
console.error(`${vm}: results too small; keeping VM for inspection`)
|
||||
pending.delete(vm)
|
||||
continue
|
||||
}
|
||||
} else {
|
||||
const tail = await sb([
|
||||
'exec',
|
||||
vm,
|
||||
'--timeout',
|
||||
'2m',
|
||||
'--',
|
||||
'bash',
|
||||
'-c',
|
||||
'tail -c 1500 /vercel/sandbox/loop.log',
|
||||
])
|
||||
console.error(`${vm}: loop ended ${state}; log tail:\n${tail}`)
|
||||
}
|
||||
try {
|
||||
await sb(['rm', vm])
|
||||
console.error(`${vm}: removed`)
|
||||
} catch (e) {
|
||||
console.error(
|
||||
`${vm}: rm failed: ${e.message.split('\n')[0].slice(0, 100)}`
|
||||
)
|
||||
}
|
||||
pending.delete(vm)
|
||||
}
|
||||
if (pending.size > 0) await new Promise((r) => setTimeout(r, 60000))
|
||||
}
|
||||
for (const [vm] of pending)
|
||||
console.error(`${vm}: deadline exceeded, left running`)
|
||||
|
||||
if (collected.length === 0) {
|
||||
console.error('nothing collected')
|
||||
process.exit(1)
|
||||
}
|
||||
writeStatus({
|
||||
phase: `done (recovered ${collected.length}/${vms.length} boots)`,
|
||||
})
|
||||
console.log(`\ncollected ${collected.length}/${vms.length} boots; analysis:`)
|
||||
const { stdout } = await execFileP(
|
||||
'node',
|
||||
[
|
||||
path.join(
|
||||
path.dirname(new URL(import.meta.url).pathname),
|
||||
'bench-analyze.mjs'
|
||||
),
|
||||
dir,
|
||||
],
|
||||
{ maxBuffer: 1 << 24 }
|
||||
)
|
||||
console.log(stdout)
|
||||
@@ -0,0 +1,574 @@
|
||||
#!/usr/bin/env node
|
||||
// Canonical store for all bench data: one SQLite file holding raw
|
||||
// measurements and artifacts, written only by this importer — no
|
||||
// derived statistics, no hand-entered numbers. The per-VM JSONL files
|
||||
// in a run dir are the wire format; importing is idempotent (a run
|
||||
// re-imports as a whole). Stats are a pure function of this DB
|
||||
// (bench-analyze.mjs reads it and nothing else).
|
||||
//
|
||||
// node bench-db.mjs import <runDir...>
|
||||
// node bench-db.mjs verify [runId]
|
||||
// node bench-db.mjs export <out.db> <runId...>
|
||||
// node bench-db.mjs ls
|
||||
import fs from 'node:fs'
|
||||
import path from 'node:path'
|
||||
import zlib from 'node:zlib'
|
||||
import crypto from 'node:crypto'
|
||||
import { execFileSync } from 'node:child_process'
|
||||
import { DatabaseSync } from 'node:sqlite'
|
||||
import { loadConfig } from './config.mjs'
|
||||
|
||||
const SCHEMA = `
|
||||
PRAGMA journal_mode=WAL;
|
||||
PRAGMA foreign_keys=ON;
|
||||
CREATE TABLE IF NOT EXISTS runs(
|
||||
run_id TEXT PRIMARY KEY,
|
||||
label TEXT,
|
||||
kind TEXT,
|
||||
started_at TEXT,
|
||||
imported_at TEXT NOT NULL,
|
||||
harness_sha TEXT,
|
||||
meta TEXT
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS samples(
|
||||
sample_id INTEGER PRIMARY KEY,
|
||||
run_id TEXT NOT NULL REFERENCES runs(run_id) ON DELETE CASCADE,
|
||||
boot INTEGER NOT NULL,
|
||||
arm TEXT NOT NULL,
|
||||
block INTEGER NOT NULL,
|
||||
run INTEGER NOT NULL,
|
||||
route TEXT NOT NULL,
|
||||
phase TEXT NOT NULL,
|
||||
fingerprint TEXT,
|
||||
version TEXT,
|
||||
cpu TEXT,
|
||||
errors INTEGER NOT NULL DEFAULT 0,
|
||||
UNIQUE(run_id, boot, arm, block, run, route, phase)
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS measurements(
|
||||
sample_id INTEGER NOT NULL REFERENCES samples(sample_id) ON DELETE CASCADE,
|
||||
metric TEXT NOT NULL,
|
||||
value REAL NOT NULL,
|
||||
UNIQUE(sample_id, metric)
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS artifacts(
|
||||
artifact_id INTEGER PRIMARY KEY,
|
||||
run_id TEXT NOT NULL REFERENCES runs(run_id) ON DELETE CASCADE,
|
||||
boot INTEGER,
|
||||
arm TEXT,
|
||||
name TEXT NOT NULL,
|
||||
kind TEXT,
|
||||
sha256 TEXT NOT NULL,
|
||||
bytes INTEGER NOT NULL,
|
||||
data BLOB NOT NULL,
|
||||
UNIQUE(run_id, boot, arm, name)
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS boots(
|
||||
run_id TEXT NOT NULL REFERENCES runs(run_id) ON DELETE CASCADE,
|
||||
boot INTEGER NOT NULL,
|
||||
vm_name TEXT,
|
||||
UNIQUE(run_id, boot)
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS source_files(
|
||||
run_id TEXT NOT NULL REFERENCES runs(run_id) ON DELETE CASCADE,
|
||||
name TEXT NOT NULL,
|
||||
sha256 TEXT NOT NULL,
|
||||
rows INTEGER NOT NULL,
|
||||
UNIQUE(run_id, name)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_samples_run ON samples(run_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_measurements_sample ON measurements(sample_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_artifacts_run ON artifacts(run_id);
|
||||
`
|
||||
|
||||
// Row fields that identify a sample; every OTHER numeric field in a
|
||||
// JSONL row is a measurement. New metrics therefore need no schema or
|
||||
// importer change.
|
||||
const IDENT = new Set([
|
||||
'vm',
|
||||
'arm',
|
||||
'block',
|
||||
'run',
|
||||
'route',
|
||||
'phase',
|
||||
'fp',
|
||||
'ver',
|
||||
'cpu',
|
||||
'errors',
|
||||
'payload',
|
||||
'round',
|
||||
])
|
||||
|
||||
export function dbPath() {
|
||||
return path.join(loadConfig({ requireScope: false }).cacheDir, 'results.db')
|
||||
}
|
||||
|
||||
export function openDb(file = dbPath()) {
|
||||
fs.mkdirSync(path.dirname(file), { recursive: true })
|
||||
const db = new DatabaseSync(file)
|
||||
db.exec(SCHEMA)
|
||||
return db
|
||||
}
|
||||
|
||||
function harnessSha() {
|
||||
try {
|
||||
return execFileSync(
|
||||
'git',
|
||||
[
|
||||
'-C',
|
||||
path.dirname(new URL(import.meta.url).pathname),
|
||||
'rev-parse',
|
||||
'HEAD',
|
||||
],
|
||||
{ encoding: 'utf8' }
|
||||
).trim()
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
function artifactKind(name) {
|
||||
if (name.endsWith('.cpuprofile')) return 'cpuprofile'
|
||||
if (name.endsWith('.json')) return 'json'
|
||||
if (name.endsWith('.log') || name.endsWith('.txt')) return 'log'
|
||||
return 'file'
|
||||
}
|
||||
|
||||
function walk(dir) {
|
||||
const out = []
|
||||
for (const e of fs.readdirSync(dir, { withFileTypes: true })) {
|
||||
const p = path.join(dir, e.name)
|
||||
if (e.isDirectory()) out.push(...walk(p))
|
||||
else if (e.isFile()) out.push(p)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Import one run dir: replaces the run's rows wholesale so re-imports
|
||||
// (recovery, added boots, new artifacts) converge on the same state.
|
||||
// A re-import that would SHRINK a run (fewer boots or samples than the
|
||||
// db already holds — e.g. a cleaned-up run dir) is refused without
|
||||
// force: claims must never silently lose data underneath them.
|
||||
export function importRun(db, dir, { force = false } = {}) {
|
||||
const runId = path.basename(path.resolve(dir))
|
||||
const files = fs
|
||||
.readdirSync(dir)
|
||||
.filter((f) => /^results-vm\d+\.jsonl$/.test(f))
|
||||
.sort()
|
||||
.map((f) => path.join(dir, f))
|
||||
if (files.length === 0) throw new Error(`no results-vm*.jsonl in ${dir}`)
|
||||
const readJson = (f) => {
|
||||
try {
|
||||
return JSON.parse(fs.readFileSync(path.join(dir, f), 'utf8'))
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
}
|
||||
const meta = readJson('meta.json')
|
||||
const status = readJson('status.json')
|
||||
const prev = db
|
||||
.prepare(
|
||||
'SELECT COUNT(DISTINCT boot) boots, COUNT(*) n FROM samples WHERE run_id = ?'
|
||||
)
|
||||
.get(runId)
|
||||
db.exec('BEGIN')
|
||||
try {
|
||||
db.prepare('DELETE FROM runs WHERE run_id = ?').run(runId)
|
||||
let kind = 'e2e'
|
||||
const insSample = db.prepare(`INSERT INTO samples
|
||||
(run_id, boot, arm, block, run, route, phase, fingerprint, version, cpu, errors)
|
||||
VALUES (?,?,?,?,?,?,?,?,?,?,?)`)
|
||||
const insMeas = db.prepare(
|
||||
'INSERT INTO measurements (sample_id, metric, value) VALUES (?,?,?)'
|
||||
)
|
||||
let samples = 0
|
||||
let pendingRows = []
|
||||
const fileInfos = []
|
||||
for (const [boot, file] of files.entries()) {
|
||||
const content = fs.readFileSync(file, 'utf8')
|
||||
let fileRows = 0
|
||||
for (const line of content.trim().split('\n')) {
|
||||
if (!line) continue
|
||||
const row = JSON.parse(line)
|
||||
const micro = row.payload !== undefined
|
||||
if (micro) kind = 'micro'
|
||||
pendingRows.push({ boot, row, micro })
|
||||
fileRows++
|
||||
}
|
||||
fileInfos.push({
|
||||
boot,
|
||||
name: path.basename(file),
|
||||
sha256: crypto.createHash('sha256').update(content).digest('hex'),
|
||||
rows: fileRows,
|
||||
})
|
||||
}
|
||||
if (
|
||||
!force &&
|
||||
prev.n > 0 &&
|
||||
(files.length < prev.boots || pendingRows.length < prev.n)
|
||||
) {
|
||||
throw new Error(
|
||||
`re-import of ${runId} would shrink it ` +
|
||||
`(${prev.boots} boots/${prev.n} samples in db, ` +
|
||||
`${files.length} boots/${pendingRows.length} samples in ${dir}) — ` +
|
||||
'pass --force only if the db copy is known bad'
|
||||
)
|
||||
}
|
||||
db.prepare(
|
||||
`INSERT INTO runs
|
||||
(run_id, label, kind, started_at, imported_at, harness_sha, meta)
|
||||
VALUES (?,?,?,?,?,?,?)`
|
||||
).run(
|
||||
runId,
|
||||
meta?.label ?? runId.replace(/^run-/, '').replace(/-[a-z0-9]+$/, ''),
|
||||
kind,
|
||||
status?.startedAt ?? null,
|
||||
new Date().toISOString(),
|
||||
harnessSha(),
|
||||
meta ? JSON.stringify(meta) : null
|
||||
)
|
||||
// Provenance: which sandbox VM produced each boot (from
|
||||
// status.json, matched by the index in the VM name), and the exact
|
||||
// bytes each boot's JSONL contributed.
|
||||
const vmByIndex = new Map()
|
||||
for (const name of Object.keys(status?.vms ?? {})) {
|
||||
const idx = name.match(/-(\d+)-[a-z0-9]+$/)?.[1]
|
||||
if (idx !== undefined) vmByIndex.set(Number(idx), name)
|
||||
}
|
||||
const insBoot = db.prepare(
|
||||
'INSERT INTO boots (run_id, boot, vm_name) VALUES (?,?,?)'
|
||||
)
|
||||
const insFile = db.prepare(
|
||||
'INSERT INTO source_files (run_id, name, sha256, rows) VALUES (?,?,?,?)'
|
||||
)
|
||||
for (const f of fileInfos) {
|
||||
const n = Number(f.name.match(/^results-vm(\d+)\.jsonl$/)?.[1])
|
||||
insBoot.run(runId, f.boot, vmByIndex.get(n) ?? null)
|
||||
insFile.run(runId, f.name, f.sha256, f.rows)
|
||||
}
|
||||
for (const { boot, row, micro } of pendingRows) {
|
||||
const { lastInsertRowid: sid } = insSample.run(
|
||||
runId,
|
||||
boot,
|
||||
row.arm,
|
||||
micro ? 0 : (row.block ?? 0),
|
||||
micro ? (row.round ?? 0) : (row.run ?? 0),
|
||||
micro ? '' : (row.route ?? ''),
|
||||
micro ? row.payload : row.phase,
|
||||
row.fp ?? null,
|
||||
row.ver ?? null,
|
||||
row.cpu ?? null,
|
||||
row.errors ?? 0
|
||||
)
|
||||
samples++
|
||||
for (const [k, v] of Object.entries(row)) {
|
||||
if (IDENT.has(k)) continue
|
||||
if (typeof v !== 'number' || !Number.isFinite(v)) continue
|
||||
insMeas.run(sid, k, v)
|
||||
}
|
||||
}
|
||||
// Artifacts: everything under prof-vm<N>/ (CPU profiles etc.),
|
||||
// stored gzipped with the sha256 of the RAW content.
|
||||
const insArt = db.prepare(`INSERT INTO artifacts
|
||||
(run_id, boot, arm, name, kind, sha256, bytes, data) VALUES (?,?,?,?,?,?,?,?)`)
|
||||
let artifacts = 0
|
||||
for (const e of fs.readdirSync(dir)) {
|
||||
const m = e.match(/^prof-vm(\d+)$/)
|
||||
if (!m) continue
|
||||
const boot = Number(m[1])
|
||||
for (const f of walk(path.join(dir, e))) {
|
||||
const rel = path.relative(path.join(dir, e), f)
|
||||
const arm = rel.match(/^prof-([^/]+)\//)?.[1] ?? null
|
||||
const raw = fs.readFileSync(f)
|
||||
insArt.run(
|
||||
runId,
|
||||
boot,
|
||||
arm,
|
||||
rel,
|
||||
artifactKind(rel),
|
||||
crypto.createHash('sha256').update(raw).digest('hex'),
|
||||
raw.length,
|
||||
zlib.gzipSync(raw)
|
||||
)
|
||||
artifacts++
|
||||
}
|
||||
}
|
||||
db.exec('COMMIT')
|
||||
return { runId, boots: files.length, samples, artifacts }
|
||||
} catch (e) {
|
||||
db.exec('ROLLBACK')
|
||||
throw e
|
||||
}
|
||||
}
|
||||
|
||||
// Rows for bench-stats.analyzeE2eRows, reconstructed from the DB —
|
||||
// the stats layer never reads run dirs.
|
||||
export function loadRows(db, runId) {
|
||||
const samples = db
|
||||
.prepare('SELECT * FROM samples WHERE run_id = ?')
|
||||
.all(runId)
|
||||
if (samples.length === 0) throw new Error(`no samples for run ${runId}`)
|
||||
const meas = db
|
||||
.prepare(
|
||||
`SELECT m.sample_id, m.metric, m.value FROM measurements m
|
||||
JOIN samples s ON s.sample_id = m.sample_id WHERE s.run_id = ?`
|
||||
)
|
||||
.all(runId)
|
||||
const byId = new Map()
|
||||
for (const s of samples) {
|
||||
byId.set(s.sample_id, {
|
||||
vm: Number(s.boot),
|
||||
arm: s.arm,
|
||||
block: Number(s.block),
|
||||
run: Number(s.run),
|
||||
route: s.route,
|
||||
phase: s.phase,
|
||||
fp: s.fingerprint,
|
||||
ver: s.version,
|
||||
errors: Number(s.errors),
|
||||
})
|
||||
}
|
||||
for (const m of meas) byId.get(m.sample_id)[m.metric] = m.value
|
||||
return [...byId.values()]
|
||||
}
|
||||
|
||||
export function runMeta(db, runId) {
|
||||
const r = db.prepare('SELECT * FROM runs WHERE run_id = ?').get(runId)
|
||||
return r ? { ...r, meta: r.meta ? JSON.parse(r.meta) : null } : null
|
||||
}
|
||||
|
||||
// Integrity checks. Everything here is mechanical: SQLite-level
|
||||
// integrity, referential health, per-run shape (one fingerprint per
|
||||
// arm, paired sample counts), and artifact hashes. Failures are
|
||||
// data-integrity violations; notes are true facts a reader must not
|
||||
// gloss over (e.g. a recovered run that is legitimately partial).
|
||||
export function verify(db, runId) {
|
||||
const problems = []
|
||||
const notes = []
|
||||
const ic = db.prepare('PRAGMA integrity_check').all()
|
||||
if (!(ic.length === 1 && ic[0].integrity_check === 'ok')) {
|
||||
problems.push(`sqlite integrity_check: ${JSON.stringify(ic)}`)
|
||||
}
|
||||
const fk = db.prepare('PRAGMA foreign_key_check').all()
|
||||
if (fk.length > 0) problems.push(`foreign_key_check: ${fk.length} violations`)
|
||||
const runs = runId
|
||||
? db.prepare('SELECT run_id FROM runs WHERE run_id = ?').all(runId)
|
||||
: db.prepare('SELECT run_id FROM runs').all()
|
||||
if (runId && runs.length === 0) problems.push(`run ${runId} not in db`)
|
||||
for (const { run_id } of runs) {
|
||||
const orphanSamples = db
|
||||
.prepare(
|
||||
`SELECT COUNT(*) c FROM samples s
|
||||
WHERE s.run_id = ? AND NOT EXISTS
|
||||
(SELECT 1 FROM measurements m WHERE m.sample_id = s.sample_id)`
|
||||
)
|
||||
.get(run_id).c
|
||||
if (orphanSamples > 0) {
|
||||
problems.push(`${run_id}: ${orphanSamples} samples with no measurements`)
|
||||
}
|
||||
const arms = db
|
||||
.prepare(
|
||||
`SELECT arm, COUNT(DISTINCT COALESCE(fingerprint,'') ||
|
||||
'/' || COALESCE(version,'')) fps, COUNT(*) n
|
||||
FROM samples WHERE run_id = ? GROUP BY arm`
|
||||
)
|
||||
.all(run_id)
|
||||
if (arms.length !== 2) {
|
||||
problems.push(`${run_id}: expected 2 arms, found ${arms.length}`)
|
||||
}
|
||||
for (const a of arms) {
|
||||
if (a.fps > 1) {
|
||||
problems.push(
|
||||
`${run_id}: arm ${a.arm} has ${a.fps} distinct fingerprints`
|
||||
)
|
||||
}
|
||||
}
|
||||
const planned = (() => {
|
||||
try {
|
||||
return JSON.parse(
|
||||
db.prepare('SELECT meta FROM runs WHERE run_id = ?').get(run_id).meta
|
||||
).vms
|
||||
} catch {
|
||||
return undefined
|
||||
}
|
||||
})()
|
||||
const gotBoots = db
|
||||
.prepare('SELECT COUNT(DISTINCT boot) c FROM samples WHERE run_id = ?')
|
||||
.get(run_id).c
|
||||
if (planned && gotBoots < planned) {
|
||||
notes.push(`${run_id}: partial run (${gotBoots}/${planned} boots)`)
|
||||
}
|
||||
if (arms.length === 2 && arms[0].n !== arms[1].n) {
|
||||
problems.push(
|
||||
`${run_id}: unpaired sample counts ` +
|
||||
`(${arms[0].arm}=${arms[0].n} vs ${arms[1].arm}=${arms[1].n})`
|
||||
)
|
||||
}
|
||||
for (const art of db
|
||||
.prepare(
|
||||
'SELECT artifact_id, name, sha256, bytes, data FROM artifacts WHERE run_id = ?'
|
||||
)
|
||||
.all(run_id)) {
|
||||
const raw = zlib.gunzipSync(art.data)
|
||||
if (
|
||||
raw.length !== Number(art.bytes) ||
|
||||
crypto.createHash('sha256').update(raw).digest('hex') !== art.sha256
|
||||
) {
|
||||
problems.push(`${run_id}: artifact ${art.name} fails hash/size check`)
|
||||
}
|
||||
}
|
||||
}
|
||||
return { failures: problems, notes }
|
||||
}
|
||||
|
||||
export function exportRuns(db, outFile, runIds) {
|
||||
fs.rmSync(outFile, { force: true })
|
||||
const out = openDb(outFile)
|
||||
const copy = (table, cols) => {
|
||||
const ins = out.prepare(`INSERT INTO ${table} (${cols.join(',')})
|
||||
VALUES (${cols.map(() => '?').join(',')})`)
|
||||
return (row) => ins.run(...cols.map((c) => row[c]))
|
||||
}
|
||||
out.exec('BEGIN')
|
||||
const copyRun = copy('runs', [
|
||||
'run_id',
|
||||
'label',
|
||||
'kind',
|
||||
'started_at',
|
||||
'imported_at',
|
||||
'harness_sha',
|
||||
'meta',
|
||||
])
|
||||
const copySample = copy('samples', [
|
||||
'sample_id',
|
||||
'run_id',
|
||||
'boot',
|
||||
'arm',
|
||||
'block',
|
||||
'run',
|
||||
'route',
|
||||
'phase',
|
||||
'fingerprint',
|
||||
'version',
|
||||
'cpu',
|
||||
'errors',
|
||||
])
|
||||
const copyMeas = copy('measurements', ['sample_id', 'metric', 'value'])
|
||||
const copyBoot = copy('boots', ['run_id', 'boot', 'vm_name'])
|
||||
const copyFile = copy('source_files', ['run_id', 'name', 'sha256', 'rows'])
|
||||
const copyArt = copy('artifacts', [
|
||||
'run_id',
|
||||
'boot',
|
||||
'arm',
|
||||
'name',
|
||||
'kind',
|
||||
'sha256',
|
||||
'bytes',
|
||||
'data',
|
||||
])
|
||||
for (const runId of runIds) {
|
||||
const run = db.prepare('SELECT * FROM runs WHERE run_id = ?').get(runId)
|
||||
if (!run) throw new Error(`run ${runId} not in db`)
|
||||
copyRun(run)
|
||||
for (const s of db
|
||||
.prepare('SELECT * FROM samples WHERE run_id = ?')
|
||||
.all(runId)) {
|
||||
copySample(s)
|
||||
for (const m of db
|
||||
.prepare('SELECT * FROM measurements WHERE sample_id = ?')
|
||||
.all(s.sample_id)) {
|
||||
copyMeas(m)
|
||||
}
|
||||
}
|
||||
for (const a of db
|
||||
.prepare('SELECT * FROM artifacts WHERE run_id = ?')
|
||||
.all(runId)) {
|
||||
copyArt(a)
|
||||
}
|
||||
for (const b of db
|
||||
.prepare('SELECT * FROM boots WHERE run_id = ?')
|
||||
.all(runId)) {
|
||||
copyBoot(b)
|
||||
}
|
||||
for (const f of db
|
||||
.prepare('SELECT * FROM source_files WHERE run_id = ?')
|
||||
.all(runId)) {
|
||||
copyFile(f)
|
||||
}
|
||||
}
|
||||
out.exec('COMMIT')
|
||||
out.close()
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------ CLI
|
||||
function isMain() {
|
||||
if (!process.argv[1]) return false
|
||||
try {
|
||||
return (
|
||||
fs.realpathSync(process.argv[1]) ===
|
||||
fs.realpathSync(new URL(import.meta.url).pathname)
|
||||
)
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
if (isMain()) {
|
||||
const [cmd, ...rest] = process.argv.slice(2)
|
||||
const db = openDb()
|
||||
const report = ({ failures, notes }, label) => {
|
||||
for (const n of notes) console.log(`note: ${n}`)
|
||||
if (failures.length) {
|
||||
console.error(`VERIFY FAILED:\n ${failures.join('\n ')}`)
|
||||
process.exit(1)
|
||||
}
|
||||
console.log(`verify: ok${label ? ` (${label})` : ''}`)
|
||||
}
|
||||
if (cmd === 'import') {
|
||||
const force = rest.includes('--force')
|
||||
const dirs = rest.filter((a) => a !== '--force')
|
||||
if (dirs.length === 0) {
|
||||
console.error('usage: bench-db.mjs import [--force] <runDir...>')
|
||||
process.exit(1)
|
||||
}
|
||||
for (const dir of dirs) {
|
||||
let r
|
||||
try {
|
||||
r = importRun(db, dir, { force })
|
||||
} catch (e) {
|
||||
console.error(`IMPORT REFUSED: ${e.message}`)
|
||||
process.exit(1)
|
||||
}
|
||||
console.log(
|
||||
`${r.runId}: ${r.boots} boots, ${r.samples} samples, ${r.artifacts} artifacts`
|
||||
)
|
||||
}
|
||||
report(verify(db))
|
||||
} else if (cmd === 'verify') {
|
||||
report(verify(db, rest[0]), rest[0] ?? 'all runs')
|
||||
} else if (cmd === 'export') {
|
||||
const [out, ...ids] = rest
|
||||
if (!out || ids.length === 0) {
|
||||
console.error('usage: bench-db.mjs export <out.db> <runId...>')
|
||||
process.exit(1)
|
||||
}
|
||||
exportRuns(db, out, ids)
|
||||
console.log(`${out}: ${ids.length} runs`)
|
||||
} else if (cmd === 'ls') {
|
||||
for (const r of db
|
||||
.prepare(
|
||||
`SELECT r.run_id, r.kind, r.started_at,
|
||||
(SELECT COUNT(DISTINCT boot) FROM samples s WHERE s.run_id = r.run_id) boots,
|
||||
(SELECT COUNT(*) FROM samples s WHERE s.run_id = r.run_id) samples,
|
||||
(SELECT COUNT(*) FROM artifacts a WHERE a.run_id = r.run_id) artifacts
|
||||
FROM runs r ORDER BY r.started_at`
|
||||
)
|
||||
.all()) {
|
||||
console.log(
|
||||
`${r.run_id} ${r.kind} boots=${r.boots} samples=${r.samples} artifacts=${r.artifacts} ${r.started_at ?? ''}`
|
||||
)
|
||||
}
|
||||
} else {
|
||||
console.error('usage: bench-db.mjs import|verify|export|ls')
|
||||
process.exit(1)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
// Shared statistics for the sandbox bench harnesses.
|
||||
//
|
||||
// Unit of analysis: the VM BOOT. Iterations within a boot share JIT
|
||||
// state, heap layout, host, and phase-sequence position, so they are
|
||||
// not independent samples (Kalibera & Jones, "Rigorous Benchmarking in
|
||||
// Reasonable Time"; JMH forks). Each boot contributes ONE delta per
|
||||
// metric (the mean of its paired within-boot deltas); confidence
|
||||
// intervals and p-values are computed across boots. Within-boot
|
||||
// statistics are printed as a diagnostic only — never as the claim.
|
||||
|
||||
// Two-sided critical values of Student's t at 97.5% (df 1..30).
|
||||
const T975 = [
|
||||
12.706, 4.303, 3.182, 2.776, 2.571, 2.447, 2.365, 2.306, 2.262, 2.228, 2.201,
|
||||
2.179, 2.16, 2.145, 2.131, 2.12, 2.11, 2.101, 2.093, 2.086, 2.08, 2.074,
|
||||
2.069, 2.064, 2.06, 2.056, 2.052, 2.048, 2.045, 2.042,
|
||||
]
|
||||
|
||||
function tCritical975(df) {
|
||||
if (df < 1) return Infinity
|
||||
if (df <= 30) return T975[df - 1]
|
||||
return 1.96 + 2.4 / df // adequate approximation past df=30
|
||||
}
|
||||
|
||||
// Two-sided p for a one-sample t test against zero, via numerical
|
||||
// integration of the t pdf (small-df accuracy is what matters here —
|
||||
// boots are few).
|
||||
export function tTestP(values) {
|
||||
const n = values.length
|
||||
if (n < 2) return 1
|
||||
if (values.some((v) => !Number.isFinite(v))) return 1
|
||||
const mean = values.reduce((a, b) => a + b, 0) / n
|
||||
const sd = Math.sqrt(
|
||||
values.reduce((a, b) => a + (b - mean) ** 2, 0) / (n - 1)
|
||||
)
|
||||
if (sd === 0) return mean === 0 ? 1 : 0
|
||||
const t = Math.abs(mean / (sd / Math.sqrt(n)))
|
||||
const df = n - 1
|
||||
if (df === 1) {
|
||||
// Student t with df=1 is Cauchy; the closed form avoids the fat
|
||||
// tail truncating a numerical integration.
|
||||
return Math.min(1, Math.max(0, 1 - (2 / Math.PI) * Math.atan(t)))
|
||||
}
|
||||
if (!Number.isFinite(t) || t > 45) {
|
||||
// p underflows well past any claim threshold; also guards the
|
||||
// integration loop below, whose step size vanishes against huge t.
|
||||
return 0
|
||||
}
|
||||
const pdf = (x) => Math.exp(-((df + 1) / 2) * Math.log(1 + (x * x) / df))
|
||||
let integral = 0
|
||||
const STEP = 0.001
|
||||
for (let x = t; x < t + 60; x += STEP) integral += pdf(x + STEP / 2) * STEP
|
||||
let norm = 0
|
||||
for (let x = 0; x < 80; x += STEP) norm += pdf(x + STEP / 2) * STEP
|
||||
return Math.min(1, integral / norm)
|
||||
}
|
||||
|
||||
// Anytime-valid confidence sequence for a running mean (asymptotic CS,
|
||||
// Waudby-Smith & Ramdas). Unlike a t-CI, this interval is valid at
|
||||
// EVERY peek simultaneously, so interim displays built on it cannot
|
||||
// manufacture significance through repeated looking. Tuned to be
|
||||
// tightest around ~12 boots.
|
||||
export function confidenceSeq(values, alpha = 0.05) {
|
||||
const n = values.length
|
||||
// Below 6 samples the estimated variance is too unstable for the
|
||||
// asymptotic guarantee; show nothing rather than something wrong.
|
||||
if (n < 6) return null
|
||||
const mean = values.reduce((a, b) => a + b, 0) / n
|
||||
let sd = Math.sqrt(values.reduce((a, b) => a + (b - mean) ** 2, 0) / (n - 1))
|
||||
if (!Number.isFinite(sd)) return null
|
||||
// Small-sample variance inflation (t-style): keeps the sequence
|
||||
// honest at the n this harness actually runs (6..32 boots).
|
||||
sd *= Math.sqrt((n - 1) / Math.max(1, n - 3))
|
||||
const nOpt = 16
|
||||
const rho2 = (2 * Math.log(2 / alpha)) / nOpt
|
||||
const width =
|
||||
sd *
|
||||
Math.sqrt(
|
||||
((2 * (n * rho2 + 1)) / (n * n * rho2)) *
|
||||
Math.log((Math.sqrt(n * rho2 + 1) * 2) / alpha)
|
||||
)
|
||||
return { mean, lo: mean - width, hi: mean + width, n }
|
||||
}
|
||||
|
||||
// What each metric measures and which direction is an improvement.
|
||||
// Deltas are always relative (%), so the unit answers "% of what".
|
||||
export const METRICS = {
|
||||
rps: { unit: 'req/s', better: 'higher' },
|
||||
docKb: { unit: 'KB', better: 'lower' },
|
||||
gzipKb: { unit: 'KB', better: 'lower' },
|
||||
flightKb: { unit: 'KB', better: 'lower' },
|
||||
median: { unit: 'ms', better: 'lower' },
|
||||
mean: { unit: 'ms', better: 'lower' },
|
||||
p95: { unit: 'ms', better: 'lower' },
|
||||
p99: { unit: 'ms', better: 'lower' },
|
||||
ttfb: { unit: 'ms', better: 'lower' },
|
||||
gcMs: { unit: 'ms', better: 'lower' },
|
||||
p50: { unit: 'ms', better: 'lower' },
|
||||
rss: { unit: 'MB', better: 'lower' },
|
||||
rssHw: { unit: 'MB peak', better: 'lower' },
|
||||
}
|
||||
|
||||
// One metric in one phase: per-boot arrays of paired deltas in, verdict out.
|
||||
export function bootLevelStats(perBootDeltas) {
|
||||
const boots = perBootDeltas.filter((d) => d.length > 0)
|
||||
const bootMeans = boots.map((d) => d.reduce((a, b) => a + b, 0) / d.length)
|
||||
const n = bootMeans.length
|
||||
if (n === 0) return null
|
||||
const mean = bootMeans.reduce((a, b) => a + b, 0) / n
|
||||
const all = boots.flat()
|
||||
const result = {
|
||||
mean,
|
||||
boots: n,
|
||||
bootMeans,
|
||||
pairs: all.length,
|
||||
withinP: tTestP(all),
|
||||
}
|
||||
if (n >= 2) {
|
||||
const sd = Math.sqrt(
|
||||
bootMeans.reduce((a, b) => a + (b - mean) ** 2, 0) / (n - 1)
|
||||
)
|
||||
result.ci95 = (tCritical975(n - 1) * sd) / Math.sqrt(n)
|
||||
result.p = tTestP(bootMeans)
|
||||
} else {
|
||||
result.ci95 = Infinity
|
||||
result.p = 1
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
export function formatStat(name, candName, baseName, s) {
|
||||
if (s === null) return ` ${name.padEnd(6)} (no data)`
|
||||
const pct = (x) => `${(x * 100).toFixed(1)}%`
|
||||
const ci = s.ci95 === Infinity ? '±∞' : `±${(s.ci95 * 100).toFixed(1)}`
|
||||
const m = METRICS[name]
|
||||
const meta = m
|
||||
? ` [${m.unit}; ${s.mean > 0 === (m.better === 'higher') ? 'IMPROVEMENT' : 'regression'} if real]`
|
||||
: ''
|
||||
// Byte metrics are deterministic per build, so a ±0.0 near-zero cell
|
||||
// with p=0 is normal; the absolute values say whether it matters.
|
||||
const abs =
|
||||
s.absBase !== undefined && m?.unit === 'KB'
|
||||
? ` [${s.absBase.toFixed(1)}KB -> ${s.absCand.toFixed(1)}KB]`
|
||||
: ''
|
||||
return (
|
||||
` ${name.padEnd(6)} ${candName} vs ${baseName}: ${pct(s.mean)} ${ci}${meta}${abs} ` +
|
||||
`(boots=${s.boots}${s.boots < 3 ? ' — TOO FEW FOR CLAIMS' : ''}, p=${s.p.toFixed(4)})` +
|
||||
` perBoot ${s.bootMeans.map((m) => pct(m)).join(' ')}` +
|
||||
` [pairs=${s.pairs} within-run p=${s.withinP.toFixed(4)} — diagnostic only]`
|
||||
)
|
||||
}
|
||||
|
||||
// E2e rows: {vm, arm, block, run, route, phase, <metrics...>}. Pairs are
|
||||
// (vm, block, run); the boot is the vm. Returns nothing; prints.
|
||||
export function analyzeE2eRows(rows, baseName, candName, metrics) {
|
||||
const fps = {}
|
||||
for (const r of rows)
|
||||
(fps[r.arm] ||= new Set()).add(r.ver ? `${r.fp}/${r.ver}` : `${r.fp}`)
|
||||
console.log(
|
||||
`\nfingerprints: ${baseName}=${[...(fps[baseName] ?? [])].join(',')} ` +
|
||||
`${candName}=${[...(fps[candName] ?? [])].join(',')}`
|
||||
)
|
||||
if ((fps[baseName]?.size ?? 0) !== 1 || (fps[candName]?.size ?? 0) !== 1) {
|
||||
// A VM measured the wrong build; numbers would look official and be
|
||||
// meaningless. Refuse instead of printing stats with a warning.
|
||||
console.log(
|
||||
'!! inconsistent fingerprints within an arm — RESULTS INVALID, no stats'
|
||||
)
|
||||
process.exitCode = 1
|
||||
return false
|
||||
}
|
||||
const baseFp = [...fps[baseName]][0]?.split('/')[0]
|
||||
const candFp = [...fps[candName]][0]?.split('/')[0]
|
||||
if (baseFp !== undefined && candFp !== undefined) {
|
||||
console.log(
|
||||
baseFp === candFp
|
||||
? 'fingerprint files byte-identical between arms (A/A for those files; ' +
|
||||
'arms may still differ elsewhere — check version strings)'
|
||||
: 'arms differ (A/B mode)'
|
||||
)
|
||||
}
|
||||
|
||||
// "Absent" must be distinguishable from "identical": metrics the run
|
||||
// never captured are named at the end instead of silently missing.
|
||||
const captured = new Set()
|
||||
for (const phase of [
|
||||
...new Set(rows.map((r) => `${r.route ?? ''} ${r.phase}`)),
|
||||
].sort()) {
|
||||
console.log(`\n${phase}`)
|
||||
const inPhase = (r) => `${r.route ?? ''} ${r.phase}` === phase
|
||||
for (const metric of metrics) {
|
||||
const perBoot = []
|
||||
const baseVals = []
|
||||
const candVals = []
|
||||
for (const vm of [...new Set(rows.map((r) => r.vm))]) {
|
||||
const deltas = []
|
||||
for (const r of rows.filter(
|
||||
(x) => x.vm === vm && inPhase(x) && x.arm === candName
|
||||
)) {
|
||||
const b = rows.find(
|
||||
(x) =>
|
||||
x.vm === vm &&
|
||||
inPhase(x) &&
|
||||
x.arm === baseName &&
|
||||
x.block === r.block &&
|
||||
x.run === r.run
|
||||
)
|
||||
if (b && b[metric] > 0 && r[metric] > 0) {
|
||||
deltas.push((r[metric] - b[metric]) / b[metric])
|
||||
baseVals.push(b[metric])
|
||||
candVals.push(r[metric])
|
||||
}
|
||||
}
|
||||
perBoot.push(deltas)
|
||||
}
|
||||
const s = bootLevelStats(perBoot)
|
||||
if (s !== null && s.pairs > 0) {
|
||||
captured.add(metric)
|
||||
s.absBase = baseVals.reduce((a, b) => a + b, 0) / baseVals.length
|
||||
s.absCand = candVals.reduce((a, b) => a + b, 0) / candVals.length
|
||||
console.log(formatStat(metric, candName, baseName, s))
|
||||
}
|
||||
}
|
||||
}
|
||||
const absent = metrics.filter((m) => !captured.has(m))
|
||||
if (absent.length > 0) {
|
||||
console.log(`\nnot captured on this run: ${absent.join(', ')}`)
|
||||
}
|
||||
return true
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
#!/usr/bin/env node
|
||||
// Usage: node bench-status.mjs [--all]
|
||||
//
|
||||
// The only reliable way to know what's in flight. Session restarts silently
|
||||
// kill background launchers while their detached VMs keep measuring, and a
|
||||
// dead launcher's log file still ends with a healthy-looking progress line —
|
||||
// never infer liveness from log tails. This reads each run's status.json,
|
||||
// checks whether the recorded launcher pid is alive, and prints the recovery
|
||||
// action per run.
|
||||
import fs from 'fs'
|
||||
import path from 'path'
|
||||
import { loadConfig } from './config.mjs'
|
||||
|
||||
const CONFIG = loadConfig({ requireScope: false })
|
||||
const E2E = path.join(CONFIG.cacheDir, 'e2e')
|
||||
const all = process.argv.includes('--all')
|
||||
|
||||
const dirs = fs.existsSync(E2E)
|
||||
? fs
|
||||
.readdirSync(E2E)
|
||||
.filter((d) => d.startsWith('run-'))
|
||||
.map((d) => path.join(E2E, d))
|
||||
.sort((a, b) => fs.statSync(b).mtimeMs - fs.statSync(a).mtimeMs)
|
||||
: []
|
||||
|
||||
const DAY = 24 * 3600 * 1000
|
||||
let shown = 0
|
||||
for (const dir of dirs) {
|
||||
const sf = path.join(dir, 'status.json')
|
||||
if (!fs.existsSync(sf)) continue
|
||||
let st
|
||||
try {
|
||||
st = JSON.parse(fs.readFileSync(sf, 'utf8'))
|
||||
} catch {
|
||||
continue
|
||||
}
|
||||
const fresh = fs.statSync(sf).mtimeMs > Date.now() - DAY
|
||||
let alive = false
|
||||
if (st.pid) {
|
||||
try {
|
||||
process.kill(st.pid, 0)
|
||||
alive = true
|
||||
} catch {}
|
||||
}
|
||||
if (!all && !fresh && !alive) {
|
||||
continue
|
||||
}
|
||||
const ageMin = st.updatedAt
|
||||
? Math.round((Date.now() - new Date(st.updatedAt).getTime()) / 60000)
|
||||
: null
|
||||
const vms = Object.values(st.vms || {})
|
||||
const rows = vms.reduce((s, v) => s + (v.rows || 0), 0)
|
||||
|
||||
let action
|
||||
if (st.phase === 'done') {
|
||||
action = 'complete — bench-analyze.mjs re-reads it any time'
|
||||
} else if (alive) {
|
||||
action = 'RUNNING — leave it alone'
|
||||
} else if (
|
||||
(rows > 0 || st.phase === 'measuring') &&
|
||||
ageMin !== null &&
|
||||
ageMin > 290
|
||||
) {
|
||||
action =
|
||||
'DEAD and its VMs have expired — whatever was collected is in the ' +
|
||||
'run dir/results db; relaunch if more data is needed'
|
||||
} else if (rows > 0 || st.phase === 'measuring') {
|
||||
action =
|
||||
`DEAD but VMs may still hold data — node scripts/bench-collect.mjs ${dir}` +
|
||||
' (measurement VMs time out ~5h after launch; collect before that)'
|
||||
} else {
|
||||
action =
|
||||
'DEAD before measurement — safe to relaunch (caches make it cheap); ' +
|
||||
'check for leaked builder VMs with sandbox-sweep.mjs'
|
||||
}
|
||||
|
||||
console.log(
|
||||
`${path.basename(dir)}\n` +
|
||||
` phase=${st.phase}` +
|
||||
(st.error
|
||||
? ` error=${JSON.stringify(String(st.error).slice(0, 140))}`
|
||||
: '') +
|
||||
` pid=${st.pid ?? '?'}(${alive ? 'alive' : 'dead'})` +
|
||||
(ageMin !== null ? ` updated=${ageMin}m ago` : '') +
|
||||
` rows=${rows}${st.rowsExpected ? `/${st.rowsExpected}` : ''}\n` +
|
||||
` -> ${action}`
|
||||
)
|
||||
shown++
|
||||
}
|
||||
if (!shown) {
|
||||
console.log('no recent runs (use --all to list everything)')
|
||||
}
|
||||
@@ -0,0 +1,150 @@
|
||||
// Configuration for the sandbox-bench scripts.
|
||||
//
|
||||
// Everything machine- or team-specific lives in a config file OUTSIDE the
|
||||
// repo (never commit team/project names into the repo):
|
||||
// ~/.config/sandbox-bench/config.json
|
||||
//
|
||||
// Required fields (no defaults — ask the user once, then save):
|
||||
// team Vercel team slug the sandboxes bill to
|
||||
// project Vercel project the sandboxes attach to
|
||||
// Optional fields:
|
||||
// cacheDir where clones/artifacts/snapshot-ids live
|
||||
// (default ~/.cache/sandbox-bench)
|
||||
// reactRepo existing react checkout to use (default: auto-clone
|
||||
// into <cacheDir>/react)
|
||||
// nextRepo existing next.js checkout to use (default: auto-clone
|
||||
// into <cacheDir>/next)
|
||||
// reactRepoUrl clone/fetch URL (default https://github.com/react/react.git)
|
||||
// nextRepoUrl clone/fetch URL (default https://github.com/vercel/next.js.git)
|
||||
// vercelBin vercel CLI binary (default "vercel")
|
||||
//
|
||||
// CLI: node config.mjs show
|
||||
// node config.mjs set team=<slug> project=<name> [key=value...]
|
||||
import fs from 'node:fs'
|
||||
import os from 'node:os'
|
||||
import path from 'node:path'
|
||||
import { execFileSync } from 'node:child_process'
|
||||
|
||||
const CONFIG_DIR = path.join(os.homedir(), '.config', 'sandbox-bench')
|
||||
const CONFIG_FILE = path.join(CONFIG_DIR, 'config.json')
|
||||
|
||||
export function loadConfig({ requireScope = true } = {}) {
|
||||
let cfg = {}
|
||||
if (fs.existsSync(CONFIG_FILE)) {
|
||||
cfg = JSON.parse(fs.readFileSync(CONFIG_FILE, 'utf8'))
|
||||
}
|
||||
// Env overrides for CI or one-off use.
|
||||
cfg.team = process.env.SANDBOX_BENCH_TEAM ?? cfg.team
|
||||
cfg.project = process.env.SANDBOX_BENCH_PROJECT ?? cfg.project
|
||||
cfg.cacheDir = expandHome(
|
||||
process.env.SANDBOX_BENCH_CACHE ?? cfg.cacheDir ?? '~/.cache/sandbox-bench'
|
||||
)
|
||||
cfg.reactRepo = expandHome(
|
||||
process.env.SANDBOX_BENCH_REACT_REPO ?? cfg.reactRepo ?? ''
|
||||
)
|
||||
cfg.nextRepo = expandHome(
|
||||
process.env.SANDBOX_BENCH_NEXT_REPO ?? cfg.nextRepo ?? ''
|
||||
)
|
||||
cfg.reactRepoUrl = cfg.reactRepoUrl ?? 'https://github.com/react/react.git'
|
||||
cfg.nextRepoUrl = cfg.nextRepoUrl ?? 'https://github.com/vercel/next.js.git'
|
||||
cfg.vercelBin = cfg.vercelBin ?? 'vercel'
|
||||
if (requireScope && (!cfg.team || !cfg.project)) {
|
||||
throw new Error(
|
||||
'sandbox-bench is not configured: team and project are required.\n' +
|
||||
'Ask the user which Vercel team + project the sandbox VMs should ' +
|
||||
'run under, then save them with:\n' +
|
||||
` node ${path.relative(process.cwd(), new URL(import.meta.url).pathname)} ` +
|
||||
'set team=<team-slug> project=<project-name>\n' +
|
||||
'(Stored in ~/.config/sandbox-bench/config.json, not in the repo.)'
|
||||
)
|
||||
}
|
||||
return cfg
|
||||
}
|
||||
|
||||
export function saveConfig(patch) {
|
||||
fs.mkdirSync(CONFIG_DIR, { recursive: true })
|
||||
const existing = fs.existsSync(CONFIG_FILE)
|
||||
? JSON.parse(fs.readFileSync(CONFIG_FILE, 'utf8'))
|
||||
: {}
|
||||
const merged = { ...existing, ...patch }
|
||||
fs.writeFileSync(CONFIG_FILE, JSON.stringify(merged, null, 2) + '\n')
|
||||
return merged
|
||||
}
|
||||
|
||||
function expandHome(p) {
|
||||
if (!p) return p
|
||||
return p.startsWith('~/') ? path.join(os.homedir(), p.slice(2)) : p
|
||||
}
|
||||
|
||||
// Scope flags appended to every `vercel sandbox` call. Never widen scope
|
||||
// beyond the configured team/project: the project may be shared.
|
||||
export function sandboxScope(cfg) {
|
||||
return ['--team', cfg.team, '--project', cfg.project]
|
||||
}
|
||||
|
||||
// Repos: use the configured checkout, else clone lazily into the cache.
|
||||
function ensureRepo(cfg, checkout, url, name) {
|
||||
if (checkout && fs.existsSync(path.join(checkout, '.git'))) return checkout
|
||||
const dest = path.join(cfg.cacheDir, name)
|
||||
if (fs.existsSync(path.join(dest, '.git'))) return dest
|
||||
fs.mkdirSync(cfg.cacheDir, { recursive: true })
|
||||
console.error(`cloning ${url} into ${dest} (one-time)...`)
|
||||
execFileSync('git', ['clone', '--filter=blob:none', url, dest], {
|
||||
stdio: 'inherit',
|
||||
})
|
||||
return dest
|
||||
}
|
||||
export function ensureReactRepo(cfg) {
|
||||
return ensureRepo(cfg, cfg.reactRepo, cfg.reactRepoUrl, 'react')
|
||||
}
|
||||
export function ensureNextRepo(cfg) {
|
||||
return ensureRepo(cfg, cfg.nextRepo, cfg.nextRepoUrl, 'next')
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------ CLI
|
||||
// realpath both sides: /tmp vs /private/tmp (macOS) or any symlinked
|
||||
// invocation path must not silently skip the CLI (a no-op "config set"
|
||||
// would leave stale team/project in effect with exit 0).
|
||||
function isMain() {
|
||||
if (!process.argv[1]) return false
|
||||
try {
|
||||
return (
|
||||
fs.realpathSync(process.argv[1]) ===
|
||||
fs.realpathSync(new URL(import.meta.url).pathname)
|
||||
)
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
if (isMain()) {
|
||||
const [cmd, ...rest] = process.argv.slice(2)
|
||||
if (cmd === 'show') {
|
||||
let cfg
|
||||
try {
|
||||
cfg = loadConfig({ requireScope: false })
|
||||
} catch (e) {
|
||||
console.error(e.message)
|
||||
process.exit(1)
|
||||
}
|
||||
console.log(JSON.stringify(cfg, null, 2))
|
||||
if (!cfg.team || !cfg.project) {
|
||||
console.error('\nNOT CONFIGURED: team/project missing (see file header).')
|
||||
process.exit(2)
|
||||
}
|
||||
} else if (cmd === 'set') {
|
||||
const patch = {}
|
||||
for (const kv of rest) {
|
||||
const i = kv.indexOf('=')
|
||||
if (i < 0) {
|
||||
console.error(`bad argument "${kv}", want key=value`)
|
||||
process.exit(1)
|
||||
}
|
||||
patch[kv.slice(0, i)] = kv.slice(i + 1)
|
||||
}
|
||||
const merged = saveConfig(patch)
|
||||
console.log(`saved: ${JSON.stringify(merged, null, 2)}`)
|
||||
} else {
|
||||
console.error('usage: node config.mjs show | set key=value [key=value...]')
|
||||
process.exit(1)
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,400 @@
|
||||
#!/usr/bin/env node
|
||||
// Correctness gate for React arms before spending bench compute: each
|
||||
// arm gets a sandbox VM booted from the shared react build-env snapshot
|
||||
// (created lazily if missing), extracts the arm's tree over the
|
||||
// snapshot's node_modules, and runs the jest suite in prod mode (the
|
||||
// channel the bench measures).
|
||||
// A bench number from an arm that fails its tests is meaningless —
|
||||
// always gate hand-assembled or conflict-resolved arms.
|
||||
//
|
||||
// Usage: node sandbox-gate.mjs --arms name=ref[,name=ref...]
|
||||
// [--pattern <jest pattern>, default: full suite] [--extra "yarn flow dom-node"]
|
||||
// [--snapshot snap_x] [--dry-run]
|
||||
import { execFile, spawn } from 'node:child_process'
|
||||
import { promisify } from 'node:util'
|
||||
import fs from 'node:fs'
|
||||
import os from 'node:os'
|
||||
import path from 'node:path'
|
||||
import crypto from 'node:crypto'
|
||||
import { loadConfig, sandboxScope, ensureReactRepo } from './config.mjs'
|
||||
|
||||
const execFileP = promisify(execFile)
|
||||
const DRY_RUN = process.argv.includes('--dry-run')
|
||||
const CONFIG = loadConfig({ requireScope: !DRY_RUN })
|
||||
const REACT = ensureReactRepo(CONFIG)
|
||||
const VERCEL = CONFIG.vercelBin
|
||||
const SCOPE = CONFIG.team ? sandboxScope(CONFIG) : []
|
||||
const SETUP_VERSION = 'v1-al2023-node24-jdk21'
|
||||
const REACT_SNAP_CACHE = path.join(CONFIG.cacheDir, 'react-snap')
|
||||
|
||||
function parseArgs() {
|
||||
const args = { arms: [], snapshot: undefined, extra: [], pattern: '' }
|
||||
const argv = process.argv.slice(2)
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
if (argv[i] === '--arms') {
|
||||
for (const spec of argv[++i].split(',')) {
|
||||
const [name, ref] = spec.split('=')
|
||||
if (!name || !ref) throw new Error(`bad arm spec: ${spec}`)
|
||||
args.arms.push({ name, ref })
|
||||
}
|
||||
} else if (argv[i] === '--snapshot') {
|
||||
args.snapshot = argv[++i]
|
||||
} else if (argv[i] === '--extra') {
|
||||
// Extra gate command(s) run in the repo root after the jest gates,
|
||||
// e.g. --extra "yarn flow dom-node".
|
||||
args.extra.push(argv[++i])
|
||||
} else if (argv[i] === '--pattern') {
|
||||
args.pattern = argv[++i]
|
||||
} else if (argv[i] === '--dry-run') {
|
||||
// handled globally
|
||||
} else {
|
||||
throw new Error(`unknown arg: ${argv[i]}`)
|
||||
}
|
||||
}
|
||||
if (args.arms.length === 0) throw new Error('need --arms name=ref[,...]')
|
||||
if (args.pattern !== '' && !/^[A-Za-z0-9|_.-]+$/.test(args.pattern)) {
|
||||
// The pattern is interpolated into the remote command line; reject
|
||||
// anything that could smuggle extra jest flags (--passWithNoTests
|
||||
// would turn an empty selection into a PASS).
|
||||
throw new Error(
|
||||
`--pattern must match [A-Za-z0-9|_.-]+, got: ${args.pattern}`
|
||||
)
|
||||
}
|
||||
return args
|
||||
}
|
||||
|
||||
async function sb(args, opts = {}) {
|
||||
const scoped = ['sandbox', ...args]
|
||||
const sep = scoped.indexOf('--')
|
||||
scoped.splice(sep < 0 ? scoped.length : sep, 0, ...SCOPE)
|
||||
const { stdout, stderr } = await execFileP(VERCEL, scoped, {
|
||||
maxBuffer: 64 * 1024 * 1024,
|
||||
...opts,
|
||||
})
|
||||
// The CLI prints some results (e.g. snapshot ids) on stderr; callers
|
||||
// that parse exact values (size checks) must keep stdout-only.
|
||||
return opts.withStderr ? `${stdout}\n${stderr}` : stdout
|
||||
}
|
||||
|
||||
async function git(args) {
|
||||
const { stdout } = await execFileP('git', ['-C', REACT, ...args], {
|
||||
maxBuffer: 256 * 1024 * 1024,
|
||||
})
|
||||
return stdout.trim()
|
||||
}
|
||||
|
||||
async function rmVm(name) {
|
||||
try {
|
||||
await sb(['rm', name])
|
||||
} catch (e) {
|
||||
process.stderr.write(`warning: could not remove ${name}: ${e.message}\n`)
|
||||
}
|
||||
}
|
||||
|
||||
// Same detached-nohup-and-poll shape as sandbox-e2e.mjs runDetached:
|
||||
// exec streams drop flakily on multi-minute silences, so never hold one.
|
||||
async function runDetached(vm, tag, script, deadlineMin) {
|
||||
const local = path.join(os.tmpdir(), `gate-${vm}.sh`)
|
||||
fs.writeFileSync(
|
||||
local,
|
||||
// EXIT trap, not ERR: see sandbox-e2e.mjs runDetached.
|
||||
`trap 'code=$?; if [ $code -eq 0 ]; then echo LOOPOK; else echo LOOPFAIL; fi > /vercel/sandbox/loop.done' EXIT\nset -e\n${script}\n`
|
||||
)
|
||||
await sb(['cp', local, `${vm}:/vercel/sandbox/loop.sh`])
|
||||
fs.rmSync(local, { force: true })
|
||||
await sb([
|
||||
'exec',
|
||||
vm,
|
||||
'--timeout',
|
||||
'2m',
|
||||
'--',
|
||||
'bash',
|
||||
'-c',
|
||||
'rm -f /vercel/sandbox/loop.done /vercel/sandbox/loop.log; nohup bash /vercel/sandbox/loop.sh >/vercel/sandbox/loop.log 2>&1 & echo kicked',
|
||||
])
|
||||
let offset = 0
|
||||
let failures = 0
|
||||
const deadline = Date.now() + deadlineMin * 60_000
|
||||
while (true) {
|
||||
await new Promise((r) => setTimeout(r, 45_000))
|
||||
if (Date.now() > deadline) throw new Error(`${tag}: deadline exceeded`)
|
||||
let out
|
||||
try {
|
||||
out = await sb([
|
||||
'exec',
|
||||
vm,
|
||||
'--timeout',
|
||||
'2m',
|
||||
'--',
|
||||
'bash',
|
||||
'-c',
|
||||
`tail -c +${offset + 1} /vercel/sandbox/loop.log | head -c 200000; printf '\\n@@SIZE %s @@DONE %s\\n' "$(stat -c %s /vercel/sandbox/loop.log 2>/dev/null || echo 0)" "$(cat /vercel/sandbox/loop.done 2>/dev/null || echo no)"`,
|
||||
])
|
||||
failures = 0
|
||||
} catch (e) {
|
||||
if (++failures >= 6)
|
||||
throw new Error(
|
||||
`${tag}: ${failures} consecutive poll failures: ${e.message.slice(0, 200)}`
|
||||
)
|
||||
continue
|
||||
}
|
||||
const m = out.match(/@@SIZE (\d+) @@DONE (\S+)/)
|
||||
const body = out.slice(0, out.lastIndexOf('\n@@SIZE'))
|
||||
for (const l of body.split('\n')) {
|
||||
if (l) process.stderr.write(`[${tag}] ${l}\n`)
|
||||
}
|
||||
if (m) {
|
||||
offset = Math.min(Number(m[1]), offset + 200000)
|
||||
if (m[2] === 'LOOPOK') return
|
||||
if (m[2] === 'LOOPFAIL')
|
||||
throw new Error(
|
||||
`${tag}: remote gate failed; tail:\n${body.slice(-3000)}`
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function gateArm(arm, snapshot, tmp, extra = [], pattern = '') {
|
||||
const sha = await git(['rev-parse', arm.ref])
|
||||
const short = sha.slice(0, 12)
|
||||
const vm = `sbench-gate-${arm.name}-${Date.now().toString(36)}`
|
||||
process.stderr.write(`[${arm.name}] vm ${vm} gating ${short}\n`)
|
||||
try {
|
||||
await sb([
|
||||
'create',
|
||||
'--name',
|
||||
vm,
|
||||
'--snapshot',
|
||||
snapshot,
|
||||
'--vcpus',
|
||||
'16',
|
||||
'--timeout',
|
||||
'2h',
|
||||
'--non-persistent',
|
||||
'--network-policy',
|
||||
'allow-all',
|
||||
'--tag',
|
||||
'purpose=sandbox-bench',
|
||||
'--silent',
|
||||
])
|
||||
} catch (e) {
|
||||
if (!/vcpu/i.test(e.message)) throw e
|
||||
await sb([
|
||||
'create',
|
||||
'--name',
|
||||
vm,
|
||||
'--snapshot',
|
||||
snapshot,
|
||||
'--vcpus',
|
||||
'8',
|
||||
'--timeout',
|
||||
'2h',
|
||||
'--non-persistent',
|
||||
'--network-policy',
|
||||
'allow-all',
|
||||
'--tag',
|
||||
'purpose=sandbox-bench',
|
||||
'--silent',
|
||||
])
|
||||
}
|
||||
try {
|
||||
const src = path.join(tmp, `gate-src-${arm.name}.tgz`)
|
||||
await execFileP('bash', [
|
||||
'-c',
|
||||
`git -C ${REACT} archive ${sha} | gzip -1 > ${src}`,
|
||||
])
|
||||
await sb(['cp', src, `${vm}:/vercel/sandbox/src.tgz`])
|
||||
const srcSize = fs.statSync(src).size
|
||||
fs.rmSync(src, { force: true })
|
||||
// sandbox cp can succeed on missing remote files: size-check remotely.
|
||||
const check = await sb([
|
||||
'exec',
|
||||
vm,
|
||||
'--timeout',
|
||||
'2m',
|
||||
'--',
|
||||
'bash',
|
||||
'-c',
|
||||
`stat -c %s /vercel/sandbox/src.tgz`,
|
||||
])
|
||||
if (Number(check.trim()) !== srcSize) {
|
||||
throw new Error(
|
||||
`${arm.name}: uploaded src.tgz size ${check.trim()} != local ${srcSize}`
|
||||
)
|
||||
}
|
||||
await runDetached(
|
||||
vm,
|
||||
`gate:${arm.name}`,
|
||||
// pipefail: every gate command is piped through tail, which would
|
||||
// otherwise mask its exit status from set -e (debugged 2026-07-19:
|
||||
// a failing flow run still printed GATE PASS).
|
||||
`set -o pipefail\n` +
|
||||
`cd /vercel/sandbox/react\n` +
|
||||
`find . -mindepth 1 -maxdepth 1 ! -name node_modules -exec rm -rf {} +\n` +
|
||||
`tar -xzf ../src.tgz && rm -f ../src.tgz\n` +
|
||||
`echo "== ${arm.name} ${short} prod gate (${pattern || 'full suite'}) =="\n` +
|
||||
`yarn test --prod ${pattern} --ci 2>&1 | tail -40\n` +
|
||||
extra
|
||||
.map(
|
||||
(cmd) =>
|
||||
`echo "== ${arm.name} extra: ${cmd} =="\n${cmd} 2>&1 | tail -30\n`
|
||||
)
|
||||
.join('') +
|
||||
`echo "== ${arm.name} GATE PASS =="\n`,
|
||||
90
|
||||
)
|
||||
return { arm: arm.name, sha: short, pass: true }
|
||||
} finally {
|
||||
await rmVm(vm)
|
||||
}
|
||||
}
|
||||
|
||||
// Build-env snapshot: node_modules installed + JDK for the closure
|
||||
// compiler, keyed on yarn.lock. Created once per lockfile, reused by
|
||||
// every later gate.
|
||||
const snapshotPromises = new Map()
|
||||
|
||||
async function ensureReactBuildSnapshot(refSha) {
|
||||
// Raw bytes, exactly as the e2e harness hashes it: a trim here would
|
||||
// silently fork the cache key and build every snapshot twice.
|
||||
const lock = (
|
||||
await execFileP('git', ['-C', REACT, 'show', `${refSha}:yarn.lock`], {
|
||||
maxBuffer: 256 * 1024 * 1024,
|
||||
})
|
||||
).stdout
|
||||
const memoKey = crypto.createHash('sha256').update(lock).digest('hex')
|
||||
if (!snapshotPromises.has(memoKey)) {
|
||||
snapshotPromises.set(memoKey, ensureReactBuildSnapshotForLock(refSha, lock))
|
||||
}
|
||||
return snapshotPromises.get(memoKey)
|
||||
}
|
||||
|
||||
async function ensureReactBuildSnapshotForLock(refSha, lock) {
|
||||
const key = crypto
|
||||
.createHash('sha256')
|
||||
.update(SETUP_VERSION + lock)
|
||||
.digest('hex')
|
||||
.slice(0, 16)
|
||||
const file = path.join(REACT_SNAP_CACHE, `snap-${key}`)
|
||||
if (fs.existsSync(file)) {
|
||||
const id = fs.readFileSync(file, 'utf8').trim()
|
||||
try {
|
||||
await sb(['snapshots', 'get', id])
|
||||
return id
|
||||
} catch {}
|
||||
}
|
||||
const vm = `sbench-snapbuild-${Date.now().toString(36)}`
|
||||
console.error(
|
||||
'creating react build-env snapshot (one-time for this yarn.lock)...'
|
||||
)
|
||||
await sb([
|
||||
'create',
|
||||
'--name',
|
||||
vm,
|
||||
'--runtime',
|
||||
'node24',
|
||||
'--vcpus',
|
||||
'8',
|
||||
'--timeout',
|
||||
'45m',
|
||||
'--non-persistent',
|
||||
'--network-policy',
|
||||
'allow-all',
|
||||
'--tag',
|
||||
'purpose=sandbox-bench',
|
||||
'--silent',
|
||||
])
|
||||
try {
|
||||
const src = path.join(os.tmpdir(), `react-snap-src-${key}.tgz`)
|
||||
await execFileP('bash', [
|
||||
'-c',
|
||||
`git -C ${REACT} archive ${refSha} | gzip -1 > ${src}`,
|
||||
])
|
||||
await sb(['cp', src, `${vm}:/vercel/sandbox/src.tgz`])
|
||||
fs.rmSync(src, { force: true })
|
||||
await sb([
|
||||
'exec',
|
||||
vm,
|
||||
'--timeout',
|
||||
'10m',
|
||||
'--sudo',
|
||||
'--',
|
||||
'dnf',
|
||||
'install',
|
||||
'-y',
|
||||
'-q',
|
||||
'java-21-amazon-corretto-headless',
|
||||
])
|
||||
await runDetached(
|
||||
vm,
|
||||
'react-snap',
|
||||
`mkdir -p /vercel/sandbox/react && cd /vercel/sandbox/react && tar -xzf ../src.tgz && rm -f ../src.tgz\n` +
|
||||
`npm i -g yarn >/dev/null 2>&1\n` +
|
||||
`yarn install --frozen-lockfile --ignore-engines >/dev/null 2>&1\n` +
|
||||
`echo react env ready\n`,
|
||||
30
|
||||
)
|
||||
const out = await sb(['snapshot', vm, '--stop', '--expiration', '30d'], {
|
||||
withStderr: true,
|
||||
})
|
||||
const snapId = out.match(/snap_[A-Za-z0-9]+/)?.[0]
|
||||
if (!snapId)
|
||||
throw new Error(`could not parse snapshot id from: ${out.slice(-500)}`)
|
||||
fs.mkdirSync(REACT_SNAP_CACHE, { recursive: true })
|
||||
fs.writeFileSync(file, `${snapId}\n`)
|
||||
return snapId
|
||||
} finally {
|
||||
await rmVm(vm)
|
||||
}
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const args = parseArgs()
|
||||
if (DRY_RUN) {
|
||||
console.log(
|
||||
`[dry-run] gate plan: ${args.arms.map((a) => `${a.name}=${a.ref}`).join(', ')}`
|
||||
)
|
||||
console.log(
|
||||
`[dry-run] per arm: 1 VM from react build-env snapshot (lazy-created), ` +
|
||||
`yarn test --prod ${args.pattern} --ci` +
|
||||
(args.extra.length ? `, extra: ${args.extra.join(' && ')}` : '')
|
||||
)
|
||||
console.log('[dry-run] PASS requires: verified test counts (pipefail on).')
|
||||
process.exit(0)
|
||||
}
|
||||
// Each arm gates against ITS OWN lockfile's environment: arms with
|
||||
// different yarn.locks must not share node_modules.
|
||||
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'sandbox-gate-'))
|
||||
try {
|
||||
const results = await Promise.allSettled(
|
||||
args.arms.map(async (arm) => {
|
||||
const sha = await git(['rev-parse', arm.ref])
|
||||
const snapshot = args.snapshot ?? (await ensureReactBuildSnapshot(sha))
|
||||
return gateArm(arm, snapshot, tmp, args.extra, args.pattern)
|
||||
})
|
||||
)
|
||||
let failed = 0
|
||||
for (let i = 0; i < results.length; i++) {
|
||||
const r = results[i]
|
||||
if (r.status === 'fulfilled') {
|
||||
console.log(`GATE ${args.arms[i].name}: PASS (${r.value.sha})`)
|
||||
console.log(
|
||||
` bench this exact sha: --arms cand=${r.value.sha} (a ref can move between gate and bench)`
|
||||
)
|
||||
} else {
|
||||
failed++
|
||||
console.log(
|
||||
`GATE ${args.arms[i].name}: FAIL — ${r.reason.message.slice(0, 2500)}`
|
||||
)
|
||||
}
|
||||
}
|
||||
process.exit(failed ? 1 : 0)
|
||||
} finally {
|
||||
fs.rmSync(tmp, { recursive: true, force: true })
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((e) => {
|
||||
console.error(e)
|
||||
process.exit(1)
|
||||
})
|
||||
@@ -0,0 +1,86 @@
|
||||
// List/remove leaked bench sandboxes. Every VM the skill's scripts
|
||||
// create is named sbench-* and tagged purpose=sandbox-bench; the
|
||||
// harnesses remove their VMs in finally blocks, but a killed local
|
||||
// process leaks them until their timeout.
|
||||
//
|
||||
// The project may be shared, so removal is deliberately conservative:
|
||||
// a VM is only listed when its name matches, its tag matches, AND it is
|
||||
// older than --min-age-hours (default 3 — longer than any healthy run),
|
||||
// and removal is per exact listed name.
|
||||
//
|
||||
// node sandbox-sweep.mjs # list what would be removed
|
||||
// node sandbox-sweep.mjs --yes # remove
|
||||
// node sandbox-sweep.mjs --min-age-hours 0 # include fresh VMs (danger:
|
||||
// # only when nothing is active)
|
||||
import { execFile } from 'child_process'
|
||||
import { promisify } from 'util'
|
||||
import { loadConfig, sandboxScope } from './config.mjs'
|
||||
|
||||
const execFileP = promisify(execFile)
|
||||
const CONFIG = loadConfig()
|
||||
const SCOPE = sandboxScope(CONFIG)
|
||||
const argv = process.argv.slice(2)
|
||||
const minAgeHours = argv.includes('--min-age-hours')
|
||||
? Number(argv[argv.indexOf('--min-age-hours') + 1])
|
||||
: 3
|
||||
|
||||
// Age from the CLI's "CREATED" column ("32 minutes ago", "2 hours ago").
|
||||
function ageHours(words) {
|
||||
const i = words.findIndex((w) => w === 'ago')
|
||||
if (i < 2) return Infinity
|
||||
const n = Number(words[i - 2])
|
||||
const unit = words[i - 1]
|
||||
if (!Number.isFinite(n)) return Infinity
|
||||
if (unit.startsWith('second')) return n / 3600
|
||||
if (unit.startsWith('minute')) return n / 60
|
||||
if (unit.startsWith('hour')) return n
|
||||
return n * 24
|
||||
}
|
||||
|
||||
// Paginate: the CLI prints a cursor hint line when more pages exist.
|
||||
const found = []
|
||||
let cursor
|
||||
for (let page = 0; page < 50; page++) {
|
||||
const args = ['sandbox', 'list', '--limit', '100', ...SCOPE]
|
||||
if (cursor) args.push('--cursor', cursor)
|
||||
const { stdout, stderr } = await execFileP(CONFIG.vercelBin, args, {
|
||||
maxBuffer: 1 << 24,
|
||||
})
|
||||
for (const line of stdout.split('\n')) {
|
||||
const words = line.trim().split(/\s+/)
|
||||
const name = words[0]
|
||||
if (!/^sbench-/.test(name)) continue
|
||||
if (!line.includes('purpose:sandbox-bench')) continue
|
||||
const age = ageHours(words)
|
||||
if (age < minAgeHours) {
|
||||
console.log(
|
||||
`skipping ${name} (${age.toFixed(1)}h old < --min-age-hours ${minAgeHours})`
|
||||
)
|
||||
continue
|
||||
}
|
||||
found.push(name)
|
||||
}
|
||||
cursor = `${stdout}\n${stderr}`.match(/--cursor\s+(\S+)/)?.[1]
|
||||
if (!cursor) break
|
||||
}
|
||||
|
||||
if (!found.length) {
|
||||
console.log('no leaked bench sandboxes found')
|
||||
process.exit(0)
|
||||
}
|
||||
const doIt = argv.includes('--yes')
|
||||
let failures = 0
|
||||
for (const name of found) {
|
||||
if (doIt) {
|
||||
try {
|
||||
await execFileP(CONFIG.vercelBin, ['sandbox', 'rm', name, ...SCOPE])
|
||||
console.log(`removed ${name}`)
|
||||
} catch (e) {
|
||||
failures++
|
||||
console.error(`could not remove ${name}: ${e.message.split('\n')[0]}`)
|
||||
}
|
||||
} else {
|
||||
console.log(`would remove ${name} (pass --yes)`)
|
||||
}
|
||||
}
|
||||
process.exit(failures ? 1 : 0)
|
||||
Reference in New Issue
Block a user