Files
engine/scripts/assemble_paper.sh
zeekay 163b9071f2 bench: reproducible cross-engine dossier harness (scripts)
The count-based greedy hanzo-bench (main.rs) is already on main; this adds the
scripts that drive it. Code on main, measurement evidence on the archive branch.

- dossier.sh: idle-gated collector with a pinned SHA-256 manifest; runs hanzo-bench
  and llama-bench on the same GGUF and shapes, both engines wall-clock at sampling
  parity (decode greedy). Refuses to measure on a busy accelerator.
- bench_stats.py: every published statistic as a pure function of the raw per-rep
  samples -- mean and Student-t 95% CI, CV>5% flag, delta-method ratio with a
  WIN/LOSS/PARITY verdict, effective-bandwidth roofline, and the paper's
  numbers-as-data macros and board rows.
- battery.sh: model-matrix driver -- loops dossier.sh over a model set per backend.
- assemble_paper.sh: fold run dirs into the paper's results-data and board seams.

Evidence run directories are intentionally excluded; they remain on the bench/dossier
branch as the citable archive the paper references.
2026-07-21 22:06:01 -07:00

30 lines
1.3 KiB
Bash

#!/usr/bin/env bash
# assemble_paper.sh PAPER_DIR RUN_DIR [RUN_DIR ...]
#
# Fold per-backend harness outputs into the paper's numbers-as-data seam:
# results-data.tex = the \renewcommand macros from every run dir (one set per
# backend/phase; backends never collide)
# results-board.tex = one complete booktabs tabular of all rows, \input at float
# level by the paper (never rows into an open alignment)
# The paper is then a pure function of the committed run dirs. Idempotent.
set -euo pipefail
PAPER="$1"; shift
[ -d "$PAPER" ] || { echo "no paper dir: $PAPER" >&2; exit 2; }
DATA="$PAPER/results-data.tex"; BOARD="$PAPER/results-board.tex"
echo "% GENERATED by scripts/assemble_paper.sh -- do not hand-edit." > "$DATA"
for d in "$@"; do
[ -f "$d/results-data.tex" ] && grep '^\\renewcommand' "$d/results-data.tex" >> "$DATA" || true
done
{
echo "\\begin{tabular}{@{}llrrrrl@{}}"
echo "\\toprule"
echo "backend & phase & \$n\$ & hanzo t/s & llama t/s & ratio & verdict \\\\"
echo "\\midrule"
for d in "$@"; do [ -f "$d/rows.tex" ] && cat "$d/rows.tex" || true; done
echo "\\bottomrule"
echo "\\end{tabular}"
} > "$BOARD"
echo "wrote $DATA ($(grep -c renewcommand "$DATA") macros) and $BOARD ($(grep -c '\\\\\\\\' "$BOARD") rows)"