mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
The count-based greedy hanzo-bench (main.rs) is already on main; this adds the scripts that drive it. Code on main, measurement evidence on the archive branch. - dossier.sh: idle-gated collector with a pinned SHA-256 manifest; runs hanzo-bench and llama-bench on the same GGUF and shapes, both engines wall-clock at sampling parity (decode greedy). Refuses to measure on a busy accelerator. - bench_stats.py: every published statistic as a pure function of the raw per-rep samples -- mean and Student-t 95% CI, CV>5% flag, delta-method ratio with a WIN/LOSS/PARITY verdict, effective-bandwidth roofline, and the paper's numbers-as-data macros and board rows. - battery.sh: model-matrix driver -- loops dossier.sh over a model set per backend. - assemble_paper.sh: fold run dirs into the paper's results-data and board seams. Evidence run directories are intentionally excluded; they remain on the bench/dossier branch as the citable archive the paper references.
30 lines
1.3 KiB
Bash
30 lines
1.3 KiB
Bash
#!/usr/bin/env bash
|
|
# assemble_paper.sh PAPER_DIR RUN_DIR [RUN_DIR ...]
|
|
#
|
|
# Fold per-backend harness outputs into the paper's numbers-as-data seam:
|
|
# results-data.tex = the \renewcommand macros from every run dir (one set per
|
|
# backend/phase; backends never collide)
|
|
# results-board.tex = one complete booktabs tabular of all rows, \input at float
|
|
# level by the paper (never rows into an open alignment)
|
|
# The paper is then a pure function of the committed run dirs. Idempotent.
|
|
set -euo pipefail
|
|
PAPER="$1"; shift
|
|
[ -d "$PAPER" ] || { echo "no paper dir: $PAPER" >&2; exit 2; }
|
|
DATA="$PAPER/results-data.tex"; BOARD="$PAPER/results-board.tex"
|
|
|
|
echo "% GENERATED by scripts/assemble_paper.sh -- do not hand-edit." > "$DATA"
|
|
for d in "$@"; do
|
|
[ -f "$d/results-data.tex" ] && grep '^\\renewcommand' "$d/results-data.tex" >> "$DATA" || true
|
|
done
|
|
|
|
{
|
|
echo "\\begin{tabular}{@{}llrrrrl@{}}"
|
|
echo "\\toprule"
|
|
echo "backend & phase & \$n\$ & hanzo t/s & llama t/s & ratio & verdict \\\\"
|
|
echo "\\midrule"
|
|
for d in "$@"; do [ -f "$d/rows.tex" ] && cat "$d/rows.tex" || true; done
|
|
echo "\\bottomrule"
|
|
echo "\\end{tabular}"
|
|
} > "$BOARD"
|
|
echo "wrote $DATA ($(grep -c renewcommand "$DATA") macros) and $BOARD ($(grep -c '\\\\\\\\' "$BOARD") rows)"
|