mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
Every mention of another inference engine is rewritten to say what our code does, or the math it computes. Reference values are "the reference value from the published config" or "the recorded reference output", line citations into another engine's source become the rule itself, comparisons with other engines are dropped, and asides about other engines' bugs go. Identifiers follow: - CUDA: hanzo-paged-attn's `namespace vllm` is `pagedattn`, hanzo-engine's MoE helpers `moe_dtype` and `moe_detail`, the rotary kernels' `hanzo_rotary`; the shuffle/ldg macros are HANZO_*. - GPTQ: the shuffled-layout path (`reconstruct_exllama`, `use_exllama`) is `reconstruct_shuffled` / `is_shuffled`, on the kernel and the FFI side. - Tests: `*_matches_vllm` and friends name what they compare against (the reference fixture, the tile rule). - MiniCPM-o's vision-LLM embedding is `vlm_embedding`; the rotary custom op is `rotary-embedding`. - hanzo-bench's board scores ours against a "rival" whose name comes from the manifest: `rival_<tag>.json`, `--rival*` manifest flags, `rival t/s` columns; dossier.sh and spec_decode.sh take `--rival-bench`. - install.ps1 installs this repository's hanzo-cli, as install.sh does, with HANZO_INSTALL_NCCL / HANZO_INSTALL_NO_NCCL. - bench_http.py reads only the hanzo: metric families. Kept, because they are what they are: LICENSE, LICENSE-APACHE, NOTICE and tools/upstream-map.sh; each ported file's copyright/licence/provenance header; Cargo.toml dependency names; data identifiers (Hugging Face repo IDs, fixture file names and fixture metadata keys the tests read, on-disk paths, executable names the GPU and engine guards match on a host); model and format names. LLM.md says the rule once, under "One name: Hanzo Engine". Co-authored-by: Hanzo Dev <[email protected]>
128 lines
6.6 KiB
Bash
128 lines
6.6 KiB
Bash
#!/usr/bin/env bash
|
|
# spec_decode.sh — the ARCHITECTURE row: prompt-lookup speculative-decode multiple, per backend.
|
|
#
|
|
# Faithful port of the Metal lane's protocol (bench-protocols/spec-decode-metal.md): three
|
|
# verbatim workload classes, ngram_max=3, a gamma sweep, greedy + lossless (target verifies every
|
|
# draft -> byte-identical to baseline greedy). Reported BESIDE the kernel board, never blended --
|
|
# this is an algorithmic multiple, orthogonal to per-token kernel efficiency. Run the SAME thing
|
|
# per backend so the acceptance/multiple is byte-comparable.
|
|
#
|
|
# spec_decode.sh --model GGUF --hanzo HANZO_CLI [--rival-bench PATH] \
|
|
# [--backend-env "K=V ..."] [--gammas 4,8,16] [--ngram 3] [--reps 7] [--maxtok 200] \
|
|
# [--port 8613] [--out DIR]
|
|
#
|
|
# multiple = spec t/s / baseline t/s (same server, spec flags off = baseline). vs-rival = spec
|
|
# t/s / the rival bench's greedy decode t/s (its `-p 0 -n 256 -o json` row). Lossless is asserted
|
|
# by SHA(completion) with lookup ON == OFF.
|
|
set -euo pipefail
|
|
MODEL="" HANZO="" RB="" BENV="" GAMMAS="4,8,16" NGRAM=3 REPS=7 MAXTOK=200 PORT=8613 OUT=""
|
|
while [[ $# -gt 0 ]]; do case "$1" in
|
|
--model) MODEL="$2"; shift 2;; --hanzo) HANZO="$2"; shift 2;; --rival-bench) RB="$2"; shift 2;;
|
|
--backend-env) BENV="$2"; shift 2;; --gammas) GAMMAS="$2"; shift 2;; --ngram) NGRAM="$2"; shift 2;;
|
|
--reps) REPS="$2"; shift 2;; --maxtok) MAXTOK="$2"; shift 2;; --port) PORT="$2"; shift 2;;
|
|
--out) OUT="$2"; shift 2;; *) echo "unknown arg: $1" >&2; exit 2;;
|
|
esac; done
|
|
: "${MODEL:?--model required}" "${HANZO:?--hanzo (CLI binary) required}"
|
|
OUT="${OUT:-$(mktemp -d "${TMPDIR:-/tmp}/spec.XXXXXX")}"; mkdir -p "$OUT"
|
|
MDIR="$(cd "$(dirname "$MODEL")" && pwd)"; MFILE="$(basename "$MODEL")"
|
|
|
|
# Verbatim prompts (heredocs preserve newlines/spacing exactly -- byte-comparable across backends).
|
|
cat > "$OUT/A.txt" <<'PA'
|
|
The mission log recorded the same entry every day. Day one: all systems nominal, crew resting, oxygen stable, course unchanged. Day two: all systems nominal, crew resting, oxygen stable, course unchanged. Day three: all systems nominal, crew resting, oxygen stable, course unchanged. Day four: all systems nominal, crew resting, oxygen stable, course unchanged. Day five:
|
|
PA
|
|
cat > "$OUT/B.txt" <<'PB'
|
|
Reproduce this file exactly with no changes:
|
|
def fib(n):
|
|
a, b = 0, 1
|
|
for _ in range(n):
|
|
a, b = b, a + b
|
|
return a
|
|
|
|
def fact(n):
|
|
r = 1
|
|
for i in range(1, n+1):
|
|
r *= i
|
|
return r
|
|
|
|
Exact reproduction:
|
|
def fib(n):
|
|
PB
|
|
cat > "$OUT/C.txt" <<'PC'
|
|
Write an original science fiction story about the first human colony on a distant exoplanet. Begin now:
|
|
PC
|
|
|
|
SPID=""
|
|
cleanup() { [[ -n "$SPID" ]] && kill -TERM "$SPID" 2>/dev/null && for _ in $(seq 1 20); do kill -0 "$SPID" 2>/dev/null || break; sleep 0.5; done; kill -KILL "$SPID" 2>/dev/null || true; SPID=""; }
|
|
trap cleanup EXIT
|
|
wait_health() { local waited=0; until curl -sf "http://localhost:$PORT/health" >/dev/null 2>&1; do kill -0 "$SPID" 2>/dev/null || return 1; sleep 1; waited=$((waited+1)); ((waited>600)) && return 1; done; }
|
|
|
|
# Run one server config over all three workloads. $1=label, $2..=extra serve flags (spec on/off).
|
|
run_cfg() {
|
|
local label="$1"; shift
|
|
env $BENV "$HANZO" serve "$@" --format gguf -m "$MDIR" -f "$MFILE" -p "$PORT" > "$OUT/serve-$label.log" 2>&1 &
|
|
SPID=$!; wait_health || { echo "server '$label' no bind" >&2; tail -5 "$OUT/serve-$label.log" >&2; cleanup; return 1; }
|
|
for wl in A B C; do
|
|
python3 - "$PORT" "$OUT/$wl.txt" "$REPS" "$MAXTOK" "$label" "$wl" "$OUT/spec_raw.jsonl" <<'PY'
|
|
import json, sys, time, hashlib, urllib.request
|
|
port, pf, reps, maxtok, label, wl, out = sys.argv[1], sys.argv[2], int(sys.argv[3]), int(sys.argv[4]), sys.argv[5], sys.argv[6], sys.argv[7]
|
|
prompt = open(pf).read()
|
|
url = f"http://localhost:{port}/v1/completions"
|
|
ts, txt = [], ""
|
|
for i in range(reps):
|
|
body = json.dumps({"model":"default","prompt":prompt,"max_tokens":maxtok,"temperature":0}).encode()
|
|
t0 = time.perf_counter()
|
|
with urllib.request.urlopen(urllib.request.Request(url, data=body, headers={"content-type":"application/json"}), timeout=600) as r:
|
|
d = json.load(r)
|
|
dt = time.perf_counter() - t0
|
|
n = (d.get("usage") or {}).get("completion_tokens") or 0
|
|
txt = d["choices"][0].get("text","")
|
|
if i > 0 and dt > 0 and n > 0: # drop the first (cold) rep
|
|
ts.append(n/dt)
|
|
mean = sum(ts)/len(ts) if ts else float("nan")
|
|
rec = {"label":label,"workload":wl,"tok_s":mean,"n_scored":len(ts),
|
|
"completion_sha256":hashlib.sha256(txt.encode()).hexdigest()}
|
|
open(out,"a").write(json.dumps(rec)+"\n")
|
|
print(f" {label} {wl}: {mean:.1f} t/s (sha {rec['completion_sha256'][:8]})", file=sys.stderr)
|
|
PY
|
|
done
|
|
cleanup
|
|
}
|
|
|
|
echo "spec-decode: $MFILE (ngram=$NGRAM gammas=$GAMMAS)" >&2
|
|
: > "$OUT/spec_raw.jsonl"
|
|
RIVAL_TS="nan"
|
|
if [[ -n "$RB" && -x "$RB" ]]; then
|
|
RIVAL_TS="$(env $BENV "$RB" -m "$MODEL" -p 0 -n 256 -o json 2>/dev/null | python3 -c "import sys,json; r=[x for x in json.load(sys.stdin) if x.get('n_gen',0)>0]; print(f'{r[0][\"avg_ts\"]:.1f}' if r else 'nan')" 2>/dev/null || echo nan)"
|
|
fi
|
|
run_cfg baseline || true
|
|
IFS=',' read -ra GS <<< "$GAMMAS"
|
|
for g in "${GS[@]}"; do run_cfg "g$g" --prompt-lookup-ngram "$NGRAM" --gamma "$g" || true; done
|
|
|
|
python3 - "$OUT" "$RIVAL_TS" <<'PY'
|
|
import json, os, sys, math
|
|
out, rival = sys.argv[1], float(sys.argv[2]) if sys.argv[2] not in ("nan","") else float("nan")
|
|
rows = [json.loads(l) for l in open(os.path.join(out,"spec_raw.jsonl"))]
|
|
by = {(r["label"], r["workload"]): r for r in rows}
|
|
names = {"A":"sustained-reuse","B":"code-copy","C":"novel"}
|
|
res = {"rival_greedy_ts": rival, "workloads": []}
|
|
labels = sorted({r["label"] for r in rows if r["label"]!="baseline"})
|
|
for wl in ["A","B","C"]:
|
|
base = by.get(("baseline",wl))
|
|
if not base: continue
|
|
bt = base["tok_s"]
|
|
entry = {"workload": names[wl], "baseline_ts": bt, "gammas": {}}
|
|
for lab in labels:
|
|
r = by.get((lab,wl))
|
|
if not r: continue
|
|
mult = r["tok_s"]/bt if bt else float("nan")
|
|
vsrv = r["tok_s"]/rival if rival and not math.isnan(rival) else float("nan")
|
|
lossless = (r["completion_sha256"] == base["completion_sha256"])
|
|
entry["gammas"][lab] = {"ts": r["tok_s"], "multiple": mult, "vs_rival": vsrv, "lossless": lossless}
|
|
res["workloads"].append(entry)
|
|
best = max(entry["gammas"].values(), key=lambda x:x["multiple"], default=None)
|
|
if best:
|
|
print(f'{names[wl]:16} baseline {bt:6.1f} best mult {best["multiple"]:.2f}x vs-rival {best["vs_rival"]:.2f}x lossless={all(g["lossless"] for g in entry["gammas"].values())}')
|
|
json.dump(res, open(os.path.join(out,"spec_decode.json"),"w"), indent=2)
|
|
print("-> "+os.path.join(out,"spec_decode.json"))
|
|
PY
|