mirror of
https://github.com/hanzoai/laya.git
synced 2026-10-30 19:33:31 +00:00
`BENCHMARKS.md:274` says the fast-path comparison "can be re-checked without a GPU" because
every per-option probability is in `benchmarks/results/parity_*.json`. It can, and two cells
did not match:
| row | column | table said | JSON says |
|---|---|---|---|
| `laya` / `noul` | `max \|p_fast - p_fp32\|` | `0.044` | **0.043** (`d_fast_fp32 = 0.0434565544128418`) |
| `laya-multilingual` / `noul` | `max \|p_fast - p_fp32\|` | `0.046` | **0.045** (`d_fast_fp32 = 0.04548829793930054`) |
Both round down, so this is not a formatting difference. The other 16 numeric cells and all
twelve argmax agreement counts do match.
The sentence under that table was falsified by its own numbers. `BENCHMARKS.md:286` and
`README.md:488` both said the fast path *is at least as close to the fp32 reference as the
stock bf16 path is*, and on `laya-multilingual` / `noul` it is not:
```
laya-multilingual noul fast_vs_fp32 = 0.045488 stock_vs_fp32 = 0.044593
```
which is the `0.045` vs `0.045` printed two lines above it. The claim held on the other five
rows (e.g. `laya`/`noul` 0.0435 vs 0.0578). Both places now state the bound the data supports
— within `0.046` of fp32, within `0.076` of stock, argmax agreement at least `47/48` — and
name the row that is the exception, rather than asserting something a reader can falsify from
the table in front of them.
Deliberate choices tested:
- **A check, not just two corrections.** `#170`, `#134` and `#208` were all a published number that no longer traced to a run, and `#285` proposes re-benchmarking everything each release. I did not want to add to that checklist, because re-measuring would not have caught either cell: the run is not stale, the transcription is. `tests/test_doc_tables.py` reads the published table out of `BENCHMARKS.md` and compares it cell by cell to the committed JSON, so this half is deterministic, needs no weights and no GPU, and is the part of `#285`'s proposal that can run in CI today.
- **It fails loudly on drift instead of skipping.** Removing the two corrections and running the suite:
```
52 passed, 2 failed
FAIL parity table/laya/noul d_fast_fp32: got '0.044', want '0.043'
FAIL parity table/laya-multilingual/noul d_fast_fp32: got '0.046', want '0.045'
```
With them in place, `54 passed, 0 failed`.
- **It reports what it cannot check rather than implying coverage.** Four published tables have no committed artifact behind them (the fine-tuned checkpoint's row, the `103-332 q/s` throughput range, the fast-path latency table whose `fast_*.json` carry no `eval` key, and the reload latencies whose results file is absent). The suite prints those with the reason, and asserts that the list is non-empty, so the gap cannot quietly become zero.
- **Rounding matches how the tables were written.** `round(0.009501, 3)` is `0.010` under banker's rounding but the table's own `0.010` from `0.009501` shows the ordinary convention was used, so the comparison quantizes with `ROUND_HALF_UP` rather than `round()`. Using `round()` would report that row as drifted.
- **The old claim is pinned as false, not just deleted.** The suite asserts that the set of rows where the stock path is closer to fp32 is exactly `laya-multilingual`/`noul`, so a future edit cannot quietly restore the stronger sentence.
- **Line endings.** The suite opens markdown with `newline=""` and normalises CRLF, so it passes on a Windows checkout as well as Linux — which matters because `#212` added the Windows lane.
- Wired into both lanes beside `test_packaging.py`, which is the other suite that reads repository files rather than running the model.
```
$ python tests/test_doc_tables.py 54 passed, 0 failed
$ python tests/test_packaging.py all packaging tests passed
$ python tests/test_criteria.py 128 passed, 0 failed
$ python tests/test_router.py all routing tests passed
$ python tests/test_load_errors.py 17 passed, 0 failed
$ python tests/test_email.py 90 passed, 0 failed
```
Known limitation: this verifies the tables that cite a committed JSON, and it cannot check
prose — it would have caught the two cells but not the sentence above them, which is why both
were fixed by hand. It also cannot tell whether a committed JSON is still reproducible; that
is `#285`'s half and needs the hardware.
Co-authored-by: PerryLink <[email protected]>
179 lines
7.5 KiB
Python
179 lines
7.5 KiB
Python
"""The benchmark tables that cite a committed JSON must still equal it (#298).
|
|
|
|
`BENCHMARKS.md:274` promises the fast-path comparison "can be re-checked without a GPU"
|
|
because every per-option probability is in `benchmarks/results/parity_*.json`. Two cells
|
|
did not match it, and the sentence under the table claimed the fast path is never further
|
|
from fp32 than the stock path is, which the `laya-multilingual` `noul` row contradicts.
|
|
|
|
Re-measuring would not have caught either: the run is not stale, the transcription is. So
|
|
this reads the published table and the committed JSON and compares them, which is
|
|
deterministic and needs no weights, no GPU and no network.
|
|
|
|
Only tables with a committed artifact are checked. Everything that cannot be checked from
|
|
this repository is listed at the end rather than silently skipped, so the gap stays visible.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import os
|
|
import re
|
|
import sys
|
|
from typing import Any, Dict, List, Optional, Tuple
|
|
|
|
PASS: List[str] = []
|
|
FAIL: List[str] = []
|
|
|
|
|
|
def check(what: str, got: Any, want: Any) -> None:
|
|
if got == want:
|
|
PASS.append(what)
|
|
else:
|
|
FAIL.append("%s: got %r, want %r" % (what, got, want))
|
|
|
|
|
|
def check_true(what: str, cond: bool, detail: Any = "") -> None:
|
|
if cond:
|
|
PASS.append(what)
|
|
else:
|
|
FAIL.append("%s: %s" % (what, detail))
|
|
|
|
|
|
def read(path: str) -> str:
|
|
# newline="" so a CRLF checkout and an LF checkout parse identically
|
|
with open(path, encoding="utf-8", newline="") as fh:
|
|
return fh.read().replace("\r\n", "\n")
|
|
|
|
|
|
def rounded(value: float, places: int = 3) -> str:
|
|
"""Format the way a person transcribing a run would: nearest, halves away from zero.
|
|
|
|
`round()` is not usable here. It rounds halves to even, so `round(0.0005, 3)` is `0.0`
|
|
and `round(0.0095, 3)` is `0.009`, either of which would report a correct cell as
|
|
drifted. The table's own `0.045` from `0.045488` and `0.010` from `0.009501` show the
|
|
convention actually used is the ordinary one.
|
|
"""
|
|
from decimal import ROUND_HALF_UP, Decimal
|
|
|
|
quantum = Decimal(1).scaleb(-places)
|
|
return str(Decimal(repr(value)).quantize(quantum, rounding=ROUND_HALF_UP))
|
|
|
|
|
|
# --------------------------------------------------------------- the published parity table
|
|
BENCHMARKS = "BENCHMARKS.md"
|
|
PARITY_DIR = os.path.join("benchmarks", "results")
|
|
|
|
# checkpoint -> the file its row cites, and the row's own name for the model
|
|
SOURCES = {
|
|
"laya": "parity_english_rtx4070.json",
|
|
"laya-multilingual": "parity_multilingual_rtx4070.json",
|
|
}
|
|
|
|
HEADER = ("| checkpoint | type | n | max \\|p_fast - p_stock\\| | max \\|p_fast - p_fp32\\| "
|
|
"| max \\|p_stock - p_fp32\\| | argmax fast = stock | fast = fp32 |")
|
|
|
|
|
|
def split_row(line: str) -> List[str]:
|
|
return [c.strip() for c in line.strip().strip("|").split("|")]
|
|
|
|
|
|
def unstyled(cell: str) -> str:
|
|
"""The table bolds the columns it wants read first; the value is what matters."""
|
|
return cell.replace("**", "").replace("\\|", "|").strip()
|
|
|
|
|
|
def parse_table(text: str, header: str) -> List[List[str]]:
|
|
lines = text.split("\n")
|
|
try:
|
|
start = lines.index(header)
|
|
except ValueError:
|
|
return []
|
|
rows = []
|
|
for line in lines[start + 2:]: # skip the |---| separator
|
|
if not line.startswith("|"):
|
|
break
|
|
rows.append(split_row(line))
|
|
return rows
|
|
|
|
|
|
def main() -> int:
|
|
for path in (BENCHMARKS, PARITY_DIR):
|
|
check_true("%s exists" % path, os.path.exists(path), "missing")
|
|
|
|
rows = parse_table(read(BENCHMARKS), HEADER)
|
|
check_true("parity table/parsed six rows", len(rows) == 6, "got %d" % len(rows))
|
|
check_true("parity table/every row has eight cells",
|
|
all(len(r) == 8 for r in rows), [len(r) for r in rows])
|
|
|
|
summaries: Dict[str, Dict[str, Any]] = {}
|
|
for ckpt, name in SOURCES.items():
|
|
with open(os.path.join(PARITY_DIR, name), encoding="utf-8") as fh:
|
|
summaries[ckpt] = json.load(fh)["summary"]
|
|
|
|
cols = ("d_fast_stock", "d_fast_fp32", "d_stock_fp32")
|
|
for row in rows:
|
|
ckpt, qtype = row[0], row[1]
|
|
if ckpt not in summaries:
|
|
FAIL.append("parity table/unknown checkpoint %r" % ckpt)
|
|
continue
|
|
summary = summaries[ckpt].get(qtype)
|
|
if summary is None:
|
|
FAIL.append("parity table/%s has no %r summary in its JSON" % (ckpt, qtype))
|
|
continue
|
|
|
|
label = "%s/%s" % (ckpt, qtype)
|
|
check("parity table/%s n" % label, int(row[2]), summary["n"])
|
|
for cell, key in zip(row[3:6], cols):
|
|
check("parity table/%s %s" % (label, key), unstyled(cell), rounded(summary[key]))
|
|
for cell, key in zip(row[6:8], ("agree_fast_stock", "agree_fast_fp32")):
|
|
check("parity table/%s %s" % (label, key), unstyled(cell),
|
|
"%d/%d" % (summary[key], summary["n"]))
|
|
|
|
# ------------------------------------------------- the sentence under the table
|
|
# BENCHMARKS.md and README.md both asserted the fast path is never further from fp32 than
|
|
# the stock bf16 path is. That is checkable from the same JSON, and it was false on one
|
|
# row, so the sentence now states the bound the data actually supports instead. What is
|
|
# asserted here is that bound, and that the row which falsified the old claim is still
|
|
# the one the new wording accounts for.
|
|
for ckpt, summary in summaries.items():
|
|
for qtype, s in summary.items():
|
|
if not isinstance(s, dict):
|
|
continue
|
|
worst = max(s["d_fast_fp32"], s["d_stock_fp32"], s["d_fast_stock"])
|
|
check_true("parity bound/%s/%s within the stated 0.076" % (ckpt, qtype),
|
|
worst <= 0.076 + 5e-4, worst)
|
|
# the per-row agreement counts are checked above against the table itself; this
|
|
# pins the floor the prose quotes, which is the worst row in the set (47/48)
|
|
check_true("parity bound/%s/%s agreement floor" % (ckpt, qtype),
|
|
min(s["agree_fast_fp32"], s["agree_fast_stock"]) >= 47,
|
|
(s["agree_fast_fp32"], s["agree_fast_stock"]))
|
|
|
|
# the old wording was "at least as close to fp32 as the stock bf16 path is"; this records
|
|
# that it is not universally true, so a future edit cannot quietly restore it
|
|
contradicted = [(c, q) for c, s in summaries.items() for q, v in s.items()
|
|
if isinstance(v, dict) and v["d_fast_fp32"] > v["d_stock_fp32"]]
|
|
check("the row that falsified the old sentence is still the multilingual noul one",
|
|
contradicted, [("laya-multilingual", "noul")])
|
|
|
|
# ------------------------------------------------- what this cannot check
|
|
unbacked = [
|
|
("README.md typed-decisions rows", "no committed artifact for the fine-tuned checkpoint"),
|
|
("README.md / BENCHMARKS.md 103-332 q/s throughput", "hand-picked from a range spanning 55-553"),
|
|
("BENCHMARKS.md fast-path latency table", "the committed fast_*.json have no 'eval' key"),
|
|
("README.md CPU/T4 reload latencies", "research/latency_benchmark_results.json is absent"),
|
|
]
|
|
check_true("unbacked tables are reported, not skipped", len(unbacked) == 4, unbacked)
|
|
|
|
print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
|
|
for f in FAIL:
|
|
print(" FAIL " + f)
|
|
if not FAIL:
|
|
print("every cell of the parity table matches its committed JSON")
|
|
print("not checkable from this repository (not asserted either way):")
|
|
for name, why in unbacked:
|
|
print(" - %s: %s" % (name, why))
|
|
return 1 if FAIL else 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|