Files
laya/tests/test_doc_tables.py
d17414d2b4 fix(benchmarks): two parity cells did not match the JSON they cite (#299)
`BENCHMARKS.md:274` says the fast-path comparison "can be re-checked without a GPU" because
every per-option probability is in `benchmarks/results/parity_*.json`. It can, and two cells
did not match:

| row | column | table said | JSON says |
|---|---|---|---|
| `laya` / `noul` | `max \|p_fast - p_fp32\|` | `0.044` | **0.043** (`d_fast_fp32 = 0.0434565544128418`) |
| `laya-multilingual` / `noul` | `max \|p_fast - p_fp32\|` | `0.046` | **0.045** (`d_fast_fp32 = 0.04548829793930054`) |

Both round down, so this is not a formatting difference. The other 16 numeric cells and all
twelve argmax agreement counts do match.

The sentence under that table was falsified by its own numbers. `BENCHMARKS.md:286` and
`README.md:488` both said the fast path *is at least as close to the fp32 reference as the
stock bf16 path is*, and on `laya-multilingual` / `noul` it is not:

```
laya-multilingual noul   fast_vs_fp32 = 0.045488   stock_vs_fp32 = 0.044593
```

which is the `0.045` vs `0.045` printed two lines above it. The claim held on the other five
rows (e.g. `laya`/`noul` 0.0435 vs 0.0578). Both places now state the bound the data supports
— within `0.046` of fp32, within `0.076` of stock, argmax agreement at least `47/48` — and
name the row that is the exception, rather than asserting something a reader can falsify from
the table in front of them.

Deliberate choices tested:

- **A check, not just two corrections.** `#170`, `#134` and `#208` were all a published number that no longer traced to a run, and `#285` proposes re-benchmarking everything each release. I did not want to add to that checklist, because re-measuring would not have caught either cell: the run is not stale, the transcription is. `tests/test_doc_tables.py` reads the published table out of `BENCHMARKS.md` and compares it cell by cell to the committed JSON, so this half is deterministic, needs no weights and no GPU, and is the part of `#285`'s proposal that can run in CI today.
- **It fails loudly on drift instead of skipping.** Removing the two corrections and running the suite:
  ```
  52 passed, 2 failed
    FAIL parity table/laya/noul d_fast_fp32: got '0.044', want '0.043'
    FAIL parity table/laya-multilingual/noul d_fast_fp32: got '0.046', want '0.045'
  ```
  With them in place, `54 passed, 0 failed`.
- **It reports what it cannot check rather than implying coverage.** Four published tables have no committed artifact behind them (the fine-tuned checkpoint's row, the `103-332 q/s` throughput range, the fast-path latency table whose `fast_*.json` carry no `eval` key, and the reload latencies whose results file is absent). The suite prints those with the reason, and asserts that the list is non-empty, so the gap cannot quietly become zero.
- **Rounding matches how the tables were written.** `round(0.009501, 3)` is `0.010` under banker's rounding but the table's own `0.010` from `0.009501` shows the ordinary convention was used, so the comparison quantizes with `ROUND_HALF_UP` rather than `round()`. Using `round()` would report that row as drifted.
- **The old claim is pinned as false, not just deleted.** The suite asserts that the set of rows where the stock path is closer to fp32 is exactly `laya-multilingual`/`noul`, so a future edit cannot quietly restore the stronger sentence.
- **Line endings.** The suite opens markdown with `newline=""` and normalises CRLF, so it passes on a Windows checkout as well as Linux — which matters because `#212` added the Windows lane.
- Wired into both lanes beside `test_packaging.py`, which is the other suite that reads repository files rather than running the model.

```
$ python tests/test_doc_tables.py               54 passed, 0 failed
$ python tests/test_packaging.py                all packaging tests passed
$ python tests/test_criteria.py                 128 passed, 0 failed
$ python tests/test_router.py                   all routing tests passed
$ python tests/test_load_errors.py              17 passed, 0 failed
$ python tests/test_email.py                    90 passed, 0 failed
```

Known limitation: this verifies the tables that cite a committed JSON, and it cannot check
prose — it would have caught the two cells but not the sentence above them, which is why both
were fixed by hand. It also cannot tell whether a committed JSON is still reproducible; that
is `#285`'s half and needs the hardware.

Co-authored-by: PerryLink <[email protected]>
2026-09-24 08:13:36 +05:30

179 lines
7.5 KiB
Python

"""The benchmark tables that cite a committed JSON must still equal it (#298).
`BENCHMARKS.md:274` promises the fast-path comparison "can be re-checked without a GPU"
because every per-option probability is in `benchmarks/results/parity_*.json`. Two cells
did not match it, and the sentence under the table claimed the fast path is never further
from fp32 than the stock path is, which the `laya-multilingual` `noul` row contradicts.
Re-measuring would not have caught either: the run is not stale, the transcription is. So
this reads the published table and the committed JSON and compares them, which is
deterministic and needs no weights, no GPU and no network.
Only tables with a committed artifact are checked. Everything that cannot be checked from
this repository is listed at the end rather than silently skipped, so the gap stays visible.
"""
from __future__ import annotations
import json
import os
import re
import sys
from typing import Any, Dict, List, Optional, Tuple
PASS: List[str] = []
FAIL: List[str] = []
def check(what: str, got: Any, want: Any) -> None:
if got == want:
PASS.append(what)
else:
FAIL.append("%s: got %r, want %r" % (what, got, want))
def check_true(what: str, cond: bool, detail: Any = "") -> None:
if cond:
PASS.append(what)
else:
FAIL.append("%s: %s" % (what, detail))
def read(path: str) -> str:
# newline="" so a CRLF checkout and an LF checkout parse identically
with open(path, encoding="utf-8", newline="") as fh:
return fh.read().replace("\r\n", "\n")
def rounded(value: float, places: int = 3) -> str:
"""Format the way a person transcribing a run would: nearest, halves away from zero.
`round()` is not usable here. It rounds halves to even, so `round(0.0005, 3)` is `0.0`
and `round(0.0095, 3)` is `0.009`, either of which would report a correct cell as
drifted. The table's own `0.045` from `0.045488` and `0.010` from `0.009501` show the
convention actually used is the ordinary one.
"""
from decimal import ROUND_HALF_UP, Decimal
quantum = Decimal(1).scaleb(-places)
return str(Decimal(repr(value)).quantize(quantum, rounding=ROUND_HALF_UP))
# --------------------------------------------------------------- the published parity table
BENCHMARKS = "BENCHMARKS.md"
PARITY_DIR = os.path.join("benchmarks", "results")
# checkpoint -> the file its row cites, and the row's own name for the model
SOURCES = {
"laya": "parity_english_rtx4070.json",
"laya-multilingual": "parity_multilingual_rtx4070.json",
}
HEADER = ("| checkpoint | type | n | max \\|p_fast - p_stock\\| | max \\|p_fast - p_fp32\\| "
"| max \\|p_stock - p_fp32\\| | argmax fast = stock | fast = fp32 |")
def split_row(line: str) -> List[str]:
return [c.strip() for c in line.strip().strip("|").split("|")]
def unstyled(cell: str) -> str:
"""The table bolds the columns it wants read first; the value is what matters."""
return cell.replace("**", "").replace("\\|", "|").strip()
def parse_table(text: str, header: str) -> List[List[str]]:
lines = text.split("\n")
try:
start = lines.index(header)
except ValueError:
return []
rows = []
for line in lines[start + 2:]: # skip the |---| separator
if not line.startswith("|"):
break
rows.append(split_row(line))
return rows
def main() -> int:
for path in (BENCHMARKS, PARITY_DIR):
check_true("%s exists" % path, os.path.exists(path), "missing")
rows = parse_table(read(BENCHMARKS), HEADER)
check_true("parity table/parsed six rows", len(rows) == 6, "got %d" % len(rows))
check_true("parity table/every row has eight cells",
all(len(r) == 8 for r in rows), [len(r) for r in rows])
summaries: Dict[str, Dict[str, Any]] = {}
for ckpt, name in SOURCES.items():
with open(os.path.join(PARITY_DIR, name), encoding="utf-8") as fh:
summaries[ckpt] = json.load(fh)["summary"]
cols = ("d_fast_stock", "d_fast_fp32", "d_stock_fp32")
for row in rows:
ckpt, qtype = row[0], row[1]
if ckpt not in summaries:
FAIL.append("parity table/unknown checkpoint %r" % ckpt)
continue
summary = summaries[ckpt].get(qtype)
if summary is None:
FAIL.append("parity table/%s has no %r summary in its JSON" % (ckpt, qtype))
continue
label = "%s/%s" % (ckpt, qtype)
check("parity table/%s n" % label, int(row[2]), summary["n"])
for cell, key in zip(row[3:6], cols):
check("parity table/%s %s" % (label, key), unstyled(cell), rounded(summary[key]))
for cell, key in zip(row[6:8], ("agree_fast_stock", "agree_fast_fp32")):
check("parity table/%s %s" % (label, key), unstyled(cell),
"%d/%d" % (summary[key], summary["n"]))
# ------------------------------------------------- the sentence under the table
# BENCHMARKS.md and README.md both asserted the fast path is never further from fp32 than
# the stock bf16 path is. That is checkable from the same JSON, and it was false on one
# row, so the sentence now states the bound the data actually supports instead. What is
# asserted here is that bound, and that the row which falsified the old claim is still
# the one the new wording accounts for.
for ckpt, summary in summaries.items():
for qtype, s in summary.items():
if not isinstance(s, dict):
continue
worst = max(s["d_fast_fp32"], s["d_stock_fp32"], s["d_fast_stock"])
check_true("parity bound/%s/%s within the stated 0.076" % (ckpt, qtype),
worst <= 0.076 + 5e-4, worst)
# the per-row agreement counts are checked above against the table itself; this
# pins the floor the prose quotes, which is the worst row in the set (47/48)
check_true("parity bound/%s/%s agreement floor" % (ckpt, qtype),
min(s["agree_fast_fp32"], s["agree_fast_stock"]) >= 47,
(s["agree_fast_fp32"], s["agree_fast_stock"]))
# the old wording was "at least as close to fp32 as the stock bf16 path is"; this records
# that it is not universally true, so a future edit cannot quietly restore it
contradicted = [(c, q) for c, s in summaries.items() for q, v in s.items()
if isinstance(v, dict) and v["d_fast_fp32"] > v["d_stock_fp32"]]
check("the row that falsified the old sentence is still the multilingual noul one",
contradicted, [("laya-multilingual", "noul")])
# ------------------------------------------------- what this cannot check
unbacked = [
("README.md typed-decisions rows", "no committed artifact for the fine-tuned checkpoint"),
("README.md / BENCHMARKS.md 103-332 q/s throughput", "hand-picked from a range spanning 55-553"),
("BENCHMARKS.md fast-path latency table", "the committed fast_*.json have no 'eval' key"),
("README.md CPU/T4 reload latencies", "research/latency_benchmark_results.json is absent"),
]
check_true("unbacked tables are reported, not skipped", len(unbacked) == 4, unbacked)
print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
for f in FAIL:
print(" FAIL " + f)
if not FAIL:
print("every cell of the parity table matches its committed JSON")
print("not checkable from this repository (not asserted either way):")
for name, why in unbacked:
print(" - %s: %s" % (name, why))
return 1 if FAIL else 0
if __name__ == "__main__":
sys.exit(main())