mirror of
https://github.com/hanzoai/laya.git
synced 2026-10-30 19:33:31 +00:00
`research/eval/README.md` claimed this harness shared its `confidence == 0.0` bin
boundary with `laya.common.ece_score` and `bench_local.py`, and `test_laya_eval.py`
asserted it under the name "matches upstream". It did not:
```
laya.common.ece_score([0,0],[1,1]) = 1.0
research/scripts/bench_local.py = 1.0
research/scripts/build_benchmark_nb.py = 1.0
research/eval/laya_eval.py = 0.0 <-- this one
```
`#39` ("fix: include zero confidence in ECE") landed the `conf >= lo` boundary in
`laya/common.py`, `bench_local.py` and `build_benchmark_nb.py` at 07:03 today. The
harness was merged in `#210` at 07:07 with the pre-`#39` test, `sel = (confidence > lo)
& (confidence <= hi)`, so zero-confidence cases fell in no bin and contributed nothing.
That is this PR's own mistake, and the README documented it as agreement.
The fix is the first bin:
```python
for i, (lo, hi) in enumerate(zip(edges[:-1], edges[1:])):
sel = (confidence >= lo if i == 0 else confidence > lo) & (confidence <= hi)
```
All four implementations now agree on every case:
| case | common | laya_eval | bench_local | build_nb |
|---|---|---|---|---|
| conf exactly 0, wrong | 1.0000 | 1.0000 | 1.0000 | 1.0000 |
| conf 0 and 1 | 0.5000 | 0.5000 | 0.5000 | 0.5000 |
| ordinary | 0.1000 | 0.1000 | 0.1000 | 0.1000 |
| just above 0 | 1.0000 | 1.0000 | 1.0000 | 1.0000 |
| one bin | 0.0500 | 0.0500 | 0.0500 | 0.0500 |
| all wrong, confident | 0.9500 | 0.9500 | 0.9500 | 0.9500 |
(`build_benchmark_nb.py` defines its `ece_score` inside a generated notebook cell, so it
cannot be imported; the function text is extracted and executed to compare it.)
Deliberate choices tested:
- **Fixed the harness rather than the doc.** `#39` was a deliberate change to include zero confidence, and it is the majority position in the repo — three implementations against one. The alternative was to reword the README to say the harness bins alone, which would leave the harness incomparable with the published numbers for no reason. Comparability was the stated point of the original claim.
- **Regenerating the committed data is not needed, and I checked rather than assumed.** The change can only move a language that has a `confidence == 0.0` case. Six languages re-run at the committed settings (`--unclamped`, 100 cases, 20 options) have **zero** such cases each, and every one of their ECE values matches `cpu_51_language_sweep.json` exactly under both boundaries:
| lang | cases with `confidence == 0.0` | old ECE | new ECE | committed |
|---|---|---|---|---|
| en | 0 | 0.1789 | 0.1789 | 0.1789 |
| de | 0 | 0.5584 | 0.5584 | 0.5584 |
| km | 0 | 0.9515 | 0.9515 | 0.9515 |
| am | 0 | 0.8253 | 0.8253 | 0.8252 |
| my | 0 | 0.8605 | 0.8605 | 0.8605 |
| th | 0 | 0.8814 | 0.8814 | 0.8814 |
`am` differs from the committed 0.8252 by 0.0001 under *both* boundaries, which is the torch-version quantisation `research/eval/README.md` already documents, not this change.
- **The test now asserts the boundary by value instead of asserting the harness matches itself.** The old case was written against the harness's own arithmetic (`check(..., 0.0)`), so it passed for exactly the reason the divergence went unnoticed.
- **No CI wiring added here.** `research/eval/test_laya_eval.py` runs in no workflow, so this suite is not gated. Wiring it needs an install step in both lanes and belongs with the other lane changes, not in a correctness fix.
Known limitation: the committed `research/results/eval_english_51_languages.json` was
generated under the old boundary, so it is an artefact of the pre-fix harness. It is
unaffected for the same reason the table above holds — no case has zero confidence — and
I have left it byte-identical rather than regenerating 51 languages to reproduce the same
numbers.
Verified:
```
$ python research/eval/test_laya_eval.py 49 passed, 0 failed (47 before)
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/ research/ # clean
```
Co-authored-by: PerryLink <[email protected]>
378 lines
16 KiB
Python
378 lines
16 KiB
Python
"""laya-eval: an independent, reproducible evaluation harness for Laya checkpoints.
|
|
|
|
Why this exists
|
|
---------------
|
|
The repository's own benchmark scripts are research code: they load everything,
|
|
run all parts, and print tables. There is no small, reproducible harness a third
|
|
party can point at a Laya checkpoint to get a per-language accuracy and ECE
|
|
report plus machine-readable per-case output. This is that harness.
|
|
|
|
Provenance
|
|
----------
|
|
The prompt format, sampling and metrics follow ``research/scripts/bench_local.py``
|
|
so that results are comparable with the published tables:
|
|
|
|
* dataset ``mteb/amazon_massive_intent``, split ``test``
|
|
* sampling first ``--per-lang`` rows, ``random.Random(SEED)`` fresh per language
|
|
* options ``--n-opts`` labels: the gold one plus ``rng.sample`` of the rest
|
|
* prompt "What is the user asking for in `utterance`?"
|
|
* rendering option keys with ``_`` -> `` `` and ``.`` -> ``: ``
|
|
* metrics accuracy, macro-F1, ECE (15 bins) on the temperature-scaled softmax
|
|
|
|
Verified against ``research/results/cpu_51_language_sweep.json``: with the raw
|
|
pre-clamp temperatures, ``en`` reproduces n=100 / accuracy 0.82 / macro_f1 0.7876
|
|
/ ece 0.1789 / mean_confidence 0.9989 exactly.
|
|
|
|
Usage
|
|
-----
|
|
python -m laya_eval --model convaiinnovations/laya --langs en,de,ro
|
|
python -m laya_eval --model ./local-checkpoint --langs all --out report.json
|
|
python -m laya_eval --model convaiinnovations/laya --subfolder multilingual --langs all
|
|
|
|
Output is a JSON document with a ``config`` block, a per-language ``report`` and
|
|
a ``cases`` list carrying every individual decision, so a number in the report can
|
|
be re-derived without re-running the model.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import json
|
|
import os
|
|
import random
|
|
import sys
|
|
import time
|
|
from typing import Any, Dict, List, Optional, Sequence, Tuple
|
|
|
|
# Running this file directly puts research/eval/ on sys.path, not the repo root, so
|
|
# `import laya` would fail. Allow both `python research/eval/laya_eval.py` and
|
|
# `python -m research.eval.laya_eval` to find the package.
|
|
_REPO_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
|
if _REPO_ROOT not in sys.path:
|
|
sys.path.insert(0, _REPO_ROOT)
|
|
|
|
SEED = 13
|
|
N_OPTS = 20
|
|
PER_LANG = 100
|
|
INSTRUCTIONS = "What is the user asking for in `utterance`?"
|
|
DATASET = "mteb/amazon_massive_intent"
|
|
ECE_BINS = 15
|
|
|
|
|
|
# --------------------------------------------------------------------- sampling
|
|
def render_label(key: str) -> str:
|
|
"""Option text for one intent label, matching bench_local.py:176."""
|
|
return key.replace("_", " ").replace(".", ": ")
|
|
|
|
|
|
def build_case(text: str, gold: str, pool: Sequence[str], rng: random.Random,
|
|
n_opts: int) -> Tuple[Dict[str, Any], Dict[str, Any], int]:
|
|
"""One case: a state, a 20-option choice question, and the gold option index."""
|
|
keys = [gold] + rng.sample(list(pool), min(n_opts - 1, len(pool)))
|
|
rng.shuffle(keys)
|
|
state = {"utterance": text}
|
|
question = {
|
|
"intent": {
|
|
"type": "choice",
|
|
"instructions": INSTRUCTIONS,
|
|
"criteria": {k: render_label(k) for k in keys},
|
|
}
|
|
}
|
|
return state, question, keys.index(gold)
|
|
|
|
|
|
def build_suite(rows: Sequence[Dict[str, Any]], labels: Sequence[str],
|
|
per_lang: int, n_opts: int, seed: int = SEED):
|
|
"""Deterministic suite for one language. Returns (cases, gold, option_keys)."""
|
|
labels = sorted(labels)
|
|
rng = random.Random(seed) # fresh per language, as upstream
|
|
cases, gold, option_keys = [], [], []
|
|
for row in list(rows)[:per_lang]:
|
|
pool = [x for x in labels if x != row["label_text"]]
|
|
state, question, gold_idx = build_case(row["text"], row["label_text"],
|
|
pool, rng, n_opts)
|
|
keys = list(question["intent"]["criteria"])
|
|
cases.append((state, question))
|
|
gold.append(gold_idx)
|
|
option_keys.append(keys)
|
|
return cases, gold, option_keys
|
|
|
|
|
|
def load_language(lang: str, split: str = "test"):
|
|
"""Load one language config. Raises with a readable message if unavailable."""
|
|
from datasets import load_dataset
|
|
try:
|
|
ds = load_dataset(DATASET, lang, split=split)
|
|
except Exception as exc: # pragma: no cover - network path
|
|
raise RuntimeError(
|
|
"could not load %s config %r: %s" % (DATASET, lang, exc)
|
|
) from exc
|
|
return [{"text": r["text"], "label_text": r["label_text"]} for r in ds]
|
|
|
|
|
|
def available_languages() -> List[str]:
|
|
"""Language configs the dataset exposes, excluding the aggregate 'default'."""
|
|
from datasets import get_dataset_config_names
|
|
names = get_dataset_config_names(DATASET)
|
|
return sorted(n for n in names if n != "default")
|
|
|
|
|
|
# ---------------------------------------------------------------------- scoring
|
|
def score_cases(agent, cases) -> List[Any]:
|
|
"""Raw marker logits per case, from one collated forward pass.
|
|
|
|
Mirrors bench_local.py:54-97: build every sequence, collate, run the model
|
|
once, and slice each row to its own marker count. No temperature is applied
|
|
here so the caller can score under more than one regime from one pass.
|
|
"""
|
|
import torch
|
|
from laya.common import QTYPES, build_sequence, collate_items, render_options
|
|
|
|
max_len = agent.cfg.get("max_len", 512)
|
|
head_max_len = agent.cfg.get("head_max_len", 192)
|
|
items = []
|
|
for state, questions in cases:
|
|
for _qid, qdef in questions.items():
|
|
q = {"t": qdef["type"],
|
|
"ins": qdef["instructions"],
|
|
"crit": qdef.get("criteria")}
|
|
ids, markers = build_sequence(agent.tok, state, q, max_len, head_max_len)
|
|
if len(markers) != len(render_options(q)):
|
|
raise ValueError("marker/option count mismatch; question exceeds head_max_len")
|
|
items.append({"ids": ids, "markers": markers, "qtype": QTYPES[q["t"]]})
|
|
batch = collate_items([items], agent.tok.pad_token_id)
|
|
with torch.no_grad():
|
|
logits, _ = agent.model(
|
|
batch["input_ids"].to(agent.device),
|
|
batch["attention_mask"].to(agent.device),
|
|
batch["marker_pos"].to(agent.device),
|
|
batch["marker_mask"].to(agent.device),
|
|
batch["qtype"].to(agent.device),
|
|
)
|
|
logits = logits.float().cpu().numpy()
|
|
return [logits[i, :len(it["markers"])] for i, it in enumerate(items)]
|
|
|
|
|
|
def softmax_t(z, temperature: float = 1.0):
|
|
import numpy as np
|
|
z = np.asarray(z, dtype=float) / max(1e-3, float(temperature))
|
|
e = np.exp(z - z.max())
|
|
return e / e.sum()
|
|
|
|
|
|
def temperature_for(agent, qtype: int, k: int, unclamped: bool = False) -> float:
|
|
"""The temperature to score this bucket with.
|
|
|
|
By default this is what ``Agent`` applies, i.e. the checkpoint's bucket clamped
|
|
to ``[TEMP_MIN, TEMP_MAX]``. With ``unclamped=True`` it is the checkpoint's raw
|
|
value, which is what the committed pre-#42 sweep was produced with.
|
|
"""
|
|
from laya.common import temp_bucket
|
|
bucket = temp_bucket(qtype, k)
|
|
if unclamped:
|
|
return float(agent.temperature_by_options_raw.get(
|
|
bucket, agent.temperature_raw[qtype]))
|
|
return float(agent.temperature_by_options.get(bucket, agent.temperature[qtype]))
|
|
|
|
|
|
def ece(confidence, correct, bins: int = ECE_BINS) -> float:
|
|
"""Expected calibration error over equal-width confidence bins.
|
|
|
|
The first bin is closed at the bottom (`>= lo`), so `confidence == 0.0` is counted.
|
|
That is the boundary #39 settled in `laya.common.ece_score`,
|
|
`research/scripts/bench_local.py` and `research/scripts/build_benchmark_nb.py`. This
|
|
harness kept the pre-#39 test until the divergence was found, which made it the only
|
|
one of the four implementations that binned differently.
|
|
"""
|
|
import numpy as np
|
|
confidence = np.asarray(confidence, dtype=float)
|
|
correct = np.asarray(correct, dtype=float)
|
|
if not len(confidence):
|
|
return float("nan")
|
|
total, edges = 0.0, np.linspace(0.0, 1.0, bins + 1)
|
|
for i, (lo, hi) in enumerate(zip(edges[:-1], edges[1:])):
|
|
sel = (confidence >= lo if i == 0 else confidence > lo) & (confidence <= hi)
|
|
if sel.any():
|
|
total += sel.mean() * abs(confidence[sel].mean() - correct[sel].mean())
|
|
return float(total)
|
|
|
|
|
|
def macro_f1(gold, pred) -> float:
|
|
import numpy as np
|
|
gold, pred = np.asarray(gold), np.asarray(pred)
|
|
scores = []
|
|
for cls in sorted(set(gold.tolist()) | set(pred.tolist())):
|
|
tp = int(((pred == cls) & (gold == cls)).sum())
|
|
fp = int(((pred == cls) & (gold != cls)).sum())
|
|
fn = int(((pred != cls) & (gold == cls)).sum())
|
|
scores.append(2 * tp / max(1, 2 * tp + fp + fn))
|
|
return float(np.mean(scores)) if scores else float("nan")
|
|
|
|
|
|
def summarise(confidences, corrects, golds, preds) -> Dict[str, float]:
|
|
"""The metric block, matching bench_local.py:132-144."""
|
|
import numpy as np
|
|
confidences = np.asarray(confidences, dtype=float)
|
|
corrects = np.asarray(corrects, dtype=float)
|
|
n = len(confidences)
|
|
if not n:
|
|
return {"n": 0}
|
|
half = max(1, n // 2)
|
|
order = np.argsort(-confidences)[:half]
|
|
return {
|
|
"n": n,
|
|
"accuracy": round(float(corrects.mean()), 4),
|
|
"macro_f1": round(macro_f1(golds, preds), 4),
|
|
"ece": round(ece(confidences, corrects), 4),
|
|
"mean_confidence": round(float(confidences.mean()), 4),
|
|
"acc_at_50_coverage": round(float(corrects[order].mean()), 4),
|
|
}
|
|
|
|
|
|
# ------------------------------------------------------------------------ runner
|
|
def run_language(agent, lang: str, per_lang: int, n_opts: int, seed: int = SEED,
|
|
unclamped: bool = False) -> Dict[str, Any]:
|
|
"""Evaluate one language and return its report plus per-case records."""
|
|
import numpy as np
|
|
from laya.common import QTYPES
|
|
|
|
rows = load_language(lang)
|
|
cases, gold, option_keys = build_suite(
|
|
rows, sorted({r["label_text"] for r in rows}), per_lang, n_opts, seed)
|
|
logits = score_cases(agent, cases)
|
|
|
|
confidences, corrects, preds, records = [], [], [], []
|
|
for i, z in enumerate(logits):
|
|
k = len(z)
|
|
temperature = temperature_for(agent, QTYPES["choice"], k, unclamped)
|
|
probs = softmax_t(z, temperature)
|
|
pred = int(np.argmax(probs))
|
|
correct = int(pred == gold[i])
|
|
confidences.append(float(probs.max()))
|
|
corrects.append(float(correct))
|
|
preds.append(pred)
|
|
records.append({
|
|
"lang": lang,
|
|
"index": i,
|
|
"state": cases[i][0],
|
|
"instructions": INSTRUCTIONS,
|
|
"options": option_keys[i],
|
|
"gold_index": int(gold[i]),
|
|
"gold_label": option_keys[i][gold[i]],
|
|
"pred_index": pred,
|
|
"pred_label": option_keys[i][pred],
|
|
"probability": round(float(probs[pred]), 6),
|
|
"p_gold": round(float(probs[gold[i]]), 6),
|
|
"confidence": round(float(probs.max()), 6),
|
|
"correct": correct,
|
|
"temperature": round(float(temperature), 6),
|
|
})
|
|
|
|
report = summarise(confidences, corrects, gold, preds)
|
|
report["temperature"] = round(
|
|
float(temperature_for(agent, QTYPES["choice"], n_opts, unclamped)), 6)
|
|
return {"report": report, "cases": records}
|
|
|
|
|
|
def main(argv: Optional[Sequence[str]] = None) -> int:
|
|
parser = argparse.ArgumentParser(
|
|
prog="laya-eval",
|
|
description="Per-language accuracy and calibration report for a Laya checkpoint.")
|
|
parser.add_argument("--model", default="convaiinnovations/laya",
|
|
help="checkpoint repo id or local path")
|
|
parser.add_argument("--subfolder", default=None,
|
|
help="checkpoint subfolder, e.g. multilingual")
|
|
parser.add_argument("--device", default=None, help="cpu, cuda, mps (default: auto)")
|
|
parser.add_argument("--langs", default="en",
|
|
help="comma-separated configs, or 'all'")
|
|
parser.add_argument("--per-lang", type=int, default=PER_LANG)
|
|
parser.add_argument("--n-opts", type=int, default=N_OPTS)
|
|
parser.add_argument("--seed", type=int, default=SEED)
|
|
parser.add_argument("--out", default=None, help="write the JSON report here")
|
|
parser.add_argument("--no-cases", action="store_true",
|
|
help="omit per-case records (smaller file)")
|
|
parser.add_argument("--unclamped", action="store_true",
|
|
help="score with the checkpoint's RAW bucket temperatures instead "
|
|
"of the clamped ones Agent applies. This is what reproduces "
|
|
"the committed pre-#42 sweep")
|
|
args = parser.parse_args(argv)
|
|
|
|
if args.langs.strip().lower() == "all":
|
|
try:
|
|
langs = available_languages()
|
|
except Exception as exc:
|
|
print("could not list dataset configs: %s" % exc, file=sys.stderr)
|
|
return 2
|
|
else:
|
|
langs = [x.strip() for x in args.langs.split(",") if x.strip()]
|
|
if not langs:
|
|
print("no languages selected", file=sys.stderr)
|
|
return 2
|
|
|
|
import laya
|
|
|
|
started = time.time()
|
|
agent = laya.load(args.model, device=args.device, subfolder=args.subfolder)
|
|
agent.model.eval()
|
|
payload: Dict[str, Any] = {
|
|
"config": {
|
|
"model": args.model,
|
|
"subfolder": args.subfolder,
|
|
"device": str(agent.device),
|
|
"max_len": agent.cfg.get("max_len"),
|
|
"head_max_len": agent.cfg.get("head_max_len"),
|
|
"dataset": DATASET,
|
|
"split": "test",
|
|
"per_lang": args.per_lang,
|
|
"n_opts": args.n_opts,
|
|
"seed": args.seed,
|
|
"instructions": INSTRUCTIONS,
|
|
"temperatures": dict(agent.temperature_by_options_raw) if args.unclamped
|
|
else dict(agent.temperature_by_options),
|
|
"laya_version": getattr(laya, "__version__", "unknown"),
|
|
},
|
|
"report": {},
|
|
"cases": [],
|
|
}
|
|
|
|
for lang in langs:
|
|
t0 = time.time()
|
|
try:
|
|
out = run_language(agent, lang, args.per_lang, args.n_opts,
|
|
args.seed, args.unclamped)
|
|
except Exception as exc:
|
|
print(" %-8s FAILED: %s" % (lang, str(exc)[:110]), file=sys.stderr)
|
|
payload["report"][lang] = {"error": str(exc)[:200]}
|
|
continue
|
|
payload["report"][lang] = out["report"]
|
|
if not args.no_cases:
|
|
payload["cases"].extend(out["cases"])
|
|
r = out["report"]
|
|
print(" %-8s n=%-4d acc=%.4f macro_f1=%.4f ece=%.4f conf=%.4f (%.1fs)"
|
|
% (lang, r.get("n", 0), r.get("accuracy", float("nan")),
|
|
r.get("macro_f1", float("nan")), r.get("ece", float("nan")),
|
|
r.get("mean_confidence", float("nan")), time.time() - t0),
|
|
flush=True)
|
|
|
|
scored = {k: v for k, v in payload["report"].items() if "error" not in v and v.get("n")}
|
|
if scored:
|
|
payload["summary"] = {
|
|
"languages": len(scored),
|
|
"macro_accuracy": round(sum(v["accuracy"] for v in scored.values()) / len(scored), 4),
|
|
"macro_ece": round(sum(v["ece"] for v in scored.values()) / len(scored), 4),
|
|
"macro_f1": round(sum(v["macro_f1"] for v in scored.values()) / len(scored), 4),
|
|
"seconds": round(time.time() - started, 1),
|
|
}
|
|
print("\n macro over %d languages: acc=%.4f ece=%.4f f1=%.4f"
|
|
% (len(scored), payload["summary"]["macro_accuracy"],
|
|
payload["summary"]["macro_ece"], payload["summary"]["macro_f1"]))
|
|
|
|
if args.out:
|
|
with open(args.out, "w", encoding="utf-8") as fh:
|
|
json.dump(payload, fh, ensure_ascii=False, indent=1)
|
|
print(" wrote %s" % args.out)
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__": # pragma: no cover
|
|
raise SystemExit(main())
|