Files
laya/tests/test_load_errors.py
946ed4f792 test: cover the load-time and budget errors no suite reached (#237)
Measured by tracing every one of the 15 suites CI runs: `laya/agent.py` executes 224 of
its 464 lines, and four of the messages a user is most likely to see are never produced
by any test.

```
$ python tests/test_load_errors.py        # before this PR, these lines had zero hits
line   hit?   what it guards
126    NO     local path not found
146    NO     subfolder missing in snapshot
154    NO     rl_agent_config.json missing
165    NO     model.safetensors missing
280    NO     _check_question: definition must be a dict
350    NO     options exceed head_max_len
```

`tests/test_load_errors.py` closes the first four and `350`, in 4.7 s, offline. `280` is
left to open PR #194, which adds `tests/test_validation.py` for exactly that branch.

Why these five and not others: each is the *only* diagnostic a user gets for a common
mistake. A wrong `--model` path, a directory that is a bare encoder rather than a
checkpoint, or a prompt with more options than the head budget allows all land here. A
regression in one is a regression in the error message, and nothing would have noticed.

Deliberate choices tested:

- **A tiny local checkpoint, not the Hub.** The suite builds a 64-hidden-size encoder, a
  `WordLevel` tokenizer and a two-key config in-process, the same shape
  `tests/test_download.py` uses, so it needs no network and no download. It runs in 4.7 s.
- **`Whitespace` as the pre-tokenizer, which is load-bearing.** Without it `WordLevel`
  treats a whole label as one word and maps it to a single `[UNK]`, so the option block
  never grows and the budget guard cannot be reached. I hit this while writing the test:
  a synthetic fixture with a human-looking vocabulary produced 18-token sequences no
  matter how many options were added.
- **The budget boundary is measured, not guessed.** Against the shipped
  `max_len=512` / `head_max_len=192`, 100 options give 413 tokens and 100 markers while
  140 give 512 tokens and 126 markers, so the guard sits between them. The test pins the
  behaviour at 140 rather than asserting a threshold nobody documented.
- **Assertions are on message content, not just the exception type.** Each of the four
  load errors asserts that it names the path or file and says what to do about it. That
  is the part a rewording would break, and the part a user reads.
- **Every negative case has a positive control.** A checkpoint with the subfolder present
  still loads, and a question that fits the budget still answers, so a change that made
  `load` or `system_one` refuse more broadly would fail the suite rather than pass it.

Both kinds of assertion fail when the code they cover is removed, verified by
neutralising each guard and running the suite:

```
# with the head_max_len guard replaced by `pass`
$ python tests/test_load_errors.py
14 passed, 3 failed
  FAIL options over budget/raises ValueError None
  FAIL options over budget/names the question None
  FAIL options over budget/reports the budget None

# with the rl_agent_config.json check forced false
$ python tests/test_load_errors.py
16 passed, 1 failed
  FAIL no config/says where it comes from [Errno 2] No such file or directory:
       '...\no-config\rl_agent_config.json'
```

That second failure is the point of the guard: without it the caller gets a bare `errno`
from `open()`, with no indication that the directory is not a Laya checkpoint at all.

Verified:

```
$ python tests/test_load_errors.py          17 passed, 0 failed    (4.7 s)
$ python tests/test_download.py             OK
$ python tests/test_packaging.py            all packaging tests passed
$ python tests/test_criteria.py             72 passed, 0 failed
$ python tests/test_router.py               all routing tests passed
$ python tests/test_email.py                46 passed, 0 failed
$ python tests/test_lang_guess.py           52 passed, 0 failed
$ python tests/test_shortlist.py            98 passed, 0 failed
$ python tests/test_empty_questions.py      OK
$ python tests/test_decision_model.py       all decision model tests passed
$ python tests/test_temperature_loading.py  OK
$ python tests/test_context_manager.py      OK
$ python tests/test_criteria_normalization.py  OK
$ python tests/test_router_memory.py        OK
$ python tests/test_shortlist_cosine.py     OK
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
```

Known limitations:

- `_check_question`'s "definition must be a dict" branch (line 280) is deliberately not
  covered here, because #194 is already adding it. Until one of the two lands, that
  branch still has no test.
- The suite does not cover `_verify_compatibility` (`agent.py:59-94`), which rejects an
  incompatible checkpoint. It is the same family of gap and deserves its own tests, but
  it needs fixtures for missing config keys, missing weight prefixes and shape mismatches,
  which is a larger job than this one.
- `tests/test_load_errors.py` is not wired into any workflow. Neither is any other suite
  added today; that is #184's subject and I have not touched `ci.yml` to avoid colliding
  with it.

Co-authored-by: PerryLink <[email protected]>
2026-09-23 18:52:52 +05:30

193 lines
8.1 KiB
Python

"""The load-time and budget errors a user actually hits.
Every branch here was reachable but never executed by any suite in CI, measured with
`sys.settrace` over all 15 of them: `laya/agent.py` lines 126, 146, 154, 165 and 350 had
zero hits. They are the messages a user sees when a checkpoint is wrong or a question is
too large, so a regression in one is a regression in the only diagnostic they get.
No network: the checkpoint is a tiny local one built here, the same shape
`tests/test_download.py` uses.
Run: python tests/test_load_errors.py
"""
import json
import os
import shutil
import sys
import tempfile
from pathlib import Path
os.environ.setdefault("USE_TF", "0")
os.environ.setdefault("USE_TORCH", "1")
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
import torch # noqa: E402
from safetensors.torch import save_file # noqa: E402
from tokenizers import Tokenizer # noqa: E402
from tokenizers.models import WordLevel # noqa: E402
from tokenizers.pre_tokenizers import Whitespace # noqa: E402
from transformers import BertConfig, BertModel, PreTrainedTokenizerFast # noqa: E402
from laya import load # noqa: E402
from laya.common import DecisionModel # noqa: E402
PASS, FAIL = [], []
def check(name, got, want):
if got == want:
PASS.append(name)
else:
FAIL.append("%s:\n got %r\n want %r" % (name, got, want))
def check_true(name, cond, detail=""):
if cond:
PASS.append(name)
else:
FAIL.append("%s %s" % (name, detail))
TMP = tempfile.TemporaryDirectory()
REPO = Path(TMP.name) / "repo"
def build_checkpoint(root, max_len=64, head_max_len=32, vocab=("hello",)):
"""A loadable Laya checkpoint, small enough to build in-process."""
words = {"[PAD]": 0, "[UNK]": 1, "[CLS]": 2, "[SEP]": 3, "[MASK]": 4}
for i, word in enumerate(vocab):
words[word] = 5 + i
root.mkdir(parents=True, exist_ok=True)
config = BertConfig(vocab_size=len(words), hidden_size=64, num_hidden_layers=1,
num_attention_heads=2, intermediate_size=128)
config.save_pretrained(root / "encoder")
# `Whitespace` is required: without a pre-tokenizer, `WordLevel` sees the whole string
# as one word and every label collapses to a single `[UNK]`, so the option block never
# grows and the budget guard below cannot be reached.
word_level = Tokenizer(WordLevel(words, unk_token="[UNK]"))
word_level.pre_tokenizer = Whitespace()
tokenizer = PreTrainedTokenizerFast(
tokenizer_object=word_level,
pad_token="[PAD]", unk_token="[UNK]", cls_token="[CLS]",
sep_token="[SEP]", mask_token="[MASK]",
)
tokenizer.save_pretrained(root / "tokenizer")
model = DecisionModel(BertModel(config), head_layers=0)
save_file(model.state_dict(), root / "model.safetensors")
(root / "rl_agent_config.json").write_text(json.dumps({
"encoder": "unused/offline", "head_layers": 0, "act_costs": {"act": 0},
"max_len": max_len, "head_max_len": head_max_len,
}), encoding="utf-8")
build_checkpoint(REPO)
def load_error(path, **kw):
"""Load and return the exception, or None if it unexpectedly succeeded."""
try:
load(str(path), device="cpu", **kw)
except Exception as exc: # noqa: BLE001 -- the message is the thing under test
return exc
return None
# ------------------------------------------------------- a good checkpoint still loads
good = load(str(REPO), device="cpu")
check_true("baseline/local checkpoint loads", good is not None)
check("baseline/config round-trips", good.cfg["max_len"], 64)
del good
# ----------------------------------------------------------------- 1. missing directory
# `model_id_or_path` starting with ./ or a drive letter is treated as a local path and
# never handed to the Hub, so the user gets this message instead of a network error.
missing_dir = Path(TMP.name) / "not-a-checkpoint"
err = load_error(missing_dir)
check_true("missing path/raises FileNotFoundError", isinstance(err, FileNotFoundError), repr(err))
# The message carries the path through `!r`, so on Windows it is the backslash-escaped
# form; compare against that rather than the plain string.
check_true("missing path/names the path", repr(str(missing_dir)) in str(err), str(err))
check_true("missing path/says what to check",
"does not exist" in str(err) or "training saved" in str(err), str(err))
# -------------------------------------------------------------- 2. missing subfolder
# Reachable when a repo exists but the requested sibling checkpoint is not in it.
err = load_error(REPO, subfolder="multilingual")
check_true("missing subfolder/raises FileNotFoundError", isinstance(err, FileNotFoundError), repr(err))
check_true("missing subfolder/names the subfolder", "multilingual" in str(err), str(err))
sub = Path(TMP.name) / "sub"
build_checkpoint(sub / "multilingual")
check_true("present subfolder/loads", load_error(sub, subfolder="multilingual") is None)
# --------------------------------------------------- 3. missing rl_agent_config.json
# This is the file that makes a directory a Laya checkpoint rather than a bare encoder,
# so the message has to point at the training run that would have written it.
no_cfg = Path(TMP.name) / "no-config"
shutil.copytree(REPO, no_cfg)
os.remove(no_cfg / "rl_agent_config.json")
err = load_error(no_cfg)
check_true("no config/raises FileNotFoundError", isinstance(err, FileNotFoundError), repr(err))
check_true("no config/names the missing file", "rl_agent_config.json" in str(err), str(err))
check_true("no config/says where it comes from",
"ships with the weights" in str(err) or "training run" in str(err), str(err))
# --------------------------------------------------------- 4. missing model.safetensors
# The config is present and valid here, so this is reached only after that check passes.
no_weights = Path(TMP.name) / "no-weights"
shutil.copytree(REPO, no_weights)
os.remove(no_weights / "model.safetensors")
err = load_error(no_weights)
check_true("no weights/raises FileNotFoundError", isinstance(err, FileNotFoundError), repr(err))
check_true("no weights/names the missing file", "model.safetensors" in str(err), str(err))
# ------------------------------------------- 5. a question whose options do not fit
# `build_sequence` drops markers past `max_len`. Without this guard the model gets a
# selected index outside its own option count, which surfaces as an index error deep in
# the head rather than as a statement about the question.
#
# Measured against the shipped budget (`max_len=512`, `head_max_len=192`) with the real
# tokenizer: 100 options give 413 tokens and 100 markers, 140 give 512 tokens and 126
# markers. So the boundary sits between 100 and 140. Nothing documents that threshold,
# which is why this pins the behaviour rather than the number.
wide = Path(TMP.name) / "wide"
WORDS = ("department", "handling", "billing", "enquiries")
build_checkpoint(wide, max_len=512, head_max_len=192,
vocab=WORDS + tuple(str(i) for i in range(1, 201)))
wide_agent = load(str(wide), device="cpu")
many = {("department %d handling billing enquiries" % i): None for i in range(1, 141)}
try:
wide_agent.system_one("hello",
{"q": {"type": "choice", "instructions": "Which department?",
"criteria": many}})
_outcome = None
except Exception as exc: # noqa: BLE001
_outcome = exc
check_true("options over budget/raises ValueError", isinstance(_outcome, ValueError), repr(_outcome))
check_true("options over budget/names the question", "'q'" in str(_outcome), str(_outcome))
check_true("options over budget/reports the budget",
"head_max_len" in str(_outcome), str(_outcome))
# ...and a question that does fit still answers, so the guard is not refusing everything.
fits = {"q": {"type": "choice", "instructions": "Pick one",
"criteria": {"department": None, "billing": None}}}
try:
wide_agent.system_one("hello", fits)
_ok = True
except Exception: # noqa: BLE001
_ok = False
check_true("options within budget/still answers", _ok)
del wide_agent
TMP.cleanup()
print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
for f in FAIL:
print(" FAIL " + f)
sys.exit(1 if FAIL else 0)