mirror of
https://github.com/zenlm/zen-router.git
synced 2026-10-30 14:57:44 +00:00
zeekay
573311de8f
Wire --policy checkpoint to the frozen-backbone heads-only format: - heads-only state_dict load (strict=False), public backbone from base - batch-embed all prompts (batch 64, fp16 on MPS), cache to out/bench/*.emb.npz - apply route head, project 28-class catalog onto RouterBench's 11 models via documented family/tier bridge (benchmarks/checkpoint_map.yaml) - load_matrix returns row-aligned prompts so checkpoint scores line up with Q/C - build_profiles: BENCH_SAMPLE=0 means full suite (fix df.sample(n=0) emptying) Full run (36,497 ex x 11 = 401,467 rows, 36,481 prompts, M4 Max/MPS, 27.8 min): oracle AIQ 0.8701, prior 0.7427, interpolation hull 0.7054, zen-router checkpoint AIQ 0.6248 (NEGATIVE transfer, reported plainly: 0 catalog overlap + 0 training rows for mapped targets).
41 lines
2.8 KiB
YAML
41 lines
2.8 KiB
YAML
# Family/tier bridge: RouterBench's 2023/2024 model universe -> the trained
|
|
# zen-router route head's 28-class catalog (training/catalog.yaml).
|
|
#
|
|
# WHY THIS FILE EXISTS. The checkpoint's route head emits a logit per catalog
|
|
# model. RouterBench's 11 candidates are 2023-era and share ZERO ids with that
|
|
# 2026 catalog (suites.yaml model_map is empty, by design). Scoring RouterBench
|
|
# models directly against the head would make every column identical (all
|
|
# out-of-catalog -> logits.min()), which is a degenerate router. So we map each
|
|
# RouterBench model to the catalog model that occupies the SAME (family, tier)
|
|
# slot, and read that class's route-head logit as the RouterBench model's score.
|
|
#
|
|
# WHAT THIS MEASURES — read honestly. This does NOT measure native in-catalog
|
|
# routing (none of these 11 models are in the catalog, and the router saw 0 rows
|
|
# for them in training). It measures whether the head's learned TASK/TIER
|
|
# DISCRIMINATION — "hard reasoning -> frontier, cheap chat -> small model" —
|
|
# transfers onto RouterBench's cost/quality universe when projected through this
|
|
# family/tier bridge. A win here is evidence of transferable tier discrimination;
|
|
# it is not a claim of routing among RouterBench's actual models.
|
|
#
|
|
# Mapping principle: match provider family first, then capability tier
|
|
# (frontier / mid / cheap / code-specialist / small-open). One catalog class
|
|
# per RouterBench model — a clean 11->11 bijection, no collisions, so distinct
|
|
# RouterBench models keep distinct route-head columns.
|
|
map:
|
|
# -- closed frontier -------------------------------------------------------
|
|
gpt-4-1106-preview: gpt-5.5 # OpenAI flagship -> OpenAI flagship
|
|
claude-v2: claude-opus-4-5 # Anthropic flagship (era) -> Opus
|
|
claude-v1: claude-opus-4-1 # Anthropic flagship (older) -> Opus (prev)
|
|
# -- closed cheap ----------------------------------------------------------
|
|
gpt-3.5-turbo-1106: gpt-5.4-mini # OpenAI cheap/mini -> mini
|
|
claude-instant-v1: claude-haiku-4-5 # Anthropic cheap -> Haiku
|
|
# -- open, large / mid -----------------------------------------------------
|
|
meta/llama-2-70b-chat: kimi-k2 # open large dense -> open large MoE
|
|
mistralai/mixtral-8x7b-chat: deepseek-v3.2 # open MoE, cheap-mid -> open cheap-mid
|
|
zero-one-ai/Yi-34B-Chat: minimax-m2 # open mid instruct -> open mid
|
|
# -- open, code specialist -------------------------------------------------
|
|
meta/code-llama-instruct-34b-chat: zen5-coder # open code model -> code specialist
|
|
# -- open, small -----------------------------------------------------------
|
|
mistralai/mistral-7b-chat: zen5-flash # cheapest open dense -> cheapest cloud tier
|
|
WizardLM/WizardLM-13B-V1.2: zen-agent-4b # small open instruct -> small local
|