mirror of
https://github.com/zenlm/zen-router.git
synced 2026-10-30 14:57:44 +00:00
zeekay
7f466d8fc9
- catalog.yaml: 28 routable models (add claude-opus-4-5/4-1, sonnet-4-5, minimax-m2 from the agentic corpus; all gateway-priced). Route head 24->28. - session_common.py: shared session-extraction (DRY) -- task heuristic, catalog mapping with dated-id normalization, user/assistant pairing. - extract_agentic_dataset.py: stream zen-agentic-dataset-private, pull embedded Claude transcripts out of assistant turns, stratified cap. 9.9B-token corpus yields ~4.4k real routing rows (rest is synthetic identity SFT + git history). - extract_local_sessions.py: refactored onto session_common. - sft.py: --freeze-backbone -> precompute pooled embeddings once, train the 3 linear heads only; publish heads-only (1.2MB) so private data cannot be memorized. Backbone stays public zen-nano-0.6b. - eval_routing.py: strict=False load (heads-only), device-aware, latency + dist. - README Measured: Run A (local, full) vs Run B (combined, frozen, published) side by side. Run B route_acc 0.791 = +25.8pts over majority, 5/28 models emitted -- genuine multi-model discrimination; task_acc 0.723. All caveats. HF: https://huggingface.co/zenlm/zen-router (heads-only weights + README).
23 lines
862 B
YAML
23 lines
862 B
YAML
# Scaled MPS run: frozen backbone, heads-only, on the combined local + agentic
|
|
# corpus dataset. Backbone frozen (--freeze-backbone) => published weights are
|
|
# the three linear heads only (private-safe); embeddings are precomputed once.
|
|
base_model: /Users/a/work/zen/models/zen-nano-0.6b
|
|
output_dir: out/zen-router
|
|
data: data/routing-sft.jsonl
|
|
|
|
tasks: [code, reasoning, math, creative, vision, long_context, cheap_chat, general]
|
|
|
|
heads:
|
|
task: {type: classification, classes: 8}
|
|
route: {type: classification, classes_from: catalog}
|
|
features: {type: projection, dim: 256}
|
|
|
|
catalog: training/catalog.yaml
|
|
|
|
sft:
|
|
lr: 1.0e-3 # heads-only: a higher LR than the full-finetune run
|
|
epochs: 12 # cheap once embeddings are cached
|
|
batch_size: 32 # backbone forward batch during the one-time precompute
|
|
max_seq_len: 512
|
|
pooled: last_token
|