mirror of
https://github.com/zenlm/zen5.git
synced 2026-10-30 14:57:32 +00:00
zeekay
9ea55abee3
Complete training infrastructure for zen5, a 3.1T parameter model with complexity-aware hierarchical routing across 6 diverse model families. Only 394M params trained (0.013% of total). - Expert extraction pipeline for dense and MoE source models - ComplexityEstimator + AlignmentLayer + MoDERouter training - Benchmark suite with 24 labeled routing test cases - HuggingFace model card generation for zen5 lineup - Full architecture specification document
7 lines
82 B
Plaintext
7 lines
82 B
Plaintext
torch>=2.4
|
|
transformers>=4.48
|
|
safetensors
|
|
huggingface_hub
|
|
accelerate
|
|
bitsandbytes
|