mirror of
https://github.com/zoo-apps/gym.git
synced 2026-10-30 14:57:09 +00:00
Wing LianandGitHub
a1bdf1b0c9
* use warmup_ratio as a better default than warmup steps since it's data dependent * replace remainder of warmup_steps
Jamba
- ✅ qlora w/ deepspeed Zero-2 needs at least 2x GPUs and
- 35GiB VRAM per GPU w minimal context length
- 56GiB VRAM per GPU (w multipack enabled)
- ✅ qlora w/ deepspeed Zero-3 needs at least 2x GPUs and 67GiB VRAM (wtf?)
- ✅ qlora single-gpu, ~51GiB VRAM
- ✅ multipack
- ✅ FSDP
- ❓ 8-bit LoRA