mirror of
https://github.com/hanzoai/ml.git
synced 2026-11-19 23:50:51 +00:00
Stream a big MoE's routed experts from NVMe instead of loading them resident: per-(layer x projection) bank keeps the stacked [E,n,k] weight on disk and preads one expert's [n,k] slice on demand through a pinned hot-set -> per-bank LRU -> disk (+posix_fadvise DONTNEED so the page cache stays bounded). Auto-sizes the cache from MemAvailable (colibri cap_for_ram, auto-raises to fill big-RAM boxes) and learns/auto-pins the hottest experts via a usage sidecar (colibri AUTOPIN). - expert_stream.rs: ExpertStreamBank (fetch/pin/LRU/stats), registry + finalize() (RAM-sized caps), save_usage/learning-pin. Runtime-gated (default OFF). - QStorage::Stream variant; indexed_moe_forward gains a Stream arm sharing ONE moe_grouped_per_expert helper with the resident path (only the fetch differs). - gguf_file: TensorInfo::read_stream / Content::tensor_stream loader seam. - libc made non-optional for posix_fadvise (dropped dep:libc from mkl/accelerate). Bit-exact: streamed expert == resident bytes -> identical indexed_moe output. Gate (tests/expert_stream_tests.rs, 4/4): streamed==resident bit-for-bit; LRU cap bounds resident RAM with correctness cap-independent; pinned expert survives eviction; usage sidecar learns the hot expert.