mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
The server turned PagedAttention on for ROCm. hanzo-bench kept its own copy of that decision and turned it off there, to match llama-bench's contiguous KV. A paged cache is what lets the decode step replay as a captured graph, so the bench timed an eager path the server never runs: on a 27B hybrid on gfx1151, 0.92 tok/s where the served path replays a token in 105 ms. paged_attn_default(device) in the engine is now the one answer. The server's five per-device constants are gone, and the bench's flags only override it. Co-authored-by: Hanzo Dev <[email protected]>
hanzo-server-core
Core crate that powers hanzo server.
Documentation: https://docs.rs/hanzo-server-core/