mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
The loaders capped zen6's context at the indexer's 2,048-token budget, because only dense attention ran. Each attention layer now runs QSA, and the context is the checkpoint's 262,144, times the YaRN factor when one is declared. The GGUF and safetensors doors build the same Layer and run the same forward (Layer::forward -> qsa::forward); only the loaders differ. The index. Beside its K/V, each attention layer keeps an entry of two rows per token, one head of 128 wide: - K holds the compressed key of group j at the group's last token; - V holds the raw key. Rows per token share the K/V's slots, so paging, prefix reuse, draft rollback and sequence swaps keep the two consistent. Where the entries live: - Paged: the entries follow the attention layers at num_layers + l, read by layer l. One rule, paged_attention::IndexEntries, serves ContentConfig (GGUF) and KvLayers::indexed (safetensors). - Eager: HybridCacheConfig::side_kv caches after the n-gram pool, batched, split, trimmed and reset with the attention caches. The forward. Every forward projects the indexer and writes the index. It closes the groups that end in the chunk, from the chunk's raw keys and the open group's head read back from the index. While every query sees at most `blocks` groups, the layer runs its dense attention untouched. Past that, per sequence: - the compressed keys are scored on the device in f32, in chunks bounded by 128 MiB; - they are ranked on the host by the deterministic top-k (score descending, ties to the lower index); - consecutive queries attend the union of their selections under a mask, while that wastes at most half the scores, with K/V rows gathered once per group from either paged layout and grouped-query heads folded into rows. The two doors' indexers are the same tensors: - The checkpoint stores one `index_qk_proj` [(heads + 1) x dim, hidden] in bf16, the query heads' rows then the key head's, and its Gemma norms as w. - The GGUF splits the projection into q_proj and k_proj, and stores the norms as 1 + w in f32. - qsa::Config::new validates the shape for both doors. The safetensors manifest reads the indexer's three tensors per attention layer. The KV per token grows by the index: 30,720 bytes at bf16 (was 24,576). YaRN (Peng et al., arXiv:2309.00071) on the M-RoPE embedding: - Qwen3VLRotaryEmbedding::new_yarn blends each frequency with itself over the factor, along the linear ramp between beta_fast and beta_slow. - cos and sin carry the attention factor 0.1 ln F + 1, multiplied in f32 before the cast. - Rotations are counted over the M-RoPE table's length, `factor` times the original context. So for this model (theta 1e7, 64 rotary dims, 262,144 native) the ramp spans frequencies 16..24, where a count over the original length gives 14..22. - Yarn::mrope builds it for both doors, from the GGUF `rope.scaling.*` keys or from `rope_parameters` with rope_type "yarn". Any other rope_type is refused. - `--rope-yarn-factor F` (the GGUF model selection of serve/run and hanzo-server) writes those keys into the loaded metadata before anything reads it (Content::extend_context). So the released GGUF, converted without them, extends without rewriting a 94 GB file. Speculation. A rejected draft rewinds the index with the K/V: the zen6 MTP test rewinds with HybridCache::truncate_attention_to, as the engine does. gguf_metadata checks the zen6 arch. Tests: - select against brute force, with exact ties; - rows read back through both paged layouts and the eager cache; - attend against an f64 reference restricted to each query's tokens; - the layer, from a GGUF, against the f64 selection, in one pass and in chunks that split groups and cross the width; - the layer within the width, bit-equal to its dense attention; - the tiny models' 40-token forwards agreeing in one pass, chunks and steps; - the paged cache's index entries; - YaRN from the flag's keys; - the ramp and eight frequencies against an f32 transcription. Co-authored-by: Hanzo Dev <[email protected]>