Files
zeekayandHanzo Dev f1fa731e6d feat(zen6): QSA sparse attention past the indexer width, and YaRN on M-RoPE, on both doors
The loaders capped zen6's context at the indexer's 2,048-token budget, because only
dense attention ran. Each attention layer now runs QSA, and the context is the
checkpoint's 262,144, times the YaRN factor when one is declared. The GGUF and
safetensors doors build the same Layer and run the same forward (Layer::forward ->
qsa::forward); only the loaders differ.

The index. Beside its K/V, each attention layer keeps an entry of two rows per token,
one head of 128 wide:
- K holds the compressed key of group j at the group's last token;
- V holds the raw key.

Rows per token share the K/V's slots, so paging, prefix reuse, draft rollback and
sequence swaps keep the two consistent. Where the entries live:
- Paged: the entries follow the attention layers at num_layers + l, read by layer l.
  One rule, paged_attention::IndexEntries, serves ContentConfig (GGUF) and
  KvLayers::indexed (safetensors).
- Eager: HybridCacheConfig::side_kv caches after the n-gram pool, batched, split,
  trimmed and reset with the attention caches.

The forward. Every forward projects the indexer and writes the index. It closes the
groups that end in the chunk, from the chunk's raw keys and the open group's head read
back from the index. While every query sees at most `blocks` groups, the layer runs its
dense attention untouched. Past that, per sequence:
- the compressed keys are scored on the device in f32, in chunks bounded by 128 MiB;
- they are ranked on the host by the deterministic top-k (score descending, ties to
  the lower index);
- consecutive queries attend the union of their selections under a mask, while that
  wastes at most half the scores, with K/V rows gathered once per group from either
  paged layout and grouped-query heads folded into rows.

The two doors' indexers are the same tensors:
- The checkpoint stores one `index_qk_proj` [(heads + 1) x dim, hidden] in bf16, the
  query heads' rows then the key head's, and its Gemma norms as w.
- The GGUF splits the projection into q_proj and k_proj, and stores the norms as 1 + w
  in f32.
- qsa::Config::new validates the shape for both doors. The safetensors manifest reads
  the indexer's three tensors per attention layer. The KV per token grows by the index:
  30,720 bytes at bf16 (was 24,576).

YaRN (Peng et al., arXiv:2309.00071) on the M-RoPE embedding:
- Qwen3VLRotaryEmbedding::new_yarn blends each frequency with itself over the factor,
  along the linear ramp between beta_fast and beta_slow.
- cos and sin carry the attention factor 0.1 ln F + 1, multiplied in f32 before the
  cast.
- Rotations are counted over the M-RoPE table's length, `factor` times the original
  context. So for this model (theta 1e7, 64 rotary dims, 262,144 native) the ramp
  spans frequencies 16..24, where a count over the original length gives 14..22.
- Yarn::mrope builds it for both doors, from the GGUF `rope.scaling.*` keys or from
  `rope_parameters` with rope_type "yarn". Any other rope_type is refused.
- `--rope-yarn-factor F` (the GGUF model selection of serve/run and hanzo-server)
  writes those keys into the loaded metadata before anything reads it
  (Content::extend_context). So the released GGUF, converted without them, extends
  without rewriting a 94 GB file.

Speculation. A rejected draft rewinds the index with the K/V: the zen6 MTP test
rewinds with HybridCache::truncate_attention_to, as the engine does.

gguf_metadata checks the zen6 arch.

Tests:
- select against brute force, with exact ties;
- rows read back through both paged layouts and the eager cache;
- attend against an f64 reference restricted to each query's tokens;
- the layer, from a GGUF, against the f64 selection, in one pass and in chunks that
  split groups and cross the width;
- the layer within the width, bit-equal to its dense attention;
- the tiny models' 40-token forwards agreeing in one pass, chunks and steps;
- the paged cache's index entries;
- YaRN from the flag's keys;
- the ramp and eight frequencies against an f32 transcription.

Co-authored-by: Hanzo Dev <[email protected]>
2026-10-17 19:32:24 -07:00
..