Files
zeekayandHanzo Dev f1fa731e6d feat(zen6): QSA sparse attention past the indexer width, and YaRN on M-RoPE, on both doors
The loaders capped zen6's context at the indexer's 2,048-token budget, because only
dense attention ran. Each attention layer now runs QSA, and the context is the
checkpoint's 262,144, times the YaRN factor when one is declared. The GGUF and
safetensors doors build the same Layer and run the same forward (Layer::forward ->
qsa::forward); only the loaders differ.

The index. Beside its K/V, each attention layer keeps an entry of two rows per token,
one head of 128 wide:
- K holds the compressed key of group j at the group's last token;
- V holds the raw key.

Rows per token share the K/V's slots, so paging, prefix reuse, draft rollback and
sequence swaps keep the two consistent. Where the entries live:
- Paged: the entries follow the attention layers at num_layers + l, read by layer l.
  One rule, paged_attention::IndexEntries, serves ContentConfig (GGUF) and
  KvLayers::indexed (safetensors).
- Eager: HybridCacheConfig::side_kv caches after the n-gram pool, batched, split,
  trimmed and reset with the attention caches.

The forward. Every forward projects the indexer and writes the index. It closes the
groups that end in the chunk, from the chunk's raw keys and the open group's head read
back from the index. While every query sees at most `blocks` groups, the layer runs its
dense attention untouched. Past that, per sequence:
- the compressed keys are scored on the device in f32, in chunks bounded by 128 MiB;
- they are ranked on the host by the deterministic top-k (score descending, ties to
  the lower index);
- consecutive queries attend the union of their selections under a mask, while that
  wastes at most half the scores, with K/V rows gathered once per group from either
  paged layout and grouped-query heads folded into rows.

The two doors' indexers are the same tensors:
- The checkpoint stores one `index_qk_proj` [(heads + 1) x dim, hidden] in bf16, the
  query heads' rows then the key head's, and its Gemma norms as w.
- The GGUF splits the projection into q_proj and k_proj, and stores the norms as 1 + w
  in f32.
- qsa::Config::new validates the shape for both doors. The safetensors manifest reads
  the indexer's three tensors per attention layer. The KV per token grows by the index:
  30,720 bytes at bf16 (was 24,576).

YaRN (Peng et al., arXiv:2309.00071) on the M-RoPE embedding:
- Qwen3VLRotaryEmbedding::new_yarn blends each frequency with itself over the factor,
  along the linear ramp between beta_fast and beta_slow.
- cos and sin carry the attention factor 0.1 ln F + 1, multiplied in f32 before the
  cast.
- Rotations are counted over the M-RoPE table's length, `factor` times the original
  context. So for this model (theta 1e7, 64 rotary dims, 262,144 native) the ramp
  spans frequencies 16..24, where a count over the original length gives 14..22.
- Yarn::mrope builds it for both doors, from the GGUF `rope.scaling.*` keys or from
  `rope_parameters` with rope_type "yarn". Any other rope_type is refused.
- `--rope-yarn-factor F` (the GGUF model selection of serve/run and hanzo-server)
  writes those keys into the loaded metadata before anything reads it
  (Content::extend_context). So the released GGUF, converted without them, extends
  without rewriting a 94 GB file.

Speculation. A rejected draft rewinds the index with the K/V: the zen6 MTP test
rewinds with HybridCache::truncate_attention_to, as the engine does.

gguf_metadata checks the zen6 arch.

Tests:
- select against brute force, with exact ties;
- rows read back through both paged layouts and the eager cache;
- attend against an f64 reference restricted to each query's tokens;
- the layer, from a GGUF, against the f64 selection, in one pass and in chunks that
  split groups and cross the width;
- the layer within the width, bit-equal to its dense attention;
- the tiny models' 40-token forwards agreeing in one pass, chunks and steps;
- the paged cache's index entries;
- YaRN from the flag's keys;
- the ramp and eight frequencies against an f32 transcription.

Co-authored-by: Hanzo Dev <[email protected]>
2026-10-17 19:32:24 -07:00
..

hanzo — Blazing-Fast LLM Inference in Rust

The Rust SDK for hanzo, a high-performance LLM inference engine supporting text, multimodal, speech, image generation, and embedding models.

API Docs | GitHub | Examples | Discord

Quick Start

use hanzo::{IsqBits, ModelBuilder, TextMessages, TextMessageRole};

#[tokio::main]
async fn main() -> hanzo::error::Result<()> {
    let model = ModelBuilder::new("Qwen/Qwen3-4B")
        .with_auto_isq(IsqBits::Four)
        .build()
        .await?;

    let response = model.chat("What is Rust's ownership model?").await?;
    println!("{response}");
    Ok(())
}

Capabilities

Capability Builder Example
Any model (auto-detect) ModelBuilder examples/getting_started/text_generation/
Text generation TextModelBuilder examples/getting_started/text_generation/
Multimodal (image+text) MultimodalModelBuilder examples/getting_started/multimodal/
GGUF quantized models GgufModelBuilder examples/getting_started/gguf/
Image generation DiffusionModelBuilder examples/models/diffusion/
Speech synthesis SpeechModelBuilder examples/models/speech/
Embeddings EmbeddingModelBuilder examples/getting_started/embedding/
Structured output Model::generate_structured examples/advanced/json_schema/
Tool calling Tool, ToolChoice examples/advanced/tools/
Agents AgentBuilder examples/advanced/agent/
LoRA / X-LoRA LoraModelBuilder, XLoraModelBuilder examples/advanced/lora/
AnyMoE AnyMoeModelBuilder examples/advanced/anymoe/
MCP client McpClientConfig examples/advanced/mcp_client/

Choosing a Request Type

Type Use When Sampling
TextMessages Simple text-only chat Deterministic
MultimodalMessages Prompt includes images or audio Deterministic
RequestBuilder Tools, logprobs, custom sampling, constraints, adapters, or web search Configurable

TextMessages and MultimodalMessages convert into RequestBuilder via Into<RequestBuilder>.

Feature Flags

Flag Effect
cuda CUDA GPU support
flash-attn Flash Attention 2 kernels (requires cuda)
cudnn cuDNN acceleration (requires cuda)
nccl Multi-GPU via NCCL (requires cuda and NCCL)
metal Apple Metal GPU support
accelerate Apple Accelerate framework
mkl Intel MKL acceleration

The default feature set (no flags) builds with pure Rust — no C compiler or system libraries required.

License

MIT