Files
zeekayandHanzo Dev f1a0516cec feat(speculative): zen6 drafts with its MTP head, per sequence, continued by prompt lookup
`--mtp-model mtp-Qwen3.8-Flash-Next-{shared-,}Q8_0.gguf` drafts for zen6 (the file
says arch `qwen4exp`). The head is one attention and MoE block at `blk.48`. It sits
behind the trunk's 4-stream hyper-connections:
- the entry is `eh_proj [enorm(embed(x_{t+1})) | hnorm_s(h_t,s)]`, per stream, over
  the trunk's streams before the output mixer;
- then the block;
- then the head's own mixer and the output projection.

The chain carries the block's streams. A shared file borrows the target's embedding
and output projection; a self-contained one uses its own.

The block's attention keeps a KV over the whole sequence, as it was trained:
- each target forward writes the keys and values of the pairs it completes, per
  sequence (the pipeline names them);
- a draft step truncates at its own position, which drops pairs written for
  rejected drafts.

The trunk's logits are the same bits with a head attached.

What the head needed elsewhere:
- `MtpStep::step` names each row's sequence, and `retain` releases a head's state
  for sequences that finished. So a head can keep a KV per sequence, not per
  chain. The Qwen3.5 heads keep only their chain and ignore both.
- The row gather the GGUF pipeline did for Qwen3.5 is now
  `qwen3_5_mtp::anchor_rows`, which a second model uses as it stands.
- The per-stream RMS norm (`nextn.hnorm`) is its own type, `hyper::Norm`, and
  `Mixer` holds one. `Branch::input` gives a block's input without the
  write-back. The f64 reference is visible to other tests.
- `speculative::Extended` continues a draft by prompt lookup:
  - it runs another proposer, looks up each sequence's tokens grown by its draft,
    and continues the draft with what followed the match;
  - only an all-greedy batch continues, since a looked-up token has no draft
    distribution for the verifier;
  - the batch continues by its shortest find, so every sequence verifies at one
    width.

`--prompt-lookup-ngram N --gamma G` beside `--mtp-model` now chains: a needle of
exactly N tokens, then up to G more. Before, prompt lookup replaced the head. The
Gemma 4 assistant refuses a lookup rather than ignore it.

Tests (CPU, synthetic GGUFs):
- a first step against an f64 reference of the graph;
- a KV built through chunked prefill, a partly rejected verify and a chained step,
  against one pass;
- shared and self-contained tables;
- refusal of a file with no head, or with another vocabulary;
- greedy speculative decoding against serial, with and without prompt lookup;
- the head fixture is seed 26, which over seeds 0..63 keeps the most drafts
  accepted.

Co-authored-by: Hanzo Dev <[email protected]>
2026-10-17 19:01:29 -07:00
..

hanzo-server-core

Core crate that powers hanzo server.

Documentation: https://docs.rs/hanzo-server-core/