mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
`--mtp-model mtp-Qwen3.8-Flash-Next-{shared-,}Q8_0.gguf` drafts for zen6 (the file
says arch `qwen4exp`). The head is one attention and MoE block at `blk.48`. It sits
behind the trunk's 4-stream hyper-connections:
- the entry is `eh_proj [enorm(embed(x_{t+1})) | hnorm_s(h_t,s)]`, per stream, over
the trunk's streams before the output mixer;
- then the block;
- then the head's own mixer and the output projection.
The chain carries the block's streams. A shared file borrows the target's embedding
and output projection; a self-contained one uses its own.
The block's attention keeps a KV over the whole sequence, as it was trained:
- each target forward writes the keys and values of the pairs it completes, per
sequence (the pipeline names them);
- a draft step truncates at its own position, which drops pairs written for
rejected drafts.
The trunk's logits are the same bits with a head attached.
What the head needed elsewhere:
- `MtpStep::step` names each row's sequence, and `retain` releases a head's state
for sequences that finished. So a head can keep a KV per sequence, not per
chain. The Qwen3.5 heads keep only their chain and ignore both.
- The row gather the GGUF pipeline did for Qwen3.5 is now
`qwen3_5_mtp::anchor_rows`, which a second model uses as it stands.
- The per-stream RMS norm (`nextn.hnorm`) is its own type, `hyper::Norm`, and
`Mixer` holds one. `Branch::input` gives a block's input without the
write-back. The f64 reference is visible to other tests.
- `speculative::Extended` continues a draft by prompt lookup:
- it runs another proposer, looks up each sequence's tokens grown by its draft,
and continues the draft with what followed the match;
- only an all-greedy batch continues, since a looked-up token has no draft
distribution for the verifier;
- the batch continues by its shortest find, so every sequence verifies at one
width.
`--prompt-lookup-ngram N --gamma G` beside `--mtp-model` now chains: a needle of
exactly N tokens, then up to G more. Before, prompt lookup replaced the head. The
Gemma 4 assistant refuses a lookup rather than ignore it.
Tests (CPU, synthetic GGUFs):
- a first step against an f64 reference of the graph;
- a KV built through chunked prefill, a partly rejected verify and a chained step,
against one pass;
- shared and self-contained tables;
- refusal of a file with no head, or with another vocabulary;
- greedy speculative decoding against serial, with and without prompt lookup;
- the head fixture is seed 26, which over seeds 0..63 keeps the most drafts
accepted.
Co-authored-by: Hanzo Dev <[email protected]>
hanzo-server-core
Core crate that powers hanzo server.
Documentation: https://docs.rs/hanzo-server-core/