mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
The loaders capped zen6's context at the indexer's 2,048-token budget, because only dense attention ran. Each attention layer now runs QSA, and the context is the checkpoint's 262,144, times the YaRN factor when one is declared. The GGUF and safetensors doors build the same Layer and run the same forward (Layer::forward -> qsa::forward); only the loaders differ. The index. Beside its K/V, each attention layer keeps an entry of two rows per token, one head of 128 wide: - K holds the compressed key of group j at the group's last token; - V holds the raw key. Rows per token share the K/V's slots, so paging, prefix reuse, draft rollback and sequence swaps keep the two consistent. Where the entries live: - Paged: the entries follow the attention layers at num_layers + l, read by layer l. One rule, paged_attention::IndexEntries, serves ContentConfig (GGUF) and KvLayers::indexed (safetensors). - Eager: HybridCacheConfig::side_kv caches after the n-gram pool, batched, split, trimmed and reset with the attention caches. The forward. Every forward projects the indexer and writes the index. It closes the groups that end in the chunk, from the chunk's raw keys and the open group's head read back from the index. While every query sees at most `blocks` groups, the layer runs its dense attention untouched. Past that, per sequence: - the compressed keys are scored on the device in f32, in chunks bounded by 128 MiB; - they are ranked on the host by the deterministic top-k (score descending, ties to the lower index); - consecutive queries attend the union of their selections under a mask, while that wastes at most half the scores, with K/V rows gathered once per group from either paged layout and grouped-query heads folded into rows. The two doors' indexers are the same tensors: - The checkpoint stores one `index_qk_proj` [(heads + 1) x dim, hidden] in bf16, the query heads' rows then the key head's, and its Gemma norms as w. - The GGUF splits the projection into q_proj and k_proj, and stores the norms as 1 + w in f32. - qsa::Config::new validates the shape for both doors. The safetensors manifest reads the indexer's three tensors per attention layer. The KV per token grows by the index: 30,720 bytes at bf16 (was 24,576). YaRN (Peng et al., arXiv:2309.00071) on the M-RoPE embedding: - Qwen3VLRotaryEmbedding::new_yarn blends each frequency with itself over the factor, along the linear ramp between beta_fast and beta_slow. - cos and sin carry the attention factor 0.1 ln F + 1, multiplied in f32 before the cast. - Rotations are counted over the M-RoPE table's length, `factor` times the original context. So for this model (theta 1e7, 64 rotary dims, 262,144 native) the ramp spans frequencies 16..24, where a count over the original length gives 14..22. - Yarn::mrope builds it for both doors, from the GGUF `rope.scaling.*` keys or from `rope_parameters` with rope_type "yarn". Any other rope_type is refused. - `--rope-yarn-factor F` (the GGUF model selection of serve/run and hanzo-server) writes those keys into the loaded metadata before anything reads it (Content::extend_context). So the released GGUF, converted without them, extends without rewriting a 94 GB file. Speculation. A rejected draft rewinds the index with the K/V: the zen6 MTP test rewinds with HybridCache::truncate_attention_to, as the engine does. gguf_metadata checks the zen6 arch. Tests: - select against brute force, with exact ties; - rows read back through both paged layouts and the eager cache; - attend against an f64 reference restricted to each query's tokens; - the layer, from a GGUF, against the f64 selection, in one pass and in chunks that split groups and cross the width; - the layer within the width, bit-equal to its dense attention; - the tiny models' 40-token forwards agreeing in one pass, chunks and steps; - the paged cache's index entries; - YaRN from the flag's keys; - the ramp and eight frequencies against an f32 transcription. Co-authored-by: Hanzo Dev <[email protected]>
223 lines
8.5 KiB
Rust
223 lines
8.5 KiB
Rust
//! LOSSLESSNESS regression test for classic draft+target speculative decoding.
|
|
//!
|
|
//! The verifier corrects every draft token against the target's own distribution, so greedy (argmax)
|
|
//! decoding MUST produce the byte-for-byte identical token sequence whether or not a draft model is
|
|
//! attached. This codifies that invariant: it greedily completes the same prompts with and without a
|
|
//! self-draft (the draft IS the target GGUF, gamma>=2 so multiple drafts are verified per target
|
|
//! step) and asserts the generated token sequences are identical. This is the gate that proves
|
|
//! losslessness forever; an inequality here means the verify/accept path regressed.
|
|
//!
|
|
//! The classic draft path requires PagedAttention, which is GPU-only, so this test is a no-op unless
|
|
//! built with a GPU backend AND given a local GGUF (so CPU CI is a clean skip):
|
|
//! TEST_GGUF=/abs/path/to/Qwen3-0.6B-Q8_0.gguf \
|
|
//! cargo test -p hanzo-cli --features rocm --test draft_spec_decode_lossless \
|
|
//! -- --nocapture --test-threads=1
|
|
|
|
use std::path::{Path, PathBuf};
|
|
use std::sync::Arc;
|
|
|
|
use hanzo_engine::{
|
|
Constraint, Hanzo, ModelDType, ModelSelected, NormalRequest, Request, RequestMessage, Response,
|
|
SamplingParams, TokenSource,
|
|
};
|
|
use hanzo_server_core::server::ServerBuilder;
|
|
use tokio::sync::mpsc::channel;
|
|
|
|
const MAX_SEQ_LEN: usize = 4096;
|
|
const GAMMA: usize = 4; // >= 2 so several draft tokens are verified per target step.
|
|
const GEN_TOKENS: usize = 48;
|
|
|
|
/// PagedAttention (required by the classic draft path) is only available on a GPU backend.
|
|
const GPU: bool = cfg!(any(
|
|
feature = "cuda",
|
|
feature = "metal",
|
|
feature = "rocm",
|
|
feature = "vulkan"
|
|
));
|
|
|
|
fn model_path() -> Option<PathBuf> {
|
|
let p = std::env::var("TEST_GGUF").ok()?;
|
|
let pb = PathBuf::from(&p);
|
|
if !pb.is_file() {
|
|
panic!("TEST_GGUF={p} is not a file");
|
|
}
|
|
Some(pb)
|
|
}
|
|
|
|
fn split_dir_file(path: &Path) -> (String, String) {
|
|
let dir = path
|
|
.parent()
|
|
.map(|d| {
|
|
if d.as_os_str().is_empty() {
|
|
".".to_string()
|
|
} else {
|
|
d.to_string_lossy().to_string()
|
|
}
|
|
})
|
|
.unwrap_or_else(|| ".".to_string());
|
|
let file = path
|
|
.file_name()
|
|
.expect("gguf path has a filename")
|
|
.to_string_lossy()
|
|
.to_string();
|
|
(dir, file)
|
|
}
|
|
|
|
fn gguf(path: &Path) -> ModelSelected {
|
|
let (dir, file) = split_dir_file(path);
|
|
ModelSelected::GGUF {
|
|
tok_model_id: None, // use the GGUF's embedded tokenizer (fully offline, draft==target vocab)
|
|
quantized_model_id: dir,
|
|
quantized_filename: file,
|
|
dtype: ModelDType::Auto,
|
|
topology: None,
|
|
rope_yarn_factor: None,
|
|
max_seq_len: MAX_SEQ_LEN,
|
|
max_batch_size: 1,
|
|
}
|
|
}
|
|
|
|
/// Build a GPU engine over the GGUF with PagedAttention forced on. With `with_draft`, the same GGUF
|
|
/// is attached as a self-draft (identical tokenizer + weights), so every draft token is exactly what
|
|
/// the target would greedily emit -> high acceptance that exercises the propose/verify/cache wiring
|
|
/// while the verifier still owns correctness.
|
|
async fn load(path: &Path, with_draft: bool) -> Arc<Hanzo> {
|
|
let draft = with_draft.then(|| gguf(path));
|
|
ServerBuilder::new()
|
|
.with_model(gguf(path))
|
|
.with_max_seqs(1)
|
|
.with_no_kv_cache(false)
|
|
.with_token_source(TokenSource::None)
|
|
.with_interactive_mode(false)
|
|
.with_prefix_cache_n(0) // determinism: never serve a cached prefix
|
|
.with_cpu(false)
|
|
.set_paged_attn(Some(true)) // classic draft path requires PagedAttention
|
|
.with_paged_ctxt_len_optional(Some(MAX_SEQ_LEN)) // bound the KV cache so self-draft fits too
|
|
.with_draft_model_optional(draft, GAMMA)
|
|
.build()
|
|
.await
|
|
.expect("load GGUF model (+ optional self-draft) with paged attention")
|
|
}
|
|
|
|
/// Greedy (argmax) raw completion. Returns the per-step generated token strings (the token-level
|
|
/// fingerprint) and the detokenized text. `RequestMessage::Completion` skips the chat template so the
|
|
/// model continues the prompt verbatim and the run is reproducible.
|
|
async fn greedy_tokens(
|
|
hanzo: &Arc<Hanzo>,
|
|
prompt: &str,
|
|
max_tokens: usize,
|
|
) -> (Vec<String>, String) {
|
|
let mut sp = SamplingParams::deterministic();
|
|
sp.max_len = Some(max_tokens);
|
|
sp.top_n_logprobs = 1; // expose the chosen token per step
|
|
|
|
let sender = hanzo.get_sender(None).expect("sender");
|
|
let (tx, mut rx) = channel(64);
|
|
|
|
let req = Request::Normal(Box::new(NormalRequest {
|
|
id: hanzo.next_request_id(),
|
|
messages: RequestMessage::Completion {
|
|
text: prompt.to_string(),
|
|
echo_prompt: false,
|
|
best_of: None,
|
|
},
|
|
sampling_params: sp,
|
|
response: tx,
|
|
return_logprobs: true,
|
|
is_streaming: false,
|
|
constraint: Constraint::None,
|
|
suffix: None,
|
|
tools: None,
|
|
tool_choice: None,
|
|
logits_processors: None,
|
|
return_raw_logits: false,
|
|
web_search_options: None,
|
|
enable_code_execution: false,
|
|
code_execution_permission: None,
|
|
code_execution_approval_notifier: None,
|
|
agent_permission: None,
|
|
agent_approval_handler: None,
|
|
agent_approval_notifier: None,
|
|
session_id: None,
|
|
max_tool_rounds: None,
|
|
tool_dispatch_url: None,
|
|
model_id: None,
|
|
truncate_sequence: false,
|
|
files: None,
|
|
}));
|
|
|
|
sender.send(req).await.expect("send request");
|
|
|
|
loop {
|
|
match rx.recv().await {
|
|
Some(Response::CompletionDone(resp)) => {
|
|
let choice = resp.choices.into_iter().next().expect("one choice");
|
|
let toks = choice.logprobs.map(|l| l.tokens).unwrap_or_default();
|
|
return (toks, choice.text);
|
|
}
|
|
Some(Response::CompletionModelError(e, _)) => panic!("completion model error: {e}"),
|
|
Some(Response::InternalError(e)) => panic!("internal error: {e:?}"),
|
|
Some(Response::ValidationError(e)) => panic!("validation error: {e:?}"),
|
|
Some(Response::ModelError(e, _)) => panic!("model error: {e}"),
|
|
Some(Response::AgenticToolCallProgress { .. }) | Some(Response::File(_)) => continue,
|
|
Some(_) => {
|
|
panic!("unexpected non-terminal response variant for a non-streaming completion")
|
|
}
|
|
None => panic!("response channel closed with no completion"),
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Greedy output must be identical with and without a (self-)draft. This is the losslessness gate.
|
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
|
async fn draft_spec_decode_is_lossless_greedy() {
|
|
if !GPU {
|
|
eprintln!(
|
|
"SKIP draft_spec_decode_is_lossless_greedy: classic draft decoding needs PagedAttention \
|
|
(GPU-only); build with --features cuda|metal|rocm|vulkan"
|
|
);
|
|
return;
|
|
}
|
|
let Some(path) = model_path() else {
|
|
eprintln!("SKIP draft_spec_decode_is_lossless_greedy: set TEST_GGUF to a local GGUF");
|
|
return;
|
|
};
|
|
|
|
let prompts = [
|
|
"The capital of France is",
|
|
"1 2 3 4 5 6 7 8 9",
|
|
"Once upon a time,",
|
|
];
|
|
|
|
// Baseline: target only (paged attention, no draft).
|
|
let baseline = load(&path, false).await;
|
|
let mut without = Vec::new();
|
|
for p in prompts {
|
|
without.push(greedy_tokens(&baseline, p, GEN_TOKENS).await);
|
|
}
|
|
drop(baseline);
|
|
|
|
// With self-draft (gamma=GAMMA) over the same target + paged attention.
|
|
let drafted = load(&path, true).await;
|
|
for (p, (ref_toks, ref_text)) in prompts.into_iter().zip(without.into_iter()) {
|
|
let (toks, text) = greedy_tokens(&drafted, p, GEN_TOKENS).await;
|
|
assert!(!toks.is_empty(), "draft run produced no tokens for {p:?}");
|
|
assert_eq!(
|
|
toks, ref_toks,
|
|
"LOSSLESSNESS VIOLATED for {p:?}: draft greedy tokens differ from no-draft\n no-draft: {ref_toks:?}\n draft: {toks:?}"
|
|
);
|
|
assert_eq!(
|
|
text, ref_text,
|
|
"LOSSLESSNESS VIOLATED for {p:?}: draft greedy text differs from no-draft"
|
|
);
|
|
}
|
|
|
|
// All assertions held (a failure would have panicked above). Skip the racy ROCm-runtime atexit
|
|
// teardown (a cosmetic post-success "pure virtual method called" SIGABRT in the HIP runtime, not
|
|
// our code) by exiting cleanly here, mirroring the CLI one-shot path. A real failure still aborts
|
|
// non-zero via the panic before reaching this point.
|
|
use std::io::Write;
|
|
let _ = std::io::stdout().flush();
|
|
std::process::exit(0);
|
|
}
|