mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 23:45:50 +00:00
The engine half of the request contract. Each control is carried from SamplingParams to the sequence. - logit_bias was parsed and never applied. It is now a logits processor that adds each bias to its token's logit before sampling: -100 bans a token, and +100 all but forces it. - seed seeds an Isaac64Rng owned by the sequence: seed plus the choice index, so n > 1 does not repeat one answer. Every sampling decision of that sequence draws from it: plain decoding, the speculative anchor, both verification paths, MTP draft sampling and the draft model. A seeded sampled request replays token for token on the same drafter. Requests without a seed keep the engine's shared RNG. - serial keeps a sequence off the speculative path. A forward verifies drafts only when every sequence in it staged some. So while a serial sequence is in the batch, the driver releases any staged reservations and the whole batch samples plainly. Greedy output is unchanged, because verification under greedy decoding is exact. The verifier is now one loop for both acceptance rules. It counts drafted and accepted tokens per sequence as each draft is judged, so the response a finishing token sends already carries them. - thinking_budget caps the tokens a <think> template's think block may take. When the budget is spent and the block is still open, the engine writes the close itself, as ordinary generated tokens on the same state: the tokenized "Considering the limited time by the user..." sentence, </think> and a blank line. They count toward completion tokens and max_tokens, and the answer follows. The forced token replaces the draw in sample_sequence, so the KV cache and the reasoning parser both see it. The speculative verifier matches drafts against it, and the driver decodes plainly until the close is written. A sequence under a grammar is not forced. Harmony and Gemma channel templates have no budget. - Usage carries reasoning_tokens, draft_tokens and draft_accepted. reasoning_tokens counts through the token that closed the think block; it is every generated token when the block never closed, and 0 without one. - Stop strings match only the answer, never the think block, as each token's text is added. A single-token stop follows the same rule. The text from a stop string on never enters the completion. A tail that could still begin a stop string is held until the next token decides it, so no part of a stop string is streamed. ReasoningParser gains in_reasoning(), answered by the think-tag and Harmony parsers. - A prompt of the context length or more is refused, where the prompt length is known: "the prompt does not fit: it is N tokens and the context is L, K tokens over...". So is a max_tokens past the room left: "max_tokens W does not fit: prompt is N tokens and the context is L, leaving room for R." Both are ValidationErrors. A request that asks for truncate_sequence is still truncated first. With media, the check runs again on the expanded prompt the model reads. - On the default scheduler, a sequence finished by its prompt step stays finished: a one-token budget, or EOS as the first token. Before, it decoded again and sent a second finish chunk on a stream. - The finishing completion chunk carries usage, as the chat chunk already did. - preserve_thinking reaches the chat template; it stays undefined when unset, so the template's own default applies. The effort a template receives is chosen in one place, template_effort. The top level is xhigh for a template that names it (Qwen3.8 accepts only xhigh|medium|low and raises otherwise) and high elsewhere. An unset effort is left undefined on templates that name xhigh. ChatTemplate::default_reasoning_effort is what a template renders when a request sets none. - The request's `drafter` is checked against what is attached. Pipelines keep the SpeculativeAttachInfo they log at attach (Pipeline::drafter). SpeculativeAttachInfo::name and NAMES spell the drafters: mtp, dflash, and spec for a draft model, DSpark or prompt lookup. Hanzo::config carries the attached drafter and the tokenizer's vocabulary size, which logit_bias ids are checked against. Co-authored-by: Hanzo Dev <[email protected]>