Files
zeekayandHanzo Dev c78779c629 feat(engine): per-request sampling, thinking, stop and context controls
The engine half of the request contract. Each control is carried from
SamplingParams to the sequence.

- logit_bias was parsed and never applied. It is now a logits processor that
  adds each bias to its token's logit before sampling: -100 bans a token, and
  +100 all but forces it.
- seed seeds an Isaac64Rng owned by the sequence: seed plus the choice index,
  so n > 1 does not repeat one answer. Every sampling decision of that sequence
  draws from it: plain decoding, the speculative anchor, both verification
  paths, MTP draft sampling and the draft model. A seeded sampled request
  replays token for token on the same drafter. Requests without a seed keep
  the engine's shared RNG.
- serial keeps a sequence off the speculative path. A forward verifies drafts
  only when every sequence in it staged some. So while a serial sequence is in
  the batch, the driver releases any staged reservations and the whole batch
  samples plainly. Greedy output is unchanged, because verification under
  greedy decoding is exact. The verifier is now one loop for both acceptance
  rules. It counts drafted and accepted tokens per sequence as each draft is
  judged, so the response a finishing token sends already carries them.
- thinking_budget caps the tokens a <think> template's think block may take.
  When the budget is spent and the block is still open, the engine writes the
  close itself, as ordinary generated tokens on the same state: the tokenized
  "Considering the limited time by the user..." sentence, </think> and a blank
  line. They count toward completion tokens and max_tokens, and the answer
  follows. The forced token replaces the draw in sample_sequence, so the KV
  cache and the reasoning parser both see it. The speculative verifier matches
  drafts against it, and the driver decodes plainly until the close is
  written. A sequence under a grammar is not forced. Harmony and Gemma channel
  templates have no budget.
- Usage carries reasoning_tokens, draft_tokens and draft_accepted.
  reasoning_tokens counts through the token that closed the think block; it is
  every generated token when the block never closed, and 0 without one.
- Stop strings match only the answer, never the think block, as each token's
  text is added. A single-token stop follows the same rule. The text from a
  stop string on never enters the completion. A tail that could still begin a
  stop string is held until the next token decides it, so no part of a stop
  string is streamed. ReasoningParser gains in_reasoning(), answered by the
  think-tag and Harmony parsers.
- A prompt of the context length or more is refused, where the prompt length
  is known: "the prompt does not fit: it is N tokens and the context is L, K
  tokens over...". So is a max_tokens past the room left: "max_tokens W does
  not fit: prompt is N tokens and the context is L, leaving room for R." Both
  are ValidationErrors. A request that asks for truncate_sequence is still
  truncated first. With media, the check runs again on the expanded prompt the
  model reads.
- On the default scheduler, a sequence finished by its prompt step stays
  finished: a one-token budget, or EOS as the first token. Before, it decoded
  again and sent a second finish chunk on a stream.
- The finishing completion chunk carries usage, as the chat chunk already did.
- preserve_thinking reaches the chat template; it stays undefined when unset,
  so the template's own default applies. The effort a template receives is
  chosen in one place, template_effort. The top level is xhigh for a template
  that names it (Qwen3.8 accepts only xhigh|medium|low and raises otherwise)
  and high elsewhere. An unset effort is left undefined on templates that name
  xhigh. ChatTemplate::default_reasoning_effort is what a template renders when
  a request sets none.
- The request's `drafter` is checked against what is attached. Pipelines keep
  the SpeculativeAttachInfo they log at attach (Pipeline::drafter).
  SpeculativeAttachInfo::name and NAMES spell the drafters: mtp, dflash, and
  spec for a draft model, DSpark or prompt lookup. Hanzo::config carries the
  attached drafter and the tokenizer's vocabulary size, which logit_bias ids
  are checked against.

Co-authored-by: Hanzo Dev <[email protected]>
2026-10-16 22:17:35 -07:00
..

hanzo-bench

Deprecated: The standalone hanzo-bench package is deprecated. Use hanzo bench instead for the same functionality.

Migration:

# Old
cargo run --release --features cuda --package hanzo-bench -- plain -m model-id

# New
hanzo bench -m model-id

This is our official benchmarking application, which allows you to collect structured information about the speed of hanzo.

Note

You should replace --features ... with one of the features specified here, or remove it for pure CPU inference.

To run: cargo run --release --features ... --package hanzo-bench

Fast and easy LLM serving.

Usage: hanzo-bench [OPTIONS] <COMMAND>

Commands:
  plain        Select a plain model
  x-lora       Select an X-LoRA architecture
  lora         Select a LoRA architecture
  gguf         Select a GGUF model
  x-lora-gguf  Select a GGUF model with X-LoRA
  lora-gguf    Select a GGUF model with LoRA
  ggml         Select a GGML model
  x-lora-ggml  Select a GGML model with X-LoRA
  lora-ggml    Select a GGML model with LoRA
  help         Print this message or the help of the given subcommand(s)

Options:
  -p, --n-prompt <N_PROMPT>
          Number of prompt tokens to run [default: 512]
  -g, --n-gen <N_GEN>
          Number of generations tokens to run [default: 128]
  -c, --concurrency <CONCURRENCY>
          Number of concurrent requests to run. Default is 1
  -r, --repetitions <REPETITIONS>
          Number of times to repeat each test [default: 5]
  -n, --num-device-layers <NUM_DEVICE_LAYERS>
          Number of device layers to load and run on the device. All others will be on the CPU
  -h, --help
          Print help
  -V, --version
          Print version