Files
079edc6182 fix(engine): isolate the prefix cache per tenant and cache only written KV (#119)
Prefix blocks were hashed from token content alone, so any two requests with the
same prefix shared KV across orgs, and the sequence-level cache and its disk spill
were keyed by tokens too. Every block hash, sequence-cache key and recurrent-prefix
key now carries the request's cache salt, which the server derives from the
gateway's X-Org-Id plus an optional cache_salt body field. Salted prefixes are
dropped instead of spilled to disk.

The scheduler also cached blocks no forward had written: a prompt preempted in the
pass that admitted it, the last sampled token's slot of every finished or preempted
sequence, and sequences that ended in error. Now only written positions become
reusable.

kv_isolation drives the real scheduler against a simulated KV store and checks every
read for unwritten, foreign-context or other-tenant KV; hanzo.yml gates on it.

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-11-19 16:39:53 -05:00
..

hanzo-bench

Deprecated: The standalone hanzo-bench package is deprecated. Use hanzo bench instead for the same functionality.

Migration:

# Old
cargo run --release --features cuda --package hanzo-bench -- plain -m model-id

# New
hanzo bench -m model-id

This is our official benchmarking application, which allows you to collect structured information about the speed of hanzo.

Note

You should replace --features ... with one of the features specified here, or remove it for pure CPU inference.

To run: cargo run --release --features ... --package hanzo-bench

Fast and easy LLM serving.

Usage: hanzo-bench [OPTIONS] <COMMAND>

Commands:
  plain        Select a plain model
  x-lora       Select an X-LoRA architecture
  lora         Select a LoRA architecture
  gguf         Select a GGUF model
  x-lora-gguf  Select a GGUF model with X-LoRA
  lora-gguf    Select a GGUF model with LoRA
  ggml         Select a GGML model
  x-lora-ggml  Select a GGML model with X-LoRA
  lora-ggml    Select a GGML model with LoRA
  help         Print this message or the help of the given subcommand(s)

Options:
  -p, --n-prompt <N_PROMPT>
          Number of prompt tokens to run [default: 512]
  -g, --n-gen <N_GEN>
          Number of generations tokens to run [default: 128]
  -c, --concurrency <CONCURRENCY>
          Number of concurrent requests to run. Default is 1
  -r, --repetitions <REPETITIONS>
          Number of times to repeat each test [default: 5]
  -n, --num-device-layers <NUM_DEVICE_LAYERS>
          Number of device layers to load and run on the device. All others will be on the CPU
  -h, --help
          Print help
  -V, --version
          Print version