Files
079edc6182 fix(engine): isolate the prefix cache per tenant and cache only written KV (#119)
Prefix blocks were hashed from token content alone, so any two requests with the
same prefix shared KV across orgs, and the sequence-level cache and its disk spill
were keyed by tokens too. Every block hash, sequence-cache key and recurrent-prefix
key now carries the request's cache salt, which the server derives from the
gateway's X-Org-Id plus an optional cache_salt body field. Salted prefixes are
dropped instead of spilled to disk.

The scheduler also cached blocks no forward had written: a prompt preempted in the
pass that admitted it, the last sampled token's slot of every finished or preempted
sequence, and sequences that ended in error. Now only written positions become
reusable.

kv_isolation drives the real scheduler against a simulated KV store and checks every
read for unwritten, foreign-context or other-tenant KV; hanzo.yml gates on it.

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-11-19 16:39:53 -05:00
..

hanzo — Blazing-Fast LLM Inference in Rust

The Rust SDK for hanzo, a high-performance LLM inference engine supporting text, multimodal, speech, image generation, and embedding models.

API Docs | GitHub | Examples | Discord

Quick Start

use hanzo::{IsqBits, ModelBuilder, TextMessages, TextMessageRole};

#[tokio::main]
async fn main() -> hanzo::error::Result<()> {
    let model = ModelBuilder::new("Qwen/Qwen3-4B")
        .with_auto_isq(IsqBits::Four)
        .build()
        .await?;

    let response = model.chat("What is Rust's ownership model?").await?;
    println!("{response}");
    Ok(())
}

Capabilities

Capability Builder Example
Any model (auto-detect) ModelBuilder examples/getting_started/text_generation/
Text generation TextModelBuilder examples/getting_started/text_generation/
Multimodal (image+text) MultimodalModelBuilder examples/getting_started/multimodal/
GGUF quantized models GgufModelBuilder examples/getting_started/gguf/
Image generation DiffusionModelBuilder examples/models/diffusion/
Speech synthesis SpeechModelBuilder examples/models/speech/
Embeddings EmbeddingModelBuilder examples/getting_started/embedding/
Structured output Model::generate_structured examples/advanced/json_schema/
Tool calling Tool, ToolChoice examples/advanced/tools/
Agents AgentBuilder examples/advanced/agent/
LoRA / X-LoRA LoraModelBuilder, XLoraModelBuilder examples/advanced/lora/
AnyMoE AnyMoeModelBuilder examples/advanced/anymoe/
MCP client McpClientConfig examples/advanced/mcp_client/

Choosing a Request Type

Type Use When Sampling
TextMessages Simple text-only chat Deterministic
MultimodalMessages Prompt includes images or audio Deterministic
RequestBuilder Tools, logprobs, custom sampling, constraints, adapters, or web search Configurable

TextMessages and MultimodalMessages convert into RequestBuilder via Into<RequestBuilder>.

Feature Flags

Flag Effect
cuda CUDA GPU support
flash-attn Flash Attention 2 kernels (requires cuda)
cudnn cuDNN acceleration (requires cuda)
nccl Multi-GPU via NCCL (requires cuda and NCCL)
metal Apple Metal GPU support
accelerate Apple Accelerate framework
mkl Intel MKL acceleration

The default feature set (no flags) builds with pure Rust — no C compiler or system libraries required.

License

MIT