mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
Prefix blocks were hashed from token content alone, so any two requests with the same prefix shared KV across orgs, and the sequence-level cache and its disk spill were keyed by tokens too. Every block hash, sequence-cache key and recurrent-prefix key now carries the request's cache salt, which the server derives from the gateway's X-Org-Id plus an optional cache_salt body field. Salted prefixes are dropped instead of spilled to disk. The scheduler also cached blocks no forward had written: a prompt preempted in the pass that admitted it, the last sampled token's slot of every finished or preempted sequence, and sequences that ended in error. Now only written positions become reusable. kv_isolation drives the real scheduler against a simulated KV store and checks every read for unwritten, foreign-context or other-tenant KV; hanzo.yml gates on it. Co-authored-by: Claude Opus 5.5 <[email protected]>
hanzo — Blazing-Fast LLM Inference in Rust
The Rust SDK for hanzo, a high-performance LLM inference engine supporting text, multimodal, speech, image generation, and embedding models.
API Docs | GitHub | Examples | Discord
Quick Start
use hanzo::{IsqBits, ModelBuilder, TextMessages, TextMessageRole};
#[tokio::main]
async fn main() -> hanzo::error::Result<()> {
let model = ModelBuilder::new("Qwen/Qwen3-4B")
.with_auto_isq(IsqBits::Four)
.build()
.await?;
let response = model.chat("What is Rust's ownership model?").await?;
println!("{response}");
Ok(())
}
Capabilities
| Capability | Builder | Example |
|---|---|---|
| Any model (auto-detect) | ModelBuilder |
examples/getting_started/text_generation/ |
| Text generation | TextModelBuilder |
examples/getting_started/text_generation/ |
| Multimodal (image+text) | MultimodalModelBuilder |
examples/getting_started/multimodal/ |
| GGUF quantized models | GgufModelBuilder |
examples/getting_started/gguf/ |
| Image generation | DiffusionModelBuilder |
examples/models/diffusion/ |
| Speech synthesis | SpeechModelBuilder |
examples/models/speech/ |
| Embeddings | EmbeddingModelBuilder |
examples/getting_started/embedding/ |
| Structured output | Model::generate_structured |
examples/advanced/json_schema/ |
| Tool calling | Tool, ToolChoice |
examples/advanced/tools/ |
| Agents | AgentBuilder |
examples/advanced/agent/ |
| LoRA / X-LoRA | LoraModelBuilder, XLoraModelBuilder |
examples/advanced/lora/ |
| AnyMoE | AnyMoeModelBuilder |
examples/advanced/anymoe/ |
| MCP client | McpClientConfig |
examples/advanced/mcp_client/ |
Choosing a Request Type
| Type | Use When | Sampling |
|---|---|---|
TextMessages |
Simple text-only chat | Deterministic |
MultimodalMessages |
Prompt includes images or audio | Deterministic |
RequestBuilder |
Tools, logprobs, custom sampling, constraints, adapters, or web search | Configurable |
TextMessages and MultimodalMessages convert into RequestBuilder via Into<RequestBuilder>.
Feature Flags
| Flag | Effect |
|---|---|
cuda |
CUDA GPU support |
flash-attn |
Flash Attention 2 kernels (requires cuda) |
cudnn |
cuDNN acceleration (requires cuda) |
nccl |
Multi-GPU via NCCL (requires cuda and NCCL) |
metal |
Apple Metal GPU support |
accelerate |
Apple Accelerate framework |
mkl |
Intel MKL acceleration |
The default feature set (no flags) builds with pure Rust — no C compiler or system libraries required.
License
MIT