mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
Prefix blocks were hashed from token content alone, so any two requests with the same prefix shared KV across orgs, and the sequence-level cache and its disk spill were keyed by tokens too. Every block hash, sequence-cache key and recurrent-prefix key now carries the request's cache salt, which the server derives from the gateway's X-Org-Id plus an optional cache_salt body field. Salted prefixes are dropped instead of spilled to disk. The scheduler also cached blocks no forward had written: a prompt preempted in the pass that admitted it, the last sampled token's slot of every finished or preempted sequence, and sequences that ended in error. Now only written positions become reusable. kv_isolation drives the real scheduler against a simulated KV store and checks every read for unwritten, foreign-context or other-tenant KV; hanzo.yml gates on it. Co-authored-by: Claude Opus 5.5 <[email protected]>
Hanzo Engine
Fast, flexible LLM inference. Run 40+ model families (text, vision, video, audio, speech, image generation, and embeddings) with automatic hardware optimization, OpenAI-compatible APIs, and built-in agentic features.
Quick Start
# Install (Linux/macOS, auto-detects CUDA, Metal, MKL)
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/hanzoai/engine/master/install.sh | sh
# Install (Windows PowerShell)
irm https://raw.githubusercontent.com/hanzoai/engine/master/install.ps1 | iex
# Run a model interactively
hanzo run --isq 4 -m Qwen/Qwen3-4B
# Or serve it as an OpenAI-compatible API with a web UI
hanzo serve --ui --isq 4 -m Qwen/Qwen3-4B
New to Hanzo Engine? Follow the Getting Started tutorial for a guided walkthrough.
Choose Your Path
| I want to... | Start here |
|---|---|
| Follow a step-by-step tutorial | Getting Started |
| Run a model from the command line | CLI Reference |
| Serve models over HTTP | HTTP Server |
| Build a Python application | Python SDK |
| Build a Rust application | Rust SDK |
| Optimize performance and memory | Performance Guide |
| Choose a quantization method | Quantization Overview |
| Use agentic features (tools, search, MCP) | Agentic Features Guide |
| Find a specific model | Supported Models |
| Troubleshoot an issue | Troubleshooting |
Highlights
- Zero config:
hanzo run -m <model>auto-detects architecture, chat template, and hardware - Quantization control: ISQ, GGUF, GPTQ, AWQ, HQQ, AFQ, FP8, MXFP4, or just
--isq 4 - Auto-tuning:
hanzo tunebenchmarks your hardware and recommends optimal settings - Agentic: Server-side tool calling loop, web search, MCP client, HTTP tool dispatch
- Multi-GPU: Automatic tensor parallelism via NCCL, or cross-machine via Ring backend
- Web UI:
--uiflag for instant browser-based chat