Files
wbin90620andClaude Opus 5.5 86119f4cd1 fix(server-core, router): scope stored Responses to the caller's org; salt replicas' prefix caches per org
The Responses API's stored responses, conversation histories and background
tasks were keyed by response ID alone, so GET, DELETE, cancel and
previous_response_id served whoever held an ID. Each entry now belongs to the
org in X-Org-Id and every lookup names the caller's org; another org's ID
answers 404.

hanzo-router passed chat and completions bodies to vLLM with no cache_salt,
so every org shared the replicas' prefix cache. It now sets cache_salt from
x-org-id and the client's own salt, encoded as Hanzo Engine encodes them.

The kv-isolation gate covers both and now builds hanzo-router with its proxy
feature.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-19 23:12:03 -08:00
..
2026-09-15 09:42:19 -07:00

Hanzo Engine

Fast, flexible LLM inference. Run 40+ model families (text, vision, video, audio, speech, image generation, and embeddings) with automatic hardware optimization, OpenAI-compatible APIs, and built-in agentic features.

Quick Start

# Install (Linux/macOS, auto-detects CUDA, Metal, MKL)
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/hanzoai/engine/master/install.sh | sh

# Install (Windows PowerShell)
irm https://raw.githubusercontent.com/hanzoai/engine/master/install.ps1 | iex

# Run a model interactively
hanzo run --isq 4 -m Qwen/Qwen3-4B

# Or serve it as an OpenAI-compatible API with a web UI
hanzo serve --ui --isq 4 -m Qwen/Qwen3-4B

New to Hanzo Engine? Follow the Getting Started tutorial for a guided walkthrough.

Choose Your Path

I want to... Start here
Follow a step-by-step tutorial Getting Started
Run a model from the command line CLI Reference
Serve models over HTTP HTTP Server
Build a Python application Python SDK
Build a Rust application Rust SDK
Optimize performance and memory Performance Guide
Choose a quantization method Quantization Overview
Use agentic features (tools, search, MCP) Agentic Features Guide
Find a specific model Supported Models
Troubleshoot an issue Troubleshooting

Highlights

  • Zero config: hanzo run -m <model> auto-detects architecture, chat template, and hardware
  • Quantization control: ISQ, GGUF, GPTQ, AWQ, HQQ, AFQ, FP8, MXFP4, or just --isq 4
  • Auto-tuning: hanzo tune benchmarks your hardware and recommends optimal settings
  • Agentic: Server-side tool calling loop, web search, MCP client, HTTP tool dispatch
  • Multi-GPU: Automatic tensor parallelism via NCCL, or cross-machine via Ring backend
  • Web UI: --ui flag for instant browser-based chat

Community