mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
Hanzo CI/CD / test (linux-amd64) (push) Skipped
Hanzo CI/CD / test (linux-arm64) (push) Skipped
Hanzo CI/CD / test-advisory (push) Skipped
Hanzo CI/CD / test-amd64 (push) Successful in 6m42s
Hanzo CI/CD / cicd (push) Successful in 1m1s
Hanzo CI/CD / verify (push) Skipped
CI/CD / cicd (push) Successful in 7m50s
The Responses API's stored responses, conversation histories and background tasks were keyed by response ID alone, so GET, DELETE, cancel and previous_response_id served whoever held an ID. Each entry now belongs to the org in X-Org-Id and every lookup names the caller's org; another org's ID answers 404. hanzo-router passed chat and completions bodies to vLLM with no cache_salt, so every org shared the replicas' prefix cache. It now sets cache_salt from x-org-id and the client's own salt, encoded as Hanzo Engine encodes them. The kv-isolation gate covers both and now builds hanzo-router with its proxy feature. Co-authored-by: Claude Opus 5.5 <[email protected]>
Hanzo Engine
Fast, flexible LLM inference. Run 40+ model families (text, vision, video, audio, speech, image generation, and embeddings) with automatic hardware optimization, OpenAI-compatible APIs, and built-in agentic features.
Quick Start
# Install (Linux/macOS, auto-detects CUDA, Metal, MKL)
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/hanzoai/engine/master/install.sh | sh
# Install (Windows PowerShell)
irm https://raw.githubusercontent.com/hanzoai/engine/master/install.ps1 | iex
# Run a model interactively
hanzo run --isq 4 -m Qwen/Qwen3-4B
# Or serve it as an OpenAI-compatible API with a web UI
hanzo serve --ui --isq 4 -m Qwen/Qwen3-4B
New to Hanzo Engine? Follow the Getting Started tutorial for a guided walkthrough.
Choose Your Path
| I want to... | Start here |
|---|---|
| Follow a step-by-step tutorial | Getting Started |
| Run a model from the command line | CLI Reference |
| Serve models over HTTP | HTTP Server |
| Build a Python application | Python SDK |
| Build a Rust application | Rust SDK |
| Optimize performance and memory | Performance Guide |
| Choose a quantization method | Quantization Overview |
| Use agentic features (tools, search, MCP) | Agentic Features Guide |
| Find a specific model | Supported Models |
| Troubleshoot an issue | Troubleshooting |
Highlights
- Zero config:
hanzo run -m <model>auto-detects architecture, chat template, and hardware - Quantization control: ISQ, GGUF, GPTQ, AWQ, HQQ, AFQ, FP8, MXFP4, or just
--isq 4 - Auto-tuning:
hanzo tunebenchmarks your hardware and recommends optimal settings - Agentic: Server-side tool calling loop, web search, MCP client, HTTP tool dispatch
- Multi-GPU: Automatic tensor parallelism via NCCL, or cross-machine via Ring backend
- Web UI:
--uiflag for instant browser-based chat