mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
Every mention of another inference engine is rewritten to say what our code does, or the math it computes. Reference values are "the reference value from the published config" or "the recorded reference output", line citations into another engine's source become the rule itself, comparisons with other engines are dropped, and asides about other engines' bugs go. Identifiers follow: - CUDA: hanzo-paged-attn's `namespace vllm` is `pagedattn`, hanzo-engine's MoE helpers `moe_dtype` and `moe_detail`, the rotary kernels' `hanzo_rotary`; the shuffle/ldg macros are HANZO_*. - GPTQ: the shuffled-layout path (`reconstruct_exllama`, `use_exllama`) is `reconstruct_shuffled` / `is_shuffled`, on the kernel and the FFI side. - Tests: `*_matches_vllm` and friends name what they compare against (the reference fixture, the tile rule). - MiniCPM-o's vision-LLM embedding is `vlm_embedding`; the rotary custom op is `rotary-embedding`. - hanzo-bench's board scores ours against a "rival" whose name comes from the manifest: `rival_<tag>.json`, `--rival*` manifest flags, `rival t/s` columns; dossier.sh and spec_decode.sh take `--rival-bench`. - install.ps1 installs this repository's hanzo-cli, as install.sh does, with HANZO_INSTALL_NCCL / HANZO_INSTALL_NO_NCCL. - bench_http.py reads only the hanzo: metric families. Kept, because they are what they are: LICENSE, LICENSE-APACHE, NOTICE and tools/upstream-map.sh; each ported file's copyright/licence/provenance header; Cargo.toml dependency names; data identifiers (Hugging Face repo IDs, fixture file names and fixture metadata keys the tests read, on-disk paths, executable names the GPU and engine guards match on a host); model and format names. LLM.md says the rule once, under "One name: Hanzo Engine". Co-authored-by: Hanzo Dev <[email protected]>
Hanzo Engine
Fast, flexible LLM inference. Run 40+ model families (text, vision, video, audio, speech, image generation, and embeddings) with automatic hardware optimization, OpenAI-compatible APIs, and built-in agentic features.
Quick Start
# Install (Linux/macOS, auto-detects CUDA, Metal, MKL)
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/hanzoai/engine/master/install.sh | sh
# Install (Windows PowerShell)
irm https://raw.githubusercontent.com/hanzoai/engine/master/install.ps1 | iex
# Run a model interactively
hanzo run --isq 4 -m Qwen/Qwen3-4B
# Or serve it as an OpenAI-compatible API with a web UI
hanzo serve --ui --isq 4 -m Qwen/Qwen3-4B
New to Hanzo Engine? Follow the Getting Started tutorial for a guided walkthrough.
Choose Your Path
| I want to... | Start here |
|---|---|
| Follow a step-by-step tutorial | Getting Started |
| Run a model from the command line | CLI Reference |
| Serve models over HTTP | HTTP Server |
| Build a Python application | Python SDK |
| Build a Rust application | Rust SDK |
| Optimize performance and memory | Performance Guide |
| Choose a quantization method | Quantization Overview |
| Use agentic features (tools, search, MCP) | Agentic Features Guide |
| Find a specific model | Supported Models |
| Troubleshoot an issue | Troubleshooting |
Highlights
- Zero config:
hanzo run -m <model>auto-detects architecture, chat template, and hardware - Quantization control: ISQ, GGUF, GPTQ, AWQ, HQQ, AFQ, FP8, MXFP4, or just
--isq 4 - Auto-tuning:
hanzo tunebenchmarks your hardware and recommends optimal settings - Agentic: Server-side tool calling loop, web search, MCP client, HTTP tool dispatch
- Multi-GPU: Automatic tensor parallelism via NCCL, or cross-machine via Ring backend
- Web UI:
--uiflag for instant browser-based chat