zeekayandHanzo Dev 8c8a1f21fd fix(rocm): partitioned paged attention for contexts v1 cannot hold
ROCm had only the v1 decode kernel, which keeps one float of logits per
context token in shared memory: padded_max_context_len * 4 bytes, plus 2064
bytes of query and reduction scratch. gfx1151 gives a workgroup 64 KiB, so
past a little under 16K tokens v1 asks for more LDS than exists and the device
refuses the launch. evo's first decode after a 16K prompt faulted exactly
there (hipErrorIllegalAddress, HSA_STATUS_ERROR_INVALID_ALLOCATION) while 4K
and 8K were fine.

The refusal was also invisible. launch_v1 discarded the <<<>>> status, so the
attention output was never written and the fault surfaced in whichever kernel
ran next, under that kernel's name — a hanzo-ml JIT kernel on evo.

v2, as CUDA already has: a partition kernel per (head, seq, 512-token
partition) computes that partition's max, exp-sum and UNNORMALIZED weighted V
into f32 scratch, and a reduce kernel rescales every live partition to a common
max and divides once. That is v1's sum over tokens regrouped, sinks included
exactly once at the reduce, so the two agree to rounding. Partitions past a
sequence's device-side length return at once and are never read, so a launch
sized for a captured graph's bucket stays correct for any shorter context.
Compiled for gfx1151, the kernels hold a fixed 4112 bytes (v2) and 0 (reduce)
of LDS at any context length.

v1 keeps every shape it serves today: paged_attention_v1_fits asks the device
for its per-workgroup limit and the kernel for its static size, and v2 runs
only where v1 would not fit. Both inputs are fixed per captured decode graph,
so capture and replay always take the same kernel. Anything that cannot be
established reads as "does not fit" and goes to v2, which serves every length.

Every launcher now returns its HIP status and the caller names the kernel when
it fails.

Also a barrier in v1 between its two block reductions: they share red[], and a
warp that reached the sum's write could overwrite a slot a slower warp was
still reading for the max.

Verified off-device only — no ROCm device here:
- device and host compile for gfx1151 with -Wall, zero errors, zero warnings
  (upstream clang 22 against evo's ROCm 7.13 headers);
- a NumPy transcription of both kernels, in the cache's own layout and index
  arithmetic, matches dense float64 attention to 2.3e-7 across partition
  boundaries (512, 513, ragged), a batch of unequal lengths, sinks, softcap
  with alibi, and 6:1 GQA; unwritten partitions poisoned with NaN, so a read of
  one fails. Reading every partition, starting a partition one token late, and
  counting the sink per partition each fail it;
- the C definition, Rust declaration and Rust call of v2 match in all 25
  parameters, by name and order.
tests/rocm_paged_attention.rs gains the on-device cases (20000 tokens even and
with a dominant key in a late partition, 16385 with the key alone in the last
partition, and a 24576 capacity serving 16500 and 700); run with
`cargo test -p hanzo-paged-attn --features rocm`.

Co-authored-by: Hanzo Dev <[email protected]>
2026-09-18 22:55:06 -07:00
2026-02-11 06:13:59 -05:00
2026-09-03 11:13:16 -07:00
2026-09-15 09:42:19 -07:00
2026-09-17 23:45:16 -07:00
2026-09-17 23:45:16 -07:00
2024-10-28 11:40:42 -04:00
2026-09-15 09:42:19 -07:00

Hanzo Engine

Hanzo Engine

The native, multimodal inference engine — text, vision, audio, speech, image, and embeddings in one fast Rust binary.

| Documentation | Rust SDK | Python SDK | Discord |

GitHub stars

Hanzo Engine runs any Hugging Face model with zero config, quantizes it for your hardware, and serves it over the OpenAI and Anthropic wire formats plus a built-in web UI — one binary, from your laptop to a GPU cluster. It is the native inference layer of the Open AI Cloud.

Latest

  • NVFP4, natively: packed 4-bit weights go straight into the GEMM, with a register-resident E2M1 codebook for decode and tensor cores for prefill, so a 27B NVFP4 checkpoint loads in 31 GB instead of the 104 GB a dequantize-at-load path asks for. Quantization docs
  • Qwen3-Omni: native end-to-end omni-modal model (understand → think → speak) — text/image/video/audio in, text + 24kHz speech out, through one extensible modality pipeline. Validated against the reference weights.
  • New frontier models: MiniMax-M2 (sparse-MoE) and DeepSeek-V3.2, alongside the existing DeepSeek-V3, Kimi-K2, GLM-4, and Qwen3 families. Supported models
  • Paged-attention serving for the omni Thinker, plus a disk-first KV cache (cross-restart sessions + agent prefix reuse) for cheap long-context serving.
  • Anthropic Messages API: hanzo-engine serve now exposes an Anthropic-compatible POST /v1/messages endpoint (streaming, tool use, and Claude Code harness support) alongside the OpenAI-compatible /v1 API. Examples
  • Agentic runtime: web search, local Python code execution with model feedback, session management, and custom tool hooks. Guide
  • Gemma 4: full multimodal: text, image, video, and audio input. Guide | Video setup
  • MXFP4 ISQ quantization: MXFP4 with optimized decode kernels for faster, smaller models. Quantization docs

Why Hanzo Engine?

  • Any Hugging Face model, zero config: Just hanzo-engine run -m user/model. Architecture, quantization format, and chat template are auto-detected.
  • True multimodality: Text, vision, video, and audio, speech generation, image generation, and embeddings in one engine.
  • Smart quantization: --quant automatically selects the best quantization format at that level: using a prebuilt UQFF if one is published, otherwise applying ISQ. Docs
  • OpenAI + Anthropic wire formats: The same hanzo-engine serve process exposes OpenAI-compatible /v1 endpoints and an Anthropic-compatible Messages endpoint.
  • Built-in web UI: Served at /ui by default. Shows reasoning, code execution, plots, and files inline. Edit any message and the new branch runs with its own Python state. Pass --no-ui to disable.
  • Hardware-aware: hanzo-engine tune benchmarks your system and picks optimal quantization + device mapping.
  • Flexible SDKs: Python package and Rust crate to build your projects.
  • Native agentic support: built-in agentic loop with web search, local Python code execution with model feedback, session management, and custom tool hooks.

Quick Start

Binaries

This repository builds two programs, and neither is called hanzo:

binary crate how to get it
hanzoai hanzo-server prebuilt, attached to each release
hanzo-engine hanzo-cli built from source by install.sh

The hanzo on your PATH is the Hanzo CLI; its hanzo engine serve MODEL runs hanzo-engine serve -m MODEL. hanzoai logs a deprecation warning that names hanzo serve from hanzo-cli, which is the hanzo-engine binary.

Each release attaches hanzoai-macos-arm64.tar.gz and hanzoai-macos-amd64.tar.gz (Metal), and hanzoai-linux-amd64.tar.gz and hanzoai-linux-arm64.tar.gz (CPU only, with cosign .sig and .pem). Each tarball holds the one hanzoai binary.

install.sh needs Rust 1.88 or newer. It runs cargo install --git https://github.com/hanzoai/engine --locked hanzo-cli with the features it detects, which puts hanzo-engine in ~/.cargo/bin:

curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/hanzoai/engine/main/install.sh | sh

Windows (PowerShell):

irm https://raw.githubusercontent.com/hanzoai/engine/main/install.ps1 | iex

Manual installation & other platforms

Serve a model

curl -L https://github.com/hanzoai/engine/releases/latest/download/hanzoai-macos-arm64.tar.gz | tar xz
./hanzoai --serve-ip 127.0.0.1 --port 1234 run -m zenlm/zen-nano-0.6b

From another shell:

curl 127.0.0.1:1234/v1/models

hanzoai has no default port: give it --port, or -i for an interactive session. --serve-ip defaults to 0.0.0.0, every interface. OpenAI-compatible clients use http://127.0.0.1:1234/v1.

hanzo-engine serve -m <model> listens on 0.0.0.0:1234 unless given -p and --host, serves a web UI at /ui, and advertises itself over mDNS unless given --no-advertise. LM Studio also defaults to port 1234. hanzo-engine tune -m <model> --emit-config config.toml recommends a quantization and device map for the machine, and hanzo-engine from-config -f config.toml runs it.

Chat and embeddings on one port

multi-model serves several models on one port, and each request names one by alias. With this models.json:

{
  "chat": { "alias": "chat", "Plain": { "model_id": "zenlm/zen-nano-0.6b" } },
  "embed": { "alias": "embed", "Embedding": { "model_id": "zenlm/zen-embedding-0.6B" } }
}
./hanzoai --serve-ip 127.0.0.1 --port 1234 multi-model --config models.json
curl 127.0.0.1:1234/v1/embeddings -H 'Content-Type: application/json' \
  -d '{"model":"embed","input":"hello"}'
curl 127.0.0.1:1234/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"chat","messages":[{"role":"user","content":"hello"}]}'

Memory

Weights take about their file size in memory. Weight files of Zen models in GB (10^9 bytes), as the Hugging Face API lists them (/api/models/<repo>?blobs=true). The 16-bit column is the safetensors release; Q8_0 and Q4_K_M are GGUF files, from the -GGUF repos for the embedding models.

model parameters 16-bit Q8_0 Q4_K_M
zenlm/zen-nano-0.6b 0.60 B 1.19 0.64 0.40
zenlm/zen-embedding-0.6B 0.60 B 1.19 0.64
zenlm/zen-eco-4b-instruct 4.02 B 8.04
zenlm/zen-embedding-8B 7.57 B 15.13 4.68
zenlm/zen-vl-8b-instruct 8.77 B 17.53

That is about 2.0 GB per billion parameters at 16 bits, 1.07 GB at Q8_0 and 0.62 GB at Q4_K_M, so a 14B model needs roughly 28, 15 or 8.7 GB for weights. hanzoai --isq q8_0 or --isq q4k quantizes a 16-bit model as it loads.

The KV cache comes on top: 2 × layers × KV heads × head dim × 2 bytes per token at 16 bits. With the values in each config.json, zen-nano-0.6b (28 × 8 × 128) takes 0.11 MB per token and zen-eco-4b-instruct (36 × 8 × 128) 0.15 MB, so a 32,768-token context adds 3.8 or 4.8 GB. The automatic device map plans for --max-seq-len, 4096 tokens unless set.

Measured on a 64 GB M1 Max with vmmap -summary: zen-nano-0.6b at 16 bits served with a 2.0 GB physical footprint, and 3.6 GB with zen-embedding-0.6B loaded beside it.

On Apple Silicon the GPU budget is the larger of Metal's recommended working set and 2/3 of RAM (3/4 above 36 GB), or sysctl iogpu.wired_limit_mb when that is set (hanzo-engine/src/utils/memory_usage.rs). For the 64 GB M1 Max above the device map reported 52 GB. On a 24 GB Mac the budget starts at 16 GB, about what a 14B model at Q8_0 needs for weights alone.

Full CLI documentation

UI Demo
Web Chat UI Demo

What Makes It Fast

Performance

Quantization (full docs)

  • In-situ quantization (ISQ) of any Hugging Face model
  • GGUF (2-8 bit), GPTQ, AWQ, HQQ, FP8, BNB support
  • ⭐ Per-layer topology: Fine-tune quantization per layer for optimal quality/speed
  • ⭐ Auto-select fastest quant method for your hardware

Flexibility

Agentic Features

Full feature documentation

Supported Models

Text Models
  • Granite 4.0
  • SmolLM 3
  • DeepSeek V3
  • GPT-OSS
  • DeepSeek V2
  • Qwen 3 Next
  • Qwen 3 MoE
  • Phi 3.5 MoE
  • Qwen 3
  • GLM 4
  • GLM-4.7-Flash
  • GLM-4.7 (MoE)
  • Gemma 2
  • Qwen 2
  • Starcoder 2
  • Phi 3
  • Mixtral
  • Phi 2
  • Gemma
  • Llama
  • Mistral
Multimodal Models
  • Qwen 3.5
  • Qwen 3.5 MoE
  • Qwen 3-VL
  • Qwen 3-VL MoE
  • Gemma 3n
  • Llama 4
  • Gemma 3
  • Mistral 3
  • Phi 4 multimodal
  • Qwen 2.5-VL
  • MiniCPM-O
  • Llama 3.2 Vision
  • Qwen 2-VL
  • Idefics 3
  • Idefics 2
  • LLaVA Next
  • LLaVA
  • Phi 3V
Speech Models
  • Voxtral (ASR/speech-to-text)
  • Dia
Image Generation Models
  • FLUX
Embedding Models
  • Embedding Gemma
  • Qwen 3 Embedding

Request a new model | Full compatibility tables

Python SDK

pip install hanzo  # or hanzo-cuda, hanzo-metal, hanzo-mkl, hanzo-accelerate
from hanzo import Runner, Which, ChatCompletionRequest

runner = Runner(
    which=Which.Plain(model_id="Qwen/Qwen3-4B"),
    in_situ_quant="4",
)

res = runner.send_chat_completion_request(
    ChatCompletionRequest(
        model="default",
        messages=[{"role": "user", "content": "Hello!"}],
        max_tokens=256,
    )
)
print(res.choices[0].message.content)

Python SDK | Installation | Examples | Cookbook

Rust SDK

cargo add hanzo
use anyhow::Result;
use hanzo::{IsqType, TextMessageRole, TextMessages, MultimodalModelBuilder};

#[tokio::main]
async fn main() -> Result<()> {
    let model = MultimodalModelBuilder::new("google/gemma-4-E4B-it")
        .with_isq(IsqType::Q4K)
        .with_logging()
        .build()
        .await?;

    let messages = TextMessages::new().add_message(
        TextMessageRole::User,
        "Hello!",
    );

    let response = model.send_chat_request(messages).await?;

    println!("{:?}", response.choices[0].message.content);

    Ok(())
}

API Docs | Crate | Examples

Docker

For quick containerized deployment:

docker pull ghcr.io/hanzoai/engine:latest
docker run --gpus all -p 1234:1234 ghcr.io/hanzoai/engine:latest \
  serve -m Qwen/Qwen3-4B

Docker images

For production use, we recommend installing the CLI directly for maximum flexibility.

Documentation

For complete documentation, see the Documentation.

Quick Links:

Contributing

Contributions welcome! Please open an issue to discuss new features or report bugs. If you want to add a new model, please contact us via an issue and we can coordinate.

Credits

Tensors, kernels, and autodiff come from hanzoai/ml, the Rust compute core underneath this engine. Thank you to all contributors.

Hanzo Engine is MIT licensed. LICENSE carries the full text and every copyright line it names; NOTICE carries the Apache-2.0 attributions that ship with the vendored GPU kernels, and both travel with any binary or crate we distribute.

Hanzo — the Open AI Cloud

Open source · every language · on-chain settlement. hanzo.ai · docs.hanzo.ai

SDKs in every language — Python (flagship) · TypeScript · Go · Rust · C++ · Swift · Kotlin · umbrella

S
Description
Hanzo AI native inference engine - Rust-based LLM & embedding engine for foundational models
Readme MIT
107 MiB
Languages
Rust 76.9%
Cuda 9.8%
JavaScript 7%
Metal 2.7%
Python 1.4%
Other 1.9%