Six claim pages, one per measurement, each fetching its own rows from /v1/benchmark/throughput at read time and printing the median with its range, the repetition count, and the engine build, flags, host and date that stand behind it. A row whose quality checks failed renders "withheld" in place of a rate, so a fast run that was serving badly cannot be quoted. Drops the figures that had no run behind them: the llama.cpp comparison table in the ROCm guide, the tok/s cards and the "2-4x" feature stats on the marketing page, the backend table in gpu/README, and MATFORMER's layer-count speed estimate. Claims are reached from the footer of every page rather than the sidebar, so one is always a hop away and never quoted inline. Co-authored-by: Hanzo Dev <[email protected]>
2.4 KiB
hanzo-engine (GPU) — native Windows GPU inference, Vulkan + ROCm
Native, bridge-free GPU inference for the hanzo stack on Windows, built on the llama.cpp/ggml runtime. Two selectable backends wired in:
| Backend | Use |
|---|---|
| vulkan (default) | any GPU (AMD/NVIDIA/Intel), Win+Linux (KHR_coopmat) |
| rocm (opt-in) | AMD, long-context fast-path (tuned rocWMMA) |
Native backends, not a bridge: a WSL to D3D12 path synchronizes on every GPU call, which is why this builds on ggml's native backends rather than a hand-ported one. Throughput for these backends is published per claim, with the build, flags, spread and date behind each figure, at https://hanzoai.github.io/hanzo/benchmarks/decode-throughput/.
Serves the OpenAI-compatible API (/v1/chat/completions, /v1/embeddings) on
127.0.0.1:<port> — a drop-in for the hanzo node (point a provider's external_url
here). Runs GGUF models (the format we already produce for the Zen models).
Build & run (from the hanzo tree)
cd hanzo\engine\gpu
# 1. Assemble backends from upstream optimized binaries (+ a demo model)
.\setup.ps1 -Backend all -PullModel # vulkan (always) + rocm (AMD) into .\dist\
# 2. Run — Vulkan (default, cross-vendor)
.\hanzo-engine.ps1 -Backend vulkan -Port 36920
# or ROCm (AMD opt-in, long-context)
.\hanzo-engine.ps1 -Backend rocm -Port 36920
# add -Embedding to expose /v1/embeddings (pooling)
Then point the hanzo node at it:
POST /v2/add_llm_provider { external_url:"http://127.0.0.1:36920", model:"openai:hanzo", api_key:"x" }
Build from source (optional — needs SDKs)
setup.ps1 fetches the same optimized binaries AMD/ggml publish, so from-source
gives no perf gain. If you want to compile anyway:
# Vulkan: winget install Kitware.CMake KhronosGroup.VulkanSDK Ninja-build.Ninja
.\build-from-source.ps1 -Backend vulkan
# ROCm: install HIP SDK for Windows 7.1.1+ (gfx1151 officially Supported)
.\build-from-source.ps1 -Backend rocm
Notes
dist\(binaries) is gitignored; commit only the scripts.- The WSL Hanzo Engine ROCm backend (
../,hanzo-mlfeature/rocm-backend) stays as the native-Linux ROCm / research track — correct but bridge-capped on Windows. - Model gotcha: the Ollama
zen-nano:0.6bblob is a degenerate quant (loops, no EOS). Use a properly-quantized GGUF (canonicalggml-org/Qwen3-0.6B-GGUF, or re-quantize Zen).