Files
zeekayandHanzo Dev 51b1849ba1 docs: every throughput figure renders from the run that produced it
Six claim pages, one per measurement, each fetching its own rows from
/v1/benchmark/throughput at read time and printing the median with its
range, the repetition count, and the engine build, flags, host and date
that stand behind it. A row whose quality checks failed renders
"withheld" in place of a rate, so a fast run that was serving badly
cannot be quoted.

Drops the figures that had no run behind them: the llama.cpp comparison
table in the ROCm guide, the tok/s cards and the "2-4x" feature stats on
the marketing page, the backend table in gpu/README, and MATFORMER's
layer-count speed estimate.

Claims are reached from the footer of every page rather than the sidebar,
so one is always a hop away and never quoted inline.

Co-authored-by: Hanzo Dev <[email protected]>
2026-09-16 02:51:11 -07:00
..

hanzo-engine (GPU) — native Windows GPU inference, Vulkan + ROCm

Native, bridge-free GPU inference for the hanzo stack on Windows, built on the llama.cpp/ggml runtime. Two selectable backends wired in:

Backend Use
vulkan (default) any GPU (AMD/NVIDIA/Intel), Win+Linux (KHR_coopmat)
rocm (opt-in) AMD, long-context fast-path (tuned rocWMMA)

Native backends, not a bridge: a WSL to D3D12 path synchronizes on every GPU call, which is why this builds on ggml's native backends rather than a hand-ported one. Throughput for these backends is published per claim, with the build, flags, spread and date behind each figure, at https://hanzoai.github.io/hanzo/benchmarks/decode-throughput/.

Serves the OpenAI-compatible API (/v1/chat/completions, /v1/embeddings) on 127.0.0.1:<port> — a drop-in for the hanzo node (point a provider's external_url here). Runs GGUF models (the format we already produce for the Zen models).

Build & run (from the hanzo tree)

cd hanzo\engine\gpu

# 1. Assemble backends from upstream optimized binaries (+ a demo model)
.\setup.ps1 -Backend all -PullModel        # vulkan (always) + rocm (AMD) into .\dist\

# 2. Run — Vulkan (default, cross-vendor)
.\hanzo-engine.ps1 -Backend vulkan -Port 36920
#    or ROCm (AMD opt-in, long-context)
.\hanzo-engine.ps1 -Backend rocm   -Port 36920
#    add -Embedding to expose /v1/embeddings (pooling)

Then point the hanzo node at it:

POST /v2/add_llm_provider { external_url:"http://127.0.0.1:36920", model:"openai:hanzo", api_key:"x" }

Build from source (optional — needs SDKs)

setup.ps1 fetches the same optimized binaries AMD/ggml publish, so from-source gives no perf gain. If you want to compile anyway:

# Vulkan: winget install Kitware.CMake KhronosGroup.VulkanSDK Ninja-build.Ninja
.\build-from-source.ps1 -Backend vulkan
# ROCm: install HIP SDK for Windows 7.1.1+ (gfx1151 officially Supported)
.\build-from-source.ps1 -Backend rocm

Notes

  • dist\ (binaries) is gitignored; commit only the scripts.
  • The WSL Hanzo Engine ROCm backend (../, hanzo-ml feature/rocm-backend) stays as the native-Linux ROCm / research track — correct but bridge-capped on Windows.
  • Model gotcha: the Ollama zen-nano:0.6b blob is a degenerate quant (loops, no EOS). Use a properly-quantized GGUF (canonical ggml-org/Qwen3-0.6B-GGUF, or re-quantize Zen).