Every mention of another inference engine is rewritten to say what our code does, or the math it computes. Reference values are "the reference value from the published config" or "the recorded reference output", line citations into another engine's source become the rule itself, comparisons with other engines are dropped, and asides about other engines' bugs go. Identifiers follow: - CUDA: hanzo-paged-attn's `namespace vllm` is `pagedattn`, hanzo-engine's MoE helpers `moe_dtype` and `moe_detail`, the rotary kernels' `hanzo_rotary`; the shuffle/ldg macros are HANZO_*. - GPTQ: the shuffled-layout path (`reconstruct_exllama`, `use_exllama`) is `reconstruct_shuffled` / `is_shuffled`, on the kernel and the FFI side. - Tests: `*_matches_vllm` and friends name what they compare against (the reference fixture, the tile rule). - MiniCPM-o's vision-LLM embedding is `vlm_embedding`; the rotary custom op is `rotary-embedding`. - hanzo-bench's board scores ours against a "rival" whose name comes from the manifest: `rival_<tag>.json`, `--rival*` manifest flags, `rival t/s` columns; dossier.sh and spec_decode.sh take `--rival-bench`. - install.ps1 installs this repository's hanzo-cli, as install.sh does, with HANZO_INSTALL_NCCL / HANZO_INSTALL_NO_NCCL. - bench_http.py reads only the hanzo: metric families. Kept, because they are what they are: LICENSE, LICENSE-APACHE, NOTICE and tools/upstream-map.sh; each ported file's copyright/licence/provenance header; Cargo.toml dependency names; data identifiers (Hugging Face repo IDs, fixture file names and fixture metadata keys the tests read, on-disk paths, executable names the GPU and engine guards match on a host); model and format names. LLM.md says the rule once, under "One name: Hanzo Engine". Co-authored-by: Hanzo Dev <[email protected]>
7.9 KiB
Fast-local replica fabric (Qwen3-30B-A3B) — runbook
3 independent full replicas of Qwen3-30B-A3B-Instruct-2507-Q4_K_M (18.5GB, 3B active MoE),
one per box, load-balanced behind hanzo-router. Goal: N parallel agents, each at full GPU speed.
Front endpoint (what the user points a client at)
export ANTHROPIC_BASE_URL=http://10.0.0.144:1234 # evo, hanzo-router
export ANTHROPIC_API_KEY=local # engine ignores auth (dev)
hanzo code
OpenAI clients: http://10.0.0.144:1234/v1. Both /v1/messages (Anthropic) and
/v1/chat/completions (OpenAI) proxy through, SSE streamed byte-for-byte.
Model-id gotcha (why the router rewrites)
Each engine serves ONE model and strict-matches its served id or default; ANY other
model id -> HTTP 500. Claude Code sends claude-* ids, so the router pins
--upstream-model default on every forwarded request (routing/affinity still key on the
client's original model). Code: hanzo-router 0.1.1, branch feat/router-upstream-model.
Build: cd ~/work/hanzo/engine && PATH=/usr/bin:$HOME/.cargo/bin:$PATH CC=/usr/bin/gcc \ cargo build --release -p hanzo-router --features proxy (the ~/.local/bin/cc Claude shim
shadows gcc and breaks the link with unknown option -m64 — force /usr/bin first).
Per-box serve commands (native hanzo serve, all bind :8080, model id = default)
GGUF paths differ per box; config+tokenizer live in a local qwen3-30b-meta/ dir
(config.json, generation_config.json, tokenizer.json, tokenizer_config.json + a symlink to
the gguf) so -m <dir> loads locally with NO HuggingFace fetch (HF 404s offline).
spark (CUDA GB10), on spark:
hanzo serve -m Qwen/Qwen3-30B-A3B-Instruct-2507 --format gguf
-f ~/models/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf -p 8080 --host 0.0.0.0 -n 0:48
# (spark has the HF repo cached, so it uses the repo id directly)
evo (ROCm Strix Halo gfx1151), on evo — ONE ROCm proc only (concurrent kfd init wedges):
LD_LIBRARY_PATH=/opt/rocm/lib:/opt/rocm/core-7.13/lib
~/work/hanzo/engine-ring/target/release/hanzo serve
-m ~/models/qwen3-30b-meta --format gguf -f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf
-p 8080 --host 127.0.0.1 -n 0:48 --pa-context-len 131072
# -n 0:48 forces all 48 layers onto rocm[0] (auto-mapper misreads APU unified mem -> CPU offload = 4 tok/s trap)
# --pa-context-len 131072 caps KV to ~24GB (default grabs 81GB -> only 7GB free)
# engine-ring = hanzo 1.7.28 (ROCm-linked); ldd shows libamdhip. engine-rocm is 1.7.21.
# KNOWN evo-ONLY bug (ROCm/gfx1151, NOT config): the last prefill position's logits
# are slightly off, so the FIRST generated token's argmax flips on *borderline* prompts
# (e.g. "reply with only: pong" -> "system\npong"). Realistic agent/code prompts are
# clean (measured 7/8; only the contrived edge case leaked). Present in 1.7.21 AND 1.7.28,
# with Q4K_FALLBACK=1 and with --paged-attn off -> it's the ROCm forward pass, not a knob.
# Signature == the CUDA MoE topk "last-row collapse" bug fixed in ml 0.11.17; the HIP topk
# path needs the same fix. FIX PATH: port that fix to the ROCm MoE routing kernel + rebuild
# engine-rocm. Until then evo is a fast (60 tok/s) replica that occasionally glitches token 1.
# To exclude evo: drop its --replica from the router (spark+dbc alone = 2 clean replicas).
dbc (Metal M4 Max), on dbc:
~/work/hanzo/engine-metal/target/release/hanzo serve
-m /Users/a/models/qwen3-30b-meta --format gguf -f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf
-p 8080 --host 0.0.0.0 -n 0:48 --paged-attn on --pa-context-len 131072
# paged-attn is auto-OFF on Metal; force on so KV is bounded + block-shared
Router (on evo)
~/work/hanzo/engine/target/release/hanzo-router --host 0.0.0.0 --port 1234 \
--model qwen3-30b-a3b --upstream-model default \
--replica http://192.168.77.2:8080 `# spark, direct 2.5GbE link` \
--replica http://127.0.0.1:8080 `# evo local` \
--replica http://10.0.0.132:8080 `# dbc over LAN`
Admin: GET /v1/replicas (health+inflight), POST /v1/replicas (add), GET /health.
Prefix-affinity (conversation sticks to one replica) + least-loaded spill + /health eviction/restore.
Measured (Qwen3-30B-A3B Q4_K_M, 256-tok decode)
single-box: spark(CUDA) 48 tok/s, evo(ROCm) 60, dbc(Metal) 72. Through router single-stream: 58-63 tok/s. 3 concurrent agents (1 per box): ~186 tok/s aggregate, each 62-68 tok/s = the parallel win (3x). Beyond 3, agents double up per box and per-agent decode drops (MoE: distinct experts per seq -> more weight reads/step; evo/APU worst because it also hosts the router/console and shares memory bandwidth). Spread across boxes is the design; the router does exactly this. Sweet spot = <=3 concurrent (one full-speed agent/GPU).
Net topology (from evo, where the router runs)
evo 10.0.0.144 / 192.168.77.1(->spark) ; spark 10.0.0.242 / 192.168.77.2 ; dbc 10.0.0.132.
Live PIDs (this session): spark serve 3895887 | evo serve 2920376 | dbc serve 21351 | router 2883009
All nohup+disown (survive session end). memguard.sh is a dead one-shot (already tripped);
current config is memory-bounded, no guard needed. To kill a serve: kill -TERM <pid> by exact
PID -- NEVER pkill -f "release/hanzo serve" (the pattern self-matches your own shell -> kills it).
Zen 5.8 / Qwen 3.8 Blackwell Cluster Topology (Live Production)
Upstream Engines
- Spark (
:30000): Blackwell GB10, modelRadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead: chunked prefill at ~1,700 tok/s, CUDA graphs, native Anthropic/v1/messagestool_use parsing. - Evo (
:8080): Strix Halo gfx1151, modelQwen3.8-27B-Q6_K.gguf.
Native Rust Router (:1235, on Spark)
- Service:
hanzo-router.serviceon Spark. - Binary:
~/.local/bin/hanzo-router, compiled with--features proxy. - Config:
~/.config/hanzo/router-pool.yaml - Features:
- Models served:
default,qwen3.8,zen-coder,zen5.8,zen5.8-coder,qwen/qwen3.8-27b. - Probes:
/v1/models(Anthropic and OpenAI model discovery),/health(backend liveness). No/api/path: Claude Code's fire-and-forgetHEAD /api/hellowarm-up passes through to a replica and 404s, and the client ignores the result. - Normalization: Auto-rewrites
output_config.effortandreasoning_effortfrom "high"/"max" to "medium" to ensure chat templates with discrete thinking tiers (xhigh, medium, low) do not throw TemplateErrors. - Fallback: Unrecognized models (e.g.
claude-*) automatically route to primary coder pool (zen5.8).
- Models served:
Claude Code Usage
Point Claude Code directly to the router on Spark:
export ANTHROPIC_BASE_URL="http://<spark>:1235" # address in the private record
export ANTHROPIC_API_KEY="dummy"
claude
Or via the public cloud gateway https://api.hanzo.ai (routed to the AI Lab grant cluster).
Python Coderouter Decommissioning
- Python
coderouter.servicehas been permanently disabled and stopped on all cluster nodes. - Zero runtime Python router dependency; pure Rust
hanzo-routerhandles all load balancing, streaming, session affinity, and wire normalization. - Mid-conversation non-leading system messages are normalized in
src/proxy.rs(prefixed with[System]:as user role) and in spark's chat template, eliminating 400 Jinja errors from Claude Code / dev agents. - Public ingress routes on
https://api.hanzo.ai(/v1/models,/v1/messages) route via Traefik tohanzo-router:1235. - Isolated Claude Code profile
~/.claude-hanzoconfigured; aliashanzo-codeavailable in~/.zshrcto prevent collisions between personalclaude.aiOAuth subscriptions and custom API routing. - Sub-agent and role-based steering:
hanzo-routerinfers agent roles from wire models and query sources. Sub-agents, titles, fast summaries, andhaiku/flashrequests automatically route to Evo (AMD Strix Halo APU,Qwen3.8-27B-Q6_K.ggufvia Vulkan RADV with 300ms TTFT), while main agent implementation andopus/sonnetqueries route to Spark (DGX Spark Blackwell GB10 NVFP4).