Files
zeekayandHanzo Dev 42e34c2d41 refactor: name only Hanzo Engine in code, comments and docs
Every mention of another inference engine is rewritten to say what our code
does, or the math it computes. Reference values are "the reference value
from the published config" or "the recorded reference output", line
citations into another engine's source become the rule itself, comparisons
with other engines are dropped, and asides about other engines' bugs go.

Identifiers follow:
- CUDA: hanzo-paged-attn's `namespace vllm` is `pagedattn`, hanzo-engine's
  MoE helpers `moe_dtype` and `moe_detail`, the rotary kernels'
  `hanzo_rotary`; the shuffle/ldg macros are HANZO_*.
- GPTQ: the shuffled-layout path (`reconstruct_exllama`, `use_exllama`) is
  `reconstruct_shuffled` / `is_shuffled`, on the kernel and the FFI side.
- Tests: `*_matches_vllm` and friends name what they compare against (the
  reference fixture, the tile rule).
- MiniCPM-o's vision-LLM embedding is `vlm_embedding`; the rotary custom op
  is `rotary-embedding`.
- hanzo-bench's board scores ours against a "rival" whose name comes from
  the manifest: `rival_<tag>.json`, `--rival*` manifest flags, `rival t/s`
  columns; dossier.sh and spec_decode.sh take `--rival-bench`.
- install.ps1 installs this repository's hanzo-cli, as install.sh does,
  with HANZO_INSTALL_NCCL / HANZO_INSTALL_NO_NCCL.
- bench_http.py reads only the hanzo: metric families.

Kept, because they are what they are: LICENSE, LICENSE-APACHE, NOTICE and
tools/upstream-map.sh; each ported file's copyright/licence/provenance
header; Cargo.toml dependency names; data identifiers (Hugging Face repo
IDs, fixture file names and fixture metadata keys the tests read, on-disk
paths, executable names the GPU and engine guards match on a host); model
and format names.

LLM.md says the rule once, under "One name: Hanzo Engine".


Co-authored-by: Hanzo Dev <[email protected]>
2026-10-16 20:59:50 -07:00

7.9 KiB

Fast-local replica fabric (Qwen3-30B-A3B) — runbook

3 independent full replicas of Qwen3-30B-A3B-Instruct-2507-Q4_K_M (18.5GB, 3B active MoE), one per box, load-balanced behind hanzo-router. Goal: N parallel agents, each at full GPU speed.

Front endpoint (what the user points a client at)

export ANTHROPIC_BASE_URL=http://10.0.0.144:1234   # evo, hanzo-router
export ANTHROPIC_API_KEY=local                      # engine ignores auth (dev)
hanzo code

OpenAI clients: http://10.0.0.144:1234/v1. Both /v1/messages (Anthropic) and /v1/chat/completions (OpenAI) proxy through, SSE streamed byte-for-byte.

Model-id gotcha (why the router rewrites)

Each engine serves ONE model and strict-matches its served id or default; ANY other model id -> HTTP 500. Claude Code sends claude-* ids, so the router pins --upstream-model default on every forwarded request (routing/affinity still key on the client's original model). Code: hanzo-router 0.1.1, branch feat/router-upstream-model. Build: cd ~/work/hanzo/engine && PATH=/usr/bin:$HOME/.cargo/bin:$PATH CC=/usr/bin/gcc \ cargo build --release -p hanzo-router --features proxy (the ~/.local/bin/cc Claude shim shadows gcc and breaks the link with unknown option -m64 — force /usr/bin first).

Per-box serve commands (native hanzo serve, all bind :8080, model id = default)

GGUF paths differ per box; config+tokenizer live in a local qwen3-30b-meta/ dir (config.json, generation_config.json, tokenizer.json, tokenizer_config.json + a symlink to the gguf) so -m <dir> loads locally with NO HuggingFace fetch (HF 404s offline).

spark (CUDA GB10), on spark: hanzo serve -m Qwen/Qwen3-30B-A3B-Instruct-2507 --format gguf
-f ~/models/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf -p 8080 --host 0.0.0.0 -n 0:48 # (spark has the HF repo cached, so it uses the repo id directly)

evo (ROCm Strix Halo gfx1151), on evo — ONE ROCm proc only (concurrent kfd init wedges): LD_LIBRARY_PATH=/opt/rocm/lib:/opt/rocm/core-7.13/lib
~/work/hanzo/engine-ring/target/release/hanzo serve
-m ~/models/qwen3-30b-meta --format gguf -f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf
-p 8080 --host 127.0.0.1 -n 0:48 --pa-context-len 131072 # -n 0:48 forces all 48 layers onto rocm[0] (auto-mapper misreads APU unified mem -> CPU offload = 4 tok/s trap) # --pa-context-len 131072 caps KV to ~24GB (default grabs 81GB -> only 7GB free) # engine-ring = hanzo 1.7.28 (ROCm-linked); ldd shows libamdhip. engine-rocm is 1.7.21.

# KNOWN evo-ONLY bug (ROCm/gfx1151, NOT config): the last prefill position's logits
# are slightly off, so the FIRST generated token's argmax flips on *borderline* prompts
# (e.g. "reply with only: pong" -> "system\npong"). Realistic agent/code prompts are
# clean (measured 7/8; only the contrived edge case leaked). Present in 1.7.21 AND 1.7.28,
# with Q4K_FALLBACK=1 and with --paged-attn off -> it's the ROCm forward pass, not a knob.
# Signature == the CUDA MoE topk "last-row collapse" bug fixed in ml 0.11.17; the HIP topk
# path needs the same fix. FIX PATH: port that fix to the ROCm MoE routing kernel + rebuild
# engine-rocm. Until then evo is a fast (60 tok/s) replica that occasionally glitches token 1.
# To exclude evo: drop its --replica from the router (spark+dbc alone = 2 clean replicas).

dbc (Metal M4 Max), on dbc: ~/work/hanzo/engine-metal/target/release/hanzo serve
-m /Users/a/models/qwen3-30b-meta --format gguf -f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf
-p 8080 --host 0.0.0.0 -n 0:48 --paged-attn on --pa-context-len 131072 # paged-attn is auto-OFF on Metal; force on so KV is bounded + block-shared

Router (on evo)

~/work/hanzo/engine/target/release/hanzo-router --host 0.0.0.0 --port 1234 \
  --model qwen3-30b-a3b --upstream-model default \
  --replica http://192.168.77.2:8080   `# spark, direct 2.5GbE link` \
  --replica http://127.0.0.1:8080      `# evo local` \
  --replica http://10.0.0.132:8080     `# dbc over LAN`

Admin: GET /v1/replicas (health+inflight), POST /v1/replicas (add), GET /health. Prefix-affinity (conversation sticks to one replica) + least-loaded spill + /health eviction/restore.

Measured (Qwen3-30B-A3B Q4_K_M, 256-tok decode)

single-box: spark(CUDA) 48 tok/s, evo(ROCm) 60, dbc(Metal) 72. Through router single-stream: 58-63 tok/s. 3 concurrent agents (1 per box): ~186 tok/s aggregate, each 62-68 tok/s = the parallel win (3x). Beyond 3, agents double up per box and per-agent decode drops (MoE: distinct experts per seq -> more weight reads/step; evo/APU worst because it also hosts the router/console and shares memory bandwidth). Spread across boxes is the design; the router does exactly this. Sweet spot = <=3 concurrent (one full-speed agent/GPU).

Net topology (from evo, where the router runs)

evo 10.0.0.144 / 192.168.77.1(->spark) ; spark 10.0.0.242 / 192.168.77.2 ; dbc 10.0.0.132.

Live PIDs (this session): spark serve 3895887 | evo serve 2920376 | dbc serve 21351 | router 2883009

All nohup+disown (survive session end). memguard.sh is a dead one-shot (already tripped); current config is memory-bounded, no guard needed. To kill a serve: kill -TERM <pid> by exact PID -- NEVER pkill -f "release/hanzo serve" (the pattern self-matches your own shell -> kills it).


Zen 5.8 / Qwen 3.8 Blackwell Cluster Topology (Live Production)

Upstream Engines

  • Spark (:30000): Blackwell GB10, model RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead: chunked prefill at ~1,700 tok/s, CUDA graphs, native Anthropic /v1/messages tool_use parsing.
  • Evo (:8080): Strix Halo gfx1151, model Qwen3.8-27B-Q6_K.gguf.

Native Rust Router (:1235, on Spark)

  • Service: hanzo-router.service on Spark.
  • Binary: ~/.local/bin/hanzo-router, compiled with --features proxy.
  • Config: ~/.config/hanzo/router-pool.yaml
  • Features:
    • Models served: default, qwen3.8, zen-coder, zen5.8, zen5.8-coder, qwen/qwen3.8-27b.
    • Probes: /v1/models (Anthropic and OpenAI model discovery), /health (backend liveness). No /api/ path: Claude Code's fire-and-forget HEAD /api/hello warm-up passes through to a replica and 404s, and the client ignores the result.
    • Normalization: Auto-rewrites output_config.effort and reasoning_effort from "high"/"max" to "medium" to ensure chat templates with discrete thinking tiers (xhigh, medium, low) do not throw TemplateErrors.
    • Fallback: Unrecognized models (e.g. claude-*) automatically route to primary coder pool (zen5.8).

Claude Code Usage

Point Claude Code directly to the router on Spark:

export ANTHROPIC_BASE_URL="http://<spark>:1235"   # address in the private record
export ANTHROPIC_API_KEY="dummy"
claude

Or via the public cloud gateway https://api.hanzo.ai (routed to the AI Lab grant cluster).

Python Coderouter Decommissioning

  • Python coderouter.service has been permanently disabled and stopped on all cluster nodes.
  • Zero runtime Python router dependency; pure Rust hanzo-router handles all load balancing, streaming, session affinity, and wire normalization.
  • Mid-conversation non-leading system messages are normalized in src/proxy.rs (prefixed with [System]: as user role) and in spark's chat template, eliminating 400 Jinja errors from Claude Code / dev agents.
  • Public ingress routes on https://api.hanzo.ai (/v1/models, /v1/messages) route via Traefik to hanzo-router:1235.
  • Isolated Claude Code profile ~/.claude-hanzo configured; alias hanzo-code available in ~/.zshrc to prevent collisions between personal claude.ai OAuth subscriptions and custom API routing.
  • Sub-agent and role-based steering: hanzo-router infers agent roles from wire models and query sources. Sub-agents, titles, fast summaries, and haiku/flash requests automatically route to Evo (AMD Strix Halo APU, Qwen3.8-27B-Q6_K.gguf via Vulkan RADV with 300ms TTFT), while main agent implementation and opus/sonnet queries route to Spark (DGX Spark Blackwell GB10 NVFP4).