Files
zeekay 4954b05910 Drop the router's /api/hello probe
Claude Code's HEAD $ANTHROPIC_BASE_URL/api/hello is a warm-up whose
result it ignores; it now passes through and 404s like any path the
router does not own.

Claude-Session: https://claude.ai/code/session_01TUxvfuta1Z44KVNYuAz71F
2026-09-24 15:01:47 -07:00

8.2 KiB

Fast-local replica fabric (Qwen3-30B-A3B) — runbook

3 independent full replicas of Qwen3-30B-A3B-Instruct-2507-Q4_K_M (18.5GB, 3B active MoE), one per box, load-balanced behind hanzo-router. Goal: N parallel agents, each at full GPU speed.

Front endpoint (what the user points a client at)

export ANTHROPIC_BASE_URL=http://10.0.0.144:1234   # evo, hanzo-router
export ANTHROPIC_API_KEY=local                      # engine ignores auth (dev)
hanzo code

OpenAI clients: http://10.0.0.144:1234/v1. Both /v1/messages (Anthropic) and /v1/chat/completions (OpenAI) proxy through, SSE streamed byte-for-byte.

Model-id gotcha (why the router rewrites)

Each engine serves ONE model and strict-matches its served id or default; ANY other model id -> HTTP 500. Claude Code sends claude-* ids, so the router pins --upstream-model default on every forwarded request (routing/affinity still key on the client's original model). Code: hanzo-router 0.1.1, branch feat/router-upstream-model. Build: cd ~/work/hanzo/engine && PATH=/usr/bin:$HOME/.cargo/bin:$PATH CC=/usr/bin/gcc \ cargo build --release -p hanzo-router --features proxy (the ~/.local/bin/cc Claude shim shadows gcc and breaks the link with unknown option -m64 — force /usr/bin first).

Per-box serve commands (native hanzo serve, all bind :8080, model id = default)

GGUF paths differ per box; config+tokenizer live in a local qwen3-30b-meta/ dir (config.json, generation_config.json, tokenizer.json, tokenizer_config.json + a symlink to the gguf) so -m <dir> loads locally with NO HuggingFace fetch (HF 404s offline).

spark (CUDA GB10), on spark: hanzo serve -m Qwen/Qwen3-30B-A3B-Instruct-2507 --format gguf
-f ~/models/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf -p 8080 --host 0.0.0.0 -n 0:48 # (spark has the HF repo cached, so it uses the repo id directly)

evo (ROCm Strix Halo gfx1151), on evo — ONE ROCm proc only (concurrent kfd init wedges): LD_LIBRARY_PATH=/opt/rocm/lib:/opt/rocm/core-7.13/lib
~/work/hanzo/engine-ring/target/release/hanzo serve
-m ~/models/qwen3-30b-meta --format gguf -f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf
-p 8080 --host 127.0.0.1 -n 0:48 --pa-context-len 131072 # -n 0:48 forces all 48 layers onto rocm[0] (auto-mapper misreads APU unified mem -> CPU offload = 4 tok/s trap) # --pa-context-len 131072 caps KV to ~24GB (default grabs 81GB -> only 7GB free) # engine-ring = hanzo 1.7.28 (ROCm-linked); ldd shows libamdhip. engine-rocm is 1.7.21.

# KNOWN evo-ONLY bug (ROCm/gfx1151, NOT config): the last prefill position's logits
# are slightly off, so the FIRST generated token's argmax flips on *borderline* prompts
# (e.g. "reply with only: pong" -> "system\npong"). Realistic agent/code prompts are
# clean (measured 7/8; only the contrived edge case leaked). Present in 1.7.21 AND 1.7.28,
# with Q4K_FALLBACK=1 and with --paged-attn off -> it's the ROCm forward pass, not a knob.
# Signature == the CUDA MoE topk "last-row collapse" bug fixed in ml 0.11.17; the HIP topk
# path needs the same fix. FIX PATH: port that fix to the ROCm MoE routing kernel + rebuild
# engine-rocm. Until then evo is a fast (60 tok/s) replica that occasionally glitches token 1.
# To exclude evo: drop its --replica from the router (spark+dbc alone = 2 clean replicas).

dbc (Metal M4 Max), on dbc: ~/work/hanzo/engine-metal/target/release/hanzo serve
-m /Users/a/models/qwen3-30b-meta --format gguf -f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf
-p 8080 --host 0.0.0.0 -n 0:48 --paged-attn on --pa-context-len 131072 # paged-attn is auto-OFF on Metal; force on so KV is bounded + block-shared

Router (on evo)

~/work/hanzo/engine/target/release/hanzo-router --host 0.0.0.0 --port 1234 \
  --model qwen3-30b-a3b --upstream-model default \
  --replica http://192.168.77.2:8080   `# spark, direct 2.5GbE link` \
  --replica http://127.0.0.1:8080      `# evo local` \
  --replica http://10.0.0.132:8080     `# dbc over LAN`

Admin: GET /v1/replicas (health+inflight), POST /v1/replicas (add), GET /health. Prefix-affinity (conversation sticks to one replica) + least-loaded spill + /health eviction/restore.

Measured (Qwen3-30B-A3B Q4_K_M, 256-tok decode)

single-box: spark(CUDA) 48 tok/s, evo(ROCm) 60, dbc(Metal) 72. Through router single-stream: 58-63 tok/s. 3 concurrent agents (1 per box): ~186 tok/s aggregate, each 62-68 tok/s = the parallel win (3x). Beyond 3, agents double up per box and per-agent decode drops (MoE: distinct experts per seq -> more weight reads/step; evo/APU worst because it also hosts the router/console and shares memory bandwidth). Spread across boxes is the design; the router does exactly this. Sweet spot = <=3 concurrent (one full-speed agent/GPU).

Net topology (from evo, where the router runs)

evo 10.0.0.144 / 192.168.77.1(->spark) ; spark 10.0.0.242 / 192.168.77.2 ; dbc 10.0.0.132.

Live PIDs (this session): spark serve 3895887 | evo serve 2920376 | dbc serve 21351 | router 2883009

All nohup+disown (survive session end). memguard.sh is a dead one-shot (already tripped); current config is memory-bounded, no guard needed. To kill a serve: kill -TERM <pid> by exact PID -- NEVER pkill -f "release/hanzo serve" (the pattern self-matches your own shell -> kills it).


Zen 5.8 / Qwen 3.8 Blackwell Cluster Topology (Live Production)

Upstream Engines

  • Spark (:30000): Blackwell GB10 running SGLang (hanzo-sglang.service):
    • Model: RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead
    • Flags: --host 0.0.0.0 --port 30000 --mem-fraction-static 0.65 --max-running-requests 8 --reasoning-parser qwen3 --tool-call-parser qwen3_coder
    • Radix KV cache with chunked prefill at ~1,700 tok/s, CUDA graph acceleration, native Anthropic /v1/messages tool_use parsing.
  • Evo (:8080): Strix Halo gfx1151 running llama-server:
    • Model: Qwen3.8-27B-Q6_K.gguf

Native Rust Router (:1235, on Spark)

  • Service: hanzo-router.service on Spark.
  • Binary: ~/.local/bin/hanzo-router, compiled with --features proxy.
  • Config: ~/.config/hanzo/router-pool.yaml
  • Features:
    • Models served: default, qwen3.8, zen-coder, zen5.8, zen5.8-coder, qwen/qwen3.8-27b.
    • Probes: /v1/models (Anthropic and OpenAI model discovery), /health (backend liveness). No /api/ path: Claude Code's fire-and-forget HEAD /api/hello warm-up passes through to a replica and 404s, and the client ignores the result.
    • Normalization: Auto-rewrites output_config.effort and reasoning_effort from "high"/"max" to "medium" to ensure chat templates with discrete thinking tiers (xhigh, medium, low) do not throw TemplateErrors.
    • Fallback: Unrecognized models (e.g. claude-*) automatically route to primary coder pool (zen5.8).

Claude Code Usage

Point Claude Code directly to the router on Spark:

export ANTHROPIC_BASE_URL="http://<spark>:1235"   # address in the private record
export ANTHROPIC_API_KEY="dummy"
claude

Or via the public cloud gateway https://api.hanzo.ai (routed to the AI Lab grant cluster).

Python Coderouter Decommissioning

  • Python coderouter.service has been permanently disabled and stopped on all cluster nodes.
  • Zero runtime Python router dependency; pure Rust hanzo-router handles all load balancing, streaming, session affinity, and wire normalization.
  • Mid-conversation non-leading system messages are normalized in src/proxy.rs (prefixed with [System]: as user role) and in SGLang chat templates, eliminating 400 Jinja errors from Claude Code / dev agents.
  • Public ingress routes on https://api.hanzo.ai (/v1/models, /v1/messages) route via Traefik to hanzo-router:1235.
  • Isolated Claude Code profile ~/.claude-hanzo configured; alias hanzo-code available in ~/.zshrc to prevent collisions between personal claude.ai OAuth subscriptions and custom API routing.
  • Sub-agent and role-based steering: hanzo-router infers agent roles from wire models and query sources. Sub-agents, titles, fast summaries, and haiku/flash requests automatically route to Evo (AMD Strix Halo APU, Qwen3.8-27B-Q6_K.gguf via Vulkan RADV with 300ms TTFT), while main agent implementation and opus/sonnet queries route to Spark (DGX Spark Blackwell GB10 NVFP4).