mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
LLM.md section 3b quoted numbers from a script that lived in a home directory and counted chunks, not tokens. Under MTP or DFlash one chunk carries several tokens, and vLLM v0.29 streams thinking as `reasoning` where hanzo sends `reasoning_content`, so chunk counting measured steps and missed vLLM's thinking entirely. bench_http.py counts token chunks by content, reasoning, reasoning_content or tool_calls, takes decode tokens from usage (a stream without usage fails), and reports TTFT, decode rate, per-step gaps and a concurrent aggregate as JSON lines. It also carries the /metrics check and the profile window used by later tests. bytes_per_token.py gives two labeled numbers: the computed weight, KV and state bytes per token from safetensors headers (19.44 GB of 27B weights, matching SGLang's figure), and the measured SM-to-L2 bytes from an nsys gb20y-top export. GB10 has no DRAM counter, so the measured figure is a cross-check, never a substitute. Claude-Session: https://claude.ai/code/session_019ABm6hF8RNyUFpbjQWqfxY