Files
zeekayandHanzo Dev 42e34c2d41 refactor: name only Hanzo Engine in code, comments and docs
Every mention of another inference engine is rewritten to say what our code
does, or the math it computes. Reference values are "the reference value
from the published config" or "the recorded reference output", line
citations into another engine's source become the rule itself, comparisons
with other engines are dropped, and asides about other engines' bugs go.

Identifiers follow:
- CUDA: hanzo-paged-attn's `namespace vllm` is `pagedattn`, hanzo-engine's
  MoE helpers `moe_dtype` and `moe_detail`, the rotary kernels'
  `hanzo_rotary`; the shuffle/ldg macros are HANZO_*.
- GPTQ: the shuffled-layout path (`reconstruct_exllama`, `use_exllama`) is
  `reconstruct_shuffled` / `is_shuffled`, on the kernel and the FFI side.
- Tests: `*_matches_vllm` and friends name what they compare against (the
  reference fixture, the tile rule).
- MiniCPM-o's vision-LLM embedding is `vlm_embedding`; the rotary custom op
  is `rotary-embedding`.
- hanzo-bench's board scores ours against a "rival" whose name comes from
  the manifest: `rival_<tag>.json`, `--rival*` manifest flags, `rival t/s`
  columns; dossier.sh and spec_decode.sh take `--rival-bench`.
- install.ps1 installs this repository's hanzo-cli, as install.sh does,
  with HANZO_INSTALL_NCCL / HANZO_INSTALL_NO_NCCL.
- bench_http.py reads only the hanzo: metric families.

Kept, because they are what they are: LICENSE, LICENSE-APACHE, NOTICE and
tools/upstream-map.sh; each ported file's copyright/licence/provenance
header; Cargo.toml dependency names; data identifiers (Hugging Face repo
IDs, fixture file names and fixture metadata keys the tests read, on-disk
paths, executable names the GPU and engine guards match on a host); model
and format names.

LLM.md says the rule once, under "One name: Hanzo Engine".


Co-authored-by: Hanzo Dev <[email protected]>
2026-10-16 20:59:50 -07:00
..
2026-09-24 15:01:47 -07:00
2026-09-24 15:01:47 -07:00

Hanzo Engine Web Chat App

Deprecated: The standalone hanzo-web-chat binary is deprecated. Use hanzo serve --ui instead for the same functionality.

Migration:

# Old
cargo run --release --features cuda --bin hanzo-web-chat -- --text-model Qwen/Qwen3-4B

# New
hanzo serve --ui -m Qwen/Qwen3-4B

The new built-in UI provides the same features and is accessible at /ui when running the server.

A minimal, fast, and modern web chat interface for Hanzo Engine, supporting text, multimodal, and speech models with drag-and-drop image and file upload, markdown rendering, and multi-model selection.

Demonstration

Features

  • Multi-model Support: Choose from multiple loaded models. See multi-model docs for advanced configuration.
  • Text-to-speech Model Support: Generate speech from text prompts and download as WAV files.
  • Speech Input Model Support: Upload audio inputs for multimodal models.
  • File Upload: Upload and work with files.
  • Drag & Drop Image Upload: Instantly preview and send images to multimodal models.
  • Markdown Output: Responses are rendered in markdown, including code blocks.
  • Copy Button: One-click copying for all code snippets.
  • Responsive UI: Clean layout for desktop and mobile.
  • Generation Settings: Configure temperature, top_p, top_k, max_tokens, and repetition penalty directly in the UI.
  • System Prompt: Set a custom system prompt for all conversations.
  • Settings Persistence: User settings are saved to localStorage and persist across sessions.
  • Web Search: Optional web search integration (when enabled on server).
  • Keyboard Shortcuts:
    • Ctrl+Enter (or Cmd+Enter on Mac) to send messages.
    • Button click also sends.

Quickstart

  1. Build the app.

Note: choose the features based on this guide.

cargo run --release --features <specify feature(s) here> --bin hanzo-web-chat -- \
  --text-model Qwen/Qwen3-4B \
  --multimodal-model google/gemma-4-E4B-it \
  --speech-model nari-labs/Dia-1.6B
  • At least one model is required (text, multimodal, or speech).
  • Multiple --text-model, --multimodal-model, or --speech-model can be specified.
  • --port is optional (defaults to 1234).

  1. Access the app!

CLI Options

Options:
  --isq <ISQ>                    In-situ quantization to apply. Defaults to Q6K on CPU, AFQ6 on Metal.
                                 Options: Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q8_1, Q2K, Q3K, Q4K, Q5K, Q6K, Q8K,
                                          HQQ1, HQQ2, HQQ3, HQQ4, HQQ8, AFQ2, AFQ4, AFQ6, AFQ8
  --text-model <MODEL>           Text-only models (HuggingFace ID or local path). Can be repeated.
  --multimodal-model <MODEL>         Multimodal models (HuggingFace ID or local path). Can be repeated.
  --speech-model <MODEL>         Speech/TTS models (HuggingFace ID or local path). Can be repeated.
  --enable-search                Enable web search tool (requires embedding model)
  --search-embedding-model <M>   Built-in search embedding model (e.g., embedding_gemma)
  -p, --port <PORT>              Port to listen on (default: 1234)
  --host <HOST>                  IP address to serve on (default: 0.0.0.0)
  --cpu                          Use CPU only (disable GPU acceleration)
  --temperature <TEMP>           Default temperature for generation (0.0-2.0). Default: 0.7
  --top-p <TOP_P>                Default top_p for generation (0.0-1.0). Default: 0.9
  --top-k <TOP_K>                Default top_k for generation. Default: 40
  --max-tokens <MAX>             Default max tokens to generate. Default: 2048
  --repetition-penalty <PEN>     Default repetition penalty (1.0 = no penalty). Default: 1.1
  --system-prompt <PROMPT>       Default system prompt for all chats
  -h, --help                     Print help
  -V, --version                  Print version

Examples

Basic usage with a text model:

cargo run --release --features cuda --bin hanzo-web-chat -- \
  --text-model meta-llama/Llama-3.2-3B-Instruct

With custom generation defaults:

cargo run --release --features cuda --bin hanzo-web-chat -- \
  --text-model Qwen/Qwen3-4B \
  --temperature 0.8 \
  --max-tokens 4096 \
  --system-prompt "You are a helpful coding assistant."

Multiple models with web search:

cargo run --release --features cuda --bin hanzo-web-chat -- \
  --text-model Qwen/Qwen3-4B \
  --multimodal-model google/gemma-4-E4B-it \
  --enable-search \
  --search-embedding-model embedding_gemma

API Endpoints

Method Endpoint Description
GET /ws WebSocket connection for streaming chat
GET /v1/settings Get server default settings
GET /v1/list_models List available models
POST /v1/select_model Switch active model
GET /v1/list_chats List saved chats
POST /v1/new_chat Create new chat
POST /v1/load_chat Load chat history
POST /v1/delete_chat Delete chat
POST /v1/rename_chat Rename chat
POST /v1/upload_image Upload image (multimodal models)
POST /v1/upload_text Upload text/code file
POST /v1/upload_audio Upload audio file
POST /v1/generate_speech Generate speech (TTS models)

WebSocket Message Format

Messages sent to the WebSocket can include generation parameters:

{
  "content": "Your message here",
  "generation_params": {
    "temperature": 0.8,
    "top_p": 0.95,
    "top_k": 50,
    "max_tokens": 2048,
    "repetition_penalty": 1.1
  },
  "web_search_options": {
    "search_context_size": "medium"
  }
}

To update the system prompt:

{
  "set_system_prompt": "You are a helpful assistant."
}

Development

Frontend

  • Edit static/index.html and supporting JS/CSS.
  • Uses marked.js for markdown rendering.
  • Settings stored in localStorage under hanzo_settings.

Backend

  • See main.rs for server and model loading logic.
  • Models are managed via CLI arguments; hot reloading is not supported.
  • Generation parameters can be configured via CLI and overridden per-message.