mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
Every mention of another inference engine is rewritten to say what our code does, or the math it computes. Reference values are "the reference value from the published config" or "the recorded reference output", line citations into another engine's source become the rule itself, comparisons with other engines are dropped, and asides about other engines' bugs go. Identifiers follow: - CUDA: hanzo-paged-attn's `namespace vllm` is `pagedattn`, hanzo-engine's MoE helpers `moe_dtype` and `moe_detail`, the rotary kernels' `hanzo_rotary`; the shuffle/ldg macros are HANZO_*. - GPTQ: the shuffled-layout path (`reconstruct_exllama`, `use_exllama`) is `reconstruct_shuffled` / `is_shuffled`, on the kernel and the FFI side. - Tests: `*_matches_vllm` and friends name what they compare against (the reference fixture, the tile rule). - MiniCPM-o's vision-LLM embedding is `vlm_embedding`; the rotary custom op is `rotary-embedding`. - hanzo-bench's board scores ours against a "rival" whose name comes from the manifest: `rival_<tag>.json`, `--rival*` manifest flags, `rival t/s` columns; dossier.sh and spec_decode.sh take `--rival-bench`. - install.ps1 installs this repository's hanzo-cli, as install.sh does, with HANZO_INSTALL_NCCL / HANZO_INSTALL_NO_NCCL. - bench_http.py reads only the hanzo: metric families. Kept, because they are what they are: LICENSE, LICENSE-APACHE, NOTICE and tools/upstream-map.sh; each ported file's copyright/licence/provenance header; Cargo.toml dependency names; data identifiers (Hugging Face repo IDs, fixture file names and fixture metadata keys the tests read, on-disk paths, executable names the GPU and engine guards match on a host); model and format names. LLM.md says the rule once, under "One name: Hanzo Engine". Co-authored-by: Hanzo Dev <[email protected]>
Hanzo Engine Web Chat App
Deprecated: The standalone
hanzo-web-chatbinary is deprecated. Usehanzo serve --uiinstead for the same functionality.Migration:
# Old cargo run --release --features cuda --bin hanzo-web-chat -- --text-model Qwen/Qwen3-4B # New hanzo serve --ui -m Qwen/Qwen3-4BThe new built-in UI provides the same features and is accessible at
/uiwhen running the server.
A minimal, fast, and modern web chat interface for Hanzo Engine, supporting text, multimodal, and speech models with drag-and-drop image and file upload, markdown rendering, and multi-model selection.
Features
- Multi-model Support: Choose from multiple loaded models. See multi-model docs for advanced configuration.
- Text-to-speech Model Support: Generate speech from text prompts and download as WAV files.
- Speech Input Model Support: Upload audio inputs for multimodal models.
- File Upload: Upload and work with files.
- Drag & Drop Image Upload: Instantly preview and send images to multimodal models.
- Markdown Output: Responses are rendered in markdown, including code blocks.
- Copy Button: One-click copying for all code snippets.
- Responsive UI: Clean layout for desktop and mobile.
- Generation Settings: Configure temperature, top_p, top_k, max_tokens, and repetition penalty directly in the UI.
- System Prompt: Set a custom system prompt for all conversations.
- Settings Persistence: User settings are saved to localStorage and persist across sessions.
- Web Search: Optional web search integration (when enabled on server).
- Keyboard Shortcuts:
Ctrl+Enter(orCmd+Enteron Mac) to send messages.- Button click also sends.
Quickstart
- Build the app.
Note: choose the features based on this guide.
cargo run --release --features <specify feature(s) here> --bin hanzo-web-chat -- \
--text-model Qwen/Qwen3-4B \
--multimodal-model google/gemma-4-E4B-it \
--speech-model nari-labs/Dia-1.6B
- At least one model is required (text, multimodal, or speech).
- Multiple
--text-model,--multimodal-model, or--speech-modelcan be specified. --portis optional (defaults to 1234).
- Access the app!
- Open http://localhost:1234 (or your chosen port).
CLI Options
Options:
--isq <ISQ> In-situ quantization to apply. Defaults to Q6K on CPU, AFQ6 on Metal.
Options: Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q8_1, Q2K, Q3K, Q4K, Q5K, Q6K, Q8K,
HQQ1, HQQ2, HQQ3, HQQ4, HQQ8, AFQ2, AFQ4, AFQ6, AFQ8
--text-model <MODEL> Text-only models (HuggingFace ID or local path). Can be repeated.
--multimodal-model <MODEL> Multimodal models (HuggingFace ID or local path). Can be repeated.
--speech-model <MODEL> Speech/TTS models (HuggingFace ID or local path). Can be repeated.
--enable-search Enable web search tool (requires embedding model)
--search-embedding-model <M> Built-in search embedding model (e.g., embedding_gemma)
-p, --port <PORT> Port to listen on (default: 1234)
--host <HOST> IP address to serve on (default: 0.0.0.0)
--cpu Use CPU only (disable GPU acceleration)
--temperature <TEMP> Default temperature for generation (0.0-2.0). Default: 0.7
--top-p <TOP_P> Default top_p for generation (0.0-1.0). Default: 0.9
--top-k <TOP_K> Default top_k for generation. Default: 40
--max-tokens <MAX> Default max tokens to generate. Default: 2048
--repetition-penalty <PEN> Default repetition penalty (1.0 = no penalty). Default: 1.1
--system-prompt <PROMPT> Default system prompt for all chats
-h, --help Print help
-V, --version Print version
Examples
Basic usage with a text model:
cargo run --release --features cuda --bin hanzo-web-chat -- \
--text-model meta-llama/Llama-3.2-3B-Instruct
With custom generation defaults:
cargo run --release --features cuda --bin hanzo-web-chat -- \
--text-model Qwen/Qwen3-4B \
--temperature 0.8 \
--max-tokens 4096 \
--system-prompt "You are a helpful coding assistant."
Multiple models with web search:
cargo run --release --features cuda --bin hanzo-web-chat -- \
--text-model Qwen/Qwen3-4B \
--multimodal-model google/gemma-4-E4B-it \
--enable-search \
--search-embedding-model embedding_gemma
API Endpoints
| Method | Endpoint | Description |
|---|---|---|
| GET | /ws |
WebSocket connection for streaming chat |
| GET | /v1/settings |
Get server default settings |
| GET | /v1/list_models |
List available models |
| POST | /v1/select_model |
Switch active model |
| GET | /v1/list_chats |
List saved chats |
| POST | /v1/new_chat |
Create new chat |
| POST | /v1/load_chat |
Load chat history |
| POST | /v1/delete_chat |
Delete chat |
| POST | /v1/rename_chat |
Rename chat |
| POST | /v1/upload_image |
Upload image (multimodal models) |
| POST | /v1/upload_text |
Upload text/code file |
| POST | /v1/upload_audio |
Upload audio file |
| POST | /v1/generate_speech |
Generate speech (TTS models) |
WebSocket Message Format
Messages sent to the WebSocket can include generation parameters:
{
"content": "Your message here",
"generation_params": {
"temperature": 0.8,
"top_p": 0.95,
"top_k": 50,
"max_tokens": 2048,
"repetition_penalty": 1.1
},
"web_search_options": {
"search_context_size": "medium"
}
}
To update the system prompt:
{
"set_system_prompt": "You are a helpful assistant."
}
Development
Frontend
- Edit static/index.html and supporting JS/CSS.
- Uses marked.js for markdown rendering.
- Settings stored in localStorage under
hanzo_settings.
Backend
- See main.rs for server and model loading logic.
- Models are managed via CLI arguments; hot reloading is not supported.
- Generation parameters can be configured via CLI and overridden per-message.