Hanzo Edge - On-Device AI Inference Runtime
Category: Hanzo Ecosystem
Related Skills: hanzo/hanzo-engine.md, hanzo/hanzo-llm-gateway.md, hanzo/hanzo-candle.md, hanzo/zenlm.md
Overview
Hanzo Edge is an on-device AI inference runtime for running Zen models and any GGUF model locally on macOS, Linux, Web (WASM), and embedded devices. Zero cloud dependency, full data privacy, zero network latency, works completely offline. Rust workspace with three crates. Live at edge.hanzo.ai.
Why Hanzo Edge?
- On-device: Data never leaves the device, works offline
- Cross-platform: macOS (Metal), Linux (CPU/CUDA), Web (WASM), embedded (ARM)
- OpenAI-compatible: Local API server as drop-in replacement
- GGUF native: First-class quantized model support (Q4_K, Q5_K, Q8_0)
- Streaming: Token-by-token via SSE and callbacks
- HuggingFace Hub: Automatic model download and caching
Tech Stack
- Language: Rust (edition 2021)
- ML Backend: Candle (candle-core, candle-nn, candle-transformers v0.9)
- HTTP Server: Axum 0.7
- CLI: Clap 4
- WASM: wasm-bindgen 0.2, web-sys 0.3
- Tokenizer: HuggingFace tokenizers 0.20
- Async: Tokio 1
OSS Base
Repo: hanzoai/edge. Built on Hanzo ML (Candle fork) for tensor operations.
When to use
- Running AI models locally with full data privacy
- Mobile, embedded, or offline inference
- In-browser AI via WebAssembly
- Low-latency inference without network round-trips
- Prototyping with local OpenAI-compatible API
Edge vs Engine
|
Edge |
Engine |
| Where |
On-device (local CPU/GPU) |
Cloud GPU clusters |
| Latency |
Zero network overhead |
Network round-trip |
| Privacy |
Data never leaves device |
Data sent to cloud |
| Models |
Quantized GGUF (Q4/Q5/Q8) |
Full-precision (FP16/BF16) |
| Best for |
Mobile, embedded, offline, privacy |
Production serving, large models, scale |
Hard requirements
- Rust toolchain (stable)
- wasm-pack for WASM builds
- CUDA toolkit for NVIDIA GPU acceleration (optional)
- Xcode Command Line Tools for Metal on macOS (optional)
Quick reference
| Item |
Value |
| Docs |
https://edge.hanzo.ai |
| Crates.io |
hanzo-edge, hanzo-edge-core, hanzo-edge-wasm |
| Image |
ghcr.io/hanzoai/edge:latest |
| API Port |
8080 (local serve) |
| License |
Apache-2.0 |
| Repo |
github.com/hanzoai/edge |
One-file quickstart
Run a model locally
Docker
Core Concepts
Architecture
Crates
| Crate |
Description |
Install |
hanzo-edge-core |
Core inference runtime and Model trait |
cargo add hanzo-edge-core |
hanzo-edge |
CLI binary with run, serve, bench, info |
cargo install hanzo-edge |
hanzo-edge-wasm |
Browser WASM module with streaming |
wasm-pack build edge-wasm |
Feature Flags
| Feature |
Description |
Build |
cpu |
CPU backend (default) |
cargo build --release |
metal |
Metal backend for macOS/iOS |
cargo build --release --features metal |
cuda |
CUDA backend for NVIDIA GPUs |
cargo build --release --features cuda |
CLI Subcommands
API Server Endpoints
| Method |
Path |
Description |
POST |
/v1/chat/completions |
Chat completion (streaming + non-streaming) |
POST |
/v1/completions |
Text completion |
GET |
/v1/models |
List loaded models |
GET |
/health |
Health check |
Rust SDK Usage
WASM (Browser)
Zen Models for Edge
| Model |
Params |
Quantized Size |
Use Case |
zenlm/zen3-nano |
600M |
~400MB (Q4_K_M) |
Ultra-lightweight, embedded, IoT |
zenlm/zen-eco |
4B |
~2.5GB (Q4_K_M) |
General purpose, mobile, tablets |
zenlm/zen4-mini |
8B |
~5GB (Q4_K_M) |
High quality, desktop and laptop |
Building from Source
Platform Support
| Platform |
Backend |
Status |
| macOS (Apple Silicon) |
Metal |
Production |
| macOS (Intel) |
CPU / Accelerate |
Production |
| Linux x86_64 |
CPU |
Production |
| Linux x86_64 |
CUDA |
Production |
| Linux ARM64 |
CPU |
Production |
| Web (WASM) |
CPU |
Stable |
| iOS |
Metal / CoreML |
Planned |
| Android |
Vulkan / NNAPI |
Planned |
Related Skills
hanzo/hanzo-engine.md - Cloud GPU inference (complementary to Edge)
hanzo/hanzo-llm-gateway.md - Unified LLM proxy for 100+ providers
hanzo/hanzo-candle.md - Rust ML framework (tensor ops, used by Edge)
hanzo/zenlm.md - Zen model family
Last Updated: 2026-03-13
Category: Hanzo Ecosystem
Related: edge, inference, wasm, on-device, gguf, rust
Prerequisites: Rust toolchain, GGUF model understanding