mirror of
https://github.com/hanzoai/engine.git
synced 2026-11-20 14:17:52 +00:00
google/embeddinggemma-2 (EmbeddingGemma2Model, Apache-2.0) loads as `embeddinggemma2`, from its config's architecture or by name: the text backbone of transformers 5.19, with Gemma 4 norms, a value norm with no scale, attention scale 1, full layers at head dim 512 and one KV head, an inclusive sliding window, projection-only per-layer embeddings, a per-layer scalar and the 512 -> 768 projection on every token. The vision and audio towers are never read. Against sentence-transformers 6.1.0 in F32, over 64 texts in 14 languages up to 1,256 tokens, every vector's cosine is at least 0.9999993 at 768, 512, 256 and 128 dimensions, and every token count matches. To serve it: - Pooling reads sentence-transformers 6's config (embedding_dimension, pooling_mode) beside the older flags. Mean pooling is a mean, where it was a sum; every sentence-transformers mode pools, where all but mean and last-token panicked; and what it cannot do fails at load. - /v1/embeddings takes `dimensions`: the leading values, normalized again, as OpenAI and truncate_dim do. - A multi-model entry takes `revision`, so a deployment pins the weights its indexes were built from. Claude-Session: https://claude.ai/code/session_01SCPMGSQsLNF2YtHFL4bwyp
Hanzo Engine
Fast, flexible LLM inference. Run 40+ model families (text, vision, video, audio, speech, image generation, and embeddings) with automatic hardware optimization, OpenAI-compatible APIs, and built-in agentic features.
Quick Start
# Install (Linux/macOS, auto-detects CUDA, Metal, MKL)
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/hanzoai/engine/master/install.sh | sh
# Install (Windows PowerShell)
irm https://raw.githubusercontent.com/hanzoai/engine/master/install.ps1 | iex
# Run a model interactively
hanzo run --isq 4 -m Qwen/Qwen3-4B
# Or serve it as an OpenAI-compatible API with a web UI
hanzo serve --ui --isq 4 -m Qwen/Qwen3-4B
New to Hanzo Engine? Follow the Getting Started tutorial for a guided walkthrough.
Choose Your Path
| I want to... | Start here |
|---|---|
| Follow a step-by-step tutorial | Getting Started |
| Run a model from the command line | CLI Reference |
| Serve models over HTTP | HTTP Server |
| Build a Python application | Python SDK |
| Build a Rust application | Rust SDK |
| Optimize performance and memory | Performance Guide |
| Choose a quantization method | Quantization Overview |
| Use agentic features (tools, search, MCP) | Agentic Features Guide |
| Find a specific model | Supported Models |
| Troubleshoot an issue | Troubleshooting |
Highlights
- Zero config:
hanzo run -m <model>auto-detects architecture, chat template, and hardware - Quantization control: ISQ, GGUF, GPTQ, AWQ, HQQ, AFQ, FP8, MXFP4, or just
--isq 4 - Auto-tuning:
hanzo tunebenchmarks your hardware and recommends optimal settings - Agentic: Server-side tool calling loop, web search, MCP client, HTTP tool dispatch
- Multi-GPU: Automatic tensor parallelism via NCCL, or cross-machine via Ring backend
- Web UI:
--uiflag for instant browser-based chat