Files
zeekay 183a461e09 feat(embeddings): serve EmbeddingGemma 2
google/embeddinggemma-2 (EmbeddingGemma2Model, Apache-2.0) loads as
`embeddinggemma2`, from its config's architecture or by name: the text
backbone of transformers 5.19, with Gemma 4 norms, a value norm with no
scale, attention scale 1, full layers at head dim 512 and one KV head,
an inclusive sliding window, projection-only per-layer embeddings, a
per-layer scalar and the 512 -> 768 projection on every token.  The
vision and audio towers are never read.

Against sentence-transformers 6.1.0 in F32, over 64 texts in 14
languages up to 1,256 tokens, every vector's cosine is at least
0.9999993 at 768, 512, 256 and 128 dimensions, and every token count
matches.

To serve it:

- Pooling reads sentence-transformers 6's config (embedding_dimension,
  pooling_mode) beside the older flags.  Mean pooling is a mean, where
  it was a sum; every sentence-transformers mode pools, where all but
  mean and last-token panicked; and what it cannot do fails at load.
- /v1/embeddings takes `dimensions`: the leading values, normalized
  again, as OpenAI and truncate_dim do.
- A multi-model entry takes `revision`, so a deployment pins the
  weights its indexes were built from.

Claude-Session: https://claude.ai/code/session_01SCPMGSQsLNF2YtHFL4bwyp
2026-10-08 20:17:40 -07:00
..
2026-09-15 09:42:19 -07:00

Hanzo Engine

Fast, flexible LLM inference. Run 40+ model families (text, vision, video, audio, speech, image generation, and embeddings) with automatic hardware optimization, OpenAI-compatible APIs, and built-in agentic features.

Quick Start

# Install (Linux/macOS, auto-detects CUDA, Metal, MKL)
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/hanzoai/engine/master/install.sh | sh

# Install (Windows PowerShell)
irm https://raw.githubusercontent.com/hanzoai/engine/master/install.ps1 | iex

# Run a model interactively
hanzo run --isq 4 -m Qwen/Qwen3-4B

# Or serve it as an OpenAI-compatible API with a web UI
hanzo serve --ui --isq 4 -m Qwen/Qwen3-4B

New to Hanzo Engine? Follow the Getting Started tutorial for a guided walkthrough.

Choose Your Path

I want to... Start here
Follow a step-by-step tutorial Getting Started
Run a model from the command line CLI Reference
Serve models over HTTP HTTP Server
Build a Python application Python SDK
Build a Rust application Rust SDK
Optimize performance and memory Performance Guide
Choose a quantization method Quantization Overview
Use agentic features (tools, search, MCP) Agentic Features Guide
Find a specific model Supported Models
Troubleshoot an issue Troubleshooting

Highlights

  • Zero config: hanzo run -m <model> auto-detects architecture, chat template, and hardware
  • Quantization control: ISQ, GGUF, GPTQ, AWQ, HQQ, AFQ, FP8, MXFP4, or just --isq 4
  • Auto-tuning: hanzo tune benchmarks your hardware and recommends optimal settings
  • Agentic: Server-side tool calling loop, web search, MCP client, HTTP tool dispatch
  • Multi-GPU: Automatic tensor parallelism via NCCL, or cross-machine via Ring backend
  • Web UI: --ui flag for instant browser-based chat

Community