mirror of
https://github.com/zenlm/zen-docs.git
synced 2026-10-30 14:58:13 +00:00
zeekay
8ef8f3e801
Updated 8 existing stubs and added 4 new model pages: - zen-nano: 0.6B edge model specs, throughput table, benchmarks - zen-scribe: 4B writing model, multilingual, use cases - zen-omni: 30B hypermodal Thinker-Talker MoE, speech-to-speech - zen-vl: 4B/8B/30B VL family with function calling and agent variants - zen-embedding: MTEB #1 multilingual (70.58), MRL support - zen-coder: 4B/31B/480B MoE family, FIM, 358 languages - zen-designer: 235B VL MoE, document parsing, GUI interaction - zen-max: 1T+ MoE, 256K context, 200+ tool calls, heavy mode - zen-pro: 8B instruct/thinking/agent variants - zen-reranker: 8B dual-task, native 7680-dim for DSO - zen-translator: real-time translation pipeline with voice cloning - zen-guard: removed upstream model reference No upstream model names in any page (Zen MoDE branding throughout).
115 lines
3.6 KiB
Plaintext
115 lines
3.6 KiB
Plaintext
---
|
|
title: Zen Omni
|
|
description: 30B hypermodal model with text, vision, and audio understanding plus speech synthesis
|
|
---
|
|
|
|
# Zen Omni
|
|
|
|
Zen Omni is a 30B hypermodal language model from the Zen MoDE family, built on a Thinker-Talker MoE architecture. With 3B active parameters via sparse activation, it supports 119 languages for text, 19 languages for speech input, and 10 languages for speech output — enabling real-time speech-to-speech translation, multimodal understanding, and audio generation.
|
|
|
|
## Specifications
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| Architecture | Zen MoDE Thinker-Talker (MoE) |
|
|
| Total Parameters | 30B |
|
|
| Active Parameters | 3B (MoE sparse activation) |
|
|
| Context Length | 32,768 tokens |
|
|
| Text Languages | 119 |
|
|
| Speech Input | 19 languages |
|
|
| Speech Output | 10 languages |
|
|
| License | Apache 2.0 |
|
|
|
|
## Model Variants
|
|
|
|
| Variant | Description |
|
|
|---------|-------------|
|
|
| zen-omni | Base multimodal model |
|
|
| zen-omni-instruct | Instruction-following for chat and tasks |
|
|
| zen-omni-thinking | Extended chain-of-thought reasoning |
|
|
| zen-omni-captioner | Audio and visual captioning |
|
|
|
|
## Architecture
|
|
|
|
Zen Omni uses a Thinker-Talker design:
|
|
|
|
- **Input Encoders** — Audio encoder (32 layers, 1280 dim), Vision encoder (27 layers, 1152 dim), Text embeddings (151,936 vocab)
|
|
- **Thinker** — 48 transformer layers, 128 MoE experts, 8 active per token, cross-modal attention fusion
|
|
- **Talker** — Streaming speech synthesis via Code2Wav codec (16 quantizers, 2048 codebook, 24kHz)
|
|
|
|
## Quick Start
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoProcessor
|
|
|
|
model_id = "zenlm/zen-omni"
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
model_id,
|
|
torch_dtype="auto",
|
|
device_map="auto"
|
|
)
|
|
processor = AutoProcessor.from_pretrained(model_id)
|
|
|
|
messages = [
|
|
{"role": "system", "content": "You are Zen, a helpful AI assistant."},
|
|
{"role": "user", "content": "Explain quantum computing in simple terms."}
|
|
]
|
|
|
|
text = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
|
|
inputs = processor(text=text, return_tensors="pt").to(model.device)
|
|
outputs = model.generate(**inputs, max_new_tokens=512)
|
|
response = processor.decode(outputs[0], skip_special_tokens=True)
|
|
print(response)
|
|
```
|
|
|
|
## Speech-to-Speech Translation
|
|
|
|
```python
|
|
import librosa
|
|
import soundfile as sf
|
|
|
|
source_audio, sr = librosa.load("japanese_speech.wav", sr=16000)
|
|
|
|
messages = [
|
|
{"role": "user", "content": [
|
|
{"type": "audio", "audio": source_audio},
|
|
{"type": "text", "text": "Translate this Japanese speech to English and speak the translation."}
|
|
]}
|
|
]
|
|
|
|
inputs = processor(messages, return_tensors="pt").to(model.device)
|
|
outputs = model.generate(**inputs, max_new_tokens=2048, return_audio=True)
|
|
|
|
sf.write("english_translation.wav", outputs.audio[0], 24000)
|
|
```
|
|
|
|
## Use Cases
|
|
|
|
- Real-time speech-to-speech translation across 19+ languages
|
|
- Multimodal question answering (image, audio, text)
|
|
- Audio captioning and transcription
|
|
- Complex reasoning with extended thinking mode
|
|
- Video dubbing pipeline (integrates with zen-dub)
|
|
|
|
## Formats
|
|
|
|
| Format | Size | RAM Required |
|
|
|--------|------|-------------|
|
|
| SafeTensors (BF16) | ~60 GB | 80 GB+ |
|
|
| MLX 4-bit | ~15 GB | 20 GB |
|
|
| GGUF Q4_K_M | ~15 GB | 20 GB |
|
|
|
|
## Performance
|
|
|
|
| Benchmark | Score |
|
|
|-----------|-------|
|
|
| Speech Translation (BLEU) | 42.3 (En/Ja bidirectional) |
|
|
| Image Understanding (VQA) | 78.2% |
|
|
| Audio Transcription (WER) | 4.2% |
|
|
| Cross-Modal Reasoning | 85.1% (MMLU multimodal) |
|
|
|
|
## Get Started
|
|
|
|
- [HuggingFace](https://huggingface.co/zenlm/zen-omni)
|
|
- [GitHub](https://github.com/zenlm/zen-omni)
|