mirror of
https://github.com/zenlm/zen-docs.git
synced 2026-10-30 14:58:13 +00:00
zeekay
8ef8f3e801
Updated 8 existing stubs and added 4 new model pages: - zen-nano: 0.6B edge model specs, throughput table, benchmarks - zen-scribe: 4B writing model, multilingual, use cases - zen-omni: 30B hypermodal Thinker-Talker MoE, speech-to-speech - zen-vl: 4B/8B/30B VL family with function calling and agent variants - zen-embedding: MTEB #1 multilingual (70.58), MRL support - zen-coder: 4B/31B/480B MoE family, FIM, 358 languages - zen-designer: 235B VL MoE, document parsing, GUI interaction - zen-max: 1T+ MoE, 256K context, 200+ tool calls, heavy mode - zen-pro: 8B instruct/thinking/agent variants - zen-reranker: 8B dual-task, native 7680-dim for DSO - zen-translator: real-time translation pipeline with voice cloning - zen-guard: removed upstream model reference No upstream model names in any page (Zen MoDE branding throughout).
107 lines
3.6 KiB
Plaintext
107 lines
3.6 KiB
Plaintext
---
|
|
title: Zen VL
|
|
description: Vision-language model family (4B, 8B, 30B) with native function calling and visual agent capabilities
|
|
---
|
|
|
|
# Zen VL
|
|
|
|
Zen VL is a family of vision-language models built on the Zen VL architecture, available at three scales. All variants include vision understanding, OCR across 32 languages, and multimodal reasoning. Agent variants add native function calling for GUI interaction, web navigation, and structured output extraction from images.
|
|
|
|
## Model Family
|
|
|
|
| Model | Parameters | Context | Type |
|
|
|-------|------------|---------|------|
|
|
| [zen-vl-4b-instruct](https://huggingface.co/zenlm/zen-vl-4b-instruct) | 4B | 256K | Base VL |
|
|
| [zen-vl-4b-agent](https://huggingface.co/zenlm/zen-vl-4b-agent) | 4B | 256K | VL + Function Calling |
|
|
| [zen-vl-8b-instruct](https://huggingface.co/zenlm/zen-vl-8b-instruct) | 8B | 256K | Base VL |
|
|
| [zen-vl-8b-agent](https://huggingface.co/zenlm/zen-vl-8b-agent) | 8B | 256K | VL + Function Calling |
|
|
| [zen-vl-30b-instruct](https://huggingface.co/zenlm/zen-vl-30b-instruct) | 31B MoE | 256K | Base VL |
|
|
| [zen-vl-30b-agent](https://huggingface.co/zenlm/zen-vl-30b-agent) | 31B MoE | 256K | VL + Function Calling |
|
|
|
|
## Capabilities
|
|
|
|
- **Visual understanding** — Image analysis, video comprehension, spatial reasoning
|
|
- **OCR** — 32 languages and writing systems
|
|
- **Multimodal reasoning** — STEM, math, code from images
|
|
- **Function calling** (agent variants) — Tool use with visual context, structured JSON output
|
|
- **GUI interaction** — Web navigation, element recognition, parameter extraction from screenshots
|
|
|
|
## Quick Start
|
|
|
|
```python
|
|
from transformers import AutoModelForVision2Seq, AutoProcessor
|
|
from PIL import Image
|
|
import torch
|
|
|
|
model = AutoModelForVision2Seq.from_pretrained(
|
|
"zenlm/zen-vl-4b-instruct",
|
|
torch_dtype=torch.bfloat16,
|
|
device_map="auto",
|
|
trust_remote_code=True
|
|
)
|
|
processor = AutoProcessor.from_pretrained(
|
|
"zenlm/zen-vl-4b-instruct",
|
|
trust_remote_code=True
|
|
)
|
|
|
|
image = Image.open("path/to/image.jpg")
|
|
|
|
messages = [
|
|
{"role": "user", "content": "Describe this image in detail."}
|
|
]
|
|
|
|
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
|
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
|
|
|
|
outputs = model.generate(**inputs, max_new_tokens=300)
|
|
print(processor.batch_decode(outputs, skip_special_tokens=True)[0])
|
|
```
|
|
|
|
## Function Calling (Agent Models)
|
|
|
|
```python
|
|
model = AutoModelForVision2Seq.from_pretrained(
|
|
"zenlm/zen-vl-4b-agent",
|
|
torch_dtype=torch.bfloat16,
|
|
device_map="auto",
|
|
trust_remote_code=True
|
|
)
|
|
|
|
image = Image.open("screenshot.png")
|
|
|
|
messages = [
|
|
{"role": "user", "content": "Click the login button in this screenshot."}
|
|
]
|
|
|
|
# Model outputs structured function call:
|
|
# {
|
|
# "thinking": "I see a login button at coordinates...",
|
|
# "function_call": {
|
|
# "name": "click_element",
|
|
# "arguments": {"x": 120, "y": 200, "element_type": "button"}
|
|
# }
|
|
# }
|
|
```
|
|
|
|
## Use Cases
|
|
|
|
- Document parsing (PDFs, invoices, forms, charts)
|
|
- Visual question answering
|
|
- GUI automation and web navigation
|
|
- Code generation from UI screenshots
|
|
- Multimodal agent workflows
|
|
|
|
## Architecture
|
|
|
|
Zen VL uses the Zen VL architecture with:
|
|
|
|
- **DeepStack** — Hierarchical vision encoder for fine-grained detail
|
|
- **Interleaved-MRoPE** — Multi-resolution position encoding for video
|
|
- **Text-Timestamp Alignment** — Temporal event localization in video
|
|
- **256K context** (expandable to 1M)
|
|
|
|
## Get Started
|
|
|
|
- [HuggingFace](https://huggingface.co/zenlm/zen-vl-4b-instruct)
|
|
- [GitHub](https://github.com/zenlm/zen-vl)
|