zen-vl/paper/outline.md

Zen VL: Vision-Language Models with Integrated Function Calling

Paper Outline

Abstract (150-200 words)

• Problem: Vision-language models lack native function calling capabilities
• Solution: Zen VL family with integrated tool use and visual agents
• Methodology: Fine-tuning Qwen3-VL with identity + function calling datasets
• Results: State-of-the-art visual agent performance across 4B/8B/30B scales
• Impact: First open VL models with native function calling

1. Introduction (~2 pages)

1.1 Motivation

• Vision-language models are powerful but lack agentic capabilities
• Gap between visual understanding and tool use
• Need for models that can extract function parameters from visual context

1.2 Contributions

1. Zen VL Family: Three scales (4B, 8B, 30B) of vision-language models
2. Function Calling: First open VL models with native tool use
3. Multimodal Identity: Novel fine-tuning methodology preserving identity across modalities
4. Benchmarks: Comprehensive evaluation on visual agent tasks
5. Open Source: All models, code, and datasets released

1.3 Paper Organization

Brief overview of sections

2. Related Work (~3 pages)

2.1 Vision-Language Models

• CLIP, BLIP, Flamingo
• LLaVA, InstructBLIP
• Qwen-VL, Qwen2-VL
• Qwen3-VL (our base)

2.2 Function Calling in LLMs

• GPT-3.5/4 function calling
• Gorilla, ToolLLM
• Agent frameworks (AutoGPT, BabyAGI)

2.3 Visual Agents

• WebAgent, Mind2Web
• GUI automation (CogAgent, SeeClick)
• Multimodal agents

2.4 Model Identity and Alignment

• Instruction fine-tuning
• Identity preservation
• Multimodal alignment

3. Methodology (~4 pages)

3.1 Base Architecture

• Qwen3-VL overview
• Interleaved-MRoPE
• DeepStack vision encoder
• Text-Timestamp Alignment

3.2 Training Pipeline

Stage 1: Identity Fine-tuning

• Text-only identity examples
• Visual identity examples
• Multimodal consistency

Stage 2: Function Calling

• Dataset construction
• Visual parameter extraction
• Structured output generation

Stage 3: Agent Optimization

• GUI interaction examples
• Tool use with visual context
• Multi-step task completion

3.3 Dataset Creation

Identity Dataset

- 100 text identity examples
- 50 visual capability examples
- 20 multimodal reasoning examples
Total: ~170 examples

Function Calling Dataset

- 50 image analysis function calls
- 30 GUI interaction examples
- 20 code generation from visuals
- 15 form filling examples
Total: ~115 examples (with augmentation: ~500+)

3.4 Training Details

• Hyperparameters
• Optimization strategy
• Hardware requirements
• Training time

4. Experimental Setup (~2 pages)

4.1 Models

• zen-vl-4b (4B params)
• zen-vl-8b (9B params)
• zen-vl-30b (31B params, MoE)

4.2 Baselines

• Qwen3-VL-Instruct (base)
• GPT-4V
• Claude 3.5 Sonnet
• Gemini 1.5 Pro

4.3 Benchmarks

Visual Understanding

• VQAv2, TextVQA
• COCO Captioning
• OCRBench

Visual Agents

• OSWorld (GUI interaction)
• Mind2Web (web navigation)
• Custom function calling benchmark

General Capability

• MMMU (multimodal understanding)
• MathVista (visual reasoning)
• MMBench

4.4 Evaluation Metrics

• Task success rate
• Function calling accuracy
• Parameter extraction F1
• Human evaluation (quality, safety)

5. Results (~4 pages)

5.1 Visual Understanding Performance

Table 1: VQA and OCR benchmarks

• zen-vl vs baselines
• Across model sizes

5.2 Function Calling Performance

Table 2: Function calling accuracy

• Tool selection accuracy
• Parameter extraction F1
• Structured output validity

Figure 1: Function calling examples

• Image → Function call visualization
• Parameter extraction accuracy

5.3 Visual Agent Performance

Table 3: OSWorld and Mind2Web results

• Task completion rate
• Step-by-step accuracy
• Error analysis

Figure 2: GUI interaction examples

• Screenshot → Action sequence
• Success/failure cases

5.4 Scaling Analysis

Figure 3: Performance vs model size

• 4B vs 8B vs 30B
• Different task categories
• Efficiency-performance tradeoff

5.5 Ablation Studies

Table 4: Ablation results

• Without identity fine-tuning
• Without function calling dataset
• Different training strategies

5.6 Qualitative Analysis

Figure 4: Example generations

• Identity consistency
• Function calling quality
• Error modes

6. Discussion (~2 pages)

6.1 Key Findings

• Identity fine-tuning is crucial for multimodal models
• Visual context significantly improves function calling
• Scaling benefits vary by task type

6.2 Comparison with Proprietary Models

• Competitive with GPT-4V on visual agent tasks
• Open weights enable customization
• Edge deployment for 4B model

6.3 Limitations

• Dataset size and diversity
• Video understanding depth
• Long-form visual reasoning
• Evaluation benchmark coverage

6.4 Broader Impact

Positive

• Open research in visual agents
• Accessibility of multimodal AI
• Edge deployment capabilities

Negative

• Potential misuse (deepfakes, surveillance)
• Bias in visual understanding
• Privacy concerns

7. Future Work (~1 page)

7.1 Technical Improvements

• Larger, more diverse datasets
• Video-native training
• Longer context windows
• Tool learning and discovery

7.2 New Capabilities

• Multi-image reasoning
• Temporal grounding
• 3D understanding
• Embodied agents

7.3 Open Research Questions

• Optimal identity-capability balance
• Visual reasoning emergence
• Tool use generalization

8. Conclusion (~0.5 pages)

• Summary of contributions
• Zen VL enables visual agent capabilities at multiple scales
• Open release accelerates research
• Future directions

---

Appendix

A. Model Details

• Full architecture specifications
• Hyperparameter tables
• Training curves

B. Dataset Examples

• Identity examples with images
• Function calling examples
• Tool definitions

C. Evaluation Details

• Benchmark descriptions
• Evaluation protocols
• Prompt templates

D. Additional Results

• Per-category breakdowns
• More ablations
• Error analysis

E. Ethical Considerations

• Dataset collection ethics
• Model safety testing
• Release guidelines

---

Figures & Tables Summary

Figures (6-8 total):

1. Function calling visualization
2. GUI interaction examples
3. Scaling analysis charts
4. Qualitative examples
5. (Optional) Training dynamics
6. (Optional) Attention visualizations

Tables (5-7 total):

1. VQA/OCR benchmarks
2. Function calling accuracy
3. Visual agent performance
4. Ablation study results
5. (Optional) Dataset statistics
6. (Optional) Hyperparameter comparison
7. (Optional) Compute requirements

---

Target Venue

Primary: NeurIPS 2025, ICLR 2026, CVPR 2026 Secondary: EMNLP 2025, ACL 2026 Workshops: Multimodal agents, Function calling, Vision-language

Estimated Length

• Main paper: 8-10 pages (NeurIPS/ICLR format)
• Appendix: 4-6 pages
• Total: 12-16 pages

Timeline

• Month 1: Complete all experiments
• Month 2: Draft sections 1-3
• Month 3: Draft sections 4-6
• Month 4: Results analysis, figures, tables
• Month 5: Writing refinement
• Month 6: Internal review, revisions
• Month 7: Submit to venue

Collaborators

• Hanzo AI Team
• (Optional) Academic collaborators
• (Optional) Qwen team acknowledgment