zenlm/zen-dub-live

Fork 0

mirror of https://github.com/zenlm/zen-dub-live.git synced 2026-10-30 14:58:12 +00:00

Files

T

zeekay e8c4b62d54 Add HuggingFace model card

2025-12-01 12:00:51 -08:00

4.3 KiB

Raw Permalink Blame History

license, tags, language, library_name, pipeline_tag

license

Zen-Dub-Live

Real-Time Speech-to-Speech Translation and Lip-Synchronized Video Dubbing

Zen-Dub-Live is a streaming pipeline that combines speech-to-speech translation with neural lip-sync rendering. The system achieves glass-to-glass latency of 2.5-3.5 seconds while maintaining broadcast-quality audio and video output.

Key Features

Ultra-Low Latency: 2.5-3.5 second end-to-end latency
Broadcast Quality: Production-ready audio and video output
Real-Time Lip-Sync: Neural VAE-based mouth manipulation
Streaming Architecture: Built for live broadcast workflows
Anchor Voices: Consistent voice identity across translations

Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                        Zen-Dub-Live Pipeline                            │
├─────────────────────────────────────────────────────────────────────────┤
│  Stage 1: Capture     →  Stage 2: Translate  →  Stage 3: Synthesize    │
│  (Audio + Video)         (Zen Omni S2S)         (Anchor Voice + TTS)   │
│                                                                         │
│                       →  Stage 4: Render                                │
│                          (Zen Dub Lip-Sync)                            │
└─────────────────────────────────────────────────────────────────────────┘

Components

Component	Model	Purpose
Translation	zen-omni	Speech-to-speech translation
Lip-Sync	zen-dub	Neural lip manipulation
VAD	Silero VAD	Voice activity detection

Installation

pip install zen-dub-live

Or with all dependencies:

pip install "zen-dub-live[all]"

Quick Start

Live Dubbing

# Dub webcam to RTMP stream
zen-dub-live live --source en --target es --output rtmp://localhost/live/stream

File Dubbing

# Dub a video file
zen-dub-live dub video.mp4 --source en --target es -o dubbed.mp4

Python API

import asyncio
from zen_dub_live import ZenDubLivePipeline, PipelineConfig

config = PipelineConfig(
    source_language="en",
    target_language="es",
)

pipeline = ZenDubLivePipeline(config)

async def main():
    async for segment in pipeline.run():
        print(f"Latency: {segment.latency:.2f}s")

asyncio.run(main())

Performance

Metric	Target	Achieved
E2E Latency	< 3.5s	2.4s
Video FPS	30+	32
Audio Quality	MOS 4.0+	4.1
Lip-Sync (LSE-D)	< 8.0	7.8

Hardware Requirements

Minimum (1080p @ 30fps):

NVIDIA RTX 4090 or A100 40GB
32GB RAM
100 Mbps network

Recommended (4K @ 60fps):

2x NVIDIA A100 80GB
64GB RAM
1 Gbps network

zen-omni - Multimodal translation model
zen-dub - Neural lip-sync model
zen-nano - Edge inference model

License

Apache 2.0

Citation

@software{zen_dub_live,
  title = {Zen-Dub-Live: Real-Time Speech-to-Speech Translation and Lip-Synchronized Video Dubbing},
  author = {Hanzo AI},
  year = {2025},
  url = {https://github.com/zenlm/zen-dub-live}
}

Hanzo AI | https://hanzo.ai | Techstars '17

4.3 KiB Raw Permalink Blame History