video/tests/qa/QA_PLAN.md

Quality Validation Plan — Phase 3.5 + G3.11

Purpose

Run every tool with real API keys, inspect outputs (see images, listen to audio, watch video), find gaps, fix them. This is the gate before calling Phase 3.5 "Verified."

Test Scripts (all ready to run)

Script	Tools Tested	API Keys Used	Est. Cost
test_01_tts.py	elevenlabs_tts (ElevenLabs)	ELEVENLABS_API_KEY	~$0.02
test_02_image_gen.py	image_gen (DALL-E 3 + FLUX)	OPENAI_API_KEY, FAL_AI_API_KEY	~$0.15
test_03_music.py	music_gen (ElevenLabs)	ELEVENLABS_API_KEY	~$0.10
test_04_audio_mix.py	audio_mixer	None (ffmpeg only)	$0
test_05_video_compose.py	video_compose	None (ffmpeg only)	$0
test_06_video_stitch.py	video_stitch	None (ffmpeg only)	$0
test_07_playbook_intelligence.py	playbook_loader.py functions	None (pure Python)	$0
test_08_end_to_end.py	Full animated-explainer pipeline	None (ffmpeg fixtures)	$0

Inspection Protocol

For each output:

1. Audio files: Use ffprobe for format/duration/channels, then LISTEN (play in media player or use Whisper to verify content matches prompt)
2. Image files: Use ffprobe for dimensions, then VIEW (open image, check composition, text readability, style match)
3. Video files: Use ffprobe for resolution/fps/duration/codec, then WATCH (check A/V sync, transitions, subtitle timing)
4. Design intelligence: Run against all 3 playbooks, verify contrast ratios match manual calculation, check CVD warnings are accurate

Known Risk Areas

Area	Risk	How to Validate
TTS voice selection	Default voice may not match playbook mood	Test with multiple voice IDs, compare against playbook voice_style
Image gen consistency	DALL-E/FLUX outputs vary wildly per prompt	Test with playbook image_prompt_prefix prepended
Music duration alignment	Music may not match narration duration	Compare music.duration vs tts.duration, check padding/looping
Audio ducking timing	Ducking may cut music too aggressively	Inspect waveform: music should duck ~6dB under speech, recover smoothly
Video stitch transitions	Crossfade may flicker with mismatched codecs	Test with both matching and mismatched clips, check auto_normalize
Subtitle burn-in	Font size/position may clip on mobile formats	Test with 9:16 (TikTok) and 16:9 (YouTube) profiles
Remotion render	Components may fail with real data	Build a test composition with all 8 components, render at 1080p
Playbook contrast	Edge cases in dark-on-dark or light-on-light themes	Test with all 3 playbooks + a deliberately low-contrast custom one

Run Order

cd C:/Users/ishan/Documents/OpenMontage

# Phase 1: Individual tools (can run in parallel)
python tests/qa/test_01_tts.py
python tests/qa/test_02_image_gen.py
python tests/qa/test_03_music.py

# Phase 2: Composition (depends on Phase 1 outputs)
python tests/qa/test_04_audio_mix.py
python tests/qa/test_05_video_compose.py
python tests/qa/test_06_video_stitch.py

# Phase 3: Intelligence validation (no API calls)
python tests/qa/test_07_playbook_intelligence.py

# Phase 4: Full pipeline
python tests/qa/test_08_end_to_end.py

Success Criteria

• All 3 TTS samples: clear speech, correct content, no artifacts, ≥44.1kHz
• All 4 images: match prompt intent, correct dimensions, no watermarks, good composition
• Both music tracks: match mood prompt, correct duration (±2s), no abrupt cuts
• Audio mix: speech clearly above music, ducking smooth, no clipping
• Video compose: A/V sync within 50ms, correct resolution, playable in VLC
• Video stitch: smooth transitions, no frame drops, PIP correctly positioned
• Playbook intelligence: all 3 playbooks pass a11y, contrast ratios within 0.1 of manual calc
• End-to-end: 60-second explainer renders without errors, all stages checkpoint correctly