video/skills/pipelines/talking-head/scene-director.md

Scene Director — Talking Head Pipeline

When to Use

You have a script (from transcription) and raw footage. Your job is to watch the footage, understand the content, and propose a creative enhancement plan — then build a scene plan that transforms raw talking-head footage into an engaging, visually rich video.

You are not just a processor. You are a creative director. Your job is to figure out what the speaker is saying and propose visual enhancements that make the content more engaging and easier to understand.

Prerequisites

Layer	Resource	Purpose
Schema	schemas/artifacts/scene_plan.schema.json	Artifact validation
Prior artifacts	Script, Brief	Section timing and context
Tools	frame_sampler (optional)	Extract representative frames
Tools	face_tracker (optional)	Analyze speaker face position for reframing
Tools	silence_cutter (optional)	Detect silence for jump cut planning

Process

Step 0: Footage Analysis

Before watching the content, analyze the raw footage to understand the physical setup.

1. Sample 5 frames from the footage using ffmpeg (evenly spaced across the duration):

   ffmpeg -i <footage> -vf "select='not(mod(n\,TOTAL_FRAMES/5))'" -vsync vfr -frames:v 5 frame_%02d.png
   

2. Run visual_qa or histogram analysis on each sampled frame to detect:

   • Background type: Green screen / blue screen / natural background. Green/blue screens show a dominant narrow-band color spike in the histogram. Use visual_qa with prompt "Is this a green screen or blue screen background?" for confirmation.
   • Speaker position: Center, left, or right of frame. Estimate the approximate bounding box (e.g., "speaker occupies center 40% of frame, from x=30% to x=70%").
   • Lighting quality: Even studio lighting, harsh shadows, backlit, mixed color temperature. Note any issues that may affect chroma keying.

3. Green screen detected?

   • If yes, note that green_screen_processor tool will be needed in the compose stage.
   • Record the detected screen color (green or blue) and estimated uniformity.
   • The compose-director will use this to run chroma key removal and composite onto an animated background.

4. Measure speaker safe zone:

   • From the speaker's bounding box, determine where graphics can be placed WITHOUT overlapping the speaker.
   • For a centered speaker: left panel and right panel are safe for overlays.
   • For a left-positioned speaker: right panel is the primary safe zone.
   • For a right-positioned speaker: left panel is the primary safe zone.
   • Upper third and lower third are generally safe regardless of speaker position.

Record all findings in the scene plan metadata for downstream stages.

Step 1: Watch & Listen — Understand the Content

This is the most important step. Do not skip it.

Read the full transcript carefully. Understand:

• What is the speaker's main topic?
• What are the key concepts they explain?
• Where do they use numbers, statistics, or data?
• Where do they compare things (A vs B, before/after, old vs new)?
• Where do they list items (3 tips, 5 steps, etc.)?
• Where do they introduce technical terms or jargon?
• Where are the section transitions (topic changes)?
• What is the emotional arc (excitement, serious, humorous)?

If frame_sampler is available, extract 5-8 representative frames to see the speaker's setup, background, lighting, and gestures.

Step 2: Propose Creative Overlays

Based on your content analysis, propose on-screen graphics that will appear alongside the speaker at key moments. These are Remotion components that get composited on top of or next to the talking-head footage.

Available overlay types:

Overlay Type	Remotion Component	Best For
Key term definition	text_card	When the speaker introduces a technical term — show the term + short definition
Statistic/number	stat_card	When the speaker mentions a number or percentage — animate it on screen
Comparison	comparison	When the speaker compares two things (A vs B) — show side-by-side
Data chart	bar_chart / pie_chart / line_chart	When the speaker references data or rankings
KPI dashboard	kpi_grid	When multiple numbers are mentioned together
Progress indicator	progress_bar	When the speaker describes a process or percentage
Section title	hero_title	At major topic transitions — show the new section title
Callout/quote	callout	When the speaker makes a key point worth emphasizing
Lower third	text_card	Speaker identification at the start

Remotion Component Constraints:

Component	Min Width	720px Portrait?	Value Type
comparison	900px	NO -> use 2x stat_card	string
kpi_grid	720px	YES	numeric ONLY (no "15+")
bar_chart	500px	YES	numeric
stat_card	300px	YES	string OK
callout	400px	YES	string
hero_title	400px	YES	string
line_chart	500px	YES	numeric
progress_bar	600px	YES	numeric
stat_reveal	300px	YES	string OK

When a component's minimum width exceeds the available space (e.g., comparison at 900px won't fit in a 720px portrait frame), substitute with the recommended alternative. For comparison, use two stat_card components shown sequentially instead.

For kpi_grid, values MUST be numeric (e.g., 4.8, 73, 2400). String values like "15+" or "$4.8B" will cause rendering errors. Use stat_card for string-formatted numbers.

Overlay planning rules:

• Don't over-overlay. 3-6 overlays per minute of final video is the sweet spot. More than that is distracting.
• Time overlays to speech. Each overlay should appear when the speaker says the relevant words, not before or after.
• Keep overlays brief. 3-5 seconds each. They support the speaker, not compete with them.
• Vary the types. Don't use 5 text_cards in a row — mix in charts, comparisons, callouts.
• Use overlays at natural pauses. When the speaker pauses for emphasis, that's a good overlay moment.
• Match the vibe. Professional talk = clean stat cards and charts. Casual talk = callouts and bold key terms.

Step 3: Present Your Plan to the User

MANDATORY: Present your enhancement plan before proceeding.

Format your proposal clearly:

## Enhancement Plan for [Video Title/Topic]

**Content Summary:** [1-2 sentences about what the speaker covers]

**Proposed Overlays:**

| Time | Type | Content | Why |
|------|------|---------|-----|
| 0:05 | lower_third | "Speaker Name — Title" | Speaker intro |
| 0:22 | text_card | "Agentic AI: software that acts autonomously" | Key term definition |
| 0:45 | comparison | Traditional Software vs Agentic Software | Speaker is comparing the two |
| 1:10 | stat_card | "73% of developers..." | Statistic mentioned |
| 1:35 | bar_chart | [Framework popularity data] | Speaker references rankings |
| 2:00 | callout | "The key insight is..." | Speaker's main takeaway |

**Enhancement Chain:**
- Silence removal: ~X seconds of dead air detected
- Speed: 1.25x (user requested)
- Face + eye enhancement
- Animated captions (bottom of frame)
- Background music: [recommendation]

**Estimated final duration:** ~Xs (from Xs raw)

IMPORTANT: When outputting the overlay plan, ALSO generate the actual Remotion JSON props file (greenscreen-bg.json) -- do not just describe scenes in prose. The JSON props file should be a complete, valid input for the Remotion TalkingHead composition, including all overlay definitions, timing, colors, and content. Save it to <project>/public/demo-props/ or the project's props directory.

Wait for user approval before proceeding. The user may:

• Approve as-is
• Add/remove overlays
• Change overlay content
• Adjust the enhancement plan

Step 4: Analyze Footage (if tools available)

Face tracking — If face_tracker is available, run it on the raw footage:

face_tracker.execute({
    "input_path": "<raw_footage>",
    "sample_fps": 5
})

This outputs per-frame face bounding boxes. Use this data to:

• Decide if reframing is needed (e.g. speaker is off-center for vertical crop)
• Identify sections where the speaker moves significantly (needs dynamic crop)
• Note face position for auto_reframe in the compose stage
• Determine overlay safe zones — where to place graphics without occluding the face

Silence detection — If silence_cutter is available, run in mark mode:

silence_cutter.execute({
    "input_path": "<raw_footage>",
    "mode": "mark",
    "silence_threshold_db": -35,
    "min_silence_duration": 0.5
})

This outputs silence/speech segment timestamps. Use this to:

• Plan which segments should be jump-cut or sped up
• Identify dead air, false starts, and long pauses
• Estimate the final video duration after cuts

Step 5: Plan Base Scenes

For talking-head, the base is simple: one scene per script section, all type talking_head. The raw footage IS the scene.

Step 6: Build Overlay Scenes

For each approved overlay from Step 3, create an overlay scene entry:

{
  "id": "overlay_1",
  "type": "overlay",
  "overlay_type": "text_card",
  "start_seconds": 22.0,
  "duration_seconds": 4.0,
  "content": {
    "text": "Agentic AI",
    "subtext": "Software that acts autonomously toward goals",
    "backgroundColor": "<theme_background>",
    "accentColor": "<theme_accent>"
  },
  "position": "lower_third"
}

Overlay position options:

• lower_third — bottom 30% of frame (safest, doesn't occlude face)
• upper_third — top 30% of frame (good for titles)
• side_panel — left or right 40% (for charts/comparisons, speaker shifts to other side)
• full_overlay — brief full-screen graphic (1-2s max, for dramatic emphasis)

Step 7: Plan Reframing & Cuts

If the target platform requires a different aspect ratio (e.g. Instagram Reels = 9:16):

• Note auto_reframe should be applied in the compose stage
• Record the target aspect ratio in the scene plan
• If face tracking data shows significant speaker movement, note that dynamic crop is needed

If silence detection found segments to cut:

• Record the recommended cut mode (remove or speed_up) in the scene plan
• Note padding preferences (default 0.08s to avoid clipping words)

Step 8: Build Scene Plan

Assemble the full scene plan with:

• Base scenes (one per script section, type talking_head)
• Overlay scenes (from Step 6, type overlay)
• Enhancement chain decisions (silence cut mode, speed factor, reframe target)
• Music recommendation
• Estimated final duration

Step 9: Self-Evaluate

Criterion	Question
Content understanding	Did I actually understand what the speaker is talking about?
Overlay relevance	Does every overlay directly relate to what's being said at that moment?
Overlay density	Am I in the 3-6 per minute range? Not too sparse, not too cluttered?
Overlay variety	Am I using different types, not just text_cards?
Timing	Are overlays timed to the speaker's words, not arbitrary moments?
Coverage	Every script section has a scene?
Feasibility	Can all overlays be rendered with available Remotion components?
User approved	Did the user approve the enhancement plan?

Step 10: Submit

Validate the scene_plan against the schema and persist via checkpoint.