mirror of
https://github.com/hanzoai/video.git
synced 2026-10-30 18:18:27 +00:00
calesthio
54155f132f
All of these changes came out of running the P1 and P2 documentary-montage
audits end-to-end and watching specific things break. Grouping them into
one commit because they share a theme: making the corpus builder and its
stock source adapters robust enough that a real brief can produce a
real corpus without special-casing.
corpus_builder.py + new clip_cache.py + test_clip_cache.py
Phase 1 of the shared-corpus architecture. Adds
~/.openmontage/clips_cache/ — a process-safe, LRU-evicted cache of
downloaded clip files keyed by clip_id. Before each candidate download,
corpus_builder asks the cache whether the bytes already exist on disk
from a previous run; on a hit it hard-links (or copies on cross-drive)
the blob into the caller's corpus dir and skips the network fetch. On
a miss it downloads as usual and ingests the fresh file. Motivation:
re-running the P1 audit after every tool fix was re-downloading gigs
of archive.org footage that had already been fetched in the prior run.
Cache faults never block the pipeline — they degrade gracefully to
normal downloads. The cache bubbles counters into the corpus_builder
return payload so the production report can show hit/miss/bytes-saved.
Default 20 GB cap, overridable via OPENMONTAGE_CACHE_MAX_GB.
Full test coverage: try_link, ingest, stats, LRU eviction, manifest
persistence, lock behavior. 23 tests, tmp_path-scoped.
stock_sources/archive_org.py
1. Three-strategy query cascade (phrase_prox_10 → distinctive_and →
distinctive_or). Motivation: natural-language documentary queries
against archive.org Solr were zeroing out — "1950s family watching
television" returned 0 hits because Solr's default multi-term AND
over-intersects. Walks strict to loose and returns the first
non-empty strategy. Stop words, source hints ("prelinger",
"archive", "footage"), and year tokens ("1950s") are excluded from
the distinctive-token picks since they don't correlate with
Prelinger title tokens.
2. 150 MB per-rendition size cap. archive.org routinely hosts
multi-hundred-megabyte h.264 masters and one 2 GB Prelinger item
poisons corpus build wall-time and disk. Within a format bucket
we now pick the largest rendition under the cap; if nothing fits
we fall through to the next format rather than dropping the item.
3. 180 s default max-duration ceiling when the caller hasn't set one
— archive.org is the only source that routinely hosts feature-
length material and a naive fan-out pulls them into corpora that
only ever want a few seconds per clip.
stock_sources/wikimedia.py
Parallel 3-strategy cascade (full → top2_or → single_best). Motivation:
Commons CirrusSearch also defaults to AND across multi-word queries;
our first P2 diagnostic pass returned 0 video results for 10/10
queries. Same stop-word / source-hint / year-token stripping as
archive_org so the two adapters stay symmetric.
test_stock_source_adapters.py
Rewrote the wikimedia query-builder tests against the new cascade
API. Added coverage for multi-word fallback + source-hint/year
stripping.
video_compose.py
Two small fixes for the Remotion renderer on Windows:
1. Resolve output_path to absolute before invoking the CLI so the
binary can write wherever the caller asked regardless of cwd.
2. Pass cwd=composer_dir to run_command so npx can find the local
Remotion binary under node_modules/.bin. Without this, Windows
npx returns "could not determine executable to run" because it
resolves .bin relative to the process cwd rather than the
script's parent.
73 lines
2.9 KiB
Python
73 lines
2.9 KiB
Python
from tools.video.stock_sources import all_sources
|
|
from tools.video.stock_sources.unsplash import _build_download_url, _orientation_for_unsplash
|
|
from tools.video.stock_sources.wikimedia import (
|
|
_build_search_queries,
|
|
_kind_from_mime,
|
|
_meta_value,
|
|
)
|
|
|
|
|
|
def test_stock_source_autodiscovery_includes_new_sources():
|
|
names = {source.name for source in all_sources()}
|
|
assert "wikimedia" in names
|
|
assert "unsplash" in names
|
|
|
|
|
|
def test_wikimedia_search_query_respects_kind():
|
|
# The cascade's first ("full") query should always carry the
|
|
# filetype filter for video/image kinds. "any" drops the prefix.
|
|
video_cascade = _build_search_queries("rain city", "video")
|
|
assert video_cascade[0][0] == "full"
|
|
assert video_cascade[0][1].startswith("filetype:video")
|
|
|
|
image_cascade = _build_search_queries("rain city", "image")
|
|
assert image_cascade[0][0] == "full"
|
|
assert image_cascade[0][1].startswith("filetype:image")
|
|
|
|
any_cascade = _build_search_queries("rain city", "any")
|
|
assert any_cascade[0][0] == "full"
|
|
assert any_cascade[0][1] == "rain city"
|
|
|
|
|
|
def test_wikimedia_cascade_falls_back_on_multi_word():
|
|
# Multi-word query should produce a 3-stage cascade: full, top2_or,
|
|
# single_best. Tokens are picked by length, so "television" beats
|
|
# "family" and "watching".
|
|
cascade = _build_search_queries(
|
|
"1950s family watching television", "video"
|
|
)
|
|
labels = [label for label, _ in cascade]
|
|
assert labels == ["full", "top2_or", "single_best"]
|
|
assert cascade[1][1] == "filetype:video television watching"
|
|
assert cascade[2][1] == "filetype:video television"
|
|
|
|
|
|
def test_wikimedia_cascade_strips_source_hints_and_years():
|
|
# "prelinger" is a source hint (redundant on Commons) and "1955" is
|
|
# a year — both are excluded from distinctive-token picks.
|
|
cascade = _build_search_queries(
|
|
"Prelinger 1955 housewife kitchen", "video"
|
|
)
|
|
# Full query keeps the source hint + year (first attempt is strict).
|
|
assert cascade[0][1] == "filetype:video Prelinger 1955 housewife kitchen"
|
|
# Distinctive picks do NOT include prelinger or 1955.
|
|
joined = " ".join(sq for _, sq in cascade[1:])
|
|
assert "housewife" in joined
|
|
assert "kitchen" in joined
|
|
assert "prelinger" not in joined.lower()
|
|
assert "1955" not in joined
|
|
|
|
|
|
def test_wikimedia_kind_and_metadata_helpers():
|
|
assert _kind_from_mime("video/webm", "File:foo.webm") == "video"
|
|
assert _kind_from_mime("image/jpeg", "File:foo.jpg") == "image"
|
|
assert _meta_value({"Artist": {"value": "<a href='/wiki/User:Test'>Test User</a>"}}, "Artist") == "Test User"
|
|
|
|
|
|
def test_unsplash_helpers_preserve_query_params():
|
|
assert _orientation_for_unsplash("square") == "squarish"
|
|
url = _build_download_url("https://images.unsplash.com/photo-123?ixid=abc", 1920)
|
|
assert "ixid=abc" in url
|
|
assert "w=1920" in url
|
|
assert "fm=jpg" in url
|