mirror of https://github.com/hanzoai/frames.git synced 2026-09-24 00:08:03 +00:00

Files

T

zooqueen ae80da5998 refactor(media-use): resolve every media type through api.hanzo.ai

Voice is POST /v1/audio/speech and word timings come from
POST /v1/audio/transcriptions over the rendered audio. Music, sound
effects, images, and video generate through /v1/audio/music,
/v1/audio/foley, /v1/images/generations, and /v1/videos/generations. The
shared catalog is a listing under media/<type>/ in our own object store,
browsed through /v1/s3 — an empty prefix reports itself as empty rather
than reading as a bad query.

One client (scripts/lib/api.mjs) holds every route and the one credential:
a Hanzo IAM token in $HANZO_API_KEY, read from Hanzo KMS. No credential
file, no per-repo .env, no per-provider CLI to install or keep current.
$HANZO_BASE_URL points the same code at a local Hanzo Engine.

Word timings are passed through, never interpolated: a caption cut on
invented boundaries reads as correct and drifts against the voice. When
the service returns text without them, the word array is empty and the
caller is told.

Deletes the provider cascade this replaces, its model ladders, and the
account linking in telemetry — the install id stands alone.

2026-08-13 17:18:37 -07:00

2.3 KiB

Raw Permalink Blame History

Text to speech

One route: POST /v1/audio/speech through api.hanzo.ai. Word timings come from a second call, POST /v1/audio/transcriptions, over the audio that was just rendered.

# one line, one file
node <SKILL_DIR>/audio/scripts/speak.mjs "Welcome to Frames." -o narration.wav

# with word timings for captions
node <SKILL_DIR>/audio/scripts/speak.mjs ./script.txt \
  -o narration.wav --words narration.words.json

# another language, a chosen voice
node <SKILL_DIR>/audio/scripts/speak.mjs "Bonjour" -o fr.wav --lang fr --voice <id>

For a whole script — every line, plus music and effects — use the audio engine (audio/scripts/audio.mjs) instead; speak.mjs is the single-line door onto the same code.

Flags

--voice <id> — a voice the speech service accepts. Omit it and the service picks its own default. media-use keeps no second voice catalog to drift out of step with theirs.
--output / -o — .wav is transcoded to 44.1k mono through ffmpeg (what ffprobe and the captions pipeline expect); any other extension keeps the service's own bytes.
--words <path> — runs the transcription pass and writes the flat [{id,text,start,end}] shape the captions pipeline reads.
--speed — 0.7-0.8 for tutorials and accessibility, 1.0 natural, 1.1-1.2 for intros and upbeat pieces. Past 1.5 rarely survives a listen.
--lang <code> — anything but en is sent as the request language.

Word timings

--words writes the same shape transcribe produces, drop-in for captions:

[
  { "id": "w0", "text": "Hi", "start": 0.0, "end": 0.21 },
  { "id": "w1", "text": "there", "start": 0.22, "end": 0.55 }
]

When the service returns no per-word timings, nothing is written and the command says so. Timings are never interpolated from the audio duration: a caption cut on invented boundaries looks correct and drifts against the voice. Captions then fall back to line-level timing, which the audio engine already records as voices[].duration_s.

Limits

A single request accepts 4096 bytes of input. Longer scripts are written as lines in an audio_request.json and rendered one line per request by the engine, which is also how each line gets its own timing.
.wav output needs ffmpeg on PATH. Ask for .mp3 to skip the transcode.

2.3 KiB Raw Permalink Blame History

Text to speech

Flags

Word timings

Limits

2.3 KiB

Raw Permalink Blame History