Files
AIDDSPandGitHub 4283b1c7ad test(qa): audio generation and transcription suite (#149)
Adds qa/audio: TTS container validators for mp3, wav, opus, aac, flac and pcm, a transcript timestamp audit, generated fixtures and live lanes for /v1/audio/speech and /v1/audio/transcriptions. Offline tests run in CI; live lanes skip without an opt-in or key.

Fixes #133

Claude-Session: https://claude.ai/code/session_015XYKLix9PSyn6Jw2HWKprk
2026-11-12 19:19:21 -08:00
..

qa/audio — Audio Generation & Transcription QA (#133)

Reproducible test suite for the gateway's OpenAI-compatible speech surface: POST /v1/audio/speech (Kokoro, pinned as zen-voice-mini) and POST /v1/audio/transcriptions (+ /transcriptions/public, Parakeet/Whisper, pinned as zen-scribe). Stdlib only — same shape as qa/crossmodel (#138) and qa/multimodal (#135): offline checks run in CI without a network or a credential, live lanes skip with their reason and never pass silently.

Coverage map

What #133 asks to test Offline (CI, always) Live (opt-in)
TTS across formats mp3 wav opus aac flac pcm every container is built byte-for-byte and walked structurally (mkMP3/mkOGGOpus/mkFLAC/mkaacFrames, TestSniffAndValidate); corrupt variants must fail with the right error (TestCorruptContainersFail) full response_format matrix against the real gateway: valid container, honest label, audible signal, exact format back (TestLiveSpeechFormats)
Voices, speeds request shape pins voice/speed and their omission rules (TestSpeechBodyWireShape) named voices must not 5xx; speed=0.8 vs 1.3 must move the duration (TestLiveSpeechVoicesSpeeds)
Streaming audio chunks streamed reads reassemble to byte-identical, still-valid containers (TestSpeechStreamedReassembles) 4 KiB chunk consumption, Content-Length agreement, container validity (TestLiveSpeechStreaming)
STT multilingual, accents, technical terms the phrase corpus is self-consistent by construction — every sentence contains its own pinned keywords (TestPhrasesCorpusIsSelfConsistent: en, en-tech, ru, uk, es, fr, de, zh, ja) synthesize each phrase, transcribe it back, demand real words: English rows must reproduce ≥ half their keywords, every language ≥ one (TestLiveTranscribeRoundTrip)
Word/segment timestamps (verbose_json, timestamp_granularities) the checker is proven against canned conforming and nine divergent bodies — overlaps, escaping words, nulls, drift, disagreement between text/segments/words (TestCheckTimingsCatchDivergence); the form must carry the bracketed array spelling (TestTranscribeFormWireShape) round trip and long lanes request word and segment and audit every timestamp (stt-timings findings)
Long recordings, chunk boundaries spliced fixtures sum exactly (TestConcatWAVSumsDurations); a 2.4 s transcript over 60 s of audio is flagged as truncation (TestCheckCoverageCatchesTruncation) ~730 chars → ~minute of audio, transcribed end to end with coverage demanded at both ends (TestLiveLongRecording)
Silence / interruption handling generated digital silence (Silence), speech-shaped onsets at known seconds (SpeechLike) silence must not become a hallucinated paragraph; the public door answers anonymous uploads cleanly (TestLiveSilenceHandling, TestLivePublicTranscriptions)
Loudness / output consistency RMS/peak over decoded PCM: a half-scale sine reads 0.354, silence reads 0 (TestLevel); "valid container full of silence" fails the format lane, not CI's eye per-format loudness floor on synthesis responses (tts-loudness findings)
Mislabelled containers (the audio/opus-carrying-MP3 class from audio_api.go history) bytes are sniffed without consulting the label; the exact historical mislabel is refused (TestMislabelCaught) every 200 response: Content-Type must name what the bytes actually are (tts-formats-label)

Files

File Role
audio.go env/lane constants, pinned SKUs, gateway root resolution (shared env names with #138/#135)
formats.go container sniffing + structural validators for all six formats: MP3 frame chain + Xing count vs truncation, FLAC STREAMINFO walk, Ogg page CRC-32 recomputation + pre-skip duration math, ADTS walk, WAVE chunk chain, s16le loudness
fixtures.go every WAV fixture generated at run time — sine, silence, speech-shaped envelope, deterministic noise, spliced long recordings; the nine-language phrase corpus with pinned keywords
transcript.go verbose_json decode + the timestamp audit: monotonicity, word-in-segment containment, duration-vs-container drift, text/segments/words agreement, coverage at both ends
client.go minimal OpenAI speech/transcribe/public client + chunked stream consumer
report.go live findings → HANZO_QA_REPORT JSON artifact + Markdown twin (the request/response trace the acceptance criteria ask for)
*_test.go offline proofs above, then the gated live lanes

Run it

# CI: no network, no credential — the whole structural suite
go test ./qa/audio -count=1

# key-free public lane (the one that runs against api.hanzo.ai today)
HANZO_QA_LIVE=1 go test ./qa/audio -run TestLivePublicTranscriptions -v

# full keyed matrix — format matrix, round trips, long recording, streaming
HANZO_QA_LIVE=1 HANZO_API_KEY=sk-... \
  HANZO_QA_REPORT=/tmp/audio-report.json go test ./qa/audio -run TestLive -v
# writes /tmp/audio-report.json + /tmp/audio-report.md (findings table)

HANZO_API_URL retargets the gateway; the two pinned SKUs (zen-voice-mini, zen-scribe) are checked against /v1/models first, so a catalog move fails loudly instead of silently testing nothing.

Findings on api.hanzo.ai today (key-free run, 2026-11-10)

HANZO_QA_LIVE=1 go test ./qa/audio -run TestLivePublicTranscriptions reproduces:

Check Today Expected
anonymous 3 s tone → /v1/audio/transcriptions/public 200, decodable transcript — the door works without a credential —
anonymous 8 bytes of garbage → same door HTTP 502, body error code: 502 (plain text) HTTP 400 + OpenAI's JSON error envelope, before anything is forwarded

Undecodable audio reaches the upstream instead of being refused at the edge: a transcriber decode failure today costs an upstream round trip and surfaces as a 5xx with a non-JSON body, which makes anonymous uploads a capacity lever and breaks every SDK's error parsing (the envelope half is #138's standing finding — the status-code half belongs here). Where to fix it: after parseTranscribeForm in controllers/audio_api.go / zap_audio.go, sniff the file part with the same container rules as this suite (formats.go) and answer 400 invalid_file_format when nothing decodes, instead of forwarding bytes the speech service will choke on.

Not covered here

The keyed lanes (TTS format matrix, voice/speed probes, streaming, nine-language round trips, long recording, silence) need HANZO_API_KEY; without one they skip with their reason — the PR that claims them should paste its report.md from a keyed run. Compressed-container loudness is structural only: mp3/opus/aac arrive valid but are not PCM-decoded with stdlib alone (SignalLevel reports ok=false rather than faking a level) — add a pure-Go decoder if "silence inside a valid MP3" ever shows up in production.