Adds qa/imagegen: image metadata validation (MIME sniffing, exact dimensions), a contract check on the fal submit body and the OpenAI images reply, and live lanes for resolution, prompt adherence, character consistency and text rendering. Offline tests run in CI; live lanes skip without an opt-in or key. Fixes #131 Claude-Session: https://claude.ai/code/session_015XYKLix9PSyn6Jw2HWKprk
qa/imagegen — Image Generation Consistency QA (#131)
Reproducible test suite for the gateway's image surface: POST /v1/images/generations across the supported fal-hosted diffusion SKUs
(zen3-image → fal-ai/flux/schnell, zen3-image-sdxl → fal-ai/fast-sdxl).
Stdlib + golang.org/x/image/webp (already a direct dep) — same shape as
qa/audio (#133), qa/crossmodel (#138) and qa/multimodal (#135): offline
checks run in CI without a network or a credential, live lanes skip with their
reason and never pass silently.
Coverage map
| What #131 asks to test | Offline (CI, always) | Live (opt-in) |
|---|---|---|
| Prompt adherence: complex compositions, spatial relations, style constraints, negative prompts | the fal submit checker demands the prompt arrive byte-identical and rejects a mutated one (TestCheckFalSubmitCatchesDivergence/prompt-mutated) |
four-prompt corpus (spatial / style / negative / multi-subject) each must return a valid decodable artifact at the requested size (TestLivePromptAdherence) |
| Editing, inpainting, image-to-image accuracy | the relay routes only /v1/images/generations — no edits door exists (finding below) |
POST /v1/images/edits must answer 404, anything else is a finding (TestLiveEditingDoor) |
| Character / subject consistency across sequential shots | n-clamp contract pinned (TestClampN, num_images divergence caught at 13→10) |
same persona twice + an n=2 batch: every artifact must validate and be byte-distinct from the others (TestLiveCharacterConsistency) |
| Embedded text rendering, short & multi-word strings | — (typography legibility cannot be judged offline) | pinned sign prompts "HELLO" / "OPEN ALL NIGHT" must return valid artifacts; URLs logged for human review (TestLiveTextRendering) |
| Resolution & aspect ratio: square / landscape / portrait, output dimensions match the request | ValidateImage proven against four real containers (PNG/JPEG/GIF/WEBP) built at 1024², 1792×1024, 1024×1792 and 640×480 — and against their corrupt twins: empty payload, HTML error page, truncated PNG stream, a square artifact handed back for a portrait request (TestFixturesSelfConsistent, TestValidateImageCatchesMismatch, TestCorruptPayloadsFailEachTheirOwnWay) |
full 2-models × 3-shapes matrix: download the artifact, sniff MIME from bytes, decode dimensions, demand exact match with the request (TestLiveResolutionAspect) |
| Parameter mismatches & silent ignores (the acceptance criteria's trace demand) | fal's structured image_size object enforced — a bare "WxH" string, a silently dropped size, swapped {width,height}, and a junk size sent anyway each fail with a named reason; gateway replies audited for empty data, payload-less entries, invalid base64, zero created |
response_format=b64_json requested, URL observed coming back → recorded as a documented silent ignore with the code citation (TestLiveResponseFormatSilentIgnore) |
| Image metadata verification across supported models (dimensions, MIME, non-empty payload) | metadata.go is that checker; its proofs are the offline tests above |
every live lane pipes artifacts through it before logging a pass |
Files
| File | Role |
|---|---|
imagegen.go |
env/lane constants, pinned SKUs (zen3-image, zen3-image-sdxl from controllers/model_pricing.go), gateway root resolution — env names shared with #133/#138/#135 |
metadata.go |
magic-byte MIME sniffing (PNG/JPEG/GIF/WEBP, never trusting a label) + ValidateImage: non-empty payload → recognized MIME → decodable header → exact requested dimensions; aspect classification |
fixtures.go |
every container generated byte-for-byte at run time (no binary blobs in git): deterministic-noise PNG/JPEG/GIF at exact sizes, hand-built VP8X WebP, truncation helper, HTML-error-page fixture |
params.go |
the contract checkers: fal async submit audit (CheckFalSubmit) and OpenAI images reply audit (CheckGatewayReply), plus ParseSize/ClampN mirroring model.doaiImageSize and the relay's [1,10] clamp |
client.go |
minimal gateway client (/images/generations, raw POST for door probes) + artifact fetch |
report.go |
live findings → HANZO_QA_REPORT JSON artifact + Markdown twin (the request/response trace the acceptance criteria ask for) |
*_test.go |
offline proofs above, then the gated live lanes |
Run it
# CI: no network, no credential — the whole structural suite
go test ./qa/imagegen -count=1
# Live lanes: real generations, artifacts downloaded and measured.
# ~14 generations ≈ $0.75 of credits at the pinned per-image prices.
HANZO_API_KEY=hz-… HANZO_QA_LIVE=1 HANZO_QA_REPORT=/tmp/imagegen-131.json \
go test ./qa/imagegen -count=1 -run 'Live' -v
# → /tmp/imagegen-131.json + /tmp/imagegen-131.md: every verdict, prompt hash,
# artifact URL, byte count, sniffed MIME and decoded dimensions.
# Against a staging gateway instead of production (with or without /v1):
HANZO_API_URL=https://staging.hanzo.ai HANZO_API_KEY=… HANZO_QA_LIVE=1 go test ./qa/imagegen -run Live -v
Reproduction steps & findings
Every finding names the exact command, model ID and prompt so a maintainer can
re-run it in one line. All four were verified against main at 4fdd6c03.
response_formatis parsed but silently ignored. Repro:POST /v1/images/generations {"model":"zen3-image","prompt":"a paper boat on a pond","size":"","response_format":"b64_json"}→ the reply carries"url", never"b64_json". Cause:imagesGenerationsRequestparsesresponse_format(controllers/images_api.go) butmodel.ImageGenRequesthas no such field, so it is never forwarded; fal always answers with URLs. Live lane records this asparam-ignore / response_format=b64_json → documented.- Unparseable
sizefalls back to the model default without an error. Repro:"size":"square_hd"(or"1024","0x0") →model.doaiImageSizereturns nil,image_sizeis omitted from the fal submit, and the request succeeds at whatever the model default is — the caller is never told their parameter was dropped. The suite pins this contract (TestParseSize,CheckFalSubmit's "must be omitted" branch) so a future silent behavior change fails loudly. - No editing/inpainting door.
routers/router.goroutes only/v1/images/generations; OpenAI-stylePOST /v1/images/edits(mask inpainting, image-to-image reference) answers 404. Any #131 "editing accuracy" workflow therefore lives upstream at fal and is not testable through the relay until a door exists. Laneediting/edits-doorrecords the 404 as documented truth; a non-404 fails the lane. noutside [1,10] is clamped, not rejected.n=0,n=-3andn=20all succeed (clamped to 1 / 1 / 10 before any cost math,controllers/images_api.go); billing follows the clamp. Pinned byTestClampN+ the submit checker's 13→10 divergence case.
What "consistency" cannot be proven by machine
Pixel-level judgments — did the fox stay the same fox across shots, is the neon sign legible — ride the artifact URLs each live lane logs. The suite guarantees everything checkable without eyes (payload validity, MIME, exact dimensions, verbatim prompt transit, parameter honesty) and leaves the human half one click away in the report.