Files
AIDDSPandGitHub f76ac05989 test(qa): image generation consistency suite (#151)
Adds qa/imagegen: image metadata validation (MIME sniffing, exact dimensions), a contract check on the fal submit body and the OpenAI images reply, and live lanes for resolution, prompt adherence, character consistency and text rendering. Offline tests run in CI; live lanes skip without an opt-in or key.

Fixes #131

Claude-Session: https://claude.ai/code/session_015XYKLix9PSyn6Jw2HWKprk
2026-11-12 19:19:32 -08:00
..

qa/imagegen — Image Generation Consistency QA (#131)

Reproducible test suite for the gateway's image surface: POST /v1/images/generations across the supported fal-hosted diffusion SKUs (zen3-image → fal-ai/flux/schnell, zen3-image-sdxl → fal-ai/fast-sdxl). Stdlib + golang.org/x/image/webp (already a direct dep) — same shape as qa/audio (#133), qa/crossmodel (#138) and qa/multimodal (#135): offline checks run in CI without a network or a credential, live lanes skip with their reason and never pass silently.

Coverage map

What #131 asks to test Offline (CI, always) Live (opt-in)
Prompt adherence: complex compositions, spatial relations, style constraints, negative prompts the fal submit checker demands the prompt arrive byte-identical and rejects a mutated one (TestCheckFalSubmitCatchesDivergence/prompt-mutated) four-prompt corpus (spatial / style / negative / multi-subject) each must return a valid decodable artifact at the requested size (TestLivePromptAdherence)
Editing, inpainting, image-to-image accuracy the relay routes only /v1/images/generations — no edits door exists (finding below) POST /v1/images/edits must answer 404, anything else is a finding (TestLiveEditingDoor)
Character / subject consistency across sequential shots n-clamp contract pinned (TestClampN, num_images divergence caught at 13→10) same persona twice + an n=2 batch: every artifact must validate and be byte-distinct from the others (TestLiveCharacterConsistency)
Embedded text rendering, short & multi-word strings — (typography legibility cannot be judged offline) pinned sign prompts "HELLO" / "OPEN ALL NIGHT" must return valid artifacts; URLs logged for human review (TestLiveTextRendering)
Resolution & aspect ratio: square / landscape / portrait, output dimensions match the request ValidateImage proven against four real containers (PNG/JPEG/GIF/WEBP) built at 1024², 1792×1024, 1024×1792 and 640×480 — and against their corrupt twins: empty payload, HTML error page, truncated PNG stream, a square artifact handed back for a portrait request (TestFixturesSelfConsistent, TestValidateImageCatchesMismatch, TestCorruptPayloadsFailEachTheirOwnWay) full 2-models × 3-shapes matrix: download the artifact, sniff MIME from bytes, decode dimensions, demand exact match with the request (TestLiveResolutionAspect)
Parameter mismatches & silent ignores (the acceptance criteria's trace demand) fal's structured image_size object enforced — a bare "WxH" string, a silently dropped size, swapped {width,height}, and a junk size sent anyway each fail with a named reason; gateway replies audited for empty data, payload-less entries, invalid base64, zero created response_format=b64_json requested, URL observed coming back → recorded as a documented silent ignore with the code citation (TestLiveResponseFormatSilentIgnore)
Image metadata verification across supported models (dimensions, MIME, non-empty payload) metadata.go is that checker; its proofs are the offline tests above every live lane pipes artifacts through it before logging a pass

Files

File Role
imagegen.go env/lane constants, pinned SKUs (zen3-image, zen3-image-sdxl from controllers/model_pricing.go), gateway root resolution — env names shared with #133/#138/#135
metadata.go magic-byte MIME sniffing (PNG/JPEG/GIF/WEBP, never trusting a label) + ValidateImage: non-empty payload → recognized MIME → decodable header → exact requested dimensions; aspect classification
fixtures.go every container generated byte-for-byte at run time (no binary blobs in git): deterministic-noise PNG/JPEG/GIF at exact sizes, hand-built VP8X WebP, truncation helper, HTML-error-page fixture
params.go the contract checkers: fal async submit audit (CheckFalSubmit) and OpenAI images reply audit (CheckGatewayReply), plus ParseSize/ClampN mirroring model.doaiImageSize and the relay's [1,10] clamp
client.go minimal gateway client (/images/generations, raw POST for door probes) + artifact fetch
report.go live findings → HANZO_QA_REPORT JSON artifact + Markdown twin (the request/response trace the acceptance criteria ask for)
*_test.go offline proofs above, then the gated live lanes

Run it

# CI: no network, no credential — the whole structural suite
go test ./qa/imagegen -count=1

# Live lanes: real generations, artifacts downloaded and measured.
# ~14 generations ≈ $0.75 of credits at the pinned per-image prices.
HANZO_API_KEY=hz-… HANZO_QA_LIVE=1 HANZO_QA_REPORT=/tmp/imagegen-131.json \
  go test ./qa/imagegen -count=1 -run 'Live' -v
# → /tmp/imagegen-131.json + /tmp/imagegen-131.md: every verdict, prompt hash,
#   artifact URL, byte count, sniffed MIME and decoded dimensions.

# Against a staging gateway instead of production (with or without /v1):
HANZO_API_URL=https://staging.hanzo.ai HANZO_API_KEY=… HANZO_QA_LIVE=1 go test ./qa/imagegen -run Live -v

Reproduction steps & findings

Every finding names the exact command, model ID and prompt so a maintainer can re-run it in one line. All four were verified against main at 4fdd6c03.

  1. response_format is parsed but silently ignored. Repro: POST /v1/images/generations {"model":"zen3-image","prompt":"a paper boat on a pond","size":"","response_format":"b64_json"} → the reply carries "url", never "b64_json". Cause: imagesGenerationsRequest parses response_format (controllers/images_api.go) but model.ImageGenRequest has no such field, so it is never forwarded; fal always answers with URLs. Live lane records this as param-ignore / response_format=b64_json → documented.
  2. Unparseable size falls back to the model default without an error. Repro: "size":"square_hd" (or "1024", "0x0") → model.doaiImageSize returns nil, image_size is omitted from the fal submit, and the request succeeds at whatever the model default is — the caller is never told their parameter was dropped. The suite pins this contract (TestParseSize, CheckFalSubmit's "must be omitted" branch) so a future silent behavior change fails loudly.
  3. No editing/inpainting door. routers/router.go routes only /v1/images/generations; OpenAI-style POST /v1/images/edits (mask inpainting, image-to-image reference) answers 404. Any #131 "editing accuracy" workflow therefore lives upstream at fal and is not testable through the relay until a door exists. Lane editing/edits-door records the 404 as documented truth; a non-404 fails the lane.
  4. n outside [1,10] is clamped, not rejected. n=0, n=-3 and n=20 all succeed (clamped to 1 / 1 / 10 before any cost math, controllers/images_api.go); billing follows the clamp. Pinned by TestClampN + the submit checker's 13→10 divergence case.

What "consistency" cannot be proven by machine

Pixel-level judgments — did the fox stay the same fox across shots, is the neon sign legible — ride the artifact URLs each live lane logs. The suite guarantees everything checkable without eyes (payload validity, MIME, exact dimensions, verbatim prompt transit, parameter honesty) and leaves the human half one click away in the report.