Adds qa/crossmodel: one identical request per model family over /v1/chat/completions and /v1/messages, checking sync answers, SSE layout, tool calls, structured outputs and error envelopes. Offline checks run in CI; live lanes skip without an opt-in or key. Fixes #138 Claude-Session: https://claude.ai/code/session_015XYKLix9PSyn6Jw2HWKprk
Cross-model API compatibility suite (#138)
One request shape, sent to every major model family over both wire formats the
gateway serves — OpenAI-compatible POST /v1/chat/completions and Anthropic-compatible
POST /v1/messages — with the response contract asserted on each answer:
| Scenario | What it asserts |
|---|---|
openai_sync |
chat.completion shape, choices[0].message, a mapped finish_reason, usage token counts |
openai_stream |
SSE layout: data: {…} chunks of chat.completion.chunk, delta.content, one terminal finish_reason, closing data: [DONE] |
openai_tools |
tools + tool_choice: finish_reason: tool_calls, tool_calls[].{id,type,function.name}, function.arguments parses as JSON |
openai_tools_parallel |
parallel_tool_calls: true yields two distinct calls in one turn |
openai_tool_followup |
multi-turn role: "tool" result round-trips and the answer comes back |
openai_stream_tools |
delta.tool_calls fragments carry an index and reassemble into complete, parseable calls |
openai_json_object / openai_json_schema |
response_format returns parseable JSON carrying the required keys |
anthropic_sync / anthropic_stream |
type: message, text content blocks, a mapped stop_reason, usage.input_tokens/output_tokens; the documented event sequence message_start → content_block_start → content_block_delta(text_delta) → content_block_stop → message_delta → message_stop |
| error envelopes | {error:{message,type,param,code}} on /v1/chat/completions, {type:"error",error:{type,message}} on /v1/messages, for 401 / 4xx-invalid-input / 404-unknown-model |
The matrix rows (Matrix in matrix.go) are the families named in #138. Ids are
pinned from the live catalog so TestCatalog_Live — which costs no credits, since
GET /v1/models is public — fails the moment a row drifts or a capability flag changes.
Running it
go test ./qa/crossmodel -count=1 # contract checks: no network, no key
HANZO_QA_LIVE=1 go test ./qa/crossmodel -run Live # public live lane: catalog + error envelopes, no key
HANZO_API_KEY=hzo_xxx HANZO_QA_REPORT=/tmp/matrix.json \
go test ./qa/crossmodel -run Live -timeout 30m -v # the full matrix; writes matrix.json + matrix.md
HANZO_API_URL retargets the gateway (default https://api.hanzo.ai, staging works too);
HANZO_API_KEY is the only credential needed. hanzo/zen is a free catalog model, so a
full keyed run can be kept off the bill.
A CI lane looks like this (the standing gate in hanzo.yml compiles and vets, which already
covers this package; the live lane is what needs a secret):
crossmodel:
name: Cross-model API compatibility (#138)
runs-on: [hanzo-build-linux-amd64]
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
steps:
- uses: actions/checkout@v4
with: {fetch-depth: 1}
- uses: actions/setup-go@v5
with: {go-version-file: go.mod}
- run: HANZO_QA_LIVE=1 go test ./qa/crossmodel -run Live -count=1 # key-free public checks
- env:
HANZO_API_KEY: ${{ secrets.HANZO_QA_KEY }}
HANZO_QA_REPORT: matrix.json
run: go test ./qa/crossmodel -run Live -timeout 40m -v
- uses: actions/upload-artifact@v4
with: {name: compatibility-matrix, path: matrix.*}
Findings on api.hanzo.ai (2026-11-09, key-free run)
Reproduce with the HANZO_QA_LIVE=1 line above; every item below is asserted by
TestPublicErrorEnvelopes_Live / TestCatalog_Live.
- 401 on
/v1/chat/completionsis not an OpenAI error envelope. Body is the console envelope{"status":"error","msg":"authentication required: …","data":null,"data2":null}. The official SDKs readerror.message, so a first unauthenticated call dies on an undefined field instead of the actionable message. Status code (401) is correct. - 401 on
/v1/messagesis not an Anthropic error envelope either — same console body, while Anthropic clients key off top-leveltype:"error"pluserror.type. - An unknown route 404s as
text/plain; charset=utf-8, bodynot found— no JSON, so no client can parse it at all. - Catalog capability flags are omitted when false (
supports_tools,supports_vision,supports_reasoningappear only on the entries where they are true), whilefreeandpremiumare always serialized. A matrix consumer cannot tell "false" from "unknown". - Kai has no chat surface at all:
hanzo/kaiis cataloged asendpoint: /v1/decisions,outputs: ["decision"], so the identical chat payload #138 asks for cannot reach it. The matrix marks every Kai cellskipwith that reason rather than pretending parity. Zen (hanzo/zen) and Enso (enso-flash) do answer/v1/chat/completionsbut are not flaggedsupports_tools, so their tool cells skip too — 511 models listed, 354 of them tool-capable.
Where the envelope fix belongs: once a request reaches a controller, both shapes are already
right (controllers/anthropic_api.go anthropicErrorBody, and routers/filter_balance.go
for model_not_found). The gap is the filter layer in front of it — denyUnauthorized in
routers/base.go, called from routers/authz_filter.go, answers every /v1/* credential
failure with the console envelope. One branch that picks the envelope by request path closes
findings 1 and 2; finding 3 is the router's no-match handler.