Live 2026-10-27: enso-ultra refused (its Anthropic key out of credit) fell to the pool's nemotron-3-ultra:free, relayed as it answered, and forced calls came back as text. enso-free is the engine's free lane, which judges a free answer and reads a forced call written out as text as the call; a refused Enso tier now asks it before the pool's own routes.
121 KiB
ai — agent guide
hanzoai/ai is the canonical AI control plane for the Hanzo platform: model
hub, native Go model routing, RAG, and MCP/A2A management. It speaks the
OpenAI-compatible /v1 API, routes 66+ models to upstream providers, and meters
every request. Renamed from hanzoai/cloud (HIP-0106); mounts as the ai
subsystem inside hanzoai/cloud. In prod it runs as cloud-api on hanzo-k8s,
fronted by hanzoai/gateway at api.hanzo.ai.
Canonical role. This is a Hanzo service/infra repo — one impl, one place.
It is NOT an SDK; SDKs link out to it. Completeness order across languages is
Python → Rust → C++ → Go. Canonical spec: ~/work/hanzo/SDK-ARCHITECTURE.md.
Brand rules (hard — enforce in every edit).
- Never call this an "LLM gateway" and never position it against LiteLLM — it is a full AI cloud / control plane, not a proxy.
/v1/only, never an/api/prefix.- Zen models are Hanzo's own family (
owned_by: hanzo) — never name upstream models. - Voice: "Hanzo — the Open AI Cloud." Modern, crisp, developer-first.
Install / run.
go build -race -ldflags "-extldflags '-static'" # build
./cloud-api-server # run (env-configured)
go test -v $(go list ./...) -tags skipCi # test (requires MySQL)
docker compose up # local stack
Key entry points. main.go (entry) · bootstrap.go (boot + replica
assertion) · routers/router.go (all /v1 routes) · controllers/ (HTTP
handlers) · object/init.go (init + LLM provider seeding) · model/ (provider
integrations) · object/kms.go (secret resolution) · web/ (React admin UI).
Architecture
Full-stack application:
- Backend: Go 1.26 on
github.com/zap-proto/zip(over fiber v3 / fasthttp), MySQL/MariaDB/PostgreSQL - Frontend: React + Ant Design v5, located in
web/(the folder is the React app; the in-house Go router that used to share the name is gone) - Auth: Hanzo IAM SSO integration
Directory Structure
cloud/
├── main.go # Entry point
├── go.mod # Go module
├── conf/app.conf # Beego config
├── controllers/ # HTTP handlers (60+ files)
├── object/ # Business logic & data models (85+ files)
├── model/ # AI model provider integrations (38+ files)
├── routers/ # Routes and middleware filters
├── embedding/ # Embedding provider implementations
├── agent/ # MCP agent support
├── split/ # Text splitting strategies
├── txt/ # Document parsing (PDF, CSV, XLSX, PPTX)
├── storage/ # File storage abstractions
├── util/ # Utilities
└── web/ # React frontend
├── src/
└── package.json
Build & Run
Backend
# Build
go build -race -ldflags "-extldflags '-static'"
# Cross-platform build (linux amd64/arm64/riscv64)
./build.sh
Frontend
cd web
yarn install
yarn start # Dev server
yarn run build # Production build
Docker
docker compose up
Testing
# Go tests (requires MySQL)
go test -v $(go list ./...) -tags skipCi
# Frontend tests
cd web && yarn test
Linting
# Go - uses golangci-lint with gofumpt
golangci-lint run
# Frontend
cd web && yarn lint
Key Conventions
Backend (Go)
- Framework:
zip. AnApiControllerIS its request — it embeds*zip.Ctx, so there is no context to marry to a controller and no recorder to read afterwards. Controllers handle HTTP, objects hold the business logic. - New AI providers go in
model/(implement the provider interface) - Database access via the native store in
object/ - Route registration: CRUD resources are GENERATED from the one table in
routers/resources.go;routers/router.goholds only what is NOT a resource (the OpenAI-compatible surface and a few singletons). See "The /v1 resource surface" below. - i18n strings: avoid duplicate keys across frontend and backend
Five things zip decides that the old router did not
Each cost a real defect during the migration, and each is invisible: the code compiles, the types check, and the behaviour is wrong.
-
A WRITE CARRIES ITS STATUS.
c.JSON(code, v)/c.Bytes(code, b)set the status as they write, so setting one beforehand and writing after replaces it.c.Statusis for a response whose body is written by something that takes no status — a stream — or that has no body at all. Everything else passes the status to the one write. A refusal that arrives as 200 is worse than an outage: the client reads it and believes it. -
A STREAMED BODY IS PRODUCED AFTER THE HANDLER RETURNS. fasthttp drains the writer handed to
SendStreamWriterwhile it serialises the response, so the callback has NOT run when the call comes back — measured, not assumed. Anything the stream reads must outlive the handler (hand the upstream body to the callback; adeferin the enclosing function closes it first and the client gets nothing), and anything the stream learns must be used inside it (token counts read outside are still zero, so a streamed answer bills no completion).AND THE CTX IN THERE IS NOT THE REQUEST'S — this line used to say it was, and production refuted it. The claim was that fasthttp finishes writing before fiber releases the request, so reading
cfrom the callback was safe. Measured on api.hanzo.ai:DefaultCtx.App()nil-dereferenced underrecordFamilyUsage -> billingOrg -> Ctx.Header, twelve times an hour. It does not fail the one request either — a SIGSEGV takes the whole plugin process, so every other call in flight dies with it and each one answersmount /v1: http: read response: EOF. Read what the stream needs from the request BEFORE the handler returns and let it travel by value;whenceinzen_client.gois that shape, andrecordFamilyUsagetakes no receiver so there is nothing to reach through. -
RequestURIIS SET BY A SERVER, NOT BY A CALLER. The fiber adaptor reads that field, so a request CONSTRUCTED in process — which is every request crossing ZAP — arrives at the router as/and 404s.controllers.targetwraps a handler to fill it in from the URL; both ZAP installers go through it. -
A BUFFERED WRITE IS NOT A DELIVERY. Bytes leave at the flush. A
Writethat returns n > 0 has reached a buffer, and an uncheckedFlushreports a broken connection as success — which is howAnthropicWriter.StreamSentcame to tell the failover loop an answer had reached a client it never reached. -
A PRINCIPAL IS A VERIFIED TOKEN, and there is nothing else to seed. No session store exists;
GetSessionClaimsparses the credential the request presents. So a test that needs a caller PRESENTS one:presenting(visit(method, target), authtest.Bearer(t, user)).internal/authtestis the ONE installer of the verifier's certificate — two of them race over the same global. Corollary: any predicate that used to mean "there is a session" now fires for ANY verified token, a sibling brand's included.
Frontend (React)
- UI components use Ant Design v5
- i18n via i18next; translation files in
web/src/locales/ - State via React Context/Hooks (no Redux)
- API calls go through
web/src/backend/helper modules
Commits & PRs
- Branch naming:
claude/<name> - Main branch:
main - Follows semantic versioning (
.releaserc.json) - PR titles should be semantic (feat/fix/chore/docs/refactor)
CI/CD (GitHub Actions)
.github/workflows/build.yml runs:
- Go unit tests (with MySQL service)
- Frontend build (Node.js 20 + Yarn)
- Go binary build
- golangci-lint
- Semantic release + Docker push to
ghcr.io/hanzoai/cloud
Configuration
- Backend config:
conf/app.confread by the nativeconf.AppConfigini reader (env-first; absent in the embedded runtime, which runs env-only) - Database: MySQL 8.0+ or MariaDB
- Default Docker DB: PostgreSQL via
compose.yml
Important Files
| File | Purpose |
|---|---|
main.go |
Application entry point |
routers/router.go |
All API route definitions |
object/init.go |
Application initialization + LLM provider seeding |
conf/app.conf |
Runtime configuration |
web/src/App.js |
Frontend root component |
web/src/backend/ |
API client helpers |
hedge — ask several providers, keep the first answer
hedge.Race(ctx, dst, attempts...) is the primitive behind fast mode. Tail
latency is not the average: a provider that usually answers in 400ms occasionally
takes eight seconds, and a request cannot know which it drew until it has already
waited. Asking three and keeping the first cuts that tail.
No provider changes, and none are needed. Every provider already writes into
an io.Writer, so Race hands each attempt a different one and only lets the
winner's reach the caller. ModelProvider is untouched — the seam was already
the right shape.
FIRST BYTE WINS, the only definition of fastest a stream can act on. A provider that will finish sooner but has not started is not the winner: the caller waits on bytes, not completion, and by the time completion is knowable the response has arrived and there was nothing to race.
Two cases decide whether hedging helps or hurts, and both are pinned by tests verified against a deliberately broken implementation:
- An EMPTY write does not claim. A provider flushing a zero-length chunk while still waiting upstream would otherwise win having said nothing, leaving the caller on the SLOWEST attempt with the others cancelled — the opposite of the point.
- A FAST FAILURE is not a win.
Racereturns when the attempt that CLAIMED the stream finishes, not when the first attempt returns; an upstream 503 in 1ms produced no bytes, so the caller is still waiting on whoever answers.
Losers are cancelled — that is what stops the upstream call, which is what stops
the cost — and Race does not wait for them to unwind, because waiting on a
provider we are abandoning hands it exactly the latency this removes.
One ledger path each, because the two losers are opposite cases
ask.fan (controllers/hedged.go) carries a request into fast mode: how many to
ask, where the winner's bytes go, and how to make and adopt a writer. serve()
dispatches on it, so a caller sets a NUMBER rather than choosing a code path, and
a fan too narrow to race falls back to the cascade with its retry and demotion
intact. Nothing sets it yet — the mechanism ships inert, and
TestWithoutAFanNothingChanges is what says so.
A raced loser and a failover refusal wear the same shape and mean the opposite thing, which is the whole design:
- A REFUSAL errored. No tokens, no upstream charge —
recordRefusalswrites a"failover"row that bills nothing, and that is correct. - A RACED LOSER was CANCELLED mid-generation. The vendor produced tokens and charged us. Filing it as a refusal records a real charge as free and makes fast mode invisible in the ledger the team reads.
So a beaten provider never appears in the attempts race() returns — the caller
records those as refusals — and settles through fan.bill instead.
Each attempt needs its OWN writer. The loser's token count is on ITS writer,
and one writer shared across attempts holds exactly one answer: the winner's. A
shared writer therefore bills every loser for the winner's tokens AND glues its
half-sentence to the winner's answer on the client's wire — both failures show up
in TestEachRacerIsBilledFromItsOwnWriter, verified against a deliberately
shared fork. OpenAIWriter.fork/adopt are that pair: forks copy the dialect
and start the buffers empty; the winner's is adopted so the non-streaming body
and the tail chunk come from the provider that actually served.
THE BILL ARRIVES LATE, and that is the shape rather than a wrinkle. Race
returns when the winner finishes, not when the losers do — so at that moment a
loser has typically not even started, and reading its outcome is both a missing
bill and a data race. (The first draft did exactly this; the test caught it.)
settle therefore runs on its own goroutine, and fan.bill must close over
VALUES: by the time it fires the handler has returned and, on fasthttp, the
request has been recycled.
Cancellation is a WRITE ERROR, not a context. QueryText takes no context,
so a loser's only stop signal is hedge.ErrLost on its next write. One that
checks its write error stops there; one that does not runs to completion and
bills us for all of it. Either way the tokens are real — which is the argument
for billing them.
The switch, and why it is decided with the reservation
A request asks with "fast": true in the body, or X-Fast: 1 for callers with
no browser in the way — the body form exists because a page cannot send a custom
header past the edge's CORS allow-list, exactly as retrieval works
(controllers/fast.go).
What a race costs is settled WITH the hold, never after. widthFor reserves
est × fastWidth before anything is spent, because N attempts is N completions
and a hold sized for one is a promise the ledger cannot keep — the losers' rows
land later and take the balance past the point the gate said it would stop.
fastWidth is 2: the tail this cuts is one provider drawing a slow response
while another would have answered normally, and a second opinion removes most of
it for one extra completion rather than two.
Two refusals that are deliberately NOT refusals:
- Cannot afford the race → serve the ordinary request. The caller asked to go faster, not to be turned away, and a new way for a funded account to be told no is worse than quietly being normal speed.
- Nobody to bill → never race. The public lane makes no reservation, so hedging there spends N completions on a stranger and settles none.
Still open: the toggle in the products. hanzo.chat follows web_search;
hanzo.app follows base and must resolve its in-flight merge on the three
AI-path files first. The API accepts the field today, so either can ship the
control without waiting on the gateway.
The /v1 resource surface — ONE table, no compound routes
The published OpenAPI description is GENERATED from that same table:
go run ./cmd/openapi -spec ../openapi/ai/openapi.yaml
It writes only the region between the # BEGIN/END generated markers in the
canonical spec (hanzoai/openapi, OpenAPI 3.1) — the inference paths above it are
hand-authored with real OpenAI request/response schemas and stay that way.
cmd/openapi -verify and TestSpecMatchesTheTable fail when the spec drifts from
the table, so the contract customers and SDKs read cannot describe a surface this
service does not serve.
The old swagger/ directory is DELETED. It was a second description, and it was
wrong: basePath: /api with 135 /<verb>-<noun> paths, a base path this service
has never served. Nothing referenced it, nothing embedded it, and no test noticed
when it stopped being true — which is the argument for generating the replacement
rather than hand-maintaining it.
Every CRUD route is generated from routers/resources.go. There is no second
registration path, and no route is written by hand.
GET /v1/<ns>/<resource> list POST /v1/<ns>/<resource> create
GET /v1/<ns>/<resource>/<owner>/<name> read PATCH …/<owner>/<name> update
DELETE …/<owner>/<name> delete
GET /v1/<ns>/<resource>/global cross-tenant list (where a resource has one)
The namespace is ai for every row, and for every other route this service
serves. HIP-0139 §3.1: a capability answers at ONE address, /v1/<name>, and a
second top-level address is a second capability or a misfiled route — never an
alias. So /v1/router/*, /v1/org/settings, /v1/finetune/*, /v1/memory/*,
/v1/rag/*, /v1/docs/ingest, /v1/feedback and /v1/traffic/globe are all
under /v1/ai/… now (ingest under /v1/ai/rag/, since it writes the index rag
reads). The groupings in the table — retrieval, chat, models, content, compute,
work, ops — are how it is READ, not a second prefix.
Two families stay outside /v1/ai because a protocol, not this repo, fixes
their spelling (§3.2): the OpenAI/Anthropic wire (/v1/chat/completions,
/v1/completions, /v1/embeddings, /v1/messages, /v1/models,
/v1/images/*, /v1/audio/*, /v1/videos/*, /v1/responses, /v1/rerank),
and POST /v1/embed, the multipart form hanzo.chat posts.
/v1/ai/<noun> is a resource collection address. Nothing may register a ZAP
gateway PREFIX there: the registry matches on path alone, so a prefix at a
collection swallows that collection's :owner/:name members and answers with
the LIST handler — a wrong answer, not an error. Deeper addresses
(/v1/ai/rag/query, /v1/ai/router/defaults) are routes, not collections, and
are safe. TestNoGatewayPrefixOverTheResourceNamespace holds the line; it is
why /v1/ai/feedback has no gateway prefix and reaches routers.App through the
fallback instead.
Three things about this that are easy to get wrong:
The member key is <owner>/<name> — two path segments, not one. Every object
is keyed by that pair (util.GetOwnerAndNameFromIdWithError requires EXACTLY two
tokens, so a name containing a slash was never valid — the two-segment URL is
lossless, not a narrowing). They are recomposed into id in the one place route
params are bound (web.Router.ServeHTTP), so handlers keep reading
c.Input().Get("id") unchanged. A single :id segment cannot work: Go decodes
%2F back into a separator before routing.
/v1/iam belongs to the IAM service, not to this one. cloud proxies that whole
subtree away (cloud/iam_edge.go). Anything registered there is shadowed and
unreachable — it fails as a wrong answer, not an error.
TestNoResourceClaimsForeignNamespace holds that line.
The policy filters key on names, not paths. authz_filter and filter_balance
hold hand-curated sets (super-admin endpoints, balance exemptions, demo-mode
allowances). Those names are DERIVED from the same table by policyKey, never
restated — so a policy entry cannot drift from the route it governs. Do not
hand-edit a path into those maps.
Adding a resource is one table entry. The tests will tell you if you got it wrong: every controller method the table names is checked to exist by reflection, every route must emit a policy key, no path may contain a verb, and an action whose handler reads no id must sit on the collection (a member action's route could never match).
An OpenAI-COMPATIBLE vendor is upstream Local, never OpenAI
Provider.Type IS the dialect declaration, and the two OpenAI-shaped upstreams are
not interchangeable:
| type | client | dialect | price from |
|---|---|---|---|
OpenAI |
NewOpenAiModelProvider |
/v1/responses — OpenAI's own surface |
CalculateOpenAIModelPrice, a table of what OpenAI RETAILS |
Local |
NewLocalModelProvider |
/v1/chat/completions (local.go, CreateChatCompletionStream) |
the provider ROW (inputPricePerThousandTokens), plus compatibleProvider for the API it speaks |
DigitalOcean already resolves to the Local client; so should every vendor that
merely SPEAKS OpenAI. Typing one OpenAI sends it to the Responses API, and every
compatible vendor shims that differently.
Measured, one provider (the Hugging Face router), one credential, one URL —
typed OpenAI: meta-llama/Llama-3.3-70B-Instruct answered, while
zai-org/GLM-5.2, deepseek-ai/DeepSeek-V4-Flash-0731 and moonshotai/Kimi-K3
returned an EMPTY message with a 200 and zero usage. All four answer directly, in
both dialects. Retyped Local, all four answer through the plane, reasoning
models included, with no <think> leaking into content — and no code change.
The reasoning streams carry response.reasoning_text.delta, which the Responses
loop does not case on (it handles only response.reasoning_summary_text.delta).
So the rule is a ROW, not a patch: an OpenAI-compatible vendor is Local with its
own URL and its own price. Reasoning stays a FIELD the dialect carries, never
something sniffed out of the content stream.
Three things still dispatch on substrings of the model NAME, and a model in
none of the lists gets wrong behaviour rather than an error — getOpenAiModelType
(which dialect), InlinesReasoning (strings.Contains(name, "deepseek"), directly
under a comment saying which SKU inlines is "a property of the catalog (HIP-0039)
… never a list here"), and the retail price table. Each is a place a new model
silently misbehaves; the Local row above is what they were compensating for.
LLM serving path (OpenAI-compatible /v1) — READ THIS
The ai module is mounted by hanzoai/cloud (HIP-0106). In prod it runs as
cloud-api in hanzo-k8s, fronted by gateway (api.hanzo.ai). Both
hanzo.chat (LibreChat, baseURL https://api.hanzo.ai/v1) and hanzo.app
(build) consume this exact surface.
Request flow for a completion:
gateway(api.hanzo.ai) validates the IAM JWT, mints identity headers, forwards/v1/chat/completions→cloud-api:8000./v1/modelsis also proxied here; admin endpoints (/v1/get-providers,/v1/update-provider,/v1/*-model-route) are NOT gateway-exposed (use the directapi.cloud.hanzo.aiingress + a session).controllers.ApiController.ListModels(/v1/models) requires a valid Bearer token — unauthenticated returns 401 with nodataarray (this looks like "0 models"; it is not — authenticate to see the real list).- Model → provider routing:
resolveModelRouteForOrgresolves in order DB routes (/v1/*-model-route, per-org → global "admin") → YAML config (conf/models.yaml, thecloud-api-modelsConfigMap — runtime source of truth in prod) → staticcontrollers/model_routes.gomap. The YAML wins, so changing routing live = edit the ConfigMap (then/v1/reload-model-configor restart). The static map is the fallback when no YAML is present. A models.yaml entry withalias_of: <id>is an alias: not a route, not listed.AliasFilterrewrites a POST body naming it to name<id>ahead of every other reader, so it is routed, gated and billed as<id>. An alias of a zen or enso SKU belongs in that family's catalog instead: its/v1/modelsstates it (aliases) and the family route resolves it, after this chain, so a row here for the same id wins./v1/decisionssends the service the route'supstreamand answers naming the id asked for. - Provider records live in the DB (
object/init.goinitLLMProviders), keyed by name:do-ai(DigitalOcean GenAI, the primary — backs OpenAI/ Anthropic/Llama/DeepSeek/Qwen/GLM/Kimi viainference.do-ai.run),fireworks,openai-direct,zen. Each call doesobject.GetModelProviderByName(route.providerName)→ readsClientSecret. - Provider keys:
ClientSecretiskms://SECRET_NAME.ResolveProviderSecret(object/kms.go) resolves it env-var-first (os.Getenv(SECRET_NAME)), falling back to a real KMS call — BUT only runs whenkms != nil, which needsKMS_CLIENT_IDorKMS_SERVICE_TOKENset. So the working prod recipe is: setKMS_CLIENT_ID(flips kms on) + provideDO_AI_API_KEY/OPENAI_API_KEY/ANTHROPIC_API_KEY/FIREWORKS_API_KEYas env (from a K8s secret sourced from KMS-managed values). The env-first path means no live KMS dependency on the hot path. - Provider re-seed self-heals
ClientSecret/ProviderUrl/State/Type/SubTypeon every boot from theinitLLMProviderstable — so fixing a stale key or upstream URL = edit the seed table and restart (no manual DB edit).
Vendor video (video_spec.go, video_vendor.go). The resale family lists
OpenRouter's video models too (lists: /v1/models, /v1/videos/models, one catalog; a
second list that answers 404 lists nothing): each entry with price SKUs and durations
is a clip SKU (zenModel.Video, outputs: ["video"], Variable, both paid floors, its
card the SKUs × markup); an editor or upscaler (no durations) is not listed.
/v1/models carries its video object (durations, resolutions, aspect ratios, sizes,
frame images, audio). POST /v1/videos/generations for one is admitted against that
object (400 names what it takes), reserved at its ceiling (the dearest SKU rate that
applies to the resolution, audio and images asked, × seconds, × markup: clipQuoteNano,
which the paid lane's quote reads too), and posted to OpenRouter's POST /v1/videos on
the first account not cooling. The job is a row (object.MediaJob) and a watcher
(clipJob.watch, every clipPoll): done, it copies the file into the host's object
store (object.Assets, Hanzo S3 from cloud), reads duration and frame size from the
MP4 (mp4.go), and bills once what the vendor stated (usage.cost × markup; the quote
when none is stated), on the create's context; failed, it releases the hold and bills
nothing. GET /v1/videos/{id} answers the row (Accept: text/event-stream streams each
change), /content 302s to a fresh signed URL, both held to the payer's subject.
ResumeVideos (bootstrap) watches every unsettled row a process before left, billing
it to the payer's cash. A listed media model with no token price (Lyria) is perCall:
Variable, priced, never free.
Vendor voice (voice_vendor.go, audio_api.go vendorVoice). The same family
reads OpenRouter's speech list (/v1/models?output_modalities=speech): each
speech-only entry is a voice SKU (zenModel.Voice: its voices, its per-character and
per-output rates; Variable and both paid floors unless the vendor serves it free).
POST /v1/audio/speech for one takes the speech admission, reserves the most its text
can cost (voiceCeiling: every character, and the longest the speech can run at its
output rate), posts OpenRouter's /v1/audio/speech on the first account not cooling
(orSend, which clipCall uses too), answers the audio, and bills
GET /v1/generation's total_cost × markup, else the listed per-character rate, else
the reservation.
Async video (/v1/videos) — Sora-style, NOT sync. Text-to-video generation
takes minutes, so /v1/videos/generations is ASYNC (the sync handler used to
hold the request ~104s → console /ai proxy timed out → browser 502 while the
generation actually succeeded). The lifecycle is three fast requests:
POST /v1/videos/generations (create → returns a video_<uuid> job id +
queued object immediately), GET /v1/videos/{id} (one status poll), and
GET /v1/videos/{id}/content (stream the finished MP4). zen3-video* route to
the spark-video provider (our GB10, spark-video.hanzo.ai/v1), whose OWN
/v1/videos API is already the same async create→poll→download shape the three
model.{Create,Retrieve,Download}VideoDOAI primitives proxy. Metering is per-user
and lands EXACTLY ONCE on completion: create RESERVES the per-video budget and
carries the *budgetHold in the in-pod videoJobStore (valid — the pod is
single-replica, the balance ledger's own invariant); the first request that
observes completed (poll or download) settles the hold + records the single
Commerce usage event (videoJob.markCompleted, idempotent); a failed job releases
the hold and bills nothing; a reaper releases abandoned/wedged jobs' holds (TTL)
so budget never leaks. Every poll/download re-runs the shared auth+routing policy
and enforces OWNERSHIP (caller subject == job subject, else 404 — never confirm
another user's job); the provider key is re-resolved per request, never stored.
The gateway passes every /v1/* sub-path through to cloud-api, so the poll/
content routes need no gateway change; the console /ai proxy allow-list adds the
v1/videos/{id}[/content] pattern (hanzoai/console#92).
zen models: branded, owned_by: hanzo, premium: true. Route to do-ai
upstreams (qwen3+/glm/kimi/deepseek — same working key, no GPU). Each zen model
declares its public owner in owned_by; the identity is injected at call time
via zenIdentityPrompt (hip-00NN). Do NOT route zen to Fireworks serverless
paths (accounts/fireworks/models/*) — that account returns 404 "not deployed"
for most of them.
Billing gate (routers/filter_balance.go + openai_api.go): non-premium
models need balance > 0; premium models need balance > starter credit. Balance
comes from commerce at /v1/billing/balance?user=<org>¤cy=usd. No
principal bypasses it: the gate always consults the balance, and a lookup that
fails returns 500 rather than free access. To verify premium/zen, credit the org.
/v1/responses turns a namespace tool (Codex declares each MCP server's tools as
one) into one chat function per tool it holds, named <namespace>__<name> and hashed
to fit 64 characters (flatName); a call comes back with the namespace and name
the client routes by (unflat), and a namespaced call in the history is named the
same way. Sending the namespace as one empty function left a model one schemaless
tool per server, and Codex refused every call it made (unsupported call: mcp__hanzo).
The chat relay — the caller's body, as written (controllers/forward.go)
Every non-family /v1/chat/completions (and /v1/responses, which becomes one)
whose route row relays — an OpenAI-compatible chat address, not Anthropic's
dialect — is sent the CALLER'S OWN BODY. Plain text, tools and images take the
one path. The relay writes four things and only these: model (each provider's
own id), the completion ceiling when the hold covers less than the caller asked
(under the caller's key; when they named none, max_completion_tokens to an
OpenAI row and max_tokens elsewhere), stream_options.include_usage on a
stream, and retrieved knowledge as one leading system message. ours
(fast/retrieval/retrieval_store) is read here and not sent on. unpriced
(models, provider, transforms, route, plugins, web_search_options,
service_tier: fields that buy something on OUR account the SKU price does not
cover) is REFUSED 400 by name on every path, before any vendor is asked, never
stripped. Nothing else is touched: every turn, temperature 0, seed,
response_format, stop, vendor extras. chatRequest holds response_format and stop raw because go-openai
cannot decode a json_schema or a string stop, which used to 400 before routing.
It fails over the way ask.cascade does — same candidates, retryTransient,
cooled, announce, recordRefusals, exhausted — and every decision is made
on the STATUS, before a byte is written. A stream is judged by its opening frames
(opening, the family's rule): an error inside a 200 moves the request; the
first answer frame commits it, and nothing after that moves it. dial is the
seam the cascade tests script. Billing is by the SKU, with
prompt_tokens_details.cached_tokens split out and priced as cached.
One prompt, on every path. usageRecord.PromptTokens is EVERY input token;
CacheReadTokens and CacheWriteTokens are parts of it, read from and written to the
upstream's prompt cache. Each usage shape is read into that one split (sniffZenUsage
into tokens{fresh, cached, written}, AnthropicUsage.billed into tally.prompt()):
OpenAI counts prompt_tokens_details.cached_tokens inside prompt_tokens, OpenRouter
its cache_write_tokens there too, Anthropic counts cache_read_input_tokens and
cache_creation_input_tokens beside input_tokens. Pricing bills the input rate on the
part read fresh (prompt less reads and writes), reads at the cache-read rate and writes at
the cache-write rate: a family SKU's cache_read / cache_write (a resold model's
input_cache_read / input_cache_write times the markup), else the input rate; the
table path's CacheReadPerMillion / cacheWriteRate. Before, the family and relay
paths recorded the fresh part as the prompt while the table path read the whole, so
cache_read > prompt on most cached rows, and no path recorded a cache write: a write,
priced above input, billed as a fresh read. Held by TestEveryUsageShapeCountsThePromptWhole
and TestACacheWriteBillsAtTheWriteRate.
A relayed stream OWNS ITS HOLD. The callback settles the real cost after the
handler has returned, and budgetHold.settle is one-shot, so the handler's
deferred release would win and the local ledger would never see a stream's spend.
forward reports that it handed the hold to a stream and the handler lets go;
the callback releases it on the way out if it ends without settling. The text
pipeline and the family pipe still release before their streams settle.
The relay does not race. A relayed attempt is committed at its status, so a
fast request takes the cascade; the reservation widthFor made covers it.
Anthropic-type rows and endpoint-less rows keep the older paths
(proxyToolRequestAnthropic, the QueryText cascade). An Anthropic-type FALLBACK on
a relayed route is passed over, not converted.
A family service: what ai reads from it (controllers/zen_client.go, zen_media.go)
Zen and Enso are served OUTSIDE ai, by a family service at ZEN_URL / ENSO_URL
(the Go zen binary today; the Rust one next). ai authenticates, gates, meters and
forwards; the service owns which upstream answers. The contract, one address each:
GET /v1/models:{"object":"list","paid":<bool>,"aliases":{"<id>":"<sku>"}, "data":[...]}.paidis the catalog's line, the platform's paid switch (zen's only).aliasesare the other ids the family answers to (claude-zen→zen6,hanzoai/zen→hanzo/zen): routed, priced and gated as their SKU, sent upstream as the SKU, answered wearing the alias, never listed; one that spells a listed SKU is that SKU. Eachdatarow:id,owned_by,mode(chat | embedding | rerank | image | audio | video),context_window,pricing{input, output, cache_read, cache_write} as decimal strings (per 1M tokens; per unit for media),pricing_tiers,access(waitlist),min_tier,funding(prepaidwhere any path spends a prepaid balance),plan,capabilities.vision.POST /v1/chat/completions,/v1/messages,/v1/embeddings: the body ai sends (familyBody),X-Org-Idthe org the call is for (a machine credential acting for a person: that person's org,tenant),X-Hanzo-Fronted-By: ai(ai bills; the family meters nothing),X-Hanzo-Spend+TE: trailersfor a plan's paid rungs.POST /v1/rerank,/v1/images/generations,/v1/audio/{voice,music,foley},/v1/videos/generations: buffered, billed per unit (zenMedia, the one media relay for HTTP and ZAP).- Back, on a fronted call:
X-Hanzo-Served(the Hanzo SKU),X-Hanzo-Arm(the upstream),X-Hanzo-Provider,X-Hanzo-Free: truewhen the rung bills nothing (the call then bills nothing),X-Hanzo-Failover(each arm that failed,<upstream> (<provider>): <reason>, one field an arm or joined by"; "), andX-Hanzo-Cogs: what the request cost Hanzo upstream in USD, every arm it ran included — a header on a whole answer, a trailer (declared inTrailer:) on a stream. A committed plan answer addsX-Hanzo-Cost-Boundand theX-Hanzo-Costtrailer (plan.go). GET /v1/admin/zen/catalog,PUTthe same,GET /v1/admin/zen/keys,GET /v1/admin/zen/free, underAuthorization: Bearer <ZEN_ADMIN_TOKEN>: the catalog a SuperAdmin edits at/v1/ai/router/catalog(routing.go), for zen whenever no zen is linked into the process. The engine (enso) answers the same under/v1/admin(/v1/admin/catalog,/keys,/free), and routing.go asks each family at its own.
What ai does with it: every priced answer settles EXACTLY, in nano-dollars
(retailNano, unitNano, rounded up, never floored to a cent): the hold
(settleNano), the debit (BilledNanoExact) and the row read one number, so an
embedding at $0.01/MTok bills $0.00001 a thousand tokens and a $0.001 rerank bills
$0.001. A hold still reserves whole cents (centsUp). X-Hanzo-Cogs is the row's
and the debit's COGS (CostNanoExact; UsageEvent.CostUSD reads usageMargin, the
same number), and each failed arm is a "failover" row beside the answer under the
same request id, billed nothing (failoverRows), as the in-process zen filed them.
A test that drives this whole contract stands up zenService in
family_service_test.go.
A family in process (plugin/)
A family service may run in this process instead: its engine linked as a Go package
(hanzo-inc/zen lib/zen and lib/enso: the Rust engine compiled to wasm32-wasip1,
translated by wasm2go, behind an http.Handler that pumps the module, makes its upstream
calls with the keys it is opened with, and writes its answers) and registered with
plugin.Register(name, handler), once per family per process (one instance holds the
family's key pool, lane health and sessions). The address decides: ZEN_URL=plugin://zen
is the handler registered as zen, any http(s) address the service over the network.
ai's family, discovery and decision clients carry the scheme (reaching), so relay,
tells, billing, failover rows, caching and attribution are the HTTP path's, unchanged.
One transport per hop.
plugin.Transportserves the request on the registered handler under the caller's context, which also ends when the caller closes the body early, so the engine hangs up on its upstream when its caller leaves. The body crosses through a pipe (a caller that stops reading stops the handler's writes); the head is handed over at the firstWriteHeader,WriteorFlush; declared trailers (andhttp.TrailerPrefixones) arrive once the body ends; a handler that has not begun withinHead(the family client'sResponseHeaderTimeout) is ended; a panic fails the call or breaks the body.- The host that registers the engines (cloud) holds the keys (KMS) and the catalog.
Acceptance bar before a plugin serves production (owner review): (1) it answers as the native Rust service on the whole conformance set (the zen suite's 25 scripted cases byte for byte, and streaming); (2) the amd64 and arm64 builds agree (dgx arm64, evo amd64); (3) CPU, RSS, allocation rate and per-request latency measured in the cloud pod's limits, plugin against today's relay; (4) Memory64 only for a module past 4 GiB, and then growth past 4 GiB and the bounds around the 32-bit wrap tested; (5) nothing untrusted runs through wasm2go: guest code stays in the hanzoai/wasm sandbox, which refuses Memory64. wasm2go is ahead-of-time translation: it removes the VM's overhead, not the module's own execution cost.
A model asked of its maker first (controllers/native.go, native_chat.go, anthropic_chat.go)
Owner, 2026-10-09/11: "in waterfall we should try all native anthropic api keys, then
fall to OR, then free models". A resale family names makers: the APIs of the vendors
that MAKE some of its SKUs, each asked for the SKUs its prefix names (maker).
openrouterFam.makers are Anthropic's (ANTHROPIC_URL, anthropic/) and OpenAI's
(OPENAI_URL, openai/, keys OPENAI_API_KEY, _2, _3, OPENAI_API_KEYS); an
unset address asks that maker nothing. OpenRouter out of credit took every OpenAI SKU and
Enso's OpenAI rungs down on 2026-10-16; a maker's key is what keeps one empty account
from doing that again. The SKU, its catalog price and every gate are the family's,
unchanged. dispatch asks the maker first and the family's own address only once no key
answered:
-
OpenAI speaks the request's own dialect (
native.chat): a chat is sent as written under OpenAI's id (dots kept:gpt-5.4-mini), its ceiling asmax_completion_tokens, streamed withstream_options.include_usage; the stream comes back as the family's, its cost read from the usage chunk (chatTap). 429insufficient_quotaand 402 rest the key as an account out of credit; amessagesbody is the reseller's. -
An account that cannot pay is said on the host's bus (
emptied,object.PublishDrained): a maker's key, an OpenRouter key (sendKeyed) or a route row, by its NAME, once per ten minutes while it stays so. cloud puts it on the SuperAdmin's feed as asupplypulse, which admin.hanzo.ai shows as a critical incident. -
Paid only, and only the route named. A maker bills every call, so it is asked only by a request seated on the paid lane (
!FreeOnlyFor(ctx)), for the route the caller named (f == fam && s == model), where the family lists and charges for it. A free route, the free pool,enso-free/zen-free/free, a route standing in for another and every unseated request never reach it (TestTheFreeLaneNeverReachesAMaker). -
Every key, in order. The pool is
object.Accounts(AnthropicKeys, AnthropicKeyList):ANTHROPIC_API_KEY,_2,_3, then each line ofANTHROPIC_API_KEYS. A key's place is its name's (2 for_2) whatever the names before it hold. Always from the first key that can serve, never round the ring: a prompt cache is the account's, so one account leads. What a refusal does to its key:- 401: benched
nativeDead(10 min), back at once when its list reads again. - 403, and an account that cannot pay (402,
billing_error, or the vendor's 400 whose sentence STARTS "Your credit balance is too low" or "You have reached your specified API usage limits"): benchednativeDead, by time alone. A 400 that merely contains those words is a validation error quoting the caller, and benches nothing — read loosely, one crafted request benched the whole pool. - 429: benched until
Retry-After(5 s when unstated), never pastnativeBusy(1 min). - 404, 408, 5xx, 529: the next key, nothing benched.
- Anything else (400, 413, 422), an answer that ended having said nothing, and a 200
that is not the stream asked for: the request's own. No other key is asked and the
reseller is, so nothing that worked through the reseller stops working.
A stream's head is waited for
nativeHead(30 s); a vendor that cannot be reached is not asked on another key.
- 401: benched
-
What a key lists is what it is asked for.
GET /v1/modelsper key at start and everynativeEvery(5 min):anthropic/claude-haiku-4.5isclaude-haiku-4-5, or the newestclaude-haiku-4-5-<date>; a SKU with a:variantis the reseller's. A key whose list was never read is not asked. A key's standing is logged when it changes (anthropic key ANTHROPIC_API_KEY is not serving (401: …)) and counted once incloud_supply_refused{provider="anthropic"}. -
As itself. Nothing here picks another model. When no key answers, the reseller is asked for the same model; when it refuses too,
fallbackreturns nothing for a third-party SKU andcandidateshas nobody left, so the caller gets 503providers_exhausted(TestAClaudeModelNobodyServesIsRefusedNeverStoodInFor). -
Dialects.
/v1/messagesbodies go as they came (model respelled, the caller'santhropic-version, the betas the shared account serves,shared()deciding whether the body is the shared account's at all). A chat, and so/v1/responses, is translated bychatAsMessagesand read back byclaudeChat— always as a stream, which the pipe assembles for a whole answer. A sampling parameter and a way of thinking are sent only to a model whose listed capabilities take them; what the Messages API has no word for (audio, a stored file, JSON mode with no schema, an answer begun for the model) iserrUntranslatedand goes to the reseller untouched. A request with nocache_controlis asked to cache its prompt, as on the reseller's path. -
Money. The answer bills the catalog price of its tokens (
zenModel.bill). The maker states what it cost at the vendor's own price (the SKU's rates less the family's markup, an hour-long cache write at the vendor's hour rate) asX-Hanzo-Cogs, header or trailer, exactly as a family service does;billnever goes below that times the markup. That a maker gave the answer is the pipe's own knowledge (serving.maker, set from the responsedispatchwas handed): a cost or a vendor in the headers of any other answer floors no bill and relabels no row. -
A stream its caller hangs up on is billed for what the vendor had written: the vendor invoices it either way.
claudeChattranslates every line already received,tapcounts every byte it passes, each states its cost when it is closed, andsettlecloses the body before it reads the trailer.sniffZenUsageno longer takes a Messages stream's opening count (message_start, an output of one token) as the answer's own: that billed a cut stream for one token, on every family (TestAStreamItsCallerHangsUpOnIsBilledForWhatWasWritten). An answer the VENDOR breaks off endsfinish_reason: "length"and then an error frame, so the whole answer assembled from it is never taken for one that finished. -
On the books.
X-Hanzo-Arm(the vendor's id),X-Hanzo-Provider: anthropic,X-Hanzo-Key(the key's NAME) andX-Hanzo-Failover(<id> (anthropic): key 1: 401 authentication_error, one a key that refused) ride the answer in process. The usage row'soriginisapi.anthropic.com#<place>(admin lens only), its span carriesgen_ai.hanzo.via = anthropic#<place>,gen_ai.provider.name = anthropicand the vendor's id asgen_ai.response.model;providerstays the family (openrouter). Margin by route isGROUP BY originoverbilled_nano/cost_nano. Never a key.
Enso and Zen rungs are walked by their family service, not here (hanzo-inc/zen): its
catalog asks a rung's upstreams in the order written, and its Go engine takes a provider
of kind: anthropic with key_env/key_envs, so the same order there is that rung
written before the OpenRouter rung for the same model. No rung in either production
catalog names a Claude model today (universe charts/app/files/{zen,enso}/catalog.yaml),
so there is nothing to reorder.
Strict mode — served unchanged or refused (controllers/strict.go)
X-Hanzo-Strict: 1 on /v1/chat/completions asks for the request to reach the
route's FIRST row exactly as sent, or be refused 409 strict_violation with the
broken invariant in X-Hanzo-Strict-Violation: route_auto (auto-routing),
translation (a family pipe, an Anthropic/QueryText row, /v1/responses),
ceiling_unset (no max_tokens named: the relay would write one), ceiling_lowered,
rag, param_dropped, rewritten, added, not_canonical (a repeated name).
No fallback, no cooled reordering: a vendor refusal is the answer. Served, it
carries X-Hanzo-Request-Sha256 and X-Hanzo-Upstream-Sha256 — SHA-256 of the
RFC 8785 form (go-json-experiment jsontext.Canonicalize) of the client body and
the upstream body, each without model and stream_options.include_usage; they
are equal on every strict answer. /v1/messages answers strict requests in its
own file (anthropic_api.go).
A forced tool choice reaches its upstream forced (forced_test.go)
tool_choice: "required" or a named function forces a call, and every path carries it
forced or refuses it; none sends it on as a choice that forces nothing.
- Chat upstreams (a family: Enso, Zen; OpenAI as maker; the relay): sent as written,
with
parallel_tool_callsand each function'sstrict(familyBody,chatBody). - Anthropic's Messages API (
chatAsMessages): required →{"type":"any"}, named →{"type":"tool","name"},parallel_tool_calls: false→disable_parallel_tool_use,stricton the tool where the listing takes a schema; no thinking beside a forced call. - Anthropic in (
anthropicToolChoiceToOpenAI,anthropicSerial): any → required, tool → named,disable_parallel_tool_use→parallel_tool_calls: false. - Responses in (
responsesToolChoiceToChat): the flat{"type":"function","name"}andcustombecome chat's nested form, a namespaced tool by itsflatName(named with or without its namespace),allowed_toolsits mode over only the tools it allows. A hosted tool, an MCP server, an undeclared name or an unknown word is a 400. - Which model can be forced is the engine's to know. Claude Opus 5.5, Sonnet 5.5 and
Fable 5.1 answer 400 to any forced choice, thinking or not, and no listing says so; the
Enso catalog marks those rungs
unforcedand hanzo-inc/engine walks a forcing request past them. Before that, enso-pro and enso-ultra (whose live supply was Sonnet 5.5 on Anthropic's key once OpenAI and OpenRouter ran dry) answered every forced call400 the model could not serve this request: the engine masks any sentence holding:, and the vendor's istool_choice: type "tool" and "any" are not supported for this model.The engine now names the field instead. The same tiers refused anytemperaturebelow 1 (`temperature` is deprecated for this model., Claude 4.7 and later); the catalog's Claude rungs nowdrops: [temperature, top_p]. - A refused Enso tier falls to
enso-freefirst (standIn), then the pool's routes: the engine judges a free answer and reads a forced call a model wrote out as text ([[{"name":…,"parameters":{…}}]],<tool_call>blocks) as the call when it names a declared tool with schema-valid arguments; the pool's routes are relayed as they answer. With no credit anywhere, that is how enso-pro and enso-ultra still return tool_calls. - Not done here: a maker's 400 on
anthropic/claude-sonnet-5.5(resale family) hands the request to OpenRouter, so with OpenRouter out of credit the caller reads 503providers_exhausted, not the vendor's refusal.
Decisions — /v1/decisions, the one decision path
controllers/decisions.go. It reaches the decision service (KAI_URL) through one
core, decide, which the HTTP handler and the ZAP twins (zap_decisions.go) call
after resolving the principal. It serves Kai (kai and its versions, below) and Jev by
OpenRouter's vendor ids (typesafe/jev-1.13, ~typesafe/jev-latest), selected by
model; the vendor ids reach Jev itself. Nothing named Jev is ever answered by Kai
(jevNamed), and a bare jev-* has no route, so it is 400. Every refusal is
{"error":{"code","message"}}. Kai bills $0.021 and Jev its list $0.042 per
million input tokens — the same row in model_pricing.go,
conf/models.yaml and hanzoai/pricing's decisionCatalog. zenlm/zen6 and
zenlm/zen6-flash are Zen answering through the same service by a chat completion
from Hanzo's router; they bill input and output ($2.28 / $7.26 per million, a PROVISIONAL launch price the owner has not confirmed: the
published Zen chat row). decisionCostNano prices input and output for every
model; Kai's and Jev's output rows are zero, so their bills are input alone.
- One principal pays. The ledger org is what the reservation, the debit and
every handle name:
billingOrgon HTTP; on ZAP the same rule over the gateway request'sX-Org-Id(carried on ctx,orgAsked) and the JWT's signedorgs. An IAM or vendor key has no membership and pays from its own org. - A handle belongs to the org that observed it.
observe/handleare sent as<org>/<id>;unscopetakes the prefix off anything said back. An answer with no handle in it goes back byte for byte. - The service is told the org. Every call
sendmakes carriesX-Org-Id: the ledger org, the same one handles are prefixed with. The service routes by it. The request is built insend, so the caller's ownX-Org-Idnever reaches it. - One body bound, shared with the service.
decisionBodyBytes(16 MiB) is the decision service'sBODY_BYTES; change both together. A body past it is 422request_too_long, after authentication. ai's socket admits more (26 MiB), so the handler sees it; a refusal a layer RETURNS (the framework reading an over-limit body included) is worded byDialectviacontrollers.Refusing. A transport-level refusal (zip's raw fasthttp server, above the socket's own limit — or the cloud edge'sGATEWAY_BODY_LIMITin front of ai) is written by fasthttp before any handler and cannot be worded here. - The body is read key for key, and only after the credential (
fieldsOf): the size bound is the one thing asked beforevouchedaccepts the credential; then an unknown top-level field (a case variant of a known one included) or a repeated one is 400 the moment it is seen, so the read is one pass over at most ten keys and the gateway prices and scopes exactly what the service reads. The forwardedmodelis always the priced route's upstream id; a model asked in any case is filed as its route id. - A handle call holds what its handle bills.
handledremembers, per org-scoped handle, the input tokens its observe and each later call billed, and a decision over it holds at least that — its own body is a few bytes. - A hold is at least a cent, spend is carried in nano. The ledger holds cents and
a decision costs a fraction of one:
decideholdsmax(1¢, estimate)and settles withsettleNano;BalanceLedgercarries the sub-cent remainder, which counts against what is available as the cent it has begun. A cent covers a whole wire's worth of state at Kai's or Jev's price. - An LLM decision bills no more than it was admitted at. A model whose output
is priced is held at a token a body byte plus
decisionPromptTokens(1,024) anddecisionOutputTokens(4,096) of output, and is settled and debited at most that hold whoever pays (BilledNanoExact), a plan included. Counts are clamped where priced: none below 0, output at 4,096, input at 2^30. An answer whose usage is not two whole token counts is 502 and billed nothing (decisionUsage). Known loss: the service writes the state out again before the router counts it (1e15 becomes 1000000000000000.0), so a number-heavy body can cost past its hold; the excess is ours and logged (decide), about $1.76 a call at worst, until the service passes the state's bytes through. A refusal after the router spent is ours too, bounded by the service's max_tokens and 64 in flight a model. - The reply never waits on the books. The debit goes to 16 settlers through a
1024-deep queue (full: filed inline, never dropped). A debit the ledger refuses or
does not answer in
usageTimeout(5 s) is tried 3 times with backoff; a retry after a lost ack is charged once: each usage record mints oneRefthe first time it is filed (object.UsageEvent.Ref), and the host keys the ledger on it. Not the record's RequestID — a hedge's winner and its losers share that one.controllers.SettledwaitsSettleBudget(); cmd/aid stops taking requests first, and cloud's ai plugin runs it as itsShutdownhook after zip drains. - Held answers are per org.
decisioncache.gois an LRU of Kai answers for an identical body (whitespace-insensitive, order kept) from the same org within a minute, under a freshid, billed like a miss. Each org holds at most an eighth of it and evicts its own. Never across orgs. Handle requests are never held. The key names the model asked (the id the answer names, sokaiand a version asked of its own service askainever share an answer); its revision is the weights (routing.sha256), plusrouting.capability{name, sha256} when the org's capability answered. A lookup uses the capability that last answered that org (kept on its share), so once a capability answers, the base answers the org held stop being served. An answer without the field keeps the weights-only key. Restateis the wording rule: every refusal in the service's shape,Retry-After+Retry-After-Mson 402/429/529,X-Request-Idon everything; the service'sX-Request-Idpasses through. TheDialectfilter sits OUTSIDERecovered, so a panic's 500 is worded too, and matches the path in any case, trailing slash or not. The ZAP cloud wire (MsgType 100) has no header slot: its replies carry status, body and error text only, never headers folded into a body. The usage row'srequest_idcolumn is the id the caller saw (ClientRequestID); the row's ownidstays minted here. A 502 names no address.createdon/v1/modelsis a release time where one is recorded: a models.yamlreleased:(RFC 3339; kai and the Jev ids carry theirs) or an OpenRouter SKU's owncreated. A model nothing records keeps the listing's time; none is invented. The shape is unchanged (Codex decodes it).pricingon/v1/modelsnames its unit in every key:prompt/completionare USD per token as decimal strings (OpenRouter's keys and unit);input_per_million/output_per_millionare the same rates per 1M. A bareinput/outputis per token to OpenRouter- and Vercel-style readers, so no per-million figure goes there (hanzoai/pricing's own document keepsinputper 1M; it is not an OpenAI-compatible catalog).- A listed price is the billed price:
buildprices every row withgetModelPriceForOrgOK(id, ""), the lookup billing charges from, so a bare id an OpenRouter alias serves lists the SKU's retail, not its config figure. No price is listed where billing has none. - OpenRouter's routers are variable (priced
-1, e.g.openrouter/auto,typesafe/jev-router): premium, paid-floored, held at the catalog's dearest rate on each side, and billed at the answer'susage.cost× the markup (the ceiling when no cost is stated)./v1/modelslists that ceiling with"variable": truein the pricing block: the most a token bills, never unpriced or $0. A catalog with nothing priced drops them. canonical_slugidentifies;idroutes:owned_by/id(hanzo/kai), the id when already qualified, absent for an unbranded passthrough and when another route owns that spelling. A published slug is refused (400), never taken by a family prefix (zenlm/zen5starts withzen) and billed at a default. Every entry path bills and gates on the raw id, so routing a slug needs canonicalizing at the entry first.- A UI reads a model's kind from
/v1/models, never from its id: every row carriesclass(ClassOf, the policy's own answer) and, for Hanzo's,family(lineage: enso/zen fromFamilyOf, kai, zoo; Jev is TypeSafe's, premium, no family). OpenRouter rows addname(vendor lead cut),description,inputs,supports_tools/supports_reasoning(fromsupported_parameters); family rows addoutputsfrom theirmodeandsupports_vision; a models.yaml row may state anameand adescription. Class and family resolve routes, sobuildstamps them after it stores the slugs — before, the first build waits on its own lock (TestFirstBuildDoesNotWaitOnItself). - A Kai version is
kai-<first 12 lowercase hex of the weights' sha256>or a release,kai-<major>.<minor>orkai-<major>.<minor>.<patch>(each part digits, no leading zero;kaiVersion). With no route of its own it is Kai: priced, gated (DecisionAs, which the balance gate reads a decision's model through, so the plan is asked aboutkaiin class ours) and filed askai, and its answer names the version. Any otherkai-…no route names is 400. - Each Kai version may run as its own deployment:
KAI_URLS. Comma-separatedid=urlpairs (kai-1.4=http://kai-1-4.hanzo.svc:8080,kai-1.5=http://kai-1-5.hanzo.svc:8080), read once at boot (InitKaiURLs, first inBootstrap): an id that is no Kai version (kaiitself included), an id named twice, or a url that is not http(s) with a host (no credential, query or fragment) refuses the boot, naming the pair. A decision whose id is a pair's goes to that url (kaiFor: Kai's own provider, only the address moved, so the same client, timeouts,KAI_API_KEY,X-Org-Id,X-Org-Capabilities, capture and billing), asked there forkai— the one Kai that deployment runs, which is how the stock image serves its folder — and the answer is renamed to the id asked. Every other id,kaiincluded, and a version no pair names (sent as asked) go toKAI_URL./v1/modelslists each pair's id askai's row under that id (owned_by hanzo, kai's price, class and family, slughanzo/<id>); no models.yaml row is needed. Blank or unset, nothing changes. Start a version: add its deployment and its pair; kill it: drop both. - Nothing named Jev reaches Kai: a route or a models.yaml
alias_ofthat would send a Jev-named id to a Kai upstream is refused where it resolves (Canonical,decisionModel). - A yes/no question is
boolean;noulis Jev's spelling of it. One kind, two spellings, and the answer mirrors the one asked:{"type":"boolean","probability":p}or{"type":"noul","noul":p}; a boolean carries noconfidence/answer_confidence(choice and score keep theirs), and Kai v0.3.x still adds both to a noul and refusesboolean(422) until v0.4.0 serves. Kai models get the body as written. Jev knows onlynoul: for a Jev modelconsultrespells eachbooleanquestionnoul(spell) and puts its answer back as exactly{"type":"boolean","probability":p}(unspell), in place overmembers, every other byte kept. A Jev model holds no state (the service refuses observe/handle for it), so no handle crosses this. - The spec is derived.
routers/shape.goreadsvalidate:"required,min=,max="andenum:"..."tags, a type'sSchema()(Content kinds) andVariants()(a question is a boolean, a noul, a choice or a score, bytype);controllers.Answer.Refusalsstates each refusal and its headers.
AI Login Manager — universal metering + connected accounts + 1% BYO fee
ai is the centralized login manager for every user's AI providers. Every AI call
from every surface (@hanzo/dev, Hanzo Desktop, hanzo.chat, hanzo.app) routes through
THIS router at api.hanzo.ai/v1. For the caller's org the router resolves WHICH
account executes the call, runs it, and meters ONE usage event — regardless of whose
credits paid for it. There is exactly one meter and one credential store.
Two account modes (one router, one meter):
- Hanzo credits (default). No org-owned provider for the requested model →
the global built-in (
Owner == "admin") serves it. A route row (models.yaml) bills its token list price; a resold OpenRouter SKU or a maker's answer bills its cost and the platform's fee, and an Enso SKU its fixed price (A pass-through call carries the fee, below). - Connected third-party account (BYO). The org connected its OWN OpenAI /
Anthropic / Google account via
/v1/ai/connections. The router uses the org's KMS-sealed key, the customer pays that provider directly, and Hanzo bills thehanzo_byokplatform fee (1%, the host's setting,fee.go) on the provider list price for routing + metering + management.
BYO is the EXISTING BYOK seam, not new plumbing. resolveProviderForUser →
object.GetModelProviderByNameForOrg(org, name) returns the org's own provider row
when it has one (Owner == org), else the global admin built-in. providerBYO
(controllers/platform_fee.go) classifies the resolved pair: BYO iff
provider.Owner != "" && != "admin" && == user.Owner.
Metering — one path, extended not rebuilt. Every completion terminal already
calls recordUsage (→ commerce POST /v1/billing/usage) + recordTrace (→ the
hanzo.cloud_usage warehouse). This feature adds three dimensions to that ONE
event: byo, fee_cents, account. platformFeeCents(costCents, byo) = byo ? ceil(costCents × hanzo_byok%) : 0. Billed amount = byo ? fee_cents : cost_cents; the
full cost_cents is recorded on every row for analytics even when only the fee is
billed. hanzo.cloud_usage is the ONE global usage store (its organization column
is the row-level tenant filter — a tenant reads only its own org; the admin org
reads unfiltered). The rich per-org analytics lens is entitlement-gated in
hanzoai/cloud (analytics.datastore); basic own-org usage is always readable.
Connected accounts API (/v1/ai/connections, IAM-authed, org-scoped). Curated
login-manager surface over the per-org object.Provider store
(controllers/connections_api.go). Keys are sealed via object.StoreProviderSecret
→ kms://… ref (fails closed if KMS is down); the raw key is NEVER stored in a row,
returned, or logged.
GET /v1/ai/connections→[{provider, connected, account_label, updated_at}](never the key)POST /v1/ai/connections{provider, apiKey}→ seals key, activates the org row; returns metadata onlyDELETE /v1/ai/connections/:provider→ deactivates the org row (resolution reverts to Hanzo credits)
The org is the one the caller acts in (GetOrg, below). Reads are every member's;
connect, reconnect, disconnect and the OAuth start are the org's admins'
(requireConnectionAdmin → administers: authz.Claims.OrgAdmin over the signed
set, or the SuperAdmin), on HTTP and on the ZAP door (zapConnectionAdmin) alike. An
sk- key administers nothing. Pinned by controllers/acting_test.go.
Enforce "even if logged in". All AI traffic must reach this router; no surface
bills a third party off-meter. hanzo.app's workspace default was flipped
openrouter → hanzo; hanzo.chat routes via its gateway config; @hanzo/dev defaults
to api.hanzo.ai/v1 under Hanzo login. Claude/Opus for @hanzo/dev is served HERE
(Anthropic-native /v1/messages + OpenAI-compat /v1/chat/completions) against the
org's connected Anthropic account + 1% fee — no direct-to-provider bypass.
Next slice (route selection). Connections are stored/sealed/listed and the
1%-fee metering fires the instant the resolver returns an org-owned provider. But
the route table maps all models to providerName: "do-ai" (the aggregating
upstream), so a connection row named openai/anthropic/google is not yet
SELECTED for its model family. Making a connected consumer account actually serve
its models is a bounded route-table change (model-family → org-connection selection
- real upstream model-id/base-url mapping) — deliberately not in this slice.
RAG — ONE unified surface (retires the standalone chat-rag-api)
ai is the single RAG layer for the whole platform. There is exactly ONE
retrieval pipeline; the standalone hanzoai/chat-rag-api (danny-avila/rag_api
fork) is now redundant and slated for archive.
One index, one embedder, one retrieval path. Every document — doc/crawl/
github/s3 ingest AND per-file uploads from hanzo.chat — lands in the SAME
per-tenant index {owner}-{store}-docs, fanned out to BOTH Hanzo Search
(keyword, Meilisearch) and Hanzo Vector (semantic, Qdrant) via
object.IndexDocuments, and retrieved via object.SearchDocuments (hybrid RRF).
See object/search_docs.go.
File-scoped RAG is a FILTER, not a parallel store. file_id is a filterable
dimension on that one index (DocIndex.FileID; Meili filterable attr file_id;
Qdrant payload file_id). Uploaded-file retrieval = a file_id filter over the
tenant index. object/rag.go holds the file-scoped logic (RagEmbedFile,
RagQuery, DeleteRagFile, RagFileContext); chunking uses
split.RecursiveSplitProvider (parity with rag-api's
RecursiveCharacterTextSplitter, 1500/100), parsing uses
txt.GetParsedTextFromUrl (PDF/CSV/XLSX/PPTX/…). Uploaded files default to the
rag-files store so they don't pollute curated docs.
One address family (controllers/rag.go): POST /v1/ai/rag/ingest,
/v1/ai/rag/embed, /v1/ai/rag/query, /v1/ai/rag/query-multiple,
/v1/ai/rag/delete, GET /v1/ai/rag/context. Ingest and embed are two ways in
to the one index; the rest read it back.
The one exception is POST /v1/embed (controllers/rag_librechat.go) — the
MULTIPART form hanzo.chat's uploadVectors() posts. Its request line is fixed
by that client, not by us, which is why it is not a spelling of
/v1/ai/rag/embed. Everything hanzo.chat READS it reads from the native family.
Migration: hanzo.chat's RAG client calls ${origin}/v1/ai/rag/* directly.
Auth/billing reuse the doc-RAG path (resolveSearchAuth/requireIndexAuth +
recordSearchUsage); owner is the authenticated principal (tenant isolation),
never a client-supplied field.
Search / Crawl / Web-search — THREE orthogonal products, ONE way each
Three distinct products, no overlap. Do NOT add a fourth crawl path.
-
POST /v1/search= native full-text index search (SearchDocs→object.SearchDocuments). Hanzo Search (Meilisearch, keyword,hanzoai/search-goatsearchHost:7700) + Hanzo Vector (Qdrant semantic,vectorHost:6333) fused hybrid-RRF over the per-tenant index{owner}-{store}-docs. Returns{hits:[…]}. Needs a populated index — an empty/new tenant index returns{hits:[]}(no error).GET /v1/search/statsreports index size. -
POST /v1/crawl= crawl a URL, get content back — THE single canonical crawl (controllers/crawl.goCrawl→object.Crawl→CrawlWithCrawl4AI). Backed by the self-hosted Crawl4AI (hanzoai/crawl,crawl.hanzo.svc:11235, Apache-2.0). Body{url}or{urls}(merged, deduped, capped at 10); returns{results:[{url,title,description,markdown,success,metadata}]}with LLM-ready markdown (prefers Crawl4AIfit_markdown, falls back toraw_markdown). Auth:requireIndexAuth(fail-closed, rejects read-only pk-/hz_ keys) + balance gate; usage recorded viarecordSearchUsage(…, "crawl", "crawl4ai", …).object/crawl4ai.gohandles the deployed Crawl4AI 0.8.6 wire shapes:markdownis an OBJECT (MarkdownFieldpolymorphic decode), the/crawlenvelope is synchronous with a booleansuccessand nostatus/task_id(return inlineResultswhenever present), andlinks/mediavalues are heterogeneous (map[string][]map[string]interface{}). -
POST /v1/scrape= crawl-and-INDEX (ingest) — NOT a general crawl. It crawls a site and WRITES structured content into the tenant search index, returningScrapeStats(ScrapeDocs→object.ScrapeAndIndex). Sibling ofPOST /v1/index(index caller-supplied docs). Stays because its output/effect (index write) is distinct from/v1/crawl(fetch + return). -
POST /v1/scrape/previewis a DEPRECATED alias of/v1/crawl— its handler now forwards toCrawl(one implementation, no parallel path). Use/v1/crawl. -
/v1/websearch/*(lives inhanzoai/cloud, not ai) = web meta-search — a separate product: SearXNG-compat/v1/websearch/search+ a firecrawl-compat/v1/websearch/v1/scrapeleg that hanzo.chat's web-search plugin calls. That leg is a fixed EXTERNAL contract (firecrawl{success,data:{markdown}}) and reuses the SAME Crawl4AI backend — it is an integration shim for the web-search flow, not a general crawl endpoint. The general "crawl a URL" is ONLY/v1/crawl.
Security invariants (READ before touching auth/billing/scaling)
-
JWT iss/aud — every request-auth JWT path goes through
object.ParseAndValidateJWT(signature via IAM and iss/aud policy), never rawiam.ParseJwtToken.object/jwt_validate.gosources the issuer fromJWT_ISSUER/IAM_ISSUER/CLOUD_IAM_ISSUER/AUTH_ISSUER(any, defaulthttps://hanzo.id) and the audience allowlist fromGATEWAY_ALLOWED_AUDIENCES(+ folds inIAM_AUDIENCE/AUTH_AUDIENCE), mirroring the gateway. A foreign-aud token → 401. The OAuth code-exchange callback (account.goSignin) is NOT a request-auth path and is intentionally exempt. -
A delegated token is the HOST's to verify — a token whose scope names
ai:inference(iam.Confined) is a person's credential IAM delegated for model calls (iam LLM.md "Delegation"), signed by a key the public JWKS never carries.ParseAndValidateJWTsends it to the verifier the host installed (object.SetConfinedVerifier— hanzo-inc/cloud passes its identity boundary's verdict on the exact token bytes) and reads its claims from the verified bytes (iam.ClaimsOf); with no verifier installed it is refused (ErrJWTConfined). Never a header: the module asks about the token it is about to believe. -
No ZAP door serves a delegated token — the host's boundary, which keeps one to model calls, stands in front of HTTP alone. The gateway (MsgType 200), the native node (100), the ops node (110) and the forward bridge refuse one 401 before any verifier is asked, wherever in the request it rides, and so do the identity seams they share (
zapResolveAuth,zapResolveUser,zapPrincipal).controllers/zap_confined.go;TestNoZapDoorServesAConfinedToken. -
A delegated token names no app — its
audis the API (RFC 8707), socontrollers.Appsanswers none for it, as for an API key: a program that named its model gets that model or the 402 saying what pays, never limited mode's free model in its place.TestADelegatedTokenNamesNoApp. -
Retrieval is stated once —
retrieval.Asked/retrieval.Store(a leaf) is when a model call searches the org's documents into its prompt; the controllers and the host that refuses retrieval to a delegated token read the same function. -
Balance ledger — single-pod invariant —
object.GlobalBalanceLedgeris in-pod memory. Reserve/Settle are correct ONLY atreplicas: 1. Two pods double-spend (each reserves against its own cached Commerce balance). Enforced by deploystrategy: Recreate+ HPAmin=max=1AND a boot assertion (bootstrap.gopanics ifCLOUD_REPLICAS > 1, the count the cloud chart injects). The paid lane's plan holds (controllers/lane.go) are in-pod too. To scale out you MUST first move Reserve/Settle behind a Commerce-atomic conditional reserve. -
Address lanes (
address.Buckets) — IPv4 whole; IPv6 its /64, then its /48 at 16×, the wider charged only for what the narrower admitted. The router asks only the /64 before buying an IAM round trip for an unresolved key, and charges the /48 only to a caller who stays anonymous, so a neighbour cannot close a paying key's /48. A resolved key stays named past its 5-minute TTL (up to 2×) while one background re-check replaces it. The free lane's DAY holds a /48 tositeDay(256) visitors, not 16: sixteen strangers a day is a quiet carrier /48. -
Single-request reservation (
controllers/billing_reserve.go) — an uncappedmax_tokensis clamped (clampMaxTokens) and the reservation covers the larger of the clamped ceiling and the QueryText pipeline's fixed completion cap (reserveCompletionTokens/reserveCompletionFloor=4096, mirrored inmodel/openai_util.go), so actual spend can never exceed the hold on any path. -
Native
/v1/messagesis the caller's bytes (nativeRequest, HTTP and ZAP, text-only included — QueryText never serves a native upstream) — onlymodelandmax_tokens(the reserved ceiling) are spliced;anthropic-version/-betaforwarded; only content-type and ourX-Request-Idcome back.splicerefuses a top-level key repeated or case-folded (encoding/json folds case, keeps the last). The shared account serves an ALLOWLIST (sharedFields/sharedTools/sharedBetas, no heldfile_id); a BYO account is sent anything. The hold is body bytes/3.5 + ceiling; an answer without the vendor's final count bills no less (lost); upstreams get an idle deadline, not a total one; no SSE line cap. Billing: SKU price, everyusage.iterationsrun, cache writes 1.25x (5m) / 2x (1h, folded into write units until usageRecord carries a 1h count). A stream takes its hold withcarry(anthropic + family), or the handler'ssettle(0)wins. -
A tier is read through
object.TierReader, and the host owes a reader that answers — there are two consumers and they must ask in the same order:controllers.commerceFamilyTier(the per-SKU gate) androuters.TierCache.commerceTierLookup(the rate limiter). Both read the installed reader FIRST; the HTTP route is what a STANDALONE ai has instead of one, not a second way to ask.InitTierCachebuilds the cache when EITHER exists, not when the endpoint alone is set.The reader is NOT necessarily in-process. Every cloud app is its own child PROCESS, so a Go func var cannot cross that boundary: what cloud installs is
metering.Client.Tier, which is an HTTP call toGET /v1/billing/tiercarryingCOMMERCE_SERVICE_TOKENandX-Org-Id. That call has no user session, so it only succeeds where the handler resolves its tenant with billing'sreaderOrg(session OR trusted service naming an org) rather thanprincipal.OrgFromalone.balancehad that rule; the typed ops behindtierdid not, and answered401 sign in to view financeto a token the same request proved good on/v1/billing/balance. Fixed in cloudapps/billing/typed.go(payerfalls through toreaderOrg), guarded byapps/billing/tier_s2s_test.go.The failure is silent in both directions, which is why it lasted: the rate limiter maps a failed read to
TierZenFree(every paying org shaped as free) and the SKU gate maps one to unknown and ALLOWS (it stops gating). Neither logs an error anyone watches, so assert the tier a caller actually gets rather than that the lookup "worked". -
A failed call bills only what a vendor actually ran —
recordUsagebills a failure because by then the vendor has run the request and invoiced us. That premise does not hold for a refusal it can decide from the request alone (400, 401, 402, 403, 404, 413, 422, 429) or for an error carrying no status at all (nothing was dialled).controllers.ran(err)answers that, and it is NOTfaultOf: faultOf asks where the request goes NEXT, and the two come apart both ways — a 401 fails over having run nothing, a partial write stops the cascade having run plenty.spent()consultsranonly when nothing came back, so a meter the vendor reported and any text that reached the caller still settle it first. Without this the quantity billed is the prompt measured on the way out, so the charge grows with the request: 400k tokens refused for being too long billed a dollar. -
Exemption is by ROUTE, never by identity.
BALANCE_EXEMPT_USERSis read by nothing;TestEnforceBalanceGate_NoExemptBypassandTestCheckBalanceNoExemptionset it and require the formerly-listed subjects to fail closed. What is exempt is a set of paths —isBalanceExemptinrouters/filter_balance.go— plusbalanceExemptNames, three usage reads named as POLICY rather than URL so a $0 org can still see the panel telling it to add credit, and so moving the route cannot silently re-gate them. Exempt from balance is not exempt from auth. -
SuperAdmin is the host boundary's verdict, never a reading of an org name. The host installs it (
object.SetSuper; hanzo-inc/cloud installsprincipal.Super, which reads theX-User-IsAdminits boundary deletes from every request and re-mints fromauthz.Claims.Sudo— a PERSON whose own org isadmin). Every HTTP seam that turns a request's token into a principal goes throughobject.Verify(c, token), which stamps the verdict oniam.User.Sudo(json:"-", never decoded);util.IsSuperAdmin(user)reads that field and nothing else, andutil.ScopeOwner(user, requested)asks it. No code here compares an org to"admin"to grant authority; the reserved org's NAME (authz.AdminOrg) is used only to say which ROWS are the platform's (provider rows, the platform's index). What this replaced decided on its own and disagreed with the boundary: an admin-org application (noorgs,ownerclaimadmin), an admin-org person'ssk-key, and an org spelledadmin(the old parse trimmedorgs[0]) were each SuperAdmin here and nowhere else.isAdminis not part of it — an org admin (hanzo/z) administershanzoand holds no platform power. -
The org a request acts in is the signed set's answer, by cloud's rule.
GetOrg→actingOrg(controllers/org_resolver.go): the SuperAdmin selects any org; a credential with no membership set (sk-key, run key, a program's token) and a confined delegation act in the org that holds them; everyone else getsauthz.Claims.EffectiveOrg(X-Org-Id)over the verified token — the same predicate cloud'sSanitizeIdentityasks, so embedded and standalone ai agree. A membership opens its org unless it is the admin org, a brand's directory (hanzo-id), or an org the platform runs (hanzo,lux, …) held as a plain member; a set that opens nothing acts in no org. It used to answeruser.Owner(orgs[0]) whenever nothing was selected andaccount.EffectiveOrg(membership only) otherwise, so a plain member of a brand org acted in the brand's org and could repoint its shared AI connections. The money resolvers (billingOrg,openai_api.go,filter_balance.go,zap_decisions.go,zap_router-policy-stats.go) still askaccount.EffectiveOrg.With no host — standalone
aid— there is no verdict and no SuperAdmin. A ZAP door is not the boundary either:target(the one place a ZAP-built request reaches the router) deletesX-User-IsAdmin, andzapPrincipal(ctx, auth)carries a verdict only when ctx does, which only the HTTP bridge (RouterConfigBridge,vouch) sets, fromobject.Superon the request it holds. A key is never the SuperAdmin on any path, as at the host.controllers/superadmin_test.go,routers/authz_filter_test.go(asSuper). -
A tenant's URL is fetched by ONE rule:
github.com/hanzoai/outbound.proxy.Publicisoutbound.Client(judged at the dial on the address the socket connects to, so DNS rebinding is refused on the answer that would have been used; every redirect a new dial judged the same way; no proxy; the IANA special-purpose registries, so NAT6464:ff9b::a9fe:a9fe, 6to4, Teredo,fd00:ec2::254, CGNAT and 240/4 are refused). It fetches every address a request names: a message'shrefand images,/v1/ai/rag/embed'surl,/v1/ai/rag/ingest'sfiles[].url, and every page the Go scraper crawls; the browser engine (crawl4ai) is handed only whatoutbound.Hostfinds public. A tenant document goes throughtxt.GetParsedTextFromAddress, which reads http/https only —txt.GetParsedTextFromUrlreads a scheme-less string as a LOCAL PATH and is for addresses this system minted (stored files) alone. A test swapsproxy.Public; outbound has no switch.object/address_test.go,object/address_redirect_test.go,object/scrape_address_test.go,proxy/public_test.go. -
Authz gate (
routers/authz_filter.go) — platform-sensitive endpoints (superAdminEndpoints) are gated FIRST (before preview-mode + exempt reads). There is deliberately NOadminDomainHost bypass (a removed full-authz-bypass primitive). No principal → 401; wrong principal → 403. -
User redaction (
object/redact.go) —RedactUserSecretsis an allowlist projection over string/[]string fields (exposableUserFields): a field not on the list is zeroed, so a NEW upstream secret field is fail-secure. Credential struct slices (MfaAccounts/ManagedAccounts/MfaItems/FaceIds) are nilled.
Cost: known, zero, or not known — three answers, never two
An unregistered COGS used to read back as the CUSTOMER PRICE. costInputPerMillion()
returned the price when no cost rate was set, so tokenProviderCostNano computed cost
== billed and every such call recorded margin_nano == 0. For the Enso family that was
every call: 1,296 rows checked, all of them zero. It is not a rounding error — it is a
ledger reporting a business that earns nothing on everything it sells.
The shape of the fix is that cost is optional and margin follows it:
controllers/model_pricing.go—modelPrice.costed()reports whether the model states what it costs us, and it requires BOTH legs. Half a COGS is not a COGS: an input rate with no output rate prices the completion — the expensive half — at zero.costInputPerMillion()/costOutputPerMillion()now return the configured rate and nothing else; there is no path that reads one without first askingcosted().controllers/billing_nano.go—tokenProviderCostNanoandproviderCostNanoreturn*int64, nil where no cost is known.costMargin.CostNanois a pointer for the same reasonMarginNanoalready was: the row has to be able to say it does not know, and an int64 can only say zero.usageMarginreports a margin only when both sides are real.controllers/zap_native.go+object/cloud_usage.go— the warehouse column isInt64, so an unknown cost writes 0 and setsuncosted = 1beside it. Queryuncosted = 0for rows whose cost is a fact.cache_saved_nano(cacheSavedNano,controllers/billing_nano.go) — what the prompt cache took off the call's bill: cache-read tokens at the model's input price less the same at its cache-read price (cacheReadRate, shared withtokenNanoAt), from the billed table. 0 for a free, BYO, stated-amount, decision, image, video or audio call, and for a row written before the column: 0 there is "not measured", so a sum is a floor. cloud's/v1/usage/summaryreads it asllm.cacheSavedCents.controllers/telemetry.go— the span emitsprovider_costonly when there is one, the way it already treated margin. It no longer asserts a COGS it does not have.
Zero and unknown are different answers. Zero is a fact: BYO (the customer paid the upstream with their own key), a free route, speech on hardware we already own. Nil is the absence of one: image and video have a per-unit PRICE table and no vendor invoice behind it, and a token model that states no COGS has told us nothing.
Speech sells at the published rates, in nano. sttNanoPerSecond (100,000 =
$0.006/min) and ttsNanoPerChar (15,000 = $15 per 1M characters) are hanzo.ai/pricing's
Speech-to-Text and Text-to-Speech rows, keyed by every id that reaches the speech
service. Nano because a dictated sentence is seconds long and a per-call cent would bill
it at twelve times the rate. The speech routes carry no token price, so the balance gate
asks for a funded caller.
Three speech doors, one service. /v1/audio/transcriptions and /v1/audio/speech
are controllers; /v1/audio/transcript (the growing transcript) is the ZAP gateway
handler in zap_transcript.go, bound to HTTP by AudioTranscript through the
in-process bridge; /v1/voice is hanzoai/voice behind zip.AdaptNetHTTP, which takes
the upgrade's connection (zip v1.37.26+). The socket's ticket is its credential, so
voice is anonymous at the bearer filter; its gate trusts object.TrustedJWTIssuers
(IAM_URL is where IAM is reached, not what it signs as) and admits the browser origins
in VOICE_ORIGINS. The speech provider row speech is operated: the paid lane being
off never refuses it.
The visitor's mic is a public lane (controllers/public_scribe.go,
POST /v1/audio/transcriptions/public): no credential, the model assigned
(zen-scribe), the audio held to 60 s by the speech service's max_seconds (it stops
decoding just past it, stt.WithLongest), and a day per visitor in its own dayCount
keyed by Lanes (the /64 and the /48 at siteDay). PUBLIC_SCRIBE_DAILY is the
switch and the ceiling; 0 closes it. It never reaches a vendor: a zen-scribe route to
a provider that is not operated closes the lane. Nothing is kept or logged.
Where a cost is registered
object.ModelRoute.CostInPerMillion / CostOutPerMillion — the route row, which the
admin cockpit already reads and writes. It is the only place, and the configured PRICE
tables are deliberately not consulted as a substitute.
The family leg used to make that unreachable for zen/enso: familyModelPrice returned
first and short-circuited the resolver, and the family wire (zenWireModel) carries no
cost field, so an enso model could not be given a COGS at all without restating its
price. getModelPriceForOrgOK now completes an uncosted family price from
registeredCost. Price and cost resolve independently, because they are independent
facts: the family owns what we charge, the route row owns what it costs us.
What still has no COGS
The mechanism no longer lies, but a rate nobody has entered is still a rate nobody has
entered. A route row (models.yaml, the static table) states no cost until an operator
sets both legs; one leg alone is ignored by costed(). Until then the rows read
uncosted = 1, which is the honest answer and a visible one: SELECT count() FROM hanzo.cloud_usage WHERE uncosted = 1 is the backlog. A family's answer (Zen, Enso, the
OpenRouter resale) states its own cost on every call (X-Hanzo-Cogs, or the vendor's
usage cost), which wins over the rate table in usageMargin: rate math cannot express
a fan-out or a failover's cost at all, and only the family can total it.
The admin cockpit's below-cost check compared price against a "cost" that WAS the price, so its difference was always exactly zero and it could neither fire honestly nor stay quiet honestly. It now has a cost to compare against, or a flag saying there is none.
A pass-through call carries the fee; Enso sells at a fixed price (fee.go, rates.go)
Owner, 2026-10-15: a model served through the gateway bills the price of the arm that answered, passed through, and the platform's fee on top: 5.5% (OpenRouter's own rate) on Hanzo's accounts, 1% on the org's own provider key. The fee is the margin there; nothing is ever billed below cost. Enso is the exception, decided after: NO adaptive or dynamic Enso pricing. Each Enso SKU has one fixed list price, and nothing measured moves it.
-
Fees are the host's (
object.Fee, cloudapps/flagsSeedFees, edited at admin.hanzo.ai Usage policy):openrouter(5.5, the vendor's fee on its credit, part of the cost),hanzo(5.5, the platform's) andhanzo_byok(1).fee.goreads each at most once a minute (levy), a stale read answering while the next runs; a read that is no percent keeps the last good one, and a platform fee of zero is refused there and at the admin write. A host that states none bills at cost: never below it. -
Cost of record (
costed): what the family (X-Hanzo-Cogs) or the vendor's usage stated, times the fee of the costliest vendor whose credit bought any arm of it (buyers: OpenRouter for the resale family's own answers; the answering arm's provider and every failed arm's for a family's; none for a maker's, whose price is its own). It iscost_nano, on every call, Enso's included. -
Pass-through (not Enso): a resold OpenRouter SKU bills its vendor's price times
(1 + openrouter fee) × (1 + hanzo fee)(openrouterMarkup), never below its stated cost times that. A maker's answer (native.go) billsplus(cost), its cost of record times the platform's fee. A BYO call (providerBYO) bills thehanzo_byokfee alone. -
Enso's price list (
ensoPrices, the one catalog): each paid SKU is the list price of the upstream its route typically answers on (the cost the served catalog, universecharts/app/files/enso/catalog.yaml, states for its first rung) times 2 to 3, and well under Claude Fable 5.1 and GPT-6 Astra ($10 / $50 at their vendors). Per million tokens, input / output / cache read / cache write:SKU typical upstream, list × price enso-flash GPT-6 Luna, 0.11 / 0.55 / 0.011 / 0.1375 3 0.33 / 1.65 / 0.033 / 0.4125 enso-pro GPT-6.1 Sol, 2.20 / 11.00 / 0.11 / 2.75 2 4.40 / 22.00 / 0.22 / 5.50 enso-ultra Claude Sonnet 5.5, 2.20 / 11.00 / 0.11 / 2.75 2.5 5.50 / 27.50 / 0.275 / 6.875 Flash is 3x because Luna past a 272k prompt costs 0.20 / 0.75; ultra 2.5x because its Opus 5.5 arm (4.40 / 22.00) with OpenRouter's fee would sit above twice Sonnet. enso-free, the one SKU the catalog serves free, has no line and bills nothing; enso, enso-auto and hanzo/enso are no SKUs but the router over these three (Enso's router, below).
modelFamily.fixlays the list over the family's discovery, so/v1/modelslists it,zenModel.billbills it, the hold reserves it, and the price list reads it (Prices, which cloud lays over/v1/pricing): one number in every place. A SKU the service prices that the list does not is dropped from the listing and never served, said in the log at each discovery. A price change is an edit ofensoPricesand a release; the engine's own quote (its catalog margins) is not read. -
The guard (
modelFamily.guard): a SKU is sent only while every arm of its route costs us no more than the SKU bills, in every token class, at a long prompt's price where an arm charges one, and with the fee of a reseller whose credit buys it. The route is the engine's admin catalog (GET {ENSO_URL}/v1/admin/catalogunderZEN_ADMIN_TOKEN,modelFamily.route): the first read is waited for, then read again beside the requests once a minute (routeEvery); a failed read keeps the last; a first read that failed answers its failure for 15 s. What cannot be checked is not sent: no price, no route read, or no arm for the SKU is a refusal. A refused SKU answers 503 naming no upstream (its route costs more than it bills/its route cannot be checked), is asked of no other provider, and is skipped as a stand-in (pool); each misfit is aguard:error in the log at every read. After the fact, an answer on an arm the route as read does not hold, or that costs more than the SKU bills, is aguard:error and reads the route again on the next request (trail). The free SKUs pass the same guard at a price of zero, so a free SKU is never sent to a paid arm, and a pod that has never read the route serves no Enso at all.TestTheServedRoutesFitUnderTheirPricesholds the list to the served catalog's arms. -
Never silent: a call whose cost exceeds what it billed (a stand-in capped at the named model's price, a free rung that cost something) writes
loss_nanoon its row and aloss:warning. The ledger counts every one. -
The row (hanzo.cloud_usage, one a call): beside
model(the SKU),cost_nano,billed_nanoandmargin_nano, a family's rows carryservedandvendor(the arm and the provider whose account ran it),reasoning(the depth asked,depthOf, mirroring hanzo-inc/engine thinking.rs: none, low, medium, high, max, or default),reasoning_tokens,loss_nano, and on Enso'sarms: JSON, the arm that answered with its rate at the time (rate, the vendor's fee asfee) and the tokens it was paid for, then each arm that failed first and why. Failed arms keep theirfailoverrows too. -
The ledger is a report (
Prices(ctx, admin)): to a SuperAdmin, each SKU's typical upstream, its list and the multiple, its route as last read with each arm'soverclasses andrefused, and the last day'sLedgerread from the usage table on demand (object.RoutedRows, one row per SKU, arm, vendor and status, deduplicated by id first): answers, failovers, OpenRouter-bought answers (fallbacks,fallback_share), cost, billed,margin_nano(billed less cost over the answers whose cost is known;uncostedcounts the rest), losses, and the same by route. Nothing reads it to price. -
Resolution is stated (
resolve.go): a model is reached on a direct key first (its maker's API for a SKU its keys list, native.go, ahead of the family that resells it; a route row's own provider), then DigitalOcean's inference (do-ai), then OpenRouter, the fallback (openrouterTail, a route's last candidate).Paths(id, org)states that order for one model, each path'skind(direct, family, do, openrouter) and whether a key serves it. Between a row's own providers a call keeps the row's declared order (candidates), so every row ofconf/models.yamllists its direct provider beforedo-ai, held byTestEveryShippedRouteAsksADirectKeyBeforeDigitalOcean. -
Gone (v1.833.348): the from / typical / up-to quote, its depth quotes,
Price,Blend,Reprice,StartRatesand the daily measure, and/v1/models'pricing.from,pricing.up_to,pricing.fee,pricing.windowandvariableon Enso. Cloud's/v1/pricing/ensoquote andPOST /v1/pricing/syncread them and go with its next pin of this module.
The routing catalog (routing.go, /v1/ai/router/catalog, SuperAdmin): zen is
read and written at the Go zen's /v1/admin/zen/*, enso at the engine's /v1/admin/*
(until v1.833.351 enso was asked at the zen address and every read answered 404).
- An apply the guard would refuse is not sent:
guardRefusesholds every priced model the edited catalog serves tomisfitagainstensoPrices, the guard's own rule, and an apply answers 400 with the arm and classes; a proposal carries the same sentence asrefused, so it shows before anyone confirms. Without it an edit that put an arm over its model's price applied cleanly and the model answered 503 within a minute, when the guard re-read the route. - Convergence leaves a new deployment standing. Each version records
boot, thedeploymentdigest of the catalog its run of versions was made over: the catalog less every field the admin view derives (keyPresent;effectiveMargin,minTier,funding; a rung'sretailandprofit), so a key that resolves or an engine that derives another figure is no new deployment. A version made while the family served the newest keeps the newest'sboot, one made over anything else starts a run.converge(underroutingMu, one step with any apply) re-applies the newest version only when the family serves thatboot(it restarted onto the file the run was made over); a family that restarted onto a new release of its file is left serving it, warned once, and the read saysredeployed: truebesidedrift. Before, the first stored version would have been pushed over every later release of the family's file, a minute after each restart. - An edit over a restart waits. An apply made while the family serves the newest
version's
bootbut not the version itself (restarted, not yet converged) is refused 409: made from what serves, it would drop the newest version's own edits. Convergence sends the versionbare(less its derived fields, which a family that dropped or renamed one would refuse), and once it has failed on a deployment (unconverged) an edit is let through, so a version the family will no longer take never locks edits out. - The engine's admin view carries no
longprice, so neither the guard norguardRefusessees a rung's long-prompt price; both read the same view, so they agree. RoutingRead,RoutingProposeandRoutingApplyare the one implementation, exported for cloud's plane (ai_routing,ai_routing_apply), which the Enso policy registry (cloudapps/train/v1/train/policies) promotes through.
Enso's router: one tier a request, enso-free when nothing pays (depth_route.go, routers/filter_balance.go)
Owner, 2026-10-15: only enso-free is free. enso, enso-auto and hanzo/enso are the
paid router, and they are ai's, not the engine's: the engine serves leaf SKUs only, and
universe's enso catalog lists none of the three.
- The router is
router.depth(cloud-api-models, universevalues/hanzo/cloud.yaml): a row per router id, reasoning depth → tier. Up to low (and a request that states no depth) enso-flash, medium and high enso-pro, xhigh and max enso-ultra. The depth is the request's own (requestDepth:reasoning_effort,reasoning.effort, a token budget,thinking). No learned router is in the path: the engine'sPOST /routeis empty inconf/models.yamland its policy serves nothing in production. - Resolved before anything prices it (
BalanceGateFilter): a request naming a router id is rewritten to its tier and then gated, held, seated, billed and guarded exactly as a request naming the tier, at the tier's fixed price (ensoPrices). Explicit tiers and enso-free are unchanged. - Nothing pays the tier, enso-free answers at $0, never a refusal for want of
payment: the family serves the tier to no one at a price; a program in this process
made the call (
object.Caller: its meter charges for the id it named); with no usage policy, the wallet cannot cover it; with one, the paid lane's switch is off, the policy refuses the tier or names nothing, names a payer the paid lane does not seat (the free plan, a free cap, granted credit:paid_plan_required), names prepaid cash the wallet cannot cover (insufficient_balance), or the paid lane has no room (paid_lane_full). A paid plan's included usage and a payer's own prepaid cash pay, plan or none (owes). What admitting the tier took is given back; a refusal enso-free meets as well (a request window, the free allowance) is enso-free's. - The answer says so:
X-Routed-Modelnames what answered (the tier or enso-free); the chat envelope'sroutingobject saysrouter: enso,chosen, the depth and key it read,policyfunded | unfunded | n/a, and a sentence of why. A refusal for want of payment adds limited mode's headers (fallBack):X-Hanzo-Fallback: enso-free,X-Hanzo-Usage-Reason(the code),X-Hanzo-Usage: limited,X-Hanzo-Topup-Url. The usage row'smodelis the SKU that answered. - ZAP has no gate, so its chat, responses and messages handlers serve a router id as
enso-free (
unpaid). - Listed (
depthListings): each router id is a row of/v1/models, its default tier's row and price (listedAs) under its own id, the narrowest window of its tiers, and a description of where it routes. Neverfree. - Tests:
TestARouterIdIsServedOnItsTierOrItsFreeSKU(every payer case, the switch, a program's call, a tier named outright),TestDepthRoute*,TestDepthRouting*,TestTheRouterIdsAreListed,TestARouterIdListsAtItsDefaultTiersPrice.
Who pays is the host's call — object.Limits (routers/filter_balance.go)
A call names the API key it arrived on. The identity boundary in front of this
module writes X-Key-Id (<org>/<id>: the org the key is filed in and its id there; stripped on ingress, written only
from a resolved key); TenantContextFilter threads it as GenAIAttribution.Key,
usageRecord.bind copies it, and the native debit carries it as UsageEvent.Key.
The host keeps each key's own spend by it, which is what a per-key budget is held to.
On every chat path and /v1/decisions the gate asks the host's usage policy
(object.LimitFunc, cloud apps/ai/limits) with the model's Class
(controllers.ClassOf: free when it costs nothing, ours for Enso/Zen and any route
owned_by: hanzo — Kai, not the Jev ids the decision service forwards — premium
otherwise) and whether it is Priced. The
answer is a LimitGrant with Pays:
plan/free— covered:Coverputs the grant on the request locals AND context;reserveFor,enforceBalanceGateand the decision hold skip the wallet;usageRecord.bindpicks the grant up from the context, the record carriesPlan, andrecordUsagesettlesplanUse(list price; COGS for a model sold at zero) against the grant.prepaid/credits— the wallet gates as before;Cashon the grant rides the record toUsageEvent.Cashso the host draws cash only.
A LimitHit refuses by Code with no figure: 429 usage_cap_exceeded (a plan
window, the free lane included — limited mode cannot be farmed), 429
free_plan_cap, 402 plan_allowance_used, 402 paid_plan_required, 402
model_cap (the model used its share of the plan; Model names it or its
pattern, Fallback the Hanzo model that answers instead). A conversation from a
signed-in app (token aud) or a client sending X-Hanzo-Fallback: allow — and, while
the paid lane's switch is on, a chat past a paid plan's included usage
(plan_allowance_used, model_cap) from any client — is handed on instead —
model_cap to Fallback, the other two to FreeModel — marked
X-Hanzo-Fallback and X-Hanzo-Usage-Reason, and the model it lands on is asked of
the policy again (up to three asks). Refusal actions: upgrade, switch (to the
fallback), and credits ("Continue with credits", which turns on the org's
PUT /v1/ai/limits opt-in) when Credits says the payer holds what could pay,
else topup. Served calls carry X-Hanzo-Usage, X-Hanzo-Usage-Class,
X-Hanzo-Paid-By (plan|credits|free — prepaid reads credits). An unreadable
policy decides nothing (the wallet gates). ZAP twins have no gate verdict, so they
stay wallet-only.
How a refused caller raises its limit is the host's to say too (object.RaiseFunc,
SetRaise / RaiseHook). Every refusal a limit lifts — 429 allowance_spent
(freeRefused), every LimitHit (limitReached), the wallet's 402 (denied) — is
written by refuse, which asks the hook once with the person (person), subject,
namespace and the refusal's code, bounded by raiseWait (500ms; a panic or no answer
leaves the refusal as it was). A Raise adds error.tier and
error.raise {method, path, body, message} after the site's own fields, and on a 429
X-RateLimit-Limit = Raise.Limit (only when > 0), Remaining 0, Reset = the
refusal's reset, unless standing already set them. nil — no hook, or a caller the host
does not single out — is byte-identical to before; a 503 never asks. The path and the
copy are the host's: this module names neither.
Every priced endpoint asks the host, not only chat (controllers.Entitled):
chat, /v1/decisions, embeddings, rerank, speech, transcriptions (the model is a
form field), the live transcript's open, voice/music/foley, images and videos. A
push to or the close of a live transcript names no model; it is asked as the model
its open named (controllers.TranscriptModel) with Session set, and counts
nothing. The media and retrieval controllers hold through reserveFor, so a covered
call reserves no wallet, and the transcript's per-push metering settles against the
push's grant. A request the host counted carries Release; the gate calls it when
the answer is 400 or above (a failure or a refusal), so an unserved call keeps no
window, cap or share. A stream is served from its first byte.
auto, zen-router and a request naming no model are the router's to resolve
(isAutoModel), and routable is the one eligibility rule it folds into Known
and the last-resort Allow floor: with an org enabled-models allowlist, exactly
what it names; without one, only Hanzo classes (ours + free). A premium model
answers auto only when the org allowlisted it; otherwise a caller names it.
A call a program in this process makes for a customer (object.Caller, ai.For)
A sibling app's embeddings, rerank or completion bought over the host's plane (cloud
apps/ai/chat_rpc.go) reaches ai through ai.For(object.Caller{Org, Person, Project}):
the same app and filters, with the caller stated as a fasthttp user value, which only
code in this process can write. CallerFilter moves it onto the context. Then:
- Attribution is the customer's:
TenantContextFiltertakes org, person and project from the caller, never headers;billingOrgis the caller's org, so the usage row is on the customer's books;tenanttells a family the caller's org. - Billing happens once, by the program, at ai's price: the program meters the call
and charges the customer itself (cloud
metered_ai.go), so the gate covers it with aPaysCallergrant (callerPays): no wallet is asked, the deployment's whose token carried it least of all, the plan is not asked, the paid lane seats it as the customer's, and the usage event isPlan: true,PaidBy: "caller"(no debit), priced as usual for the record. The price is TOLD to the caller:Caller.Bill(object.Bill) rides the grant (LimitGrant.Bill), andrecordUsageadds each answer'susageMargin(record).BilledNanoto it — what the same answer debits any other caller, a hedge's losing provider included. The program charges exactly that; it has no rate of its own.controllers.PriceOf(model, org)is the billed row as aRate, for a program that must reserve before it calls; an unpriced model answers false, never the conservative default. Pinned byTestAProgramIsToldWhatItsCallCost,TestOnlyACallersGrantIsToldItsPrice,TestPriceOfIsTheBilledRow. - A request from the network states no caller whatever headers it carries
(
TestARequestFromTheNetworkStatesNoCaller), andForcarries no identity header in (authz.Headers,authz.Retired): the identity is the caller and the credential (TestForCarriesNoIdentityHeader).
The paid lane: ONE switch, then the plan that pays
controllers.FreeOnly is the platform's switch, and ai does not own it: it is the zen
catalog's paid line as the zen service's own discovery states it (GET ZEN_URL/v1/models, top-level "paid": true; zenFam.open), free unless the catalog,
universe charts/app/files/zen/catalog.yaml, says paid: true. Spending is an opt-in:
a catalog not yet read, or silent on it, keeps the paid lane off, and a failed refresh
keeps the last catalog's line. GitOps is the source of truth; there is no settings row,
no FREE_ONLY env and no second line (the enso service reads the same line through
ZEN_SWITCH). A cloud that still links zen in process assigns it from that zen
(aicontrollers.FreeOnly = z.Free) until its zen moves out; nothing else sets it.
A closed lane bounds the platform's cash, so it reaches a priced Hanzo SKU only where
one could be spent: an embedding no stand-in answers is sent to its family when the
catalog funds it from grants and our own compute on every path (spends: its listing
says no "funding": "prepaid"), and refused when it does not.
With the switch on, each request is on one lane (controllers/lane.go), decided once
by controllers.Seat inside routers.BalanceGateFilter from the host's grant:
paid when the grant names a paid plan (LimitGrant.Plan — cloud apps/ai/limits
planOf: an active subscription whose period was paid by card or recorded
externally, with figures from apps/flags PlanSeeds, per seat for team rungs), the
plan's included usage or the payer's prepaid/granted credit past it pays for a priced
call, and the call's estimate fits; free otherwise (no plan, a free model, a
model's free daily cap, a policy that said nothing, a call that did not fit, and a
request that never passed the gate — every ZAP handler; the forward bridge is the ZAP
route that runs the gate). routers.LaneFilter only labels a request the gate did not
seat. Every read on the request path is FreeOnlyFor(ctx) (or shut(ctx, p), which
also lets the org's own key through while the switch is on); candidates,
openrouterTail, paying, spendOf and every ZAP handler (zapLane) take its answer.
A seat holds the call's quote (controllers/quote.go, quoteOf): every body byte as
a token (a tokenizer never reads fewer than one byte per token), plus 1,024 for the
vendor's framing and 2,000 per vendor-defined tool; a decodable picture (PNG, JPEG,
GIF, WebP, ≤ 20 MiB) in an image part as its 768-px tiles at 258 tokens (at least
6,000, never below $0.01) in place of its bytes; a remote picture as an 8,000-px one; a
PDF, a remote document or a stored file as a whole 1M-token context (or the model's
declared window when larger); a prompt with cache_control at the hour-long cache-write
rate; the completion ceiling the handler sends (reserveCompletionTokens) × n, plus a
predicted output at the completion rate — at the most of list price and cost, the
platform's and the org's, ×2 for fast mode; media at unit price.
The paid lane has no ceiling of its own (owner, 2026-10-09: no paid on free; no
ceiling but what people prepay, or their usage). Nothing caps what paid-lane calls spend
together or what one org's calls spend; there is no platform day and no org share. A
seat holds, for a plan-paid call, against the plan class's LimitGrant.Spend less the
payer's calls in flight; a wallet-paid call (prepaid cash, the org's credit line) is
held against the wallet by its handler (reserveFor); one payer holds at most 8 seats
(paidLaneCalls). A plan-paid chat that does not fit at the ceiling it asked for is
sent with the highest one that fits (never below 4,096), named in
X-Hanzo-Lane-Max-Tokens (laneTokens in the handlers). A family is sent
X-Hanzo-Spend only on a seat, capped at what it holds. A seat closes exactly once,
when its answer is counted (recordUsage → seat.settle, after the plan's settle); a
fast mode loser gives nothing back (seat.lost; ask.race holds the seat until every
loser is counted); Unseat closes one that recorded nothing after its handler; a stream
whose client takes nothing for 2 minutes (seat.paced, applied in
ApiController.SendStreamWriter) and any seat after 30 minutes lapse. Race losers and a
video's later poll bill the creating request's context only on a seat (billing); off
it they bill as before. The holds live in this process, as the wallet ledger's do, under
the same single-replica invariant (bootstrap.go refuses to start at CLOUD_REPLICAS > 1). Provider-side loops — web fetch, tool search, compaction — are not served on the
shared Anthropic account. A payer the lane had no room for gets X-Hanzo-Lane-Reason: paid_lane_full; a chat for a third-party model is handed to the free model in limited
mode (never one on the org's own key, OwnKey), and any other call only the paid lane
serves is refused paid_lane_full (laneOff): 413 when the plan has less left than the
call's least, 429 when the payer's calls in flight hold the rest or it holds 8 seats.
Responses carry X-Hanzo-Lane: paid|free only while the switch is on. The enso/zen
service's own paid lane (zen lib: the catalog's paid, or the ZEN_SWITCH file's)
spends what a fronted request's X-Hanzo-Spend allows; what zen spends for callers that
reach it directly is bounded by the host's plan limiter (cloud apps/zen planSpend).
On the free lane no chat request reaches a priced route, and the routes that stand in
for it answer, named as what they are; a tool or media request for one, and every
embeddings, rerank, image, audio and video call to a provider that is not a family's own
service (paying), is refused, and so is a decision that names Jev. Only zen and enso,
whose catalogs decide for themselves, serve.
OpenRouter's list, kept in the store (controllers/listing.go, object/listing.go)
OpenRouter's /v1/models is synced into the store and the family reads its catalog
from there, never from the vendor: every process lists, routes and bills from the same
rows, and the free lane reads the same rows.
- When: a process's first read syncs unless any process did within 5 minutes
(
listingStart); then once an hour (listingEvery), fleet-wide.object.TakeLeaseis a compare-and-set on theleasetable, so one process runs an interval's sync; a failed sync gives the interval back.keepListedreads the family every TTL so the schedule does not wait on traffic. - What:
object.Listingper model,Itemthe vendor's entry verbatim (every price,context_length,top_provider.max_completion_tokens, modalities,supported_parameters,created,expiration_date),Pricingcanonical (keys sorted, exact decimals: a respelled number is no change),Free(prompt and completion zero, not a-1router),Seen,Synced,Gone. A list of nothing is refused, never synced as every model gone. - Changes:
object.ListingEventrowsnew,back,gone,price,free(priced → free),paid(free → priced), withWas/Nowprice lists, thenobject.PublishListing(the host installsSetListingPublisher: cloud puts them on its bus). A sync that seeds an empty store records its rows and tells the bus nothing. A gone model leaves the catalog and is served no longer; its row and history stay. - Billing a resold SKU (
zenModel.bill): tokens at the tier the prompt reaches (min_prompt_tokensoverrides are tiers, the dearer side of the line; an hours-of-day override is a discount and is never billed), cached tokens atinput_cache_read, all × the markup; and never below the answer's statedusage.cost× the markup, which covers what tokens do not price (cache writes, web search, a dearer provider). A variable router bills the stated cost alone. - The markup is 1 + the vendor's fee, a percent the host's operator sets
(
object.SetFees; cloud reads it from admin.hanzo.ai's usage policy, seeded at OpenRouter's 5.5%). ai keeps no fee of its own: with none set a resold model bills at its vendor's price, and a fee that is not a percent of 0 or more fails the catalog's refresh, which keeps the prices it had. Zen and Enso SKUs carry Hanzo's margin in their own catalogs; a resold model carries only the vendor's fee. - Listed:
/v1/modelsrows carryfree,expires,max_output_tokens, andpricing.rates(every vendor rate × the markup, exact, in the vendor's names and units) andpricing.overrides(each condition with its rates × the markup). - Read:
GET /v1/models/vendors/openrouter[?free=1](anonymous) answers{vendor, synced, data:[the vendor's entries]}, the shape the free lane parses;GET /v1/models/vendors/openrouter/events[?id=&limit=]the history; SuperAdminGET /v1/admin/ai/freethe free set with the last 24 h per model (answered, failed, rate) and per account. Counts areobject.Served(vendor, model, account name, UTC hour; 48 h kept): each free request ai sends on its ring (counting), each free arm a family ran on a listed vendor (X-Hanzo-Keynames the account), and each failed free arm of a failover chain (no account).
The free pool — every OpenRouter account, spent as one
The Free plan is limited usage from ONE pool every free user shares: the
OpenRouter accounts named by object.OpenRouterKeys (OPENROUTER_API_KEY,
_2, _3, resolved KMS-first), spent by controllers/family_keys.go.
- A free route turns the ring; a priced route does not. OpenRouter's free allowance is per ACCOUNT (20/min; 50/day, or 1000/day once $10 of credit was bought), so a free request starts one key further round each time. A priced request starts at the first key, the funded one. An admin row's single key is sent as before and never benched (one tenant's 402 must not bench it for all).
- A 429 is read, not guessed.
freeQuotabenches a key for free routes only when the message STARTSRate limit exceeded: free-models-per-min|day, untilX-RateLimit-Reset(header orerror.metadata.headers, ms epoch), else the window's own length; a bench is never shortened. Any other 429 on a FREE route is the model limited upstream: answered after one account (asking all three is three calls for one refusal) and the pool moves it to the next route. On a priced route it still asks the next account. FreePool()is the pool's standing: state (available|busy|exhausted) and when it refills, per process. Cloud'sai_poolplane op publishes state + refill only (no account counts) andGET /v1/allowancecarries it to the Free usage page.- A busy FREE route moves to the pool (
fallback): enso-auto on the enso service's one account answering 429 while three accounts sit idle made a shared pool a queue for one key. A busy PRICED route still stays itself. - The pool answers for itself only when an account is benched. A free-lane
request the pool could not serve gets
object.PoolRefused(429pool_busy/pool_exhausted,upgrade_url=object.PayURL, hanzo.ai/pay) only whenFreePool()is not available; a walk that failed with every account ready (502s, nothing discovered) keeps its path to the route's alternates. Busy never carries a reset past a minute; exhausted carries the refill andx-should-retry: false. A caller on a paid plan is told to pick a paid model, not to upgrade. A person's own share spent is 402allowance_spent(object.AllowanceSpent) naming the window;object.SpentFuncanswers aStanding. All three codes arebillingNotice: relayed, they are never moved to the pool. - Free is held for seconds, not minutes (
familyFreeTTL,freeTierTTL, 10s), and the rate limiter re-asks a free entry's tier: it is the one answer a payment changes, so a paying org is rated as what it bought by the time checkout brings it back.
No escape hatch: the gate refuses what it cannot decide, and every call is a row
- Fail closed. A usage policy (
object.LimitFunc) or free allowance (object.SpentFunc) that errors refuses 503usage_unavailable(object.UsageUnavailable), on free and priced models alike — an outage of the counts is never a way past them. - No configured tier.
RATE_LIMIT_TIERSis deleted: a tier is the caller's recorded subscription (commerce), never an operator mapping. - Shared counts.
object.SetCounters(QuotaFunc, TakeFunc)putsRateLimiter(per-minute buckets) andQuota(8h/week/month) on the host's store, so every replica counts together and a restart forgets nothing; with none installed they count in process (standalone). - One transcript lookup.
controllers.TranscriptModelis the only one; the rate filter and the plan gate both ask it. - Limited mode is one rule.
fallback(c, path, code): in chat, from a signed-in app orX-Hanzo-Fallback: allow, any refusal for want of payment —plan_allowance_used,paid_plan_required,model_cap,insufficient_balance(the wallet's own 402) — is answered by the free model, withX-Hanzo-Fallback(lane),X-Hanzo-Usage-Reason(the real code),X-Hanzo-Usage: limited,X-Hanzo-Topup-Url,X-Hanzo-Upgrade-Url; the handler names the concrete model inX-Hanzo-Served. Non-chat callers keep their 402. - Every served call is billed and described. A unit-priced call (Zen media, scrape,
crawl, RAG chat) carries its charge as
BilledNanoExact—Costalone was read by nothing on the money path and debited $0. A bi-encoder rerank bills the tokens it embedded. A vendor key pays as its row's owner on the ZAP wire as on HTTP.UsageEventcarriesClass,Units,CostUSDandPaidBy("plan", "hanzo" for a free model the platform absorbs, "" for the wallet) so the host writes a row for every call.