Both passages describe an escape the code refuses. BALANCE_EXEMPT_USERS is read by nothing — TestEnforceBalanceGate_NoExemptBypass and TestCheckBalanceNoExemption set it and require the listed subjects to fail closed — and what is exempt is a set of routes. IsSuperAdmin is owner == AdminOrg alone, with no list and no env override, so the doc's globalAdminOrgs=admin,hanzo told a reader the hanzo tenant org holds platform power. Co-authored-by: Hanzo Dev <[email protected]>
44 KiB
ai — agent guide
hanzoai/ai is the canonical AI control plane for the Hanzo platform: model
hub, native Go model routing, RAG, and MCP/A2A management. It speaks the
OpenAI-compatible /v1 API, routes 66+ models to upstream providers, and meters
every request. Renamed from hanzoai/cloud (HIP-0106); mounts as the ai
subsystem inside hanzoai/cloud. In prod it runs as cloud-api on hanzo-k8s,
fronted by hanzoai/gateway at api.hanzo.ai.
Canonical role. This is a Hanzo service/infra repo — one impl, one place.
It is NOT an SDK; SDKs link out to it. Completeness order across languages is
Python → Rust → C++ → Go. Canonical spec: ~/work/hanzo/SDK-ARCHITECTURE.md.
Brand rules (hard — enforce in every edit).
- Never call this an "LLM gateway" and never position it against LiteLLM — it is a full AI cloud / control plane, not a proxy.
/v1/only, never an/api/prefix.- Zen models are Hanzo's own family (
owned_by: hanzo) — never name upstream models. - Voice: "Hanzo — the Open AI Cloud." Modern, crisp, developer-first.
Install / run.
go build -race -ldflags "-extldflags '-static'" # build
./cloud-api-server # run (env-configured)
go test -v $(go list ./...) -tags skipCi # test (requires MySQL)
docker compose up # local stack
Key entry points. main.go (entry) · bootstrap.go (boot + replica
assertion) · routers/router.go (all /v1 routes) · controllers/ (HTTP
handlers) · object/init.go (init + LLM provider seeding) · model/ (provider
integrations) · object/kms.go (secret resolution) · web/ (React admin UI).
Architecture
Full-stack application:
- Backend: Go 1.26 on
github.com/zap-proto/zip(over fiber v3 / fasthttp), MySQL/MariaDB/PostgreSQL - Frontend: React + Ant Design v5, located in
web/(the folder is the React app; the in-house Go router that used to share the name is gone) - Auth: Hanzo IAM SSO integration
Directory Structure
cloud/
├── main.go # Entry point
├── go.mod # Go module
├── conf/app.conf # Beego config
├── controllers/ # HTTP handlers (60+ files)
├── object/ # Business logic & data models (85+ files)
├── model/ # AI model provider integrations (38+ files)
├── routers/ # Routes and middleware filters
├── embedding/ # Embedding provider implementations
├── agent/ # MCP agent support
├── split/ # Text splitting strategies
├── txt/ # Document parsing (PDF, CSV, XLSX, PPTX)
├── storage/ # File storage abstractions
├── util/ # Utilities
└── web/ # React frontend
├── src/
└── package.json
Build & Run
Backend
# Build
go build -race -ldflags "-extldflags '-static'"
# Cross-platform build (linux amd64/arm64/riscv64)
./build.sh
Frontend
cd web
yarn install
yarn start # Dev server
yarn run build # Production build
Docker
docker compose up
Testing
# Go tests (requires MySQL)
go test -v $(go list ./...) -tags skipCi
# Frontend tests
cd web && yarn test
Linting
# Go - uses golangci-lint with gofumpt
golangci-lint run
# Frontend
cd web && yarn lint
Key Conventions
Backend (Go)
- Framework:
zip. AnApiControllerIS its request — it embeds*zip.Ctx, so there is no context to marry to a controller and no recorder to read afterwards. Controllers handle HTTP, objects hold the business logic. - New AI providers go in
model/(implement the provider interface) - Database access via the native store in
object/ - Route registration: CRUD resources are GENERATED from the one table in
routers/resources.go;routers/router.goholds only what is NOT a resource (the OpenAI-compatible surface and a few singletons). See "The /v1 resource surface" below. - i18n strings: avoid duplicate keys across frontend and backend
Five things zip decides that the old router did not
Each cost a real defect during the migration, and each is invisible: the code compiles, the types check, and the behaviour is wrong.
-
A WRITE CARRIES ITS STATUS.
c.JSON(code, v)/c.Bytes(code, b)set the status as they write, so setting one beforehand and writing after replaces it.c.Statusis for a response whose body is written by something that takes no status — a stream — or that has no body at all. Everything else passes the status to the one write. A refusal that arrives as 200 is worse than an outage: the client reads it and believes it. -
A STREAMED BODY IS PRODUCED AFTER THE HANDLER RETURNS. fasthttp drains the writer handed to
SendStreamWriterwhile it serialises the response, so the callback has NOT run when the call comes back — measured, not assumed. Anything the stream reads must outlive the handler (hand the upstream body to the callback; adeferin the enclosing function closes it first and the client gets nothing), and anything the stream learns must be used inside it (token counts read outside are still zero, so a streamed answer bills no completion).AND THE CTX IN THERE IS NOT THE REQUEST'S — this line used to say it was, and production refuted it. The claim was that fasthttp finishes writing before fiber releases the request, so reading
cfrom the callback was safe. Measured on api.hanzo.ai:DefaultCtx.App()nil-dereferenced underrecordFamilyUsage -> billingOrg -> Ctx.Header, twelve times an hour. It does not fail the one request either — a SIGSEGV takes the whole plugin process, so every other call in flight dies with it and each one answersmount /v1: http: read response: EOF. Read what the stream needs from the request BEFORE the handler returns and let it travel by value;whenceinzen_client.gois that shape, andrecordFamilyUsagetakes no receiver so there is nothing to reach through. -
RequestURIIS SET BY A SERVER, NOT BY A CALLER. The fiber adaptor reads that field, so a request CONSTRUCTED in process — which is every request crossing ZAP — arrives at the router as/and 404s.controllers.targetwraps a handler to fill it in from the URL; both ZAP installers go through it. -
A BUFFERED WRITE IS NOT A DELIVERY. Bytes leave at the flush. A
Writethat returns n > 0 has reached a buffer, and an uncheckedFlushreports a broken connection as success — which is howAnthropicWriter.StreamSentcame to tell the failover loop an answer had reached a client it never reached. -
A PRINCIPAL IS A VERIFIED TOKEN, and there is nothing else to seed. No session store exists;
GetSessionClaimsparses the credential the request presents. So a test that needs a caller PRESENTS one:presenting(visit(method, target), authtest.Bearer(t, user)).internal/authtestis the ONE installer of the verifier's certificate — two of them race over the same global. Corollary: any predicate that used to mean "there is a session" now fires for ANY verified token, a sibling brand's included.
Frontend (React)
- UI components use Ant Design v5
- i18n via i18next; translation files in
web/src/locales/ - State via React Context/Hooks (no Redux)
- API calls go through
web/src/backend/helper modules
Commits & PRs
- Branch naming:
claude/<name> - Main branch:
main - Follows semantic versioning (
.releaserc.json) - PR titles should be semantic (feat/fix/chore/docs/refactor)
CI/CD (GitHub Actions)
.github/workflows/build.yml runs:
- Go unit tests (with MySQL service)
- Frontend build (Node.js 20 + Yarn)
- Go binary build
- golangci-lint
- Semantic release + Docker push to
ghcr.io/hanzoai/cloud
Configuration
- Backend config:
conf/app.confread by the nativeconf.AppConfigini reader (env-first; absent in the embedded runtime, which runs env-only) - Database: MySQL 8.0+ or MariaDB
- Default Docker DB: PostgreSQL via
compose.yml
Important Files
| File | Purpose |
|---|---|
main.go |
Application entry point |
routers/router.go |
All API route definitions |
object/init.go |
Application initialization + LLM provider seeding |
conf/app.conf |
Runtime configuration |
web/src/App.js |
Frontend root component |
web/src/backend/ |
API client helpers |
hedge — ask several providers, keep the first answer
hedge.Race(ctx, dst, attempts...) is the primitive behind fast mode. Tail
latency is not the average: a provider that usually answers in 400ms occasionally
takes eight seconds, and a request cannot know which it drew until it has already
waited. Asking three and keeping the first cuts that tail.
No provider changes, and none are needed. Every provider already writes into
an io.Writer, so Race hands each attempt a different one and only lets the
winner's reach the caller. ModelProvider is untouched — the seam was already
the right shape.
FIRST BYTE WINS, the only definition of fastest a stream can act on. A provider that will finish sooner but has not started is not the winner: the caller waits on bytes, not completion, and by the time completion is knowable the response has arrived and there was nothing to race.
Two cases decide whether hedging helps or hurts, and both are pinned by tests verified against a deliberately broken implementation:
- An EMPTY write does not claim. A provider flushing a zero-length chunk while still waiting upstream would otherwise win having said nothing, leaving the caller on the SLOWEST attempt with the others cancelled — the opposite of the point.
- A FAST FAILURE is not a win.
Racereturns when the attempt that CLAIMED the stream finishes, not when the first attempt returns; an upstream 503 in 1ms produced no bytes, so the caller is still waiting on whoever answers.
Losers are cancelled — that is what stops the upstream call, which is what stops
the cost — and Race does not wait for them to unwind, because waiting on a
provider we are abandoning hands it exactly the latency this removes.
One ledger path each, because the two losers are opposite cases
ask.fan (controllers/hedged.go) carries a request into fast mode: how many to
ask, where the winner's bytes go, and how to make and adopt a writer. serve()
dispatches on it, so a caller sets a NUMBER rather than choosing a code path, and
a fan too narrow to race falls back to the cascade with its retry and demotion
intact. Nothing sets it yet — the mechanism ships inert, and
TestWithoutAFanNothingChanges is what says so.
A raced loser and a failover refusal wear the same shape and mean the opposite thing, which is the whole design:
- A REFUSAL errored. No tokens, no upstream charge —
recordRefusalswrites a"failover"row that bills nothing, and that is correct. - A RACED LOSER was CANCELLED mid-generation. The vendor produced tokens and charged us. Filing it as a refusal records a real charge as free and makes fast mode invisible in the ledger the team reads.
So a beaten provider never appears in the attempts race() returns — the caller
records those as refusals — and settles through fan.bill instead.
Each attempt needs its OWN writer. The loser's token count is on ITS writer,
and one writer shared across attempts holds exactly one answer: the winner's. A
shared writer therefore bills every loser for the winner's tokens AND glues its
half-sentence to the winner's answer on the client's wire — both failures show up
in TestEachRacerIsBilledFromItsOwnWriter, verified against a deliberately
shared fork. OpenAIWriter.fork/adopt are that pair: forks copy the dialect
and start the buffers empty; the winner's is adopted so the non-streaming body
and the tail chunk come from the provider that actually served.
THE BILL ARRIVES LATE, and that is the shape rather than a wrinkle. Race
returns when the winner finishes, not when the losers do — so at that moment a
loser has typically not even started, and reading its outcome is both a missing
bill and a data race. (The first draft did exactly this; the test caught it.)
settle therefore runs on its own goroutine, and fan.bill must close over
VALUES: by the time it fires the handler has returned and, on fasthttp, the
request has been recycled.
Cancellation is a WRITE ERROR, not a context. QueryText takes no context,
so a loser's only stop signal is hedge.ErrLost on its next write. One that
checks its write error stops there; one that does not runs to completion and
bills us for all of it. Either way the tokens are real — which is the argument
for billing them.
The switch, and why it is decided with the reservation
A request asks with "fast": true in the body, or X-Fast: 1 for callers with
no browser in the way — the body form exists because a page cannot send a custom
header past the edge's CORS allow-list, exactly as retrieval works
(controllers/fast.go).
What a race costs is settled WITH the hold, never after. widthFor reserves
est × fastWidth before anything is spent, because N attempts is N completions
and a hold sized for one is a promise the ledger cannot keep — the losers' rows
land later and take the balance past the point the gate said it would stop.
fastWidth is 2: the tail this cuts is one provider drawing a slow response
while another would have answered normally, and a second opinion removes most of
it for one extra completion rather than two.
Two refusals that are deliberately NOT refusals:
- Cannot afford the race → serve the ordinary request. The caller asked to go faster, not to be turned away, and a new way for a funded account to be told no is worse than quietly being normal speed.
- Nobody to bill → never race. The public lane makes no reservation, so hedging there spends N completions on a stranger and settles none.
Still open: the toggle in the products. hanzo.chat follows web_search;
hanzo.app follows base and must resolve its in-flight merge on the three
AI-path files first. The API accepts the field today, so either can ship the
control without waiting on the gateway.
The /v1 resource surface — ONE table, no compound routes
The published OpenAPI description is GENERATED from that same table:
go run ./cmd/openapi -spec ../openapi/ai/openapi.yaml
It writes only the region between the # BEGIN/END generated markers in the
canonical spec (hanzoai/openapi, OpenAPI 3.1) — the inference paths above it are
hand-authored with real OpenAI request/response schemas and stay that way.
cmd/openapi -verify and TestSpecMatchesTheTable fail when the spec drifts from
the table, so the contract customers and SDKs read cannot describe a surface this
service does not serve.
The old swagger/ directory is DELETED. It was a second description, and it was
wrong: basePath: /api with 135 /<verb>-<noun> paths, a base path this service
has never served. Nothing referenced it, nothing embedded it, and no test noticed
when it stopped being true — which is the argument for generating the replacement
rather than hand-maintaining it.
Every CRUD route is generated from routers/resources.go. There is no second
registration path, and no route is written by hand.
GET /v1/<ns>/<resource> list POST /v1/<ns>/<resource> create
GET /v1/<ns>/<resource>/<owner>/<name> read PATCH …/<owner>/<name> update
DELETE …/<owner>/<name> delete
GET /v1/<ns>/<resource>/global cross-tenant list (where a resource has one)
The namespace is ai for every row, and for every other route this service
serves. HIP-0139 §3.1: a capability answers at ONE address, /v1/<name>, and a
second top-level address is a second capability or a misfiled route — never an
alias. So /v1/router/*, /v1/org/settings, /v1/finetune/*, /v1/memory/*,
/v1/rag/*, /v1/docs/ingest, /v1/feedback and /v1/traffic/globe are all
under /v1/ai/… now (ingest under /v1/ai/rag/, since it writes the index rag
reads). The groupings in the table — retrieval, chat, models, content, compute,
work, ops — are how it is READ, not a second prefix.
Two families stay outside /v1/ai because a protocol, not this repo, fixes
their spelling (§3.2): the OpenAI/Anthropic wire (/v1/chat/completions,
/v1/completions, /v1/embeddings, /v1/messages, /v1/models,
/v1/images/*, /v1/audio/*, /v1/videos/*, /v1/responses, /v1/rerank),
and POST /v1/embed, the multipart form hanzo.chat posts.
/v1/ai/<noun> is a resource collection address. Nothing may register a ZAP
gateway PREFIX there: the registry matches on path alone, so a prefix at a
collection swallows that collection's :owner/:name members and answers with
the LIST handler — a wrong answer, not an error. Deeper addresses
(/v1/ai/rag/query, /v1/ai/router/defaults) are routes, not collections, and
are safe. TestNoGatewayPrefixOverTheResourceNamespace holds the line; it is
why /v1/ai/feedback has no gateway prefix and reaches routers.App through the
fallback instead.
Three things about this that are easy to get wrong:
The member key is <owner>/<name> — two path segments, not one. Every object
is keyed by that pair (util.GetOwnerAndNameFromIdWithError requires EXACTLY two
tokens, so a name containing a slash was never valid — the two-segment URL is
lossless, not a narrowing). They are recomposed into id in the one place route
params are bound (web.Router.ServeHTTP), so handlers keep reading
c.Input().Get("id") unchanged. A single :id segment cannot work: Go decodes
%2F back into a separator before routing.
/v1/iam belongs to the IAM service, not to this one. cloud proxies that whole
subtree away (cloud/iam_edge.go). Anything registered there is shadowed and
unreachable — it fails as a wrong answer, not an error.
TestNoResourceClaimsForeignNamespace holds that line.
The policy filters key on names, not paths. authz_filter and filter_balance
hold hand-curated sets (super-admin endpoints, balance exemptions, demo-mode
allowances). Those names are DERIVED from the same table by policyKey, never
restated — so a policy entry cannot drift from the route it governs. Do not
hand-edit a path into those maps.
Adding a resource is one table entry. The tests will tell you if you got it wrong: every controller method the table names is checked to exist by reflection, every route must emit a policy key, no path may contain a verb, and an action whose handler reads no id must sit on the collection (a member action's route could never match).
An OpenAI-COMPATIBLE vendor is upstream Local, never OpenAI
Provider.Type IS the dialect declaration, and the two OpenAI-shaped upstreams are
not interchangeable:
| type | client | dialect | price from |
|---|---|---|---|
OpenAI |
NewOpenAiModelProvider |
/v1/responses — OpenAI's own surface |
CalculateOpenAIModelPrice, a table of what OpenAI RETAILS |
Local |
NewLocalModelProvider |
/v1/chat/completions (local.go, CreateChatCompletionStream) |
the provider ROW (inputPricePerThousandTokens), plus compatibleProvider for the API it speaks |
DigitalOcean already resolves to the Local client; so should every vendor that
merely SPEAKS OpenAI. Typing one OpenAI sends it to the Responses API, and every
compatible vendor shims that differently.
Measured, one provider (the Hugging Face router), one credential, one URL —
typed OpenAI: meta-llama/Llama-3.3-70B-Instruct answered, while
zai-org/GLM-5.2, deepseek-ai/DeepSeek-V4-Flash-0731 and moonshotai/Kimi-K3
returned an EMPTY message with a 200 and zero usage. All four answer directly, in
both dialects. Retyped Local, all four answer through the plane, reasoning
models included, with no <think> leaking into content — and no code change.
The reasoning streams carry response.reasoning_text.delta, which the Responses
loop does not case on (it handles only response.reasoning_summary_text.delta).
So the rule is a ROW, not a patch: an OpenAI-compatible vendor is Local with its
own URL and its own price. Reasoning stays a FIELD the dialect carries, never
something sniffed out of the content stream.
Three things still dispatch on substrings of the model NAME, and a model in
none of the lists gets wrong behaviour rather than an error — getOpenAiModelType
(which dialect), InlinesReasoning (strings.Contains(name, "deepseek"), directly
under a comment saying which SKU inlines is "a property of the catalog (HIP-0039)
… never a list here"), and the retail price table. Each is a place a new model
silently misbehaves; the Local row above is what they were compensating for.
LLM serving path (OpenAI-compatible /v1) — READ THIS
The ai module is mounted by hanzoai/cloud (HIP-0106). In prod it runs as
cloud-api in hanzo-k8s, fronted by gateway (api.hanzo.ai). Both
hanzo.chat (LibreChat, baseURL https://api.hanzo.ai/v1) and hanzo.app
(build) consume this exact surface.
Request flow for a completion:
gateway(api.hanzo.ai) validates the IAM JWT, mints identity headers, forwards/v1/chat/completions→cloud-api:8000./v1/modelsis also proxied here; admin endpoints (/v1/get-providers,/v1/update-provider,/v1/*-model-route) are NOT gateway-exposed (use the directapi.cloud.hanzo.aiingress + a session).controllers.ApiController.ListModels(/v1/models) requires a valid Bearer token — unauthenticated returns 401 with nodataarray (this looks like "0 models"; it is not — authenticate to see the real list).- Model → provider routing:
resolveModelRouteForOrgresolves in order DB routes (/v1/*-model-route, per-org → global "admin") → YAML config (conf/models.yaml, thecloud-api-modelsConfigMap — runtime source of truth in prod) → staticcontrollers/model_routes.gomap. The YAML wins, so changing routing live = edit the ConfigMap (then/v1/reload-model-configor restart). The static map is the fallback when no YAML is present. - Provider records live in the DB (
object/init.goinitLLMProviders), keyed by name:do-ai(DigitalOcean GenAI, the primary — backs OpenAI/ Anthropic/Llama/DeepSeek/Qwen/GLM/Kimi viainference.do-ai.run),fireworks,openai-direct,zen. Each call doesobject.GetModelProviderByName(route.providerName)→ readsClientSecret. - Provider keys:
ClientSecretiskms://SECRET_NAME.ResolveProviderSecret(object/kms.go) resolves it env-var-first (os.Getenv(SECRET_NAME)), falling back to a real KMS call — BUT only runs whenkms != nil, which needsKMS_CLIENT_IDorKMS_SERVICE_TOKENset. So the working prod recipe is: setKMS_CLIENT_ID(flips kms on) + provideDO_AI_API_KEY/OPENAI_API_KEY/ANTHROPIC_API_KEY/FIREWORKS_API_KEYas env (from a K8s secret sourced from KMS-managed values). The env-first path means no live KMS dependency on the hot path. - Provider re-seed self-heals
ClientSecret/ProviderUrl/State/Type/SubTypeon every boot from theinitLLMProviderstable — so fixing a stale key or upstream URL = edit the seed table and restart (no manual DB edit).
Async video (/v1/videos) — Sora-style, NOT sync. Text-to-video generation
takes minutes, so /v1/videos/generations is ASYNC (the sync handler used to
hold the request ~104s → console /ai proxy timed out → browser 502 while the
generation actually succeeded). The lifecycle is three fast requests:
POST /v1/videos/generations (create → returns a video_<uuid> job id +
queued object immediately), GET /v1/videos/{id} (one status poll), and
GET /v1/videos/{id}/content (stream the finished MP4). zen3-video* route to
the spark-video provider (our GB10, spark-video.hanzo.ai/v1), whose OWN
/v1/videos API is already the same async create→poll→download shape the three
model.{Create,Retrieve,Download}VideoDOAI primitives proxy. Metering is per-user
and lands EXACTLY ONCE on completion: create RESERVES the per-video budget and
carries the *budgetHold in the in-pod videoJobStore (valid — the pod is
single-replica, the balance ledger's own invariant); the first request that
observes completed (poll or download) settles the hold + records the single
Commerce usage event (videoJob.markCompleted, idempotent); a failed job releases
the hold and bills nothing; a reaper releases abandoned/wedged jobs' holds (TTL)
so budget never leaks. Every poll/download re-runs the shared auth+routing policy
and enforces OWNERSHIP (caller subject == job subject, else 404 — never confirm
another user's job); the provider key is re-resolved per request, never stored.
The gateway passes every /v1/* sub-path through to cloud-api, so the poll/
content routes need no gateway change; the console /ai proxy allow-list adds the
v1/videos/{id}[/content] pattern (hanzoai/console#92).
zen models: branded, owned_by: hanzo, premium: true. Route to do-ai
upstreams (qwen3+/glm/kimi/deepseek — same working key, no GPU). Each zen model
declares its public owner in owned_by; the identity is injected at call time
via zenIdentityPrompt (hip-00NN). Do NOT route zen to Fireworks serverless
paths (accounts/fireworks/models/*) — that account returns 404 "not deployed"
for most of them.
Billing gate (routers/filter_balance.go + openai_api.go): non-premium
models need balance > 0; premium models need balance > starter credit. Balance
comes from commerce at /v1/billing/balance?user=<org>¤cy=usd. No
principal bypasses it: the gate always consults the balance, and a lookup that
fails returns 500 rather than free access. To verify premium/zen, credit the org.
AI Login Manager — universal metering + connected accounts + 1% BYO fee
ai is the centralized login manager for every user's AI providers. Every AI call
from every surface (@hanzo/dev, Hanzo Desktop, hanzo.chat, hanzo.app) routes through
THIS router at api.hanzo.ai/v1. For the caller's org the router resolves WHICH
account executes the call, runs it, and meters ONE usage event — regardless of whose
credits paid for it. There is exactly one meter and one credential store.
Two account modes (one router, one meter):
- Hanzo credits (default). No org-owned provider for the requested model →
the global built-in (
Owner == "admin") serves it. Billed = token list-price cost (existing behavior). No surcharge. - Connected third-party account (BYO). The org connected its OWN OpenAI /
Anthropic / Google account via
/v1/ai/connections. The router uses the org's KMS-sealed key, the customer pays that provider directly, and Hanzo bills a 1% platform fee on the provider list price for routing + metering + management.
BYO is the EXISTING BYOK seam, not new plumbing. resolveProviderForUser →
object.GetModelProviderByNameForOrg(org, name) returns the org's own provider row
when it has one (Owner == org), else the global admin built-in. providerBYO
(controllers/platform_fee.go) classifies the resolved pair: BYO iff
provider.Owner != "" && != "admin" && == user.Owner.
Metering — one path, extended not rebuilt. Every completion terminal already
calls recordUsage (→ commerce POST /v1/billing/usage) + recordTrace (→ the
hanzo.cloud_usage warehouse). This feature adds three dimensions to that ONE
event: byo, fee_cents, account. platformFeeCents(costCents, byo) = byo ? ceil(costCents/100) : 0. Billed amount = byo ? fee_cents : cost_cents; the
full cost_cents is recorded on every row for analytics even when only the fee is
billed. hanzo.cloud_usage is the ONE global usage store (its organization column
is the row-level tenant filter — a tenant reads only its own org; the admin org
reads unfiltered). The rich per-org analytics lens is entitlement-gated in
hanzoai/cloud (analytics.datastore); basic own-org usage is always readable.
Connected accounts API (/v1/ai/connections, IAM-authed, org-scoped). Curated
login-manager surface over the per-org object.Provider store
(controllers/connections_api.go). Keys are sealed via object.StoreProviderSecret
→ kms://… ref (fails closed if KMS is down); the raw key is NEVER stored in a row,
returned, or logged.
GET /v1/ai/connections→[{provider, connected, account_label, updated_at}](never the key)POST /v1/ai/connections{provider, apiKey}→ seals key, activates the org row; returns metadata onlyDELETE /v1/ai/connections/:provider→ deactivates the org row (resolution reverts to Hanzo credits)
Enforce "even if logged in". All AI traffic must reach this router; no surface
bills a third party off-meter. hanzo.app's workspace default was flipped
openrouter → hanzo; hanzo.chat routes via its gateway config; @hanzo/dev defaults
to api.hanzo.ai/v1 under Hanzo login. Claude/Opus for @hanzo/dev is served HERE
(Anthropic-native /v1/messages + OpenAI-compat /v1/chat/completions) against the
org's connected Anthropic account + 1% fee — no direct-to-provider bypass.
Next slice (route selection). Connections are stored/sealed/listed and the
1%-fee metering fires the instant the resolver returns an org-owned provider. But
the route table maps all models to providerName: "do-ai" (the aggregating
upstream), so a connection row named openai/anthropic/google is not yet
SELECTED for its model family. Making a connected consumer account actually serve
its models is a bounded route-table change (model-family → org-connection selection
- real upstream model-id/base-url mapping) — deliberately not in this slice.
RAG — ONE unified surface (retires the standalone chat-rag-api)
ai is the single RAG layer for the whole platform. There is exactly ONE
retrieval pipeline; the standalone hanzoai/chat-rag-api (danny-avila/rag_api
fork) is now redundant and slated for archive.
One index, one embedder, one retrieval path. Every document — doc/crawl/
github/s3 ingest AND per-file uploads from hanzo.chat — lands in the SAME
per-tenant index {owner}-{store}-docs, fanned out to BOTH Hanzo Search
(keyword, Meilisearch) and Hanzo Vector (semantic, Qdrant) via
object.IndexDocuments, and retrieved via object.SearchDocuments (hybrid RRF).
See object/search_docs.go.
File-scoped RAG is a FILTER, not a parallel store. file_id is a filterable
dimension on that one index (DocIndex.FileID; Meili filterable attr file_id;
Qdrant payload file_id). Uploaded-file retrieval = a file_id filter over the
tenant index. object/rag.go holds the file-scoped logic (RagEmbedFile,
RagQuery, DeleteRagFile, RagFileContext); chunking uses
split.RecursiveSplitProvider (parity with rag-api's
RecursiveCharacterTextSplitter, 1500/100), parsing uses
txt.GetParsedTextFromUrl (PDF/CSV/XLSX/PPTX/…). Uploaded files default to the
rag-files store so they don't pollute curated docs.
One address family (controllers/rag.go): POST /v1/ai/rag/ingest,
/v1/ai/rag/embed, /v1/ai/rag/query, /v1/ai/rag/query-multiple,
/v1/ai/rag/delete, GET /v1/ai/rag/context. Ingest and embed are two ways in
to the one index; the rest read it back.
The one exception is POST /v1/embed (controllers/rag_librechat.go) — the
MULTIPART form hanzo.chat's uploadVectors() posts. Its request line is fixed
by that client, not by us, which is why it is not a spelling of
/v1/ai/rag/embed. Everything hanzo.chat READS it reads from the native family.
Migration: hanzo.chat's RAG client calls ${origin}/v1/ai/rag/* directly.
Auth/billing reuse the doc-RAG path (resolveSearchAuth/requireIndexAuth +
recordSearchUsage); owner is the authenticated principal (tenant isolation),
never a client-supplied field.
Search / Crawl / Web-search — THREE orthogonal products, ONE way each
Three distinct products, no overlap. Do NOT add a fourth crawl path.
-
POST /v1/search= native full-text index search (SearchDocs→object.SearchDocuments). Hanzo Search (Meilisearch, keyword,hanzoai/search-goatsearchHost:7700) + Hanzo Vector (Qdrant semantic,vectorHost:6333) fused hybrid-RRF over the per-tenant index{owner}-{store}-docs. Returns{hits:[…]}. Needs a populated index — an empty/new tenant index returns{hits:[]}(no error).GET /v1/search/statsreports index size. -
POST /v1/crawl= crawl a URL, get content back — THE single canonical crawl (controllers/crawl.goCrawl→object.Crawl→CrawlWithCrawl4AI). Backed by the self-hosted Crawl4AI (hanzoai/crawl,crawl.hanzo.svc:11235, Apache-2.0). Body{url}or{urls}(merged, deduped, capped at 10); returns{results:[{url,title,description,markdown,success,metadata}]}with LLM-ready markdown (prefers Crawl4AIfit_markdown, falls back toraw_markdown). Auth:requireIndexAuth(fail-closed, rejects read-only pk-/hz_ keys) + balance gate; usage recorded viarecordSearchUsage(…, "crawl", "crawl4ai", …).object/crawl4ai.gohandles the deployed Crawl4AI 0.8.6 wire shapes:markdownis an OBJECT (MarkdownFieldpolymorphic decode), the/crawlenvelope is synchronous with a booleansuccessand nostatus/task_id(return inlineResultswhenever present), andlinks/mediavalues are heterogeneous (map[string][]map[string]interface{}). -
POST /v1/scrape= crawl-and-INDEX (ingest) — NOT a general crawl. It crawls a site and WRITES structured content into the tenant search index, returningScrapeStats(ScrapeDocs→object.ScrapeAndIndex). Sibling ofPOST /v1/index(index caller-supplied docs). Stays because its output/effect (index write) is distinct from/v1/crawl(fetch + return). -
POST /v1/scrape/previewis a DEPRECATED alias of/v1/crawl— its handler now forwards toCrawl(one implementation, no parallel path). Use/v1/crawl. -
/v1/websearch/*(lives inhanzoai/cloud, not ai) = web meta-search — a separate product: SearXNG-compat/v1/websearch/search+ a firecrawl-compat/v1/websearch/v1/scrapeleg that hanzo.chat's web-search plugin calls. That leg is a fixed EXTERNAL contract (firecrawl{success,data:{markdown}}) and reuses the SAME Crawl4AI backend — it is an integration shim for the web-search flow, not a general crawl endpoint. The general "crawl a URL" is ONLY/v1/crawl.
Security invariants (READ before touching auth/billing/scaling)
-
JWT iss/aud — every request-auth JWT path goes through
object.ParseAndValidateJWT(signature via IAM and iss/aud policy), never rawiam.ParseJwtToken.object/jwt_validate.gosources the issuer fromJWT_ISSUER/IAM_ISSUER/CLOUD_IAM_ISSUER/AUTH_ISSUER(any, defaulthttps://hanzo.id) and the audience allowlist fromGATEWAY_ALLOWED_AUDIENCES(+ folds inIAM_AUDIENCE/AUTH_AUDIENCE), mirroring the gateway. A foreign-aud token → 401. The OAuth code-exchange callback (account.goSignin) is NOT a request-auth path and is intentionally exempt. -
Balance ledger — single-pod invariant —
object.GlobalBalanceLedgeris in-pod memory. Reserve/Settle are correct ONLY atreplicas: 1. Two pods double-spend (each reserves against its own cached Commerce balance). Enforced by deploystrategy: Recreate+ HPAmin=max=1AND a boot assertion (bootstrap.gopanics ifCLOUD_API_REPLICAS > 1). To scale out you MUST first move Reserve/Settle behind a Commerce-atomic conditional reserve. -
Single-request reservation (
controllers/billing_reserve.go) — an uncappedmax_tokensis clamped (clampMaxTokens) and the reservation covers the larger of the clamped ceiling and the QueryText pipeline's fixed completion cap (reserveCompletionTokens/reserveCompletionFloor=4096, mirrored inmodel/openai_util.go), so actual spend can never exceed the hold on any path. -
A tier is read through
object.TierReader, and the host owes a reader that answers — there are two consumers and they must ask in the same order:controllers.commerceFamilyTier(the per-SKU gate) androuters.TierCache.commerceTierLookup(the rate limiter). Both read the installed reader FIRST; the HTTP route is what a STANDALONE ai has instead of one, not a second way to ask.InitTierCachebuilds the cache when EITHER exists, not when the endpoint alone is set.The reader is NOT necessarily in-process. Every cloud app is its own child PROCESS, so a Go func var cannot cross that boundary: what cloud installs is
metering.Client.Tier, which is an HTTP call toGET /v1/billing/tiercarryingCOMMERCE_SERVICE_TOKENandX-Org-Id. That call has no user session, so it only succeeds where the handler resolves its tenant with billing'sreaderOrg(session OR trusted service naming an org) rather thanprincipal.OrgFromalone.balancehad that rule; the typed ops behindtierdid not, and answered401 sign in to view financeto a token the same request proved good on/v1/billing/balance. Fixed in cloudapps/billing/typed.go(payerfalls through toreaderOrg), guarded byapps/billing/tier_s2s_test.go.The failure is silent in both directions, which is why it lasted: the rate limiter maps a failed read to
TierZenFree(every paying org shaped as free) and the SKU gate maps one to unknown and ALLOWS (it stops gating). Neither logs an error anyone watches, so assert the tier a caller actually gets rather than that the lookup "worked". -
A failed call bills only what a vendor actually ran —
recordUsagebills a failure because by then the vendor has run the request and invoiced us. That premise does not hold for a refusal it can decide from the request alone (400, 401, 402, 403, 404, 413, 422, 429) or for an error carrying no status at all (nothing was dialled).controllers.ran(err)answers that, and it is NOTfaultOf: faultOf asks where the request goes NEXT, and the two come apart both ways — a 401 fails over having run nothing, a partial write stops the cascade having run plenty.spent()consultsranonly when nothing came back, so a meter the vendor reported and any text that reached the caller still settle it first. Without this the quantity billed is the prompt measured on the way out, so the charge grows with the request: 400k tokens refused for being too long billed a dollar. -
Exemption is by ROUTE, never by identity.
BALANCE_EXEMPT_USERSis read by nothing;TestEnforceBalanceGate_NoExemptBypassandTestCheckBalanceNoExemptionset it and require the formerly-listed subjects to fail closed. What is exempt is a set of paths —isBalanceExemptinrouters/filter_balance.go— plusbalanceExemptNames, three usage reads named as POLICY rather than URL so a $0 org can still see the panel telling it to add credit, and so moving the route cannot silently re-gate them. Exempt from balance is not exempt from auth. -
SuperAdmin —
util.IsSuperAdmin=owner == util.AdminOrg, the constant"admin". Membership in that one reserved IAM org is the whole definition: no list, no env override, andisAdminis not part of it — an org admin (hanzo/z) administershanzoand holds no platform power. -
Authz gate (
routers/authz_filter.go) — platform-sensitive endpoints (superAdminEndpoints) are gated FIRST (before preview-mode + exempt reads). There is deliberately NOadminDomainHost bypass (a removed full-authz-bypass primitive). No principal → 401; wrong principal → 403. -
User redaction (
object/redact.go) —RedactUserSecretsis an allowlist projection over string/[]string fields (exposableUserFields): a field not on the list is zeroed, so a NEW upstream secret field is fail-secure. Credential struct slices (MfaAccounts/ManagedAccounts/MfaItems/FaceIds) are nilled.
Cost: known, zero, or not known — three answers, never two
An unregistered COGS used to read back as the CUSTOMER PRICE. costInputPerMillion()
returned the price when no cost rate was set, so tokenProviderCostNano computed cost
== billed and every such call recorded margin_nano == 0. For the Enso family that was
every call: 1,296 rows checked, all of them zero. It is not a rounding error — it is a
ledger reporting a business that earns nothing on everything it sells.
The shape of the fix is that cost is optional and margin follows it:
controllers/model_pricing.go—modelPrice.costed()reports whether the model states what it costs us, and it requires BOTH legs. Half a COGS is not a COGS: an input rate with no output rate prices the completion — the expensive half — at zero.costInputPerMillion()/costOutputPerMillion()now return the configured rate and nothing else; there is no path that reads one without first askingcosted().controllers/billing_nano.go—tokenProviderCostNanoandproviderCostNanoreturn*int64, nil where no cost is known.costMargin.CostNanois a pointer for the same reasonMarginNanoalready was: the row has to be able to say it does not know, and an int64 can only say zero.usageMarginreports a margin only when both sides are real.controllers/zap_native.go+object/cloud_usage.go— the warehouse column isInt64, so an unknown cost writes 0 and setsuncosted = 1beside it. Queryuncosted = 0for rows whose cost is a fact.controllers/telemetry.go— the span emitsprovider_costonly when there is one, the way it already treated margin. It no longer asserts a COGS it does not have.
Zero and unknown are different answers. Zero is a fact: BYO (the customer paid the upstream with their own key), a free route, speech on hardware we already own. Nil is the absence of one: image and video have a per-unit PRICE table and no vendor invoice behind it, and a token model that states no COGS has told us nothing.
Where a cost is registered
object.ModelRoute.CostInPerMillion / CostOutPerMillion — the route row, which the
admin cockpit already reads and writes. It is the only place, and the configured PRICE
tables are deliberately not consulted as a substitute.
The family leg used to make that unreachable for zen/enso: familyModelPrice returned
first and short-circuited the resolver, and the family wire (zenWireModel) carries no
cost field, so an enso model could not be given a COGS at all without restating its
price. getModelPriceForOrgOK now completes an uncosted family price from
registeredCost. Price and cost resolve independently, because they are independent
facts: the family owns what we charge, the route row owns what it costs us.
What still has no COGS
The mechanism no longer lies, but a rate nobody has entered is still a rate nobody has
entered. Enso's real figures are known and live in the Enso catalog: 1.392 / 2.784
$/MTok on the deepseek-v4-pro anchor (enso, enso-ultra) and 0.112 / 0.224 on
deepseek-4-flash. Against billed 4/20 that is roughly 2.9x in / 7.2x out.
Registering them is an operator action on the route row — set both legs; one leg alone
is ignored by costed(). Until then the rows read uncosted = 1, which is the honest
answer and a visible one: SELECT count() FROM hanzo.cloud_usage WHERE uncosted = 1
is the backlog, and it should fall as rates are entered.
For enso-ultra the fan-out means COGS is the sum of the arms that ran, which only the
server can total. controllers/trace_export.go carries BilledNano/CostNano and
surfaces as CostNanoExact; a stated cost wins over the rate table in usageMargin.
Rate math cannot express a fan-out cost at all, so that is the right seam for it.
The admin cockpit's below-cost check compared price against a "cost" that WAS the price, so its difference was always exactly zero and it could neither fire honestly nor stay quiet honestly. It now has a cost to compare against, or a flag saying there is none.