Files
ai/LLM.md
zeekayandHanzo Dev bc16ef0797 Correct the bypasses LLM.md says exist
Both passages describe an escape the code refuses. BALANCE_EXEMPT_USERS is
read by nothing — TestEnforceBalanceGate_NoExemptBypass and
TestCheckBalanceNoExemption set it and require the listed subjects to fail
closed — and what is exempt is a set of routes. IsSuperAdmin is
owner == AdminOrg alone, with no list and no env override, so the doc's
globalAdminOrgs=admin,hanzo told a reader the hanzo tenant org holds
platform power.

Co-authored-by: Hanzo Dev <[email protected]>
2026-09-09 00:10:51 -07:00

44 KiB
Raw Permalink Blame History

ai — agent guide

hanzoai/ai is the canonical AI control plane for the Hanzo platform: model hub, native Go model routing, RAG, and MCP/A2A management. It speaks the OpenAI-compatible /v1 API, routes 66+ models to upstream providers, and meters every request. Renamed from hanzoai/cloud (HIP-0106); mounts as the ai subsystem inside hanzoai/cloud. In prod it runs as cloud-api on hanzo-k8s, fronted by hanzoai/gateway at api.hanzo.ai.

Canonical role. This is a Hanzo service/infra repo — one impl, one place. It is NOT an SDK; SDKs link out to it. Completeness order across languages is Python → Rust → C++ → Go. Canonical spec: ~/work/hanzo/SDK-ARCHITECTURE.md.

Brand rules (hard — enforce in every edit).

  • Never call this an "LLM gateway" and never position it against LiteLLM — it is a full AI cloud / control plane, not a proxy.
  • /v1/ only, never an /api/ prefix.
  • Zen models are Hanzo's own family (owned_by: hanzo) — never name upstream models.
  • Voice: "Hanzo — the Open AI Cloud." Modern, crisp, developer-first.

Install / run.

go build -race -ldflags "-extldflags '-static'"   # build
./cloud-api-server                                 # run (env-configured)
go test -v $(go list ./...) -tags skipCi           # test (requires MySQL)
docker compose up                                  # local stack

Key entry points. main.go (entry) · bootstrap.go (boot + replica assertion) · routers/router.go (all /v1 routes) · controllers/ (HTTP handlers) · object/init.go (init + LLM provider seeding) · model/ (provider integrations) · object/kms.go (secret resolution) · web/ (React admin UI).

Architecture

Full-stack application:

  • Backend: Go 1.26 on github.com/zap-proto/zip (over fiber v3 / fasthttp), MySQL/MariaDB/PostgreSQL
  • Frontend: React + Ant Design v5, located in web/ (the folder is the React app; the in-house Go router that used to share the name is gone)
  • Auth: Hanzo IAM SSO integration

Directory Structure

cloud/
├── main.go                 # Entry point
├── go.mod                  # Go module
├── conf/app.conf           # Beego config
├── controllers/            # HTTP handlers (60+ files)
├── object/                 # Business logic & data models (85+ files)
├── model/                  # AI model provider integrations (38+ files)
├── routers/                # Routes and middleware filters
├── embedding/              # Embedding provider implementations
├── agent/                  # MCP agent support
├── split/                  # Text splitting strategies
├── txt/                    # Document parsing (PDF, CSV, XLSX, PPTX)
├── storage/                # File storage abstractions
├── util/                   # Utilities
└── web/                    # React frontend
    ├── src/
    └── package.json

Build & Run

Backend

# Build
go build -race -ldflags "-extldflags '-static'"

# Cross-platform build (linux amd64/arm64/riscv64)
./build.sh

Frontend

cd web
yarn install
yarn start       # Dev server
yarn run build   # Production build

Docker

docker compose up

Testing

# Go tests (requires MySQL)
go test -v $(go list ./...) -tags skipCi

# Frontend tests
cd web && yarn test

Linting

# Go - uses golangci-lint with gofumpt
golangci-lint run

# Frontend
cd web && yarn lint

Key Conventions

Backend (Go)

  • Framework: zip. An ApiController IS its request — it embeds *zip.Ctx, so there is no context to marry to a controller and no recorder to read afterwards. Controllers handle HTTP, objects hold the business logic.
  • New AI providers go in model/ (implement the provider interface)
  • Database access via the native store in object/
  • Route registration: CRUD resources are GENERATED from the one table in routers/resources.go; routers/router.go holds only what is NOT a resource (the OpenAI-compatible surface and a few singletons). See "The /v1 resource surface" below.
  • i18n strings: avoid duplicate keys across frontend and backend

Five things zip decides that the old router did not

Each cost a real defect during the migration, and each is invisible: the code compiles, the types check, and the behaviour is wrong.

  • A WRITE CARRIES ITS STATUS. c.JSON(code, v) / c.Bytes(code, b) set the status as they write, so setting one beforehand and writing after replaces it. c.Status is for a response whose body is written by something that takes no status — a stream — or that has no body at all. Everything else passes the status to the one write. A refusal that arrives as 200 is worse than an outage: the client reads it and believes it.

  • A STREAMED BODY IS PRODUCED AFTER THE HANDLER RETURNS. fasthttp drains the writer handed to SendStreamWriter while it serialises the response, so the callback has NOT run when the call comes back — measured, not assumed. Anything the stream reads must outlive the handler (hand the upstream body to the callback; a defer in the enclosing function closes it first and the client gets nothing), and anything the stream learns must be used inside it (token counts read outside are still zero, so a streamed answer bills no completion).

    AND THE CTX IN THERE IS NOT THE REQUEST'S — this line used to say it was, and production refuted it. The claim was that fasthttp finishes writing before fiber releases the request, so reading c from the callback was safe. Measured on api.hanzo.ai: DefaultCtx.App() nil-dereferenced under recordFamilyUsage -> billingOrg -> Ctx.Header, twelve times an hour. It does not fail the one request either — a SIGSEGV takes the whole plugin process, so every other call in flight dies with it and each one answers mount /v1: http: read response: EOF. Read what the stream needs from the request BEFORE the handler returns and let it travel by value; whence in zen_client.go is that shape, and recordFamilyUsage takes no receiver so there is nothing to reach through.

  • RequestURI IS SET BY A SERVER, NOT BY A CALLER. The fiber adaptor reads that field, so a request CONSTRUCTED in process — which is every request crossing ZAP — arrives at the router as / and 404s. controllers.target wraps a handler to fill it in from the URL; both ZAP installers go through it.

  • A BUFFERED WRITE IS NOT A DELIVERY. Bytes leave at the flush. A Write that returns n > 0 has reached a buffer, and an unchecked Flush reports a broken connection as success — which is how AnthropicWriter.StreamSent came to tell the failover loop an answer had reached a client it never reached.

  • A PRINCIPAL IS A VERIFIED TOKEN, and there is nothing else to seed. No session store exists; GetSessionClaims parses the credential the request presents. So a test that needs a caller PRESENTS one: presenting(visit(method, target), authtest.Bearer(t, user)). internal/authtest is the ONE installer of the verifier's certificate — two of them race over the same global. Corollary: any predicate that used to mean "there is a session" now fires for ANY verified token, a sibling brand's included.

Frontend (React)

  • UI components use Ant Design v5
  • i18n via i18next; translation files in web/src/locales/
  • State via React Context/Hooks (no Redux)
  • API calls go through web/src/backend/ helper modules

Commits & PRs

  • Branch naming: claude/<name>
  • Main branch: main
  • Follows semantic versioning (.releaserc.json)
  • PR titles should be semantic (feat/fix/chore/docs/refactor)

CI/CD (GitHub Actions)

.github/workflows/build.yml runs:

  1. Go unit tests (with MySQL service)
  2. Frontend build (Node.js 20 + Yarn)
  3. Go binary build
  4. golangci-lint
  5. Semantic release + Docker push to ghcr.io/hanzoai/cloud

Configuration

  • Backend config: conf/app.conf read by the native conf.AppConfig ini reader (env-first; absent in the embedded runtime, which runs env-only)
  • Database: MySQL 8.0+ or MariaDB
  • Default Docker DB: PostgreSQL via compose.yml

Important Files

File Purpose
main.go Application entry point
routers/router.go All API route definitions
object/init.go Application initialization + LLM provider seeding
conf/app.conf Runtime configuration
web/src/App.js Frontend root component
web/src/backend/ API client helpers

hedge — ask several providers, keep the first answer

hedge.Race(ctx, dst, attempts...) is the primitive behind fast mode. Tail latency is not the average: a provider that usually answers in 400ms occasionally takes eight seconds, and a request cannot know which it drew until it has already waited. Asking three and keeping the first cuts that tail.

No provider changes, and none are needed. Every provider already writes into an io.Writer, so Race hands each attempt a different one and only lets the winner's reach the caller. ModelProvider is untouched — the seam was already the right shape.

FIRST BYTE WINS, the only definition of fastest a stream can act on. A provider that will finish sooner but has not started is not the winner: the caller waits on bytes, not completion, and by the time completion is knowable the response has arrived and there was nothing to race.

Two cases decide whether hedging helps or hurts, and both are pinned by tests verified against a deliberately broken implementation:

  • An EMPTY write does not claim. A provider flushing a zero-length chunk while still waiting upstream would otherwise win having said nothing, leaving the caller on the SLOWEST attempt with the others cancelled — the opposite of the point.
  • A FAST FAILURE is not a win. Race returns when the attempt that CLAIMED the stream finishes, not when the first attempt returns; an upstream 503 in 1ms produced no bytes, so the caller is still waiting on whoever answers.

Losers are cancelled — that is what stops the upstream call, which is what stops the cost — and Race does not wait for them to unwind, because waiting on a provider we are abandoning hands it exactly the latency this removes.

One ledger path each, because the two losers are opposite cases

ask.fan (controllers/hedged.go) carries a request into fast mode: how many to ask, where the winner's bytes go, and how to make and adopt a writer. serve() dispatches on it, so a caller sets a NUMBER rather than choosing a code path, and a fan too narrow to race falls back to the cascade with its retry and demotion intact. Nothing sets it yet — the mechanism ships inert, and TestWithoutAFanNothingChanges is what says so.

A raced loser and a failover refusal wear the same shape and mean the opposite thing, which is the whole design:

  • A REFUSAL errored. No tokens, no upstream charge — recordRefusals writes a "failover" row that bills nothing, and that is correct.
  • A RACED LOSER was CANCELLED mid-generation. The vendor produced tokens and charged us. Filing it as a refusal records a real charge as free and makes fast mode invisible in the ledger the team reads.

So a beaten provider never appears in the attempts race() returns — the caller records those as refusals — and settles through fan.bill instead.

Each attempt needs its OWN writer. The loser's token count is on ITS writer, and one writer shared across attempts holds exactly one answer: the winner's. A shared writer therefore bills every loser for the winner's tokens AND glues its half-sentence to the winner's answer on the client's wire — both failures show up in TestEachRacerIsBilledFromItsOwnWriter, verified against a deliberately shared fork. OpenAIWriter.fork/adopt are that pair: forks copy the dialect and start the buffers empty; the winner's is adopted so the non-streaming body and the tail chunk come from the provider that actually served.

THE BILL ARRIVES LATE, and that is the shape rather than a wrinkle. Race returns when the winner finishes, not when the losers do — so at that moment a loser has typically not even started, and reading its outcome is both a missing bill and a data race. (The first draft did exactly this; the test caught it.) settle therefore runs on its own goroutine, and fan.bill must close over VALUES: by the time it fires the handler has returned and, on fasthttp, the request has been recycled.

Cancellation is a WRITE ERROR, not a context. QueryText takes no context, so a loser's only stop signal is hedge.ErrLost on its next write. One that checks its write error stops there; one that does not runs to completion and bills us for all of it. Either way the tokens are real — which is the argument for billing them.

The switch, and why it is decided with the reservation

A request asks with "fast": true in the body, or X-Fast: 1 for callers with no browser in the way — the body form exists because a page cannot send a custom header past the edge's CORS allow-list, exactly as retrieval works (controllers/fast.go).

What a race costs is settled WITH the hold, never after. widthFor reserves est × fastWidth before anything is spent, because N attempts is N completions and a hold sized for one is a promise the ledger cannot keep — the losers' rows land later and take the balance past the point the gate said it would stop. fastWidth is 2: the tail this cuts is one provider drawing a slow response while another would have answered normally, and a second opinion removes most of it for one extra completion rather than two.

Two refusals that are deliberately NOT refusals:

  • Cannot afford the race → serve the ordinary request. The caller asked to go faster, not to be turned away, and a new way for a funded account to be told no is worse than quietly being normal speed.
  • Nobody to bill → never race. The public lane makes no reservation, so hedging there spends N completions on a stranger and settles none.

Still open: the toggle in the products. hanzo.chat follows web_search; hanzo.app follows base and must resolve its in-flight merge on the three AI-path files first. The API accepts the field today, so either can ship the control without waiting on the gateway.

The /v1 resource surface — ONE table, no compound routes

The published OpenAPI description is GENERATED from that same table:

go run ./cmd/openapi -spec ../openapi/ai/openapi.yaml

It writes only the region between the # BEGIN/END generated markers in the canonical spec (hanzoai/openapi, OpenAPI 3.1) — the inference paths above it are hand-authored with real OpenAI request/response schemas and stay that way. cmd/openapi -verify and TestSpecMatchesTheTable fail when the spec drifts from the table, so the contract customers and SDKs read cannot describe a surface this service does not serve.

The old swagger/ directory is DELETED. It was a second description, and it was wrong: basePath: /api with 135 /<verb>-<noun> paths, a base path this service has never served. Nothing referenced it, nothing embedded it, and no test noticed when it stopped being true — which is the argument for generating the replacement rather than hand-maintaining it.

Every CRUD route is generated from routers/resources.go. There is no second registration path, and no route is written by hand.

GET    /v1/<ns>/<resource>                 list      POST   /v1/<ns>/<resource>          create
GET    /v1/<ns>/<resource>/<owner>/<name>  read      PATCH  …/<owner>/<name>            update
                                                     DELETE …/<owner>/<name>            delete
GET    /v1/<ns>/<resource>/global          cross-tenant list (where a resource has one)

The namespace is ai for every row, and for every other route this service serves. HIP-0139 §3.1: a capability answers at ONE address, /v1/<name>, and a second top-level address is a second capability or a misfiled route — never an alias. So /v1/router/*, /v1/org/settings, /v1/finetune/*, /v1/memory/*, /v1/rag/*, /v1/docs/ingest, /v1/feedback and /v1/traffic/globe are all under /v1/ai/… now (ingest under /v1/ai/rag/, since it writes the index rag reads). The groupings in the table — retrieval, chat, models, content, compute, work, ops — are how it is READ, not a second prefix.

Two families stay outside /v1/ai because a protocol, not this repo, fixes their spelling (§3.2): the OpenAI/Anthropic wire (/v1/chat/completions, /v1/completions, /v1/embeddings, /v1/messages, /v1/models, /v1/images/*, /v1/audio/*, /v1/videos/*, /v1/responses, /v1/rerank), and POST /v1/embed, the multipart form hanzo.chat posts.

/v1/ai/<noun> is a resource collection address. Nothing may register a ZAP gateway PREFIX there: the registry matches on path alone, so a prefix at a collection swallows that collection's :owner/:name members and answers with the LIST handler — a wrong answer, not an error. Deeper addresses (/v1/ai/rag/query, /v1/ai/router/defaults) are routes, not collections, and are safe. TestNoGatewayPrefixOverTheResourceNamespace holds the line; it is why /v1/ai/feedback has no gateway prefix and reaches routers.App through the fallback instead.

Three things about this that are easy to get wrong:

The member key is <owner>/<name> — two path segments, not one. Every object is keyed by that pair (util.GetOwnerAndNameFromIdWithError requires EXACTLY two tokens, so a name containing a slash was never valid — the two-segment URL is lossless, not a narrowing). They are recomposed into id in the one place route params are bound (web.Router.ServeHTTP), so handlers keep reading c.Input().Get("id") unchanged. A single :id segment cannot work: Go decodes %2F back into a separator before routing.

/v1/iam belongs to the IAM service, not to this one. cloud proxies that whole subtree away (cloud/iam_edge.go). Anything registered there is shadowed and unreachable — it fails as a wrong answer, not an error. TestNoResourceClaimsForeignNamespace holds that line.

The policy filters key on names, not paths. authz_filter and filter_balance hold hand-curated sets (super-admin endpoints, balance exemptions, demo-mode allowances). Those names are DERIVED from the same table by policyKey, never restated — so a policy entry cannot drift from the route it governs. Do not hand-edit a path into those maps.

Adding a resource is one table entry. The tests will tell you if you got it wrong: every controller method the table names is checked to exist by reflection, every route must emit a policy key, no path may contain a verb, and an action whose handler reads no id must sit on the collection (a member action's route could never match).

An OpenAI-COMPATIBLE vendor is upstream Local, never OpenAI

Provider.Type IS the dialect declaration, and the two OpenAI-shaped upstreams are not interchangeable:

type client dialect price from
OpenAI NewOpenAiModelProvider /v1/responses — OpenAI's own surface CalculateOpenAIModelPrice, a table of what OpenAI RETAILS
Local NewLocalModelProvider /v1/chat/completions (local.go, CreateChatCompletionStream) the provider ROW (inputPricePerThousandTokens), plus compatibleProvider for the API it speaks

DigitalOcean already resolves to the Local client; so should every vendor that merely SPEAKS OpenAI. Typing one OpenAI sends it to the Responses API, and every compatible vendor shims that differently.

Measured, one provider (the Hugging Face router), one credential, one URL — typed OpenAI: meta-llama/Llama-3.3-70B-Instruct answered, while zai-org/GLM-5.2, deepseek-ai/DeepSeek-V4-Flash-0731 and moonshotai/Kimi-K3 returned an EMPTY message with a 200 and zero usage. All four answer directly, in both dialects. Retyped Local, all four answer through the plane, reasoning models included, with no <think> leaking into content — and no code change. The reasoning streams carry response.reasoning_text.delta, which the Responses loop does not case on (it handles only response.reasoning_summary_text.delta).

So the rule is a ROW, not a patch: an OpenAI-compatible vendor is Local with its own URL and its own price. Reasoning stays a FIELD the dialect carries, never something sniffed out of the content stream.

Three things still dispatch on substrings of the model NAME, and a model in none of the lists gets wrong behaviour rather than an error — getOpenAiModelType (which dialect), InlinesReasoning (strings.Contains(name, "deepseek"), directly under a comment saying which SKU inlines is "a property of the catalog (HIP-0039) … never a list here"), and the retail price table. Each is a place a new model silently misbehaves; the Local row above is what they were compensating for.

LLM serving path (OpenAI-compatible /v1) — READ THIS

The ai module is mounted by hanzoai/cloud (HIP-0106). In prod it runs as cloud-api in hanzo-k8s, fronted by gateway (api.hanzo.ai). Both hanzo.chat (LibreChat, baseURL https://api.hanzo.ai/v1) and hanzo.app (build) consume this exact surface.

Request flow for a completion:

  1. gateway (api.hanzo.ai) validates the IAM JWT, mints identity headers, forwards /v1/chat/completions → cloud-api:8000. /v1/models is also proxied here; admin endpoints (/v1/get-providers, /v1/update-provider, /v1/*-model-route) are NOT gateway-exposed (use the direct api.cloud.hanzo.ai ingress + a session).
  2. controllers.ApiController.ListModels (/v1/models) requires a valid Bearer token — unauthenticated returns 401 with no data array (this looks like "0 models"; it is not — authenticate to see the real list).
  3. Model → provider routing: resolveModelRouteForOrg resolves in order DB routes (/v1/*-model-route, per-org → global "admin") → YAML config (conf/models.yaml, the cloud-api-models ConfigMap — runtime source of truth in prod) → static controllers/model_routes.go map. The YAML wins, so changing routing live = edit the ConfigMap (then /v1/reload-model-config or restart). The static map is the fallback when no YAML is present.
  4. Provider records live in the DB (object/init.go initLLMProviders), keyed by name: do-ai (DigitalOcean GenAI, the primary — backs OpenAI/ Anthropic/Llama/DeepSeek/Qwen/GLM/Kimi via inference.do-ai.run), fireworks, openai-direct, zen. Each call does object.GetModelProviderByName(route.providerName) → reads ClientSecret.
  5. Provider keys: ClientSecret is kms://SECRET_NAME. ResolveProviderSecret (object/kms.go) resolves it env-var-first (os.Getenv(SECRET_NAME)), falling back to a real KMS call — BUT only runs when kms != nil, which needs KMS_CLIENT_ID or KMS_SERVICE_TOKEN set. So the working prod recipe is: set KMS_CLIENT_ID (flips kms on) + provide DO_AI_API_KEY/OPENAI_API_KEY/ ANTHROPIC_API_KEY/FIREWORKS_API_KEY as env (from a K8s secret sourced from KMS-managed values). The env-first path means no live KMS dependency on the hot path.
  6. Provider re-seed self-heals ClientSecret/ProviderUrl/State/Type/ SubType on every boot from the initLLMProviders table — so fixing a stale key or upstream URL = edit the seed table and restart (no manual DB edit).

Async video (/v1/videos) — Sora-style, NOT sync. Text-to-video generation takes minutes, so /v1/videos/generations is ASYNC (the sync handler used to hold the request ~104s → console /ai proxy timed out → browser 502 while the generation actually succeeded). The lifecycle is three fast requests: POST /v1/videos/generations (create → returns a video_<uuid> job id + queued object immediately), GET /v1/videos/{id} (one status poll), and GET /v1/videos/{id}/content (stream the finished MP4). zen3-video* route to the spark-video provider (our GB10, spark-video.hanzo.ai/v1), whose OWN /v1/videos API is already the same async create→poll→download shape the three model.{Create,Retrieve,Download}VideoDOAI primitives proxy. Metering is per-user and lands EXACTLY ONCE on completion: create RESERVES the per-video budget and carries the *budgetHold in the in-pod videoJobStore (valid — the pod is single-replica, the balance ledger's own invariant); the first request that observes completed (poll or download) settles the hold + records the single Commerce usage event (videoJob.markCompleted, idempotent); a failed job releases the hold and bills nothing; a reaper releases abandoned/wedged jobs' holds (TTL) so budget never leaks. Every poll/download re-runs the shared auth+routing policy and enforces OWNERSHIP (caller subject == job subject, else 404 — never confirm another user's job); the provider key is re-resolved per request, never stored. The gateway passes every /v1/* sub-path through to cloud-api, so the poll/ content routes need no gateway change; the console /ai proxy allow-list adds the v1/videos/{id}[/content] pattern (hanzoai/console#92).

zen models: branded, owned_by: hanzo, premium: true. Route to do-ai upstreams (qwen3+/glm/kimi/deepseek — same working key, no GPU). Each zen model declares its public owner in owned_by; the identity is injected at call time via zenIdentityPrompt (hip-00NN). Do NOT route zen to Fireworks serverless paths (accounts/fireworks/models/*) — that account returns 404 "not deployed" for most of them.

Billing gate (routers/filter_balance.go + openai_api.go): non-premium models need balance > 0; premium models need balance > starter credit. Balance comes from commerce at /v1/billing/balance?user=<org>&currency=usd. No principal bypasses it: the gate always consults the balance, and a lookup that fails returns 500 rather than free access. To verify premium/zen, credit the org.

AI Login Manager — universal metering + connected accounts + 1% BYO fee

ai is the centralized login manager for every user's AI providers. Every AI call from every surface (@hanzo/dev, Hanzo Desktop, hanzo.chat, hanzo.app) routes through THIS router at api.hanzo.ai/v1. For the caller's org the router resolves WHICH account executes the call, runs it, and meters ONE usage event — regardless of whose credits paid for it. There is exactly one meter and one credential store.

Two account modes (one router, one meter):

  • Hanzo credits (default). No org-owned provider for the requested model → the global built-in (Owner == "admin") serves it. Billed = token list-price cost (existing behavior). No surcharge.
  • Connected third-party account (BYO). The org connected its OWN OpenAI / Anthropic / Google account via /v1/ai/connections. The router uses the org's KMS-sealed key, the customer pays that provider directly, and Hanzo bills a 1% platform fee on the provider list price for routing + metering + management.

BYO is the EXISTING BYOK seam, not new plumbing. resolveProviderForUser → object.GetModelProviderByNameForOrg(org, name) returns the org's own provider row when it has one (Owner == org), else the global admin built-in. providerBYO (controllers/platform_fee.go) classifies the resolved pair: BYO iff provider.Owner != "" && != "admin" && == user.Owner.

Metering — one path, extended not rebuilt. Every completion terminal already calls recordUsage (→ commerce POST /v1/billing/usage) + recordTrace (→ the hanzo.cloud_usage warehouse). This feature adds three dimensions to that ONE event: byo, fee_cents, account. platformFeeCents(costCents, byo) = byo ? ceil(costCents/100) : 0. Billed amount = byo ? fee_cents : cost_cents; the full cost_cents is recorded on every row for analytics even when only the fee is billed. hanzo.cloud_usage is the ONE global usage store (its organization column is the row-level tenant filter — a tenant reads only its own org; the admin org reads unfiltered). The rich per-org analytics lens is entitlement-gated in hanzoai/cloud (analytics.datastore); basic own-org usage is always readable.

Connected accounts API (/v1/ai/connections, IAM-authed, org-scoped). Curated login-manager surface over the per-org object.Provider store (controllers/connections_api.go). Keys are sealed via object.StoreProviderSecret → kms://… ref (fails closed if KMS is down); the raw key is NEVER stored in a row, returned, or logged.

  • GET /v1/ai/connections → [{provider, connected, account_label, updated_at}] (never the key)
  • POST /v1/ai/connections {provider, apiKey} → seals key, activates the org row; returns metadata only
  • DELETE /v1/ai/connections/:provider → deactivates the org row (resolution reverts to Hanzo credits)

Enforce "even if logged in". All AI traffic must reach this router; no surface bills a third party off-meter. hanzo.app's workspace default was flipped openrouter → hanzo; hanzo.chat routes via its gateway config; @hanzo/dev defaults to api.hanzo.ai/v1 under Hanzo login. Claude/Opus for @hanzo/dev is served HERE (Anthropic-native /v1/messages + OpenAI-compat /v1/chat/completions) against the org's connected Anthropic account + 1% fee — no direct-to-provider bypass.

Next slice (route selection). Connections are stored/sealed/listed and the 1%-fee metering fires the instant the resolver returns an org-owned provider. But the route table maps all models to providerName: "do-ai" (the aggregating upstream), so a connection row named openai/anthropic/google is not yet SELECTED for its model family. Making a connected consumer account actually serve its models is a bounded route-table change (model-family → org-connection selection

  • real upstream model-id/base-url mapping) — deliberately not in this slice.

RAG — ONE unified surface (retires the standalone chat-rag-api)

ai is the single RAG layer for the whole platform. There is exactly ONE retrieval pipeline; the standalone hanzoai/chat-rag-api (danny-avila/rag_api fork) is now redundant and slated for archive.

One index, one embedder, one retrieval path. Every document — doc/crawl/ github/s3 ingest AND per-file uploads from hanzo.chat — lands in the SAME per-tenant index {owner}-{store}-docs, fanned out to BOTH Hanzo Search (keyword, Meilisearch) and Hanzo Vector (semantic, Qdrant) via object.IndexDocuments, and retrieved via object.SearchDocuments (hybrid RRF). See object/search_docs.go.

File-scoped RAG is a FILTER, not a parallel store. file_id is a filterable dimension on that one index (DocIndex.FileID; Meili filterable attr file_id; Qdrant payload file_id). Uploaded-file retrieval = a file_id filter over the tenant index. object/rag.go holds the file-scoped logic (RagEmbedFile, RagQuery, DeleteRagFile, RagFileContext); chunking uses split.RecursiveSplitProvider (parity with rag-api's RecursiveCharacterTextSplitter, 1500/100), parsing uses txt.GetParsedTextFromUrl (PDF/CSV/XLSX/PPTX/…). Uploaded files default to the rag-files store so they don't pollute curated docs.

One address family (controllers/rag.go): POST /v1/ai/rag/ingest, /v1/ai/rag/embed, /v1/ai/rag/query, /v1/ai/rag/query-multiple, /v1/ai/rag/delete, GET /v1/ai/rag/context. Ingest and embed are two ways in to the one index; the rest read it back.

The one exception is POST /v1/embed (controllers/rag_librechat.go) — the MULTIPART form hanzo.chat's uploadVectors() posts. Its request line is fixed by that client, not by us, which is why it is not a spelling of /v1/ai/rag/embed. Everything hanzo.chat READS it reads from the native family.

Migration: hanzo.chat's RAG client calls ${origin}/v1/ai/rag/* directly. Auth/billing reuse the doc-RAG path (resolveSearchAuth/requireIndexAuth + recordSearchUsage); owner is the authenticated principal (tenant isolation), never a client-supplied field.

Search / Crawl / Web-search — THREE orthogonal products, ONE way each

Three distinct products, no overlap. Do NOT add a fourth crawl path.

  • POST /v1/search = native full-text index search (SearchDocs → object.SearchDocuments). Hanzo Search (Meilisearch, keyword, hanzoai/search-go at searchHost:7700) + Hanzo Vector (Qdrant semantic, vectorHost:6333) fused hybrid-RRF over the per-tenant index {owner}-{store}-docs. Returns {hits:[…]}. Needs a populated index — an empty/new tenant index returns {hits:[]} (no error). GET /v1/search/stats reports index size.

  • POST /v1/crawl = crawl a URL, get content back — THE single canonical crawl (controllers/crawl.go Crawl → object.Crawl → CrawlWithCrawl4AI). Backed by the self-hosted Crawl4AI (hanzoai/crawl, crawl.hanzo.svc:11235, Apache-2.0). Body {url} or {urls} (merged, deduped, capped at 10); returns {results:[{url,title,description,markdown,success,metadata}]} with LLM-ready markdown (prefers Crawl4AI fit_markdown, falls back to raw_markdown). Auth: requireIndexAuth (fail-closed, rejects read-only pk-/hz_ keys) + balance gate; usage recorded via recordSearchUsage(…, "crawl", "crawl4ai", …). object/crawl4ai.go handles the deployed Crawl4AI 0.8.6 wire shapes: markdown is an OBJECT (MarkdownField polymorphic decode), the /crawl envelope is synchronous with a boolean success and no status/task_id (return inline Results whenever present), and links/media values are heterogeneous (map[string][]map[string]interface{}).

  • POST /v1/scrape = crawl-and-INDEX (ingest) — NOT a general crawl. It crawls a site and WRITES structured content into the tenant search index, returning ScrapeStats (ScrapeDocs → object.ScrapeAndIndex). Sibling of POST /v1/index (index caller-supplied docs). Stays because its output/effect (index write) is distinct from /v1/crawl (fetch + return).

  • POST /v1/scrape/preview is a DEPRECATED alias of /v1/crawl — its handler now forwards to Crawl (one implementation, no parallel path). Use /v1/crawl.

  • /v1/websearch/* (lives in hanzoai/cloud, not ai) = web meta-search — a separate product: SearXNG-compat /v1/websearch/search + a firecrawl-compat /v1/websearch/v1/scrape leg that hanzo.chat's web-search plugin calls. That leg is a fixed EXTERNAL contract (firecrawl {success,data:{markdown}}) and reuses the SAME Crawl4AI backend — it is an integration shim for the web-search flow, not a general crawl endpoint. The general "crawl a URL" is ONLY /v1/crawl.

Security invariants (READ before touching auth/billing/scaling)

  • JWT iss/aud — every request-auth JWT path goes through object.ParseAndValidateJWT (signature via IAM and iss/aud policy), never raw iam.ParseJwtToken. object/jwt_validate.go sources the issuer from JWT_ISSUER/IAM_ISSUER/CLOUD_IAM_ISSUER/AUTH_ISSUER (any, default https://hanzo.id) and the audience allowlist from GATEWAY_ALLOWED_AUDIENCES (+ folds in IAM_AUDIENCE/AUTH_AUDIENCE), mirroring the gateway. A foreign-aud token → 401. The OAuth code-exchange callback (account.go Signin) is NOT a request-auth path and is intentionally exempt.

  • Balance ledger — single-pod invariant — object.GlobalBalanceLedger is in-pod memory. Reserve/Settle are correct ONLY at replicas: 1. Two pods double-spend (each reserves against its own cached Commerce balance). Enforced by deploy strategy: Recreate + HPA min=max=1 AND a boot assertion (bootstrap.go panics if CLOUD_API_REPLICAS > 1). To scale out you MUST first move Reserve/Settle behind a Commerce-atomic conditional reserve.

  • Single-request reservation (controllers/billing_reserve.go) — an uncapped max_tokens is clamped (clampMaxTokens) and the reservation covers the larger of the clamped ceiling and the QueryText pipeline's fixed completion cap (reserveCompletionTokens / reserveCompletionFloor=4096, mirrored in model/openai_util.go), so actual spend can never exceed the hold on any path.

  • A tier is read through object.TierReader, and the host owes a reader that answers — there are two consumers and they must ask in the same order: controllers.commerceFamilyTier (the per-SKU gate) and routers.TierCache.commerceTierLookup (the rate limiter). Both read the installed reader FIRST; the HTTP route is what a STANDALONE ai has instead of one, not a second way to ask. InitTierCache builds the cache when EITHER exists, not when the endpoint alone is set.

    The reader is NOT necessarily in-process. Every cloud app is its own child PROCESS, so a Go func var cannot cross that boundary: what cloud installs is metering.Client.Tier, which is an HTTP call to GET /v1/billing/tier carrying COMMERCE_SERVICE_TOKEN and X-Org-Id. That call has no user session, so it only succeeds where the handler resolves its tenant with billing's readerOrg (session OR trusted service naming an org) rather than principal.OrgFrom alone. balance had that rule; the typed ops behind tier did not, and answered 401 sign in to view finance to a token the same request proved good on /v1/billing/balance. Fixed in cloud apps/billing/typed.go (payer falls through to readerOrg), guarded by apps/billing/tier_s2s_test.go.

    The failure is silent in both directions, which is why it lasted: the rate limiter maps a failed read to TierZenFree (every paying org shaped as free) and the SKU gate maps one to unknown and ALLOWS (it stops gating). Neither logs an error anyone watches, so assert the tier a caller actually gets rather than that the lookup "worked".

  • A failed call bills only what a vendor actually ran — recordUsage bills a failure because by then the vendor has run the request and invoiced us. That premise does not hold for a refusal it can decide from the request alone (400, 401, 402, 403, 404, 413, 422, 429) or for an error carrying no status at all (nothing was dialled). controllers.ran(err) answers that, and it is NOT faultOf: faultOf asks where the request goes NEXT, and the two come apart both ways — a 401 fails over having run nothing, a partial write stops the cascade having run plenty. spent() consults ran only when nothing came back, so a meter the vendor reported and any text that reached the caller still settle it first. Without this the quantity billed is the prompt measured on the way out, so the charge grows with the request: 400k tokens refused for being too long billed a dollar.

  • Exemption is by ROUTE, never by identity. BALANCE_EXEMPT_USERS is read by nothing; TestEnforceBalanceGate_NoExemptBypass and TestCheckBalanceNoExemption set it and require the formerly-listed subjects to fail closed. What is exempt is a set of paths — isBalanceExempt in routers/filter_balance.go — plus balanceExemptNames, three usage reads named as POLICY rather than URL so a $0 org can still see the panel telling it to add credit, and so moving the route cannot silently re-gate them. Exempt from balance is not exempt from auth.

  • SuperAdmin — util.IsSuperAdmin = owner == util.AdminOrg, the constant "admin". Membership in that one reserved IAM org is the whole definition: no list, no env override, and isAdmin is not part of it — an org admin (hanzo/z) administers hanzo and holds no platform power.

  • Authz gate (routers/authz_filter.go) — platform-sensitive endpoints (superAdminEndpoints) are gated FIRST (before preview-mode + exempt reads). There is deliberately NO adminDomain Host bypass (a removed full-authz-bypass primitive). No principal → 401; wrong principal → 403.

  • User redaction (object/redact.go) — RedactUserSecrets is an allowlist projection over string/[]string fields (exposableUserFields): a field not on the list is zeroed, so a NEW upstream secret field is fail-secure. Credential struct slices (MfaAccounts/ManagedAccounts/MfaItems/FaceIds) are nilled.

Cost: known, zero, or not known — three answers, never two

An unregistered COGS used to read back as the CUSTOMER PRICE. costInputPerMillion() returned the price when no cost rate was set, so tokenProviderCostNano computed cost == billed and every such call recorded margin_nano == 0. For the Enso family that was every call: 1,296 rows checked, all of them zero. It is not a rounding error — it is a ledger reporting a business that earns nothing on everything it sells.

The shape of the fix is that cost is optional and margin follows it:

  • controllers/model_pricing.go — modelPrice.costed() reports whether the model states what it costs us, and it requires BOTH legs. Half a COGS is not a COGS: an input rate with no output rate prices the completion — the expensive half — at zero. costInputPerMillion() / costOutputPerMillion() now return the configured rate and nothing else; there is no path that reads one without first asking costed().
  • controllers/billing_nano.go — tokenProviderCostNano and providerCostNano return *int64, nil where no cost is known. costMargin.CostNano is a pointer for the same reason MarginNano already was: the row has to be able to say it does not know, and an int64 can only say zero. usageMargin reports a margin only when both sides are real.
  • controllers/zap_native.go + object/cloud_usage.go — the warehouse column is Int64, so an unknown cost writes 0 and sets uncosted = 1 beside it. Query uncosted = 0 for rows whose cost is a fact.
  • controllers/telemetry.go — the span emits provider_cost only when there is one, the way it already treated margin. It no longer asserts a COGS it does not have.

Zero and unknown are different answers. Zero is a fact: BYO (the customer paid the upstream with their own key), a free route, speech on hardware we already own. Nil is the absence of one: image and video have a per-unit PRICE table and no vendor invoice behind it, and a token model that states no COGS has told us nothing.

Where a cost is registered

object.ModelRoute.CostInPerMillion / CostOutPerMillion — the route row, which the admin cockpit already reads and writes. It is the only place, and the configured PRICE tables are deliberately not consulted as a substitute.

The family leg used to make that unreachable for zen/enso: familyModelPrice returned first and short-circuited the resolver, and the family wire (zenWireModel) carries no cost field, so an enso model could not be given a COGS at all without restating its price. getModelPriceForOrgOK now completes an uncosted family price from registeredCost. Price and cost resolve independently, because they are independent facts: the family owns what we charge, the route row owns what it costs us.

What still has no COGS

The mechanism no longer lies, but a rate nobody has entered is still a rate nobody has entered. Enso's real figures are known and live in the Enso catalog: 1.392 / 2.784 $/MTok on the deepseek-v4-pro anchor (enso, enso-ultra) and 0.112 / 0.224 on deepseek-4-flash. Against billed 4/20 that is roughly 2.9x in / 7.2x out.

Registering them is an operator action on the route row — set both legs; one leg alone is ignored by costed(). Until then the rows read uncosted = 1, which is the honest answer and a visible one: SELECT count() FROM hanzo.cloud_usage WHERE uncosted = 1 is the backlog, and it should fall as rates are entered.

For enso-ultra the fan-out means COGS is the sum of the arms that ran, which only the server can total. controllers/trace_export.go carries BilledNano/CostNano and surfaces as CostNanoExact; a stated cost wins over the rate table in usageMargin. Rate math cannot express a fan-out cost at all, so that is the right seam for it.

The admin cockpit's below-cost check compared price against a "cost" that WAS the price, so its difference was always exactly zero and it could neither fire honestly nor stay quiet honestly. It now has a cost to compare against, or a flag saying there is none.