Every cloud_usage row for the Enso family has billed_nano == cost_nano and margin_nano == 0. The cause is not Enso-specific: costInputPerMillion and costOutputPerMillion fall back to the CUSTOMER PRICE when no COGS rate is registered, and tokenProviderCostNano computes COGS at exactly those rates, so any model without a registered COGS reports cost equal to what we charged. Filing rather than fixing. The fix is a design choice -- the serving side should push CostNanoExact, which trace_export already accepts and which is the only way to express an enso-ultra fan-out cost at all -- and it belongs with whoever owns the trace writer. Recorded because of what it breaks downstream: the admin cockpit's below-cost check compares price against a cost that IS the price, so the difference is always exactly zero and the check can neither fire nor stay quiet honestly. That is the likeliest explanation for it misfiring. A computed zero is also indistinguishable from a real zero (BYO calls legitimately record cost 0), so nothing downstream can tell "not computed" from "earned nothing". The rates themselves are correct and reconciled against production usage rows. Do not "fix" this by editing them. Co-authored-by: Hanzo Dev <[email protected]>
28 KiB
ai — agent guide
hanzoai/ai is the canonical AI control plane for the Hanzo platform: model
hub, native Go model routing, RAG, and MCP/A2A management. It speaks the
OpenAI-compatible /v1 API, routes 66+ models to upstream providers, and meters
every request. Renamed from hanzoai/cloud (HIP-0106); mounts as the ai
subsystem inside hanzoai/cloud. In prod it runs as cloud-api on hanzo-k8s,
fronted by hanzoai/gateway at api.hanzo.ai.
Canonical role. This is a Hanzo service/infra repo — one impl, one place.
It is NOT an SDK; SDKs link out to it. Completeness order across languages is
Python → Rust → C++ → Go. Canonical spec: ~/work/hanzo/SDK-ARCHITECTURE.md.
Brand rules (hard — enforce in every edit).
- Never call this an "LLM gateway" and never position it against LiteLLM — it is a full AI cloud / control plane, not a proxy.
/v1/only, never an/api/prefix.- Zen models are Hanzo's own family (
owned_by: hanzo) — never name upstream models. - Voice: "Hanzo — the Open AI Cloud." Modern, crisp, developer-first.
Install / run.
go build -race -ldflags "-extldflags '-static'" # build
./cloud-api-server # run (env-configured)
go test -v $(go list ./...) -tags skipCi # test (requires MySQL)
docker compose up # local stack
Key entry points. main.go (entry) · bootstrap.go (boot + replica
assertion) · routers/router.go (all /v1 routes) · controllers/ (HTTP
handlers) · object/init.go (init + LLM provider seeding) · model/ (provider
integrations) · object/kms.go (secret resolution) · web/ (React admin UI).
Architecture
Full-stack application:
- Backend: Go 1.26 + native web router (
github.com/hanzoai/ai/web, served asrouters.App; upstream beego dropped), MySQL/MariaDB/PostgreSQL - Frontend: React + Ant Design v5, located in
web/ - Auth: Hanzo IAM SSO integration
Directory Structure
cloud/
├── main.go # Entry point
├── go.mod # Go module
├── conf/app.conf # Beego config
├── controllers/ # HTTP handlers (60+ files)
├── object/ # Business logic & data models (85+ files)
├── model/ # AI model provider integrations (38+ files)
├── routers/ # Routes and middleware filters
├── embedding/ # Embedding provider implementations
├── agent/ # MCP agent support
├── split/ # Text splitting strategies
├── txt/ # Document parsing (PDF, CSV, XLSX, PPTX)
├── storage/ # File storage abstractions
├── util/ # Utilities
└── web/ # React frontend
├── src/
└── package.json
Build & Run
Backend
# Build
go build -race -ldflags "-extldflags '-static'"
# Cross-platform build (linux amd64/arm64/riscv64)
./build.sh
Frontend
cd web
yarn install
yarn start # Dev server
yarn run build # Production build
Docker
docker compose up
Testing
# Go tests (requires MySQL)
go test -v $(go list ./...) -tags skipCi
# Frontend tests
cd web && yarn test
Linting
# Go - uses golangci-lint with gofumpt
golangci-lint run
# Frontend
cd web && yarn lint
Key Conventions
Backend (Go)
- Framework: native web router (
github.com/hanzoai/ai/web, served asrouters.App) — controllers handle HTTP, objects contain business logic; upstream beego is dropped (no direct import — routing, config, logging, pagination are all native) - New AI providers go in
model/(implement the provider interface) - Database access via the native store in
object/ - Route registration: CRUD resources are GENERATED from the one table in
routers/resources.go;routers/router.goholds only what is NOT a resource (the OpenAI-compatible surface and a few singletons). See "The /v1 resource surface" below. - i18n strings: avoid duplicate keys across frontend and backend
Frontend (React)
- UI components use Ant Design v5
- i18n via i18next; translation files in
web/src/locales/ - State via React Context/Hooks (no Redux)
- API calls go through
web/src/backend/helper modules
Commits & PRs
- Branch naming:
claude/<name> - Main branch:
main - Follows semantic versioning (
.releaserc.json) - PR titles should be semantic (feat/fix/chore/docs/refactor)
CI/CD (GitHub Actions)
.github/workflows/build.yml runs:
- Go unit tests (with MySQL service)
- Frontend build (Node.js 20 + Yarn)
- Go binary build
- golangci-lint
- Semantic release + Docker push to
ghcr.io/hanzoai/cloud
Configuration
- Backend config:
conf/app.confread by the nativeconf.AppConfigini reader (env-first; absent in the embedded runtime, which runs env-only) - Database: MySQL 8.0+ or MariaDB
- Default Docker DB: PostgreSQL via
compose.yml
Important Files
| File | Purpose |
|---|---|
main.go |
Application entry point |
routers/router.go |
All API route definitions |
object/init.go |
Application initialization + LLM provider seeding |
conf/app.conf |
Runtime configuration |
web/src/App.js |
Frontend root component |
web/src/backend/ |
API client helpers |
The /v1 resource surface — ONE table, no compound routes
The published OpenAPI description is GENERATED from that same table:
go run ./cmd/openapi -spec ../openapi/ai/openapi.yaml
It writes only the region between the # BEGIN/END generated markers in the
canonical spec (hanzoai/openapi, OpenAPI 3.1) — the inference paths above it are
hand-authored with real OpenAI request/response schemas and stay that way.
cmd/openapi -verify and TestSpecMatchesTheTable fail when the spec drifts from
the table, so the contract customers and SDKs read cannot describe a surface this
service does not serve.
The old swagger/ directory is DELETED. It was a second description, and it was
wrong: basePath: /api with 135 /<verb>-<noun> paths, a base path this service
has never served. Nothing referenced it, nothing embedded it, and no test noticed
when it stopped being true — which is the argument for generating the replacement
rather than hand-maintaining it.
Every CRUD route is generated from routers/resources.go. There is no second
registration path, and no route is written by hand.
GET /v1/<ns>/<resource> list POST /v1/<ns>/<resource> create
GET /v1/<ns>/<resource>/<owner>/<name> read PATCH …/<owner>/<name> update
DELETE …/<owner>/<name> delete
GET /v1/<ns>/<resource>/global cross-tenant list (where a resource has one)
Namespaces are the subsystems that own the data, so a reader can tell what a route
touches from its prefix: auth (this backend's own account/session objects),
rag, chat, ai, content, compute, work, ops.
Three things about this that are easy to get wrong:
The member key is <owner>/<name> — two path segments, not one. Every object
is keyed by that pair (util.GetOwnerAndNameFromIdWithError requires EXACTLY two
tokens, so a name containing a slash was never valid — the two-segment URL is
lossless, not a narrowing). They are recomposed into id in the one place route
params are bound (web.Router.ServeHTTP), so handlers keep reading
c.Input().Get("id") unchanged. A single :id segment cannot work: Go decodes
%2F back into a separator before routing.
/v1/iam belongs to the IAM service, not to this one. cloud proxies that whole
subtree away (cloud/iam_edge.go). Anything registered there is shadowed and
unreachable — it fails as a wrong answer, not an error.
TestNoResourceClaimsForeignNamespace holds that line.
The policy filters key on names, not paths. authz_filter and filter_balance
hold hand-curated sets (super-admin endpoints, balance exemptions, demo-mode
allowances). Those names are DERIVED from the same table by policyKey, never
restated — so a policy entry cannot drift from the route it governs. Do not
hand-edit a path into those maps.
Adding a resource is one table entry. The tests will tell you if you got it wrong: every controller method the table names is checked to exist by reflection, every route must emit a policy key, no path may contain a verb, and an action whose handler reads no id must sit on the collection (a member action's route could never match).
LLM serving path (OpenAI-compatible /v1) — READ THIS
The ai module is mounted by hanzoai/cloud (HIP-0106). In prod it runs as
cloud-api in hanzo-k8s, fronted by gateway (api.hanzo.ai). Both
hanzo.chat (LibreChat, baseURL https://api.hanzo.ai/v1) and hanzo.app
(build) consume this exact surface.
Request flow for a completion:
gateway(api.hanzo.ai) validates the IAM JWT, mints identity headers, forwards/v1/chat/completions→cloud-api:8000./v1/modelsis also proxied here; admin endpoints (/v1/get-providers,/v1/update-provider,/v1/*-model-route) are NOT gateway-exposed (use the directapi.cloud.hanzo.aiingress + a session).controllers.ApiController.ListModels(/v1/models) requires a valid Bearer token — unauthenticated returns 401 with nodataarray (this looks like "0 models"; it is not — authenticate to see the real list).- Model → provider routing:
resolveModelRouteForOrgresolves in order DB routes (/v1/*-model-route, per-org → global "admin") → YAML config (conf/models.yaml, thecloud-api-modelsConfigMap — runtime source of truth in prod) → staticcontrollers/model_routes.gomap. The YAML wins, so changing routing live = edit the ConfigMap (then/v1/reload-model-configor restart). The static map is the fallback when no YAML is present. - Provider records live in the DB (
object/init.goinitLLMProviders), keyed by name:do-ai(DigitalOcean GenAI, the primary — backs OpenAI/ Anthropic/Llama/DeepSeek/Qwen/GLM/Kimi viainference.do-ai.run),fireworks,openai-direct,zen. Each call doesobject.GetModelProviderByName(route.providerName)→ readsClientSecret. - Provider keys:
ClientSecretiskms://SECRET_NAME.ResolveProviderSecret(object/kms.go) resolves it env-var-first (os.Getenv(SECRET_NAME)), falling back to a real KMS call — BUT only runs whenkms != nil, which needsKMS_CLIENT_IDorKMS_SERVICE_TOKENset. So the working prod recipe is: setKMS_CLIENT_ID(flips kms on) + provideDO_AI_API_KEY/OPENAI_API_KEY/ANTHROPIC_API_KEY/FIREWORKS_API_KEYas env (from a K8s secret sourced from KMS-managed values). The env-first path means no live KMS dependency on the hot path. - Provider re-seed self-heals
ClientSecret/ProviderUrl/State/Type/SubTypeon every boot from theinitLLMProviderstable — so fixing a stale key or upstream URL = edit the seed table and restart (no manual DB edit).
Async video (/v1/videos) — Sora-style, NOT sync. Text-to-video generation
takes minutes, so /v1/videos/generations is ASYNC (the sync handler used to
hold the request ~104s → console /ai proxy timed out → browser 502 while the
generation actually succeeded). The lifecycle is three fast requests:
POST /v1/videos/generations (create → returns a video_<uuid> job id +
queued object immediately), GET /v1/videos/{id} (one status poll), and
GET /v1/videos/{id}/content (stream the finished MP4). zen3-video* route to
the spark-video provider (our GB10, spark-video.hanzo.ai/v1), whose OWN
/v1/videos API is already the same async create→poll→download shape the three
model.{Create,Retrieve,Download}VideoDOAI primitives proxy. Metering is per-user
and lands EXACTLY ONCE on completion: create RESERVES the per-video budget and
carries the *budgetHold in the in-pod videoJobStore (valid — the pod is
single-replica, the balance ledger's own invariant); the first request that
observes completed (poll or download) settles the hold + records the single
Commerce usage event (videoJob.markCompleted, idempotent); a failed job releases
the hold and bills nothing; a reaper releases abandoned/wedged jobs' holds (TTL)
so budget never leaks. Every poll/download re-runs the shared auth+routing policy
and enforces OWNERSHIP (caller subject == job subject, else 404 — never confirm
another user's job); the provider key is re-resolved per request, never stored.
The gateway passes every /v1/* sub-path through to cloud-api, so the poll/
content routes need no gateway change; the console /ai proxy allow-list adds the
v1/videos/{id}[/content] pattern (hanzoai/console#92).
zen models: branded, owned_by: hanzo, premium: true. Route to do-ai
upstreams (qwen3+/glm/kimi/deepseek — same working key, no GPU). Each zen model
declares its public owner in owned_by; the identity is injected at call time
via zenIdentityPrompt (hip-00NN). Do NOT route zen to Fireworks serverless
paths (accounts/fireworks/models/*) — that account returns 404 "not deployed"
for most of them.
Billing gate (routers/filter_balance.go + openai_api.go): non-premium
models need balance > 0; premium models need balance > starter credit. Balance
comes from commerce at /v1/billing/balance?user=<org>¤cy=usd. Users in
BALANCE_EXEMPT_USERS (e.g. hanzo/z) bypass ALL gates — use the
[email protected] token to verify premium/zen without a credited org.
AI Login Manager — universal metering + connected accounts + 1% BYO fee
ai is the centralized login manager for every user's AI providers. Every AI call
from every surface (@hanzo/dev, Hanzo Desktop, hanzo.chat, hanzo.app) routes through
THIS router at api.hanzo.ai/v1. For the caller's org the router resolves WHICH
account executes the call, runs it, and meters ONE usage event — regardless of whose
credits paid for it. There is exactly one meter and one credential store.
Two account modes (one router, one meter):
- Hanzo credits (default). No org-owned provider for the requested model →
the global built-in (
Owner == "admin") serves it. Billed = token list-price cost (existing behavior). No surcharge. - Connected third-party account (BYO). The org connected its OWN OpenAI /
Anthropic / Google account via
/v1/ai/connections. The router uses the org's KMS-sealed key, the customer pays that provider directly, and Hanzo bills a 1% platform fee on the provider list price for routing + metering + management.
BYO is the EXISTING BYOK seam, not new plumbing. resolveProviderForUser →
object.GetModelProviderByNameForOrg(org, name) returns the org's own provider row
when it has one (Owner == org), else the global admin built-in. providerBYO
(controllers/platform_fee.go) classifies the resolved pair: BYO iff
provider.Owner != "" && != "admin" && == user.Owner.
Metering — one path, extended not rebuilt. Every completion terminal already
calls recordUsage (→ commerce POST /v1/billing/usage) + recordTrace (→ the
hanzo.cloud_usage warehouse). This feature adds three dimensions to that ONE
event: byo, fee_cents, account. platformFeeCents(costCents, byo) = byo ? ceil(costCents/100) : 0. Billed amount = byo ? fee_cents : cost_cents; the
full cost_cents is recorded on every row for analytics even when only the fee is
billed. hanzo.cloud_usage is the ONE global usage store (its organization column
is the row-level tenant filter — a tenant reads only its own org; the admin org
reads unfiltered). The rich per-org analytics lens is entitlement-gated in
hanzoai/cloud (analytics.datastore); basic own-org usage is always readable.
Connected accounts API (/v1/ai/connections, IAM-authed, org-scoped). Curated
login-manager surface over the per-org object.Provider store
(controllers/connections_api.go). Keys are sealed via object.StoreProviderSecret
→ kms://… ref (fails closed if KMS is down); the raw key is NEVER stored in a row,
returned, or logged.
GET /v1/ai/connections→[{provider, connected, account_label, updated_at}](never the key)POST /v1/ai/connections{provider, apiKey}→ seals key, activates the org row; returns metadata onlyDELETE /v1/ai/connections/:provider→ deactivates the org row (resolution reverts to Hanzo credits)
Enforce "even if logged in". All AI traffic must reach this router; no surface
bills a third party off-meter. hanzo.app's workspace default was flipped
openrouter → hanzo; hanzo.chat routes via its gateway config; @hanzo/dev defaults
to api.hanzo.ai/v1 under Hanzo login. Claude/Opus for @hanzo/dev is served HERE
(Anthropic-native /v1/messages + OpenAI-compat /v1/chat/completions) against the
org's connected Anthropic account + 1% fee — no direct-to-provider bypass.
Next slice (route selection). Connections are stored/sealed/listed and the
1%-fee metering fires the instant the resolver returns an org-owned provider. But
the route table maps all models to providerName: "do-ai" (the aggregating
upstream), so a connection row named openai/anthropic/google is not yet
SELECTED for its model family. Making a connected consumer account actually serve
its models is a bounded route-table change (model-family → org-connection selection
- real upstream model-id/base-url mapping) — deliberately not in this slice.
RAG — ONE unified surface (retires the standalone chat-rag-api)
ai is the single RAG layer for the whole platform. There is exactly ONE
retrieval pipeline; the standalone hanzoai/chat-rag-api (danny-avila/rag_api
fork) is now redundant and slated for archive.
One index, one embedder, one retrieval path. Every document — doc/crawl/
github/s3 ingest AND per-file uploads from hanzo.chat — lands in the SAME
per-tenant index {owner}-{store}-docs, fanned out to BOTH Hanzo Search
(keyword, Meilisearch) and Hanzo Vector (semantic, Qdrant) via
object.IndexDocuments, and retrieved via object.SearchDocuments (hybrid RRF).
See object/search_docs.go.
File-scoped RAG is a FILTER, not a parallel store. file_id is a filterable
dimension on that one index (DocIndex.FileID; Meili filterable attr file_id;
Qdrant payload file_id). Uploaded-file retrieval = a file_id filter over the
tenant index. object/rag.go holds the file-scoped logic (RagEmbedFile,
RagQuery, DeleteRagFile, RagFileContext); chunking uses
split.RecursiveSplitProvider (parity with rag-api's
RecursiveCharacterTextSplitter, 1500/100), parsing uses
txt.GetParsedTextFromUrl (PDF/CSV/XLSX/PPTX/…). Uploaded files default to the
rag-files store so they don't pollute curated docs.
Two HTTP projections over the one logic (controllers/rag.go,
controllers/rag_librechat.go):
- Native (canonical):
POST /v1/rag/embed,/v1/rag/query,/v1/rag/query-multiple,/v1/rag/delete,GET /v1/rag/context. - LibreChat-compat (drop-in for the retired rag-api's fixed contract):
POST /v1/embed(multipart),/v1/query,/v1/query_multiple,DELETE /v1/documents,GET /v1/documents/:file_id/context. Returns the LangChain[[{page_content,metadata},score],…]tuple shape hanzo.chat parses.
Migration: point hanzo.chat's RAG_API_URL at https://api.hanzo.ai/v1 —
no chat-repo change. Auth/billing reuse the doc-RAG path
(resolveSearchAuth/requireIndexAuth + recordSearchUsage); owner is the
authenticated principal (tenant isolation), never a client-supplied field.
Search / Crawl / Web-search — THREE orthogonal products, ONE way each
Three distinct products, no overlap. Do NOT add a fourth crawl path.
-
POST /v1/search= native full-text index search (SearchDocs→object.SearchDocuments). Hanzo Search (Meilisearch, keyword,hanzoai/search-goatsearchHost:7700) + Hanzo Vector (Qdrant semantic,vectorHost:6333) fused hybrid-RRF over the per-tenant index{owner}-{store}-docs. Returns{hits:[…]}. Needs a populated index — an empty/new tenant index returns{hits:[]}(no error).GET /v1/search/statsreports index size. -
POST /v1/crawl= crawl a URL, get content back — THE single canonical crawl (controllers/crawl.goCrawl→object.Crawl→CrawlWithCrawl4AI). Backed by the self-hosted Crawl4AI (hanzoai/crawl,crawl.hanzo.svc:11235, Apache-2.0). Body{url}or{urls}(merged, deduped, capped at 10); returns{results:[{url,title,description,markdown,success,metadata}]}with LLM-ready markdown (prefers Crawl4AIfit_markdown, falls back toraw_markdown). Auth:requireIndexAuth(fail-closed, rejects read-only pk-/hz_ keys) + balance gate; usage recorded viarecordSearchUsage(…, "crawl", "crawl4ai", …).object/crawl4ai.gohandles the deployed Crawl4AI 0.8.6 wire shapes:markdownis an OBJECT (MarkdownFieldpolymorphic decode), the/crawlenvelope is synchronous with a booleansuccessand nostatus/task_id(return inlineResultswhenever present), andlinks/mediavalues are heterogeneous (map[string][]map[string]interface{}). -
POST /v1/scrape= crawl-and-INDEX (ingest) — NOT a general crawl. It crawls a site and WRITES structured content into the tenant search index, returningScrapeStats(ScrapeDocs→object.ScrapeAndIndex). Sibling ofPOST /v1/index(index caller-supplied docs). Stays because its output/effect (index write) is distinct from/v1/crawl(fetch + return). -
POST /v1/scrape/previewis a DEPRECATED alias of/v1/crawl— its handler now forwards toCrawl(one implementation, no parallel path). Use/v1/crawl. -
/v1/websearch/*(lives inhanzoai/cloud, not ai) = web meta-search — a separate product: SearXNG-compat/v1/websearch/search+ a firecrawl-compat/v1/websearch/v1/scrapeleg that hanzo.chat's web-search plugin calls. That leg is a fixed EXTERNAL contract (firecrawl{success,data:{markdown}}) and reuses the SAME Crawl4AI backend — it is an integration shim for the web-search flow, not a general crawl endpoint. The general "crawl a URL" is ONLY/v1/crawl.
Security invariants (READ before touching auth/billing/scaling)
-
JWT iss/aud — every request-auth JWT path goes through
object.ParseAndValidateJWT(signature via IAM and iss/aud policy), never rawiam.ParseJwtToken.object/jwt_validate.gosources the issuer fromJWT_ISSUER/IAM_ISSUER/CLOUD_IAM_ISSUER/AUTH_ISSUER(any, defaulthttps://hanzo.id) and the audience allowlist fromGATEWAY_ALLOWED_AUDIENCES(+ folds inIAM_AUDIENCE/AUTH_AUDIENCE), mirroring the gateway. A foreign-aud token → 401. The OAuth code-exchange callback (account.goSignin) is NOT a request-auth path and is intentionally exempt. -
Balance ledger — single-pod invariant —
object.GlobalBalanceLedgeris in-pod memory. Reserve/Settle are correct ONLY atreplicas: 1. Two pods double-spend (each reserves against its own cached Commerce balance). Enforced by deploystrategy: Recreate+ HPAmin=max=1AND a boot assertion (bootstrap.gopanics ifCLOUD_API_REPLICAS > 1). To scale out you MUST first move Reserve/Settle behind a Commerce-atomic conditional reserve. -
Single-request reservation (
controllers/billing_reserve.go) — an uncappedmax_tokensis clamped (clampMaxTokens) and the reservation covers the larger of the clamped ceiling and the QueryText pipeline's fixed completion cap (reserveCompletionTokens/reserveCompletionFloor=4096, mirrored inmodel/openai_util.go), so actual spend can never exceed the hold on any path. -
BALANCE_EXEMPT_USERS — matched by the ONE shared
object.BalanceExemptSet(gate + controller): anowner/nameentry exempts only that exact subject; a bareownerexempts the whole org.hanzo/zdoes NOT exempt all of org hanzo. -
Global admin —
util.IsGlobalAdmin=IsAdmin && owner ∈ globalAdminOrgs. Canonical defaultdefaultGlobalAdminOrgs = {admin, built-in}(matches IAM AdminOrg + console2). The liveglobalAdminOrgsenv override is authoritative; it is currentlyadmin,hanzo(grants the hanzo TENANT org platform power) — FLAGGED for alignment toadmin,built-inonce the hanzo-org provider-config workflow is confirmed to run via theadminorg. -
Authz gate (
routers/authz_filter.go) — platform-sensitive endpoints (globalAdminEndpoints) are gated FIRST (before preview-mode + exempt reads). There is deliberately NOadminDomainHost bypass (a removed full-authz-bypass primitive). No principal → 401; wrong principal → 403. -
User redaction (
object/redact.go) —RedactUserSecretsis an allowlist projection over string/[]string fields (exposableUserFields): a field not on the list is zeroed, so a NEW upstream secret field is fail-secure. Credential struct slices (MfaAccounts/ManagedAccounts/MfaItems/FaceIds) are nilled.
OPEN DEFECT — recorded COGS is retail, so margin is structurally 0
Not fixed. Do not "fix" it by editing the enso rates; the rates are correct.
Every hanzo.cloud_usage row for the Enso family (1,296 rows checked, ClickHouse
at datastore.hanzo.svc) has billed_nano == cost_nano and margin_nano == 0.
The recorded cost is RETAIL, not what the upstream actually cost us.
The mechanism is a deliberate fallback used outside the case it was written for:
controllers/model_pricing.go:43-57—costInputPerMillion()/costOutputPerMillion()returnCostInPerMillion/CostOutPerMillionwhen set, else the customer price.controllers/billing_nano.go:75-79—tokenProviderCostNanocomputes COGS at exactly those rates, so a model with no registered COGS reports cost == billed.controllers/billing_nano.go:124-135—providerCostNanofeeds that intocostMargin, which the ledger, thecloud_usagerow, and the o11y span all read.
For Enso no COGS rate is registered, so the fallback fires on every call. The
real COGS is known and lives in the Enso catalog: 1.392 / 2.784 $/MTok on the
deepseek-v4-pro anchor (enso and enso-ultra) and 0.112 / 0.224 on
deepseek-4-flash. Against billed 4/20 the true margin is roughly 2.9x in /
7.2x out — recorded as zero.
A channel to carry the exact figure already exists and is simply not populated:
controllers/trace_export.go:40-41 takes BilledNano / CostNano with
0 = recompute from rates, surfaced as CostNanoExact in
controllers/openai_api.go:557-558. The serving side knows its per-arm cost —
for enso-ultra the fan-out means COGS is the real sum of the arms that ran, which
only the server can total — so pushing the exact value is the right seam. Falling
back to rate math cannot express a fan-out cost at all.
Consequences, in order of severity:
- The admin cockpit's below-cost check cannot work for these models. It compares price against a "cost" that IS the price, so the difference is always exactly zero: it can neither fire on a genuine below-cost sale nor stay quiet honestly. This is the likely source of its misfiring.
- Margin reporting reads 0 for the whole family, so gross margin is understated
wherever it aggregates
margin_nano. - The zero is indistinguishable from a real zero-margin sale (BYO calls
legitimately record
CostNano = 0), so nothing downstream can tell "we did not compute this" from "this genuinely earned nothing".
The fallback is defensible as a default — a new model with no COGS should not report imaginary profit — but it is silent, and silence is what let it stand. Whatever the fix, an unregistered COGS should be distinguishable from a measured one rather than quietly equal to price.