Files
Zach Kelling 098f89edfc billing: retire the Circle Gateway top-up rail
The owner's call, 2026-10-30: USDC through Circle Gateway is Circle's
own SDK (@circle-fin/x402-batching) in Hanzo's x402 gateway in front of
cloud, a deposit into Gateway once and a payment per request. The Go
rail reimplemented that protocol, so it goes now rather than later:
billing/gateway, payment/providers/circle, models/remittance, the three
/v1/billing/crypto/gateway routes, the commerce treasury route, and the
gateway networks in /v1/billing/crypto/options. Crypto top-ups by
authorization pause until the gateway takes them; the deposit-address
rail is unchanged. Hashid kinds 292 and 293 stay taken.

Testing: go build ./... and vet clean; api/billing's crypto tests pass
(TestPlanAtInterval and TestSubscribeTrial_NoChargeCardVaultedTrialing
fail on main without this change too).
Risk: removes a live payment route; cloud must drop its plane ops in
the release that pins this.
How to revert: revert this commit.
2026-10-30 07:44:06 -07:00

303 KiB
Raw Permalink Blame History

Commerce - LLM Context

Overview

Multi-tenant e-commerce platform. Standalone Go binary with embedded SQLite, migrated from Google App Engine. Zero GAE dependencies remain.

Live at: https://commerce.hanzo.ai | Version: v1.33.0

Architecture

Commerce App (Cobra CLI + Gin HTTP + Hooks + Events)
  |
  +-- User SQLite (data/users/{userID}/data.db) + sqlite-vec
  +-- Org SQLite (data/orgs/{orgID}/data.db) + sqlite-vec
  +-- PostgreSQL (alternative, pgvector)
  +-- Datastore via hanzo/datastore-go (analytics)

Commerce Admin

There is ONE Commerce admin: the Next static export in app/admin (@hanzo/commerce-dashboard), embedded in the Go binary (ui/dist, //go:embed) and served by it at /admin. It renders @hanzo/ui/product — the same component set the cloud console renders (DataTable, PageHeader, StatusTag, EmptyState, MetricCard, BackendStateCard, CommerceResource) — on @hanzo/gui. There is no commerce-local design system for the admin, and no second shell.

One artifact, one URL contract. next.config.ts sets basePath: '/admin' (from src/lib/basepath.ts), so chunks are /admin/_next/* and the client router resolves routes under the same prefix. A basePath-less export is built for / and CANNOT be served from a sub-path: its root-absolute /_next/* escapes the mount and its router 404s on its own pathname. That is why the separate commerce-admin static-container image is gone — it served the same bundle at a second origin's ROOT, i.e. one artifact with two incompatible URL contracts, and keeping it is what let the embedded copy rot.

The other three spellings are gone: the Vite SPA at frontend/ (deleted — its @hanzogui/admin file: target no longer exists), hanzoai/admin apps/admin-commerce (retired), and the dead Medusa React-Router tree under app/admin/src/routes (deleted along with the vendored Tailwind surface).

  • The catalog is data. src/lib/resources.ts is a plain list of rows — slug, backend noun, copy, column specs. src/lib/columns.tsx turns a column spec into a rendered column; src/app/(dashboard)/[resource]/page.tsx emits one static page per row via generateStaticParams, and the sidebar reads the same list. Adding a merchant surface is adding a ROW, never a directory. The module is deliberately component-free: generateStaticParams runs it in Node at build time.
  • One client. src/lib/commerce.ts calls the bare /v1/<kind> resources (/v1/product, /v1/order, /v1/c/user, /v1/collection, /v1/stocklocation, /v1/store/current) on NEXT_PUBLIC_COMMERCE_API_URL with the IAM bearer and X-Org-Id. Failures THROW with their status, so classifyBackend renders the honest state (402 → add credits, 403 → not enabled, 404 → not mounted) instead of a fabricated empty table.
  • Effects are injected. @hanzo/ui/product never imports a router or an auth module; src/app/providers.tsx supplies signIn/addCredits once through HostProvider.
  • IAM is native PKCE through @hanzo/iam; the active organization scopes every read and is the sole tenant selector.
  • @hanzo/ui comes from the registry at the range the console pins (^8.0.12) so the admin and the console cannot drift, and so a developer's build and an image build resolve the identical package. It was a link: to a sibling checkout that every image rewrote to the registry range before installing — an image built what nobody had built locally, which is how a missing @hanzogui/next-theme (imported by @hanzo/ui/product's ThemeToggle, declared by @hanzo/ui only as a devDependency) reached a release and failed next build there and only there.
  • @hanzo/ui declares none of the singletons it imports, so webpack must be told. Each consumer installs its own @hanzo/gui / @hanzogui/* and resolution is free to pick a transitive copy. The bundle then holds two @hanzogui/webs — two ThemeStateContexts — GuiProvider publishes to one, every shared component reads the other, and the admin dies on load with Missing theme. tsconfig paths already pinned this for the TYPE layer; next.config.ts's SINGLETONS alias now pins the RUNTIME layer to match. Symptom of a regression here: a blank admin with "Application error: a client-side exception".
  • The build is a real gate: turbo run typecheck build --filter=@hanzo/commerce-dashboard. next.config.ts no longer sets ignoreBuildErrors, and hanzo.yml runs the typecheck in CI.
  • @hanzo/commerce-ui (app/packages/ui, the vendored Medusa design system) is now a STOREFRONT-only dependency — store/ still renders it. The admin is fully off it.
  • The embedded copy in the Go binary is the SAME export: scripts/sync-admin-ui.sh copies app/admin/out into ui/dist for //go:embed, served at the /admin/* mount. ui/dist is BUILD OUTPUT — gitignored but for .gitkeep, and that sync is its ONE producer. It was tracked content until it shipped a retired Vite SPA: a committed bundle is a bundle nobody rebuilds. Run the producer before any go build, which is exactly what Dockerfile does.
  • A sub-path mount is not a constraint on the export, it is a PARAMETER of it: an export can be served from /admin precisely because it was BUILT for /admin (basePath). What cannot be served there is an export built for / — its root-absolute /_next/* escapes the mount. TestEmbedAdminServesItsOwnAssets asserts this, on content type: the static fallback answers every miss with index.html and a 200, so status alone cannot see the break.
  • Dockerfile.store builds the storefront and its workspace packages through the root pnpm/Turbo graph, then publishes the standalone server. Do not copy a partial package set or introduce a second package manager.

Three credentials, three distinct jobs — do NOT collapse them

An audit will read commerce as having "three competing auth systems". It has one identity system and two machine credentials, and they are different VALUES, not three ways to spell one:

  • IAM / OIDC — human identity. The only source of user identity. A validated principal is proven by X-User-Id, which is minted solely from a verified credential and never restored from client input.
  • COMMERCE_SERVICE_TOKEN — ONE global platform machine identity (the cloud→commerce S2S dispatch). Checked FIRST in TokenRequired, ahead of the IAM branch, so an S2S call never meets the per-user scope gate.
  • Published storefront token — a PER-ORG machine credential, minted by an org admin via POST /v1/store/token (org.AddToken("storefront", permission.Published)) and consumed as HANZO_COMMERCE_STOREFRONT_TOKEN by the storefront BFF. Org-bound, scoped, revocable by re-minting.

That third one rides the legacy per-org access-token path in middleware/accesstoken.go — GetWithAccessToken → tok.Verify(org.SecretKey) → tok.Permissions → org.Live. It LOOKS like dead duplication and is not: delete it and every storefront's checkout stops authenticating. IAM does not currently issue per-org, org-admin-mintable, revocable scoped keys, so folding this into IAM is a migration that needs that capability first — not a cleanup.

Multi-Tenancy

  • Namespace-based: Organization.Name IS the namespace
  • middleware.Namespace() sets context namespace for downstream datastore
  • rest.New() auto-applies namespace middleware unless DefaultNamespace = true
  • Dual auth: legacy access token (org-bound) + IAM JWT (OIDC/JWKS via hanzo.id)
  • Every org (incl. "platform") is STRICTLY scoped to its own name. The legacy "platform" -> "" (cross-org) namespace bypass was REMOVED (1.42.40, Red M1): it keyed cross-org datastore on an org-NAME string, not real platform-admin identity. Cross-org access gates on auth.IAMClaims.IsSuperAdmin() ONLY.

Gateway Trust Headers (2026-04-27)

commerced does NOT validate JWTs in-binary. The trust boundary is hanzoai/gateway. Only the gateway-minted X-* identity headers are trusted. Headers are stripped unconditionally on ingress to prevent client spoofing (see gateway/auth_middleware.go stripIdentityHeaders).

Directly-exposed edge (COMMERCE_EDGE_AUTH=true) — boundary mount point: commerce-api is reachable in-cluster at commerce.hanzo.svc:8001 WITHOUT the gateway, so it strips + re-mints identity itself via middleware.EdgeAuth. That boundary MUST be installed before any route group: gin applies engine.Use() only to routes registered AFTER the call. It is now installed in Bootstrap (server.go installIdentityBoundary, called before setupRoutes), so it covers /_/commerce/*, /v1/commerce/* AND the post-Bootstrap /v1 api.Route() bundle. The boundary NEVER 401s opaque service tokens (not JWTs), so the cloud-api → commerce per-org billing money path is untouched (require=false; COMMERCED_REQUIRE_IDENTITY is incompatible with the service-token path). Every identity header is stripped UNCONDITIONALLY. A client's X-Org-Id survives only as a SELECTION that tenancy grants or refuses against a verified JWT's signed membership set (below); an opaque bearer selects nothing. Regression: middleware/edgeauth_test.go, middleware/edgeauth_tenancy_test.go.

Header Written by the edge from Use
X-Org-Id the org the request acts in Org slug; namespace + scope
X-User-Id JWT sub User identity
X-User-Email JWT email Email; audit + notifications
X-User-IsAdmin authz.Claims.Sudo "true" iff a SuperAdmin — the ONE verdict
X-User-IsOrgAdmin authz.Claims.OrgAdmin(acting org) "true" iff admin of the org acted in
X-User-Owner authz.Claims.Home (orgs[0]) HOME org — identity and billing anchor
X-User-Permissions edge-derived bit.Field as base-10 int; 0 fails closed

The edge in front of commerce — cloud's SanitizeIdentity in production, authz/edge behind a gateway, middleware.EdgeAuth at a standalone one — strips every one of these from the request and writes them back from a verified token alone. X-User-IsGlobalAdmin and X-Roles are RETIRED and still stripped.

SuperAdmin is the edge's verdict, read in one place. auth.IAMClaims.Sudo is X-User-IsAdmin == "true" (iammiddleware.GetIAMClaims), and IAMClaims.IsSuperAdmin() returns it and nothing else. No org comparison happens in commerce. It used to: IsSuperAdmin was strings.EqualFold(homeOrg(), "admin"), with homeOrg falling back to the owner claim (the MINTING APPLICATION's org), and the money bits (Admin|Live) were granted on X-User-Owner == "admin". Cloud writes X-User-Owner for an application (its own org) and an API key (its holder's org) as well as a person, so an admin-org application or an admin-org person's sk- key could mint money and edit the catalog here while being nobody in the rest of the same process, and an org spelled Admin would have passed the fold (IAM refuses to store one: schema.OrgName). No machine is SuperAdmin (owner decision D1, until answered: no). middleware.PlatformOnly, IsPlatform, MayMintMoney, MayReadPlatform and every catalog/plan/rate/promo/invite gate read the verdict. The standalone EdgeAuth decides it with the one predicate, IAMClaims.Authz().Sudo() over the verified token's signed orgs/type, and writes X-User-Owner from Home().

Org-admin vs SuperAdmin (anti-conflation): IAMClaims.IsAdmin is the org-level admin of the org the request acts in (X-User-IsOrgAdmin), for org-scoped RBAC only. middleware.RequireAdmin is the LOOSER org-money gate — IsAdmin || IsSuperAdmin() — and is never the cross-tenant one. Never gate platform work on IsAdmin or on an org-mintable role NAME like "superadmin". Tests: auth/superadmin_test.go, auth/sudo_test.go, middleware/iammiddleware/sudo_test.go, middleware/platformonly_test.go, middleware/edgeauth_test.go.

EdgeAuth tenancy (middleware/edgeauth.go)

At the standalone edge (COMMERCE_EDGE_AUTH=true) tenancy resolves a verified token to two orgs by the rule cloud's SanitizeIdentity applies, so one token is one tenant whichever edge commerce sits behind:

  • X-User-Owner, the HOME org: a person's orgs[0] (authz.Claims.Home).
  • X-Org-Id, the org the request ACTS in: the client's selection (X-Org-Id, or ?org=<slug> on a /billing/ path, consumed either way) when the signed set admits it, else the default (authz.Claims.EffectiveOrg). A SuperAdmin may select any org; anyone else an org the set opens, never the admin org, a brand's directory, or a platform org held as a plain member.
  • A program (type: application) is its application: both are the owner claim, and it selects nothing. A person's token without orgs acts in no org.

The owner claim of a person's token is the org of the application they signed in through (the brand app every customer uses), and decides nothing: it used to set X-Org-Id and the billing subject, so every self-serve user signing in through the brand app acted in and billed to the brand's org. /billing/ requests pin user/userId/customerId (query and JSON write body) to the acting org. Tests: middleware/edgeauth_tenancy_test.go.

Call sites read identity via:

  • pkg/auth.OrgID(ctx) / UserID(ctx) / UserEmail(ctx) (preferred)
  • iammiddleware.GetIAMClaims(c) returns a real *auth.IAMClaims populated from headers (Owner=X-Org-Id, Subject=X-User-Id, Name=X-User-Id, Email=X-User-Email, IsAdmin=X-User-IsAdmin=="true", Roles split from X-Roles). Never nil; do not add nil-guards downstream.

Webhooks — the platform's, not commerce's

An org's webhooks are cloud's /v1/webhook (apps/webhook: HMAC-signed, retried, every platform event). Commerce's own per-org webhooks are being retired:

  • No route creates or edits a row (the /v1/commerce/webhook CRUD is gone).
  • The legacy sender (models/webhook/tasks) posts {"data": ...} and nothing else — the hook's AccessToken used to ride in every body — through hanzoai/outbound, so a subscriber URL is never a private address.
  • What it reaches, measured (TestEmitReadsOnlyTheSystemStore): rows in the SYSTEM store, which are sent every org's events; never a row in an org's own store, where the CRUD wrote every row (it set a namespace on a datastore whose queries ignore one). So CRUD-made rows have received nothing.
  • models/webhook/census reads both stores without creating anything, and Embed logs it once per boot: one commerce: legacy webhook subscriber line per row (store, org, id, host, a digest of the URL, events, enabled, live — never a credential) and a commerce: legacy webhook census total. Read it with kubectl logs before the sender, the model and the census are deleted.

Key Directories

commerce/
  cmd/commerce/    CLI entry point
  commerce.go      Main app framework
  db/              SQLite, Postgres, Datastore backends
  hooks/           Hook system (Base-compatible): Hook[T], TaggedHook[T], Resolver
  events/          Unified event forwarding to Datastore/Insights/Analytics
  insights/        Hanzo Insights integration + Gin middleware
  api/             HTTP handlers (store, cart, analytics, namespace, etc.)
  models/          Data models
  middleware/      HTTP middleware (auth, namespace, IAM)
  infra/           Infrastructure clients (Redis, Meilisearch, etc.)

Running

go run cmd/commerce/main.go serve --dev     # Development
./commerce serve 0.0.0.0:80                 # Production

Environment Variables

Variable Default Notes
COMMERCE_DIR ./commerce_data Data directory
COMMERCE_SECRET change-me-in-production Encryption secret
COMMERCE_HTTP 127.0.0.1:8090 Listen address
REDIS_URL - redis://[:pass@]host:port[/db] (priority over VALKEY_URL)
DATASTORE_URL - Datastore DSN
INSIGHTS_ENABLED false Hanzo Insights product analytics
ANALYTICS_ENABLED false Umami-like web analytics
KMS_URL - Standalone only: Hanzo KMS server; unset = no secret reader
KMS_ORG - Standalone only: the org the KMS identity belongs to (required with KMS_URL)
KMS_CLIENT_ID / KMS_CLIENT_SECRET - Standalone only: the KMS machine identity, exchanged at HANZO_IAM_URL
SQUARE_WEBHOOK_SIGNATURE_KEY / SQUARE_WEBHOOK_URL - The deployment's Square webhook subscription key and its exact notification URL

Analytics Endpoints

Endpoint Method Purpose
/v1/analytics/event POST Single event
/v1/analytics/events POST Batch events
/v1/analytics/identify POST User identification
/v1/analytics/ast POST astley.js page AST (JSON-LD)
/v1/analytics/pixel.gif GET Pixel tracking
/v1/analytics/ai/message POST AI message event
/v1/analytics/ai/completion POST AI completion event

SaaS Metrics God-View (api/metrics, 2026-07)

GET /v1/commerce/metrics/saas — the whole-business SaaS operations snapshot for admin.hanzo.ai's "SaaS Metrics" board. ONE cross-org walk (proven all-orgs pattern from api/billing.RunBillingCycleAllOrgs), cached 45s, folding every org's subscription ledger (models/subscription) + api-usage transaction ledger into: revenue (MRR/ARR, new/churned MRR, MRR by plan category from the embedded @hanzo/plans catalog), subscription mix (per-plan active/trialing/seats + a recent create/cancel events feed), metered pay-as-you-go revenue totals, and top customers. Params: window (7d/30d/90d/mtd/all), limit (customers cap). Bundle-child subs (ProviderType "bundle") are excluded from counts/MRR. Upgrades/downgrades are null (plan-change events are not written to models/billingevent).

Gate: middleware.RequirePlatformAdmin — the SINGLE cross-org read gate (global admin OR trusted service token; org-admin refused). api/costs.requireCostsAdmin delegates to it, so the two god-views share ONE gate definition. The route-level TokenRequired(permission.Admin) is a no-op on the IAM path — the in-handler gate is the boundary. The console reaches it through its OWN SuperAdmin-gated proxy (app/admin/saas) forwarding COMMERCE_SERVICE_TOKEN.

Deliberately NOT here (owned by the fleet o11y god-view GET /v1/admin/o11y): per-model LLM tokens/latency/error and the per-org AI-spend ranking. Commerce names those in the response gaps[] array, never fabricates. Per-model latency is captured nowhere in the stack; per-model error rate is in Datastore cloud_usage.status but not yet surfaced by the o11y topModels SQL.

Dependencies

Core: cobra, go-sqlite3, gin, hanzoai/datastore-go Infra: go-redis/v9, hanzos3/go (the house S3 client — package minio, republishes storage-go under the s3-go import path), meilisearch-go, nats.go, temporal SDK Vector: Qdrant via REST/HTTP (port 6333). No gRPC, no vector-go SDK -- plain net/http + encoding/json.

Secrets: the host's KMS, one reader (secrets)

Commerce holds no KMS client. It reads through secrets.Reader, which the host installs at embed time (EmbedConfig.Secrets): the cloud binary passes its own KMS handle (deps.KMS, KMSPeer over the internal plane), and the standalone binary (cmd/commerce) passes hanzoai/kms/sdk/go as its machine identity (KMS_URL, KMS_ORG, KMS_CLIENT_ID/SECRET, HANZO_IAM_URL). thirdparty/kms is deleted: it read /v1/kms/orgs/<org>/secrets/..., which cloud stopped serving, so per-org payment credentials never loaded — and it was off in production anyway (KMS_ENABLED unset).

One address. An org's own material is orgs/<org>/<provider>/<NAME> in the default env — cloud's OrgPath and the convention every in-process reader uses. secrets.Org(ctx, org, name) is the only way to read it, and it asks AS org: cloud's KMS (apps/kms admit) serves a tenant ref only to a call acting for that tenant, and a request's caller is the PAYER when a customer pays a brand, so the read re-points that caller with zip.ActingAs (the call names who is acting; the KMS writes no audit row of it). A raw handler passes c.Forward(), which carries the request's caller; its c.Context() carries none. A call with no caller reads nothing — cloud's KMS answers an unstated call as its own brand. org must already be a namespace.Sanitize-clean slug. The ref carries no leading slash, so the tenancy check runs. /tenants/<org>/<provider> was the old client's layout and nothing reads it now.

Who loads which credentials. payment.Hydrate(ctx, org, providers...) fills an org's integration fields from those refs. It is called where the org RECEIVES the money: storefront checkout (api/checkout), storefront subscriptions (api/subscription), storefront wire instructions, and the billing wire rail, where the receiving org is the brand that owns the host (orgs/hanzo/wire/* for pay.hanzo.ai). Hanzo's own billing (api/billing top-ups, plans, invoices, renewals, auto-recharge, saved cards) never hydrates the PAYING org: its Square account is not where that money goes. That path charges the deployment's own account through ProcessorsForOrg's environment fallback (SQUARE_*, KMS /commerce-secrets → KMSSecret → env). Moving it onto the brand org's own credential is an open owner decision.

Nothing is cached. The store is the process next door; a rotation lands on the next read. commerce seed no longer copies env credentials into KMS — a second copy of a secret is a second place for it to go stale.

Square webhooks: one route, one verifier

  • Route: POST /v1/commerce/webhooks/:provider, declared once in setupRoutes on the public /v1/commerce group, so standalone and embedded serve the same address from one declaration (cloud no longer declares it). /v1/billing/webhooks is gone. The URL is part of Square's signature: it moves only with the Square subscription (payment/orgsetup.go defaultSquareWebhookURL).
  • Verifier: payment/providers/square ValidateWebhook, reached through processor.Global(). thirdparty/square is the API client only — no HMAC, no init registration, not a PaymentProcessor — so no second Square type can be registered. Every registry holds *square.Provider (ProcessorsForOrg too).
  • The path names the processor, and it must hold its secret. Only the processor registered under :provider verifies a delivery, and only if it is a processor.WebhookReceiver that ReceivesWebhooks() — it holds the secret its deliveries are signed with (Square: key and URL; MPC: MPC_WEBHOOK_SECRET). Every provider package registers itself configured or not, and some verifiers need no secret (Solana Pay's chain lookup), so asking one of them would let a caller author an event; an unknown name or a secretless processor is refused. The receiver set is pinned to {mpc, square} by TestTheWebhookReceiversAreTheSecretHolders. payment/router, a second dispatcher that tried every processor, is deleted.
  • A clawback stays in the verifying account's org. When the event names an org (a per-org account) and the payment's receipt is another org's, the refund or dispute moves nothing (RECONCILE).
  • No freshness window; the event id is the replay guard. Square keeps an event's created_at across every retry, so the old 5-minute window refused every retry of an event first sent during a deploy, forever. Each verified event is recorded once for the whole deployment — idempotencykey in the system namespace, scope webhook, key <processor>:<event id>, an insert-if-absent swap — so a repeat is acked duplicate:true (or 409 while the first is still being applied) whatever path or org it is presented under. Every money posting is also keyed on the payment, refund or dispute id. A signed body with no event_id is refused.
  • An effect that did not land leaves the delivery open. The settlement, clawback and subscription steps return an error when a store or the ledger did not answer; the handler then answers 503 without completing the guard, so the provider redelivers and the guard is re-claimed after StartedTTL. Every effect is safe to apply twice. Unattributable money, and a receipt naming an invoice these books do not hold, are not errors: RECONCILE, refused, acknowledged. The org's billing_event audit row is written once, by the delivery that applied the event.
  • The books are the verifying account's. bookedOrg reads processor.WebhookEvent.Org (the org the Square account belongs to; the deployment's env-seeded account names none) and falls back to checkout.DefaultBrandSlug(). No header or session picks the org: a provider sends no identity, and one on a replay is the replayer's.
  • Accounted for, wherever: settlementDestination reads the payment index (models/paymentorg) first. An in-session charge writes its receipt in the PAYER's org and indexes it; the delivery resolves the processor account's org, so without the index every such payment raised a false RECONCILE.
  • Production (2026-10-19): one Square subscription, wbhk_44c9077bfece4c0c96a463ac9fe56e0f, created 2026-09-23, notification URL https://pay.hanzo.ai/v1/commerce/webhooks/square; its key is KMS org hanzo /commerce-secrets/SQUARE_WEBHOOK_SIGNATURE_KEY@prod (measured equal to Square's). Square's Events API holds nothing to replay: event search was never enabled on the account.

Checkout Sessions

POST /v1/checkout/sessions — AUTHENTICATED hosted-checkout mint (Square Payment Link). Same publishedRequired gate as every sibling checkout route (TokenRequired(Admin, Published)): a service token, a per-org Published storefront token, or an IAM principal. There is no anonymous path — an unauthenticated POST is 401 BEFORE any org resolution or Square call.

An opaque (non-JWT) bearer never selects an org. EdgeAuth (middleware/edgeauth.go) strips a client X-Org-Id and puts nothing back. TokenRequired's IAM branch ENFORCES the requested masks (len(masks)==0 || hasScope(c, permissions)), and IAMTokenRequired requires a validated principal (X-User-Id), not merely an X-Org-Id selector — the same predicate as cloud's clients/principal.Validated, needing no env flag. Regression: middleware/edgeauth_test.go, middleware/accesstoken_test.go, middleware/iammiddleware/authz_test.go.

Request: { company, providerHint, currency, customer, items, successUrl, cancelUrl, couponCode? } Response: { checkoutUrl, sessionId }

Security invariants (do not regress):

  • One org source: the org is SOLELY the authenticated principal's (authedOrg(c) = middleware.GetOrganization + KMS hydrate). There is NO request-body org/tenant field and NO ?org/X-Org-Id/X-IAM-Org override on the mint path — a caller can only ever mint for its OWN tenant.
  • Fail-closed creds: an org without its own per-tenant KMS Square creds cannot mint live — squareCheckoutClientForOrg has NO production env/platform fallback (isPlatformOrg removed). Only SANDBOX keeps the env fallback (no money at risk), behind auth, so the per-org sandbox storefront still mints.
  • Redirect allowlist: successUrl/cancelUrl are bounded to the org's own domains via checkout.AllowedCheckoutRedirect (org Websites + brand first-party hosts + brand registrable domains) — closes the open-redirect / phishing pivot on the minted link. Enforced for authed callers too.
  • Item prices are server-authoritative from the org's per-org catalog (catalogPrice); the client unitPrice is ignored. Emits checkout.started.

The legitimate storefront BFF (server-side, holds the service token or the per-org Published storefront token — mint via POST /v1/store/token) is the ONE reachable checkout entry. A browser must never call this endpoint directly with a client-chosen org.

SQLite Query Engine (2026-02-23)

Critical: PascalCase→camelCase conversion. The SQLite query builder converts Go struct field names (PascalCase) to JSON field names (camelCase) automatically via toJSONFieldName(). This is needed because json.Marshal() uses json struct tags (camelCase), but callers use Go field names in Filter()/Order() calls (legacy from Cloud Datastore migration).

  • Filter("Test=", false) → json_extract(data, '$.test') = 0 (not $.Test)
  • Filter("DestinationKind=", kind) → json_extract(data, '$.destinationKind')
  • Nested paths handled: Filter("Account.TransactionHash=", h) → $.account.transactionHash
  • Boolean false/0 handled via COALESCE: COALESCE(json_extract(...), 0) = 0 (handles NULL from omitempty)

Data directory: {COMMERCE_DIR}/orgs/{orgName}/data.db per org. System data in orgs/system/data.db.

Tenant store lifecycle lives in orm/db.Registry, not here. db.Manager used to hold a userDBs and an orgDBs map, opening a store per tenant ever touched and closing them only at shutdown — descriptors and memory grew with the tenant count, with no bound and no replication. It now delegates to ormdb.Registry[db.DB]: open on demand, bounded by Config.MaxOpenTenants (256), idle stores closed after Config.TenantIdleTTL (5m), coldest evicted first, never one that is mid-operation.

Registry.Do(ctx, tenant, fn) scopes a handle to a callback, on purpose — a handle you can keep is one someone must remember to give back. Manager.User/ Org still return a db.DB because every call site keeps one (Store.DB, the namespaced datastore resolver, the user service), but what they return holds no handle: db/tenant.go is a VALUE that borrows the tenant's store for each operation and gives it back. So a held db.DB costs four words, never a file descriptor, and two consecutive operations on it may run on different open handles — nothing needs otherwise, since anything atomic across statements is already inside RunInTransaction (one borrow).

Registry's Materialize (restore a tenant file from object storage on a local miss) and OnOpen/OnClose (bracket a handle's life) are where hanzoai/replicate attaches. Both are unset today: commerce tenant files are still local-only, and turning them on needs an object-store config, not more code here.

PVC: commerce-data (10Gi, do-block-storage) — deployment uses Recreate strategy (not RollingUpdate) because the PVC is ReadWriteOnce.

At-Rest Encryption (SQLCipher via hanzoai/cek)

Per-tenant SQLite files hold money data (balances, transactions, usage), so they can be encrypted at rest with github.com/hanzoai/cek on top of github.com/hanzoai/sqlite v0.5.0 (SQLCipher AES-256) — the SAME derived-key scheme cloud and IAM use, so there is one at-rest model platform-wide. Code: db/encryption.go.

  • Master key from KMS, ONE source: COMMERCE_KMS_MASTER_KEY (64 hex), read ONLY by ResolveMasterKey(). Injected from KMS into the environment, the same mechanism as HUSD_TREASURY_KEY. Never in git/code.
  • Derived, not wrapped: a tenant's file key is cek.DeriveKey(master, ns, "commerce") — HKDF over the master, the tenant's hanzoai/namespace name and the subsystem. It is a pure function of the tenant's name, so there is NO DEK, no sidecar, no unwrap step, no rewrap and no per-file key material to lose. A file reopens after a restart with nothing persisted beside it. Losing the master loses the data, which is the property at-rest encryption is for.
  • Why DeriveKey and not cek.Open: commerce opens a DUAL pool — a concurrent read pool and a serialized single-connection writer — against ONE file under ONE key. cek.Open hands back a single *sql.DB. The key is the shared part; the pool is commerce's.
  • Posture decided once: unset key → unencrypted (dev/CI). Set + codec linked → encrypted. Set + cgo-but-no-libsqlcipher → refuse (CodecLinked() probe), never silent plaintext — commerce's dual pool cannot use the single-writer codec envelope a non-libsqlcipher build falls back to.
  • Layout: an ORG's store is <DataDir>/orgs/<slug>/commerce.db, placed by namespace.Path, so the file's key and the file's path are two renderings of one name. This is the layout every Hanzo service shares.
  • Per-USER stores are UNENCRYPTED, and this is a real limit: namespace.Key refuses KindUser — the shared layout has no place for a user — so a per-user store cannot be named, and therefore cannot be keyed, in this scheme. An encrypted open of a user tenant fails closed with an error naming the limitation (TestEncryptedUserTenantIsRefused); the unencrypted path is unaffected. Per-user files stay at <UserDataDir>/<id>/data.db. Giving hanzoai/namespace a user layout is the prerequisite for closing this.

No migration. Databases written under the previous DEK-sidecar scheme do not open under this derivation, by design — there is no fallback, dual-read, version probe or config flag, because a second way to key a file is how two of them end up disagreeing. Wipe and recreate.

Rollout: build the image with libsqlcipher linked — add libsqlite3 to the build tags, CGO_CFLAGS=-DSQLITE_HAS_CODEC -DSQLITE_USE_URI=1 -I/usr/include/sqlcipher, CGO_LDFLAGS=-lsqlcipher, apk add sqlcipher-dev (builder) + sqlcipher-libs (runtime), and a test stage with SQLITE_REQUIRE_CODEC=1 so a mis-link fails CI — then supply a 32-byte key.

Where the key comes from: an EMBEDDER passes EmbedConfig.MasterKey (cloud hands over the KEK it already resolves, so one process has one key, not two); a STANDALONE reads COMMERCE_KMS_MASTER_KEY. One source per deployment shape, never both.

Billing API (2026-02-23)

Endpoint Method Purpose
/v1/billing/balance GET Balance by user+currency (cents)
/v1/billing/balance/all GET All currency balances
/v1/billing/usage POST Record API usage (withdraw)
/v1/billing/deposit POST Create deposit transaction (per-subject money-in / settlement)
/v1/billing/credit POST THE ONE mint-gated, org-keyed credit grant (see below)
/v1/billing/zap POST Clear balance

All require permission.Admin token (org live/test JWT). Cloud-api connects via commerceEndpoint + commerceToken env vars.

The invoice PDF (api/billing/invoice_pdf.go, pdf.go)

InvoicePDF(ctx, org, id, brand) renders one invoice; brand is the slug of the brand that bills (hanzo, lux, zoo, pars; anything else is hanzo). Cloud passes the caller's brand; the standalone route reads it from the request host (checkout.BrandSlugForHost). The page is US Letter in Zen Regular and SemiBold (embedded TrueType subsets in fonts/, OFL, credited in NOTICE) and two inks: the brand's mark (vector paths from @hanzo/logo, @hanzo/brand and @zooai/logo) and name top left, the number and a status pill top right (PAID is the one solid pill), billed to / issued / due or paid / period, the amount in words, a ruled line-item table, totals right-aligned, and the brand's legal name and address at the foot (Hanzo Industries Inc., 995 Market St, San Francisco, CA 94103; Lux and Zoo state none and close with their name and site). Long invoices continue onto numbered pages. Text is Windows-1252 (accents kept, other scripts read "?") and uncompressed, so the number and amounts stay greppable; the output is a pure function of the row. Tests: invoice_pdf_test.go.

One way to price a plan — the published catalog, applied by a DEPLOY

The plan catalog is @hanzo/plans (npm, public). scripts/fetch-plans.sh vendors it into api/billing/plans/*.json at build time, go:embed bakes it in, and plan.Seed reconciles the live models/plan rows to it on EVERY boot. So changing a price is: edit the package → publish → re-vendor + bump PinnedPlansVersion → bump commerce → bump cloud's go.mod → deploy. Every step versioned, reviewed and revertible.

There is no second way. cloud/scripts/seed-plans.sh (a human curling production with a SuperAdmin bearer, with the ladder retyped in bash) is DELETED — it was a second statement of the ladder kept in step by hand, which is exactly how the pricing page came to publish Pro at $49 while billing charged $20.

The boot seed CAN change what a plan charges. An older comment claimed the opposite ("seed values EQUAL the embed, so it changes NO charge"); that stopped being true when the seed gained the ability to correct its own prior output.

Why the seed could not correct itself (and what fixed it)

Managed was set by BOTH plan.Seed and api/plan's Create/Update, so one bit carried two facts — "authoritative" and "who decided it" — and if existing.Managed { continue } skipped every row the seed had ever written. Publishing a new catalog would therefore CREATE the plans that were missing and LEAVE every plan that had changed at its old price: half-new, half-stale.

AdminEdited is the discriminator that was missing. ONLY api/plan sets it. Seed reconciles anything else to the catalog — the rule everyone already assumed held: the package decides, an admin edit overrides. Proven end to end by api/billing/plans_converge_test.go, which plants the real production catalog (Managed rows, retired tiers, old prices) and boots it against the published one.

A draft catalog row is created unlisted (plans 1.8.17)

The boot seed makes a catalog price chargeable the moment a deploy carries it, so the catalog can publish a rung as "status": "draft" (max-50x $500 and max-100x $1,000 are). plan.Seed reads the catalog's status in BOTH directions: a draft row is created unlisted, a row already where the catalog says is left alone, active makes a row sellable and draft takes a seed-written row back off sale. An admin edit still wins (AdminEdited). A draft still resolves (lookupPlan, resolveSubscriptionPlan) so a renewal or invoice could price itself; it is on sale nowhere: planAuthorityRows, the embed fallback (listedCatalog), StaticPlans and the three purchase entrypoints all go through Listed() / the draft filter. Activating a rung is one edit in @hanzo/plans (delete the status line), a release, a re-vendor and a deploy, and it needs the owner's word because it makes a live price chargeable.

Lifecycle: archive, never delete (the Shopify product model)

plan.Status is active | draft | archived, and empty means active — every row written before the field existed has no value for it, and a zero-value meaning "draft" would blank the catalog on deploy. plan.Listed() is the ONE predicate, gating both halves of "on sale":

  • the public catalog (planAuthorityRows skips unlisted rows), and
  • the three purchase entrypoints — CreateBillingSubscription, the PATCH plan-change, and the card subscribe — which 404 an unlisted plan. Hiding a tier from the page is worth nothing if the API still sells it, and the card path refuses BEFORE the charge.

resolveSubscriptionPlan is deliberately NOT gated: it answers "which row is this" and must keep answering for an archived plan, or a renewal on a retired tier could not price itself. Retiring stops new sales; it never strands a subscriber. Seed archives what the catalog stopped publishing; an admin-created plan is exempt (it is not "missing from the catalog", it is theirs).

The wire shape IS the model shape

A plan's display envelope (features/limits/bundles/includedIn) used to live only as a packed JSON string in Metadata, with the packer private to api/billing — somewhere the admin handler could not reach. So PUT /v1/plans/entries/:slug carrying features set the price and SILENTLY DISCARDED the copy: a tier could be repriced but never re-described. Those fields are now first-class on plan.Plan (JSON-visible, datastore:"-", still persisted packed under the same one key), so whatever GET /v1/billing/plans emits, the admin CRUD accepts. planLimits is an ALIAS of plan.Limits, not a second declaration.

Gotchas this cost

  • A row from GetAll is a VALUE with no datastore binding. Update() on it writes nowhere AND RETURNS NO ERROR. Re-load through the bound point query (New(db) + Query().Filter("Slug=",…).Get()) before writing.
  • Save() must pack into Metadata, not Metadata_. The ORM stores a row as JSON of the struct and Metadata_ is json:"-", so a value written only there does not survive. Measured, not assumed.
  • paidTier counts a contact-sales plan as paid. It stores Price=0 because its price is null, not free; reading price alone made a negotiated tier self-serve, so a catalog row with a real allotment could be minted with no payment. That never fired before only because the tier with the large allotment also carried a large price — luck, not a gate.
  • A cgo build needs -tags sqlite_math_functions (hanzoai/base asserts it at compile time). Without it v1.49.26–v1.49.37 each pushed a release tag and published NO image — twelve dead releases.

The seed is only as good as what it compares

planEqual/limitsEqual decide whether the seed writes at all, so a field copied but not compared is invisible: the stored row and the published catalog disagree, the seed reports success, and nothing logs. copyInto's comment has always said the two lists must match; nothing enforced it.

TestEveryLimitCopiedIsAlsoCompared walks the struct by reflection, so the next field added to Limits fails immediately instead of two deploys later.

The catalog decides whether a plan is sold, in BOTH directions. The seed archives a slug the catalog stops publishing; it now un-archives one the catalog names again. Without the second half a retired tier could never come back — Status is carried by neither copyInto nor planEqual, so a republished row reconciled to the new values and stayed invisible. That is what happened to dev: retired at plans 1.4.16, republished at 1.4.20, still absent from the live catalog. An admin's own unlisting still wins, because AdminEdited is checked first — the package is the source, a human edit is an override.

OPEN: the list and the enforcement read different sources

ListPlans/GetPlan read planAuthorityRows — the DB, where an admin edit lands. Everything that ENFORCES reads lookupPlan, which walks the Go catalog embed: tier resolution, trials, DNS usage, the plan-move score and the rest of plans.go. Many call sites, one source, and it is not the one the admin writes to.

So an operator editing a plan's limits at admin.hanzo.ai changes what a customer is SHOWN and not what they GET. Which of the two should win is a real design question — this file's own heading says a plan is "applied by a DEPLOY", which argues for the embed; the AdminEdited override argues the other way — and it should be answered deliberately rather than by whichever call site a reader happens to open.

A team mixes seat types on ONE subscription (plans 1.8.20)

Team is sold in two seat types one team can mix: team (standard, $25 or $240 a year) and team-premium ($100 or $960). The catalog's group ("team" on both) says they are seat types of one team, and its minSeats (2) and maxMembers (100) count every seat: 1 standard + 1 premium is a team, 1 alone is not.

The model is one row, not one per seat type: Plan/Quantity are the first seat type and subscription.Lines the others (Lines, each with its slug, price for the row's period, quantity and pending quantity). So one charge pays the period, one invoice carries one plan line per seat type (buildPeriodInvoice), one renewal moves them together (pending quantities included), and "one paid subscription per subject" (billingSubscription) and the tier resolver still see the team as one row. Two rows would have needed a second notion of which subscription governs the subject, two charges that can half-fail, and a minimum that a single cancel breaks.

  • Buy: POST /v1/billing/subscribe/card {"planId":"team","quantity":1, "lines":[{"planId":"team-premium","quantity":1}],"interval":"month"|"year"}. linesAt refuses a line outside the plan's group, the plan itself, a type named twice, zero seats, or another currency, before any charge; teamSeats is the floor and ceiling over every seat. A different mix is a different idempotency fact.
  • Change seats: PATCH lines, or ChangeSeats (cloud's plane op), pending to the renewal as quantity is. A seat type of the group the row does not hold yet is ADDED (nextLines, priced by linesAt at the row's period): it holds 0 seats now and the renewal that opens the next period charges it. So a team bought with one seat type, before premium existed or from a checkout that sent no lines, mixes later; a second sale stays refused (one paid subscription). An added seat type is Line.Full: billed outside the promotion the row was sold under (buildPeriodInvoice discounts only the rest), since the promotion was sold for what was bought. One not yet charged is taken back by asking for 0; a charged one keeps a seat. A comp's seats do not change (compSeats): a record names them. Over HTTP a member billed from the org's pool changes nothing (mayChangeWallet, the one wallet rule), and members are <org>/<name>. The renewal still writes its row back after the charge, so a change of seats made during the charge is lost to it, as a change of quantity always was; making the renewal a Swap is its own change to the engine. engine.ChangePlan refuses a team with lines: a move prices one plan's seats against another's.
  • Record: RecordIn.Lines opens a recorded team with both seat types, its price checked against every seat; extending one checks the price of every seat it holds for the period, and a payment naming other lines is refused. Each recorded period keeps its seats per seat type (periods[].lines), which periodPaid reads back as a charged invoice's plan lines, so cloud serves a recorded premium seat as premium.
  • Offers: a promotion or coupon takes its percent off a team only when it covers every seat type bought (everyLine), since the charge and the invoice discount every plan line together.
  • Read: the view's lines carry each type's seats paid for (periodPaid keys plan lines by PlanId: the row's plan id for the first type, the slug for a line); seats and chargedCents count all of them. MRR includes lines.
  • Assign: PUT /v1/billing/subscriptions/:id/seats {"assigned":[{"member", "plan"}]} (AssignSeats, admins only over HTTP: mayActForOrg) replaces who holds which seat type, never more members on a type than its seats paid now. A member is a person of the org as accounts name one, <org>/<name> folded (member); anything else is refused. It writes by Swap, so it never puts back what a renewal or a change of seats wrote meanwhile. It moves no money and bounds no usage: cloud's limiter pools a team's seats. The older members → bundle-child rows (provisionMembers) is standalone-only and never removes a row; it is not the assignment.

A plan states no AI usage

What a plan allows of AI — per model class, per session and day, per model — is cloud's usage policy (hanzo-inc/cloud apps/ai/limits), config in cloud's flags store, edited by a SuperAdmin at admin.hanzo.ai and read by customers at /v1/ai/limits. Commerce holds no second statement of it:

  • plan.Limits has no requestsPer*/tokensPerMinute field, so an admin PUT carrying one stores nothing and the seed has nothing to reconcile.
  • /v1/billing/tier and /v1/billing/usage/rollup carry no windows.
  • The public answer (ReadPlans → PlanView, which cloud forwards as GET /v1/billing/plans) narrows limits to Capacity: agents, bots, guests, members, seats and the DNS quotas. The stored row is never served verbatim.
  • A plan move (PATCH) is scored by listCents: the embed's monthly list price, per seat on a per-seat plan, resolved slug, contact-sales as the most there is. A move that raises it needs MayMintMoney; lateral and downgrade are self-serve.

One way to price a UNIT — the rate authority, edited at admin.hanzo.ai

A plan says what a SUBSCRIPTION costs. A rate says what one UNIT costs — a model per million tokens, a tool per call, storage per GB-month. They are different questions with different lifecycles, and models/rate is the second one.

Rates used to live in hanzoai/pricing as an embedded 150KB JSON: 506 priced items, where changing one number meant editing a module, cutting a tag, bumping commerce, bumping cloud and deploying. That is the right lifecycle for a plan ladder (see the section above — versioned, reviewed, revertible) and the wrong one for 506 upstream prices that move when a vendor moves theirs.

So the rate authority has NO EMBED, deliberately. There is nothing to vendor and no PinnedRatesVersion. POST /v1/commerce/rates/import loads a whole document and RECONCILES it: a matching row is left alone, a drifted one corrected, an operator-edited one skipped. Importing the same document twice is a no-op.

AdminEdited means the same thing it means for plans, and matters more here: an operator's price outranks the document it came from. Without it you would change a rate in admin.hanzo.ai, watch it apply, and find it silently reverted by the next import — which would make an editable catalog WORSE than a hardcoded one.

Identity is (product, meter), derived into one indexed Slug by Bind(). The same model name can be metered by two products at two rates, so keying on the name alone lets one product's price overwrite another's.

Rates are NANO-dollars. Cents cannot hold them: a cheap model is fractions of a cent per million tokens, and rounding to a cent prices it at zero or at 100x. The usage ledger already carries cost_nano/billed_nano/margin_nano.

models/rate is NOT models/meter. A meter is an event AGGREGATION rule (EventName, AggregationType sum/count/last, Dimensions) — what to count. A rate is what one of those units costs. They shared a name for one commit and billing/engine broke immediately.

One way to grant credit — POST /v1/billing/credit (2026-07-16)

POST /v1/billing/credit is the ONLY way credit enters an org ledger. It is mint-gated (registered through middleware.Mint, which applies middleware.PlatformOnly): only the internal service token OR a SuperAdmin (owner=="admin") reaches it — every self-service / org-owner / no-auth caller is 403/401. A client-supplied mint amount must never be self-service; that is the money-critical core (a user cannot credit itself). Body: {org, amountCents, reason, tag?, currency?, expiresAt?, idempotencyKey?}, org-keyed (the org POOL account, not a per-human subject), idempotent on idempotencyKey.

The mint gate declares itself. A mint route is registered through middleware.Mint(api) — mint.Post("/deposit", Deposit) — which BOTH applies PlatformOnly() and records the route's method + full path in the package-level registry. Registration is the single declaration, so the gate and the record cannot drift. middleware.MintRoutes() []MintRoute exports that surface (after mounting) for cross-service checks: cloud's /v1/billing bridge forwards with the admin COMMERCE_SERVICE_TOKEN, which satisfies MayMintMoney, so its billingForwardable allowlist must stay DISJOINT from MintRoutes() — an assertion cloud can now make against this API instead of hand-copying the list. Routes registered through util/rest's deferred table (api/affiliate) carry the same PlatformOnly chain explicitly but are not self-declaring.

Consolidated three prior handlers into this one: deleted self-service GrantStarterCredit (old /credit), PostMyWelcome (/me/welcome), and GrantStarter (/grant-starter). "Starter credit" is now a PARAMETERIZED call (tag=starter-credit, amountCents=500, expiry=+365d) driven by the billing/credit constants (reduced to data-only). The mint-surface guard (api/billing/mint_surface_test.go) proves /credit is the sole gated credit entry point.

Backing = injected double-entry ledger (billing/creditledger). Commerce runs embedded in the cloud binary, where the AI spend-gate reads cloud's native the ledger. Commerce must NOT import cloud, so the host injects a creditledger.CreditLedger via EmbedConfig.Ledger (embed.go → creditledger.Set). When set, Credit and GetBalance route to it — a credit lands in the SAME per-org account the gate reads (one ledger, no split). When nil (standalone), both fall back to commerce's own datastore (tagged Deposit). The interface is compiler-enforced; cloud implements a ~40-line ledger adapter.

type CreditLedger interface {
    Credit(ctx, CreditInput) (txID string, balanceCents int64, err error)
    Balance(ctx, org, currency string) (availableCents int64, err error)
}
type CreditInput struct { Org, Currency, Reason, Tag, IdempotencyKey string; AmountCents int64; ExpiresAt *time.Time }

A plan coupon runs for months (engine.Grant)

A plan coupon (api/promo, models/coupon with Filter == "plan") is a percent off the first Months billing cycles of one purchase (Months 0 reads as 1, the first month; 1..36). The sale grants the subscription the coupon (engine.Grant: CouponCode/Percent/Months/Plans, CouponFrom = the period the sale holds, which on a trial is the first paid period after it). An invoice, the sale's first included, takes the percent while fewer than Months cycles were invoiced in [CouponFrom, its period) (engine.Billed: distinct plan-fee periods, voided ones aside), the row bills monthly (engine.BillsMonthly; the sale refuses a coupon on anything else), and the coupon names the row's plan and every seat type sold (engine.couponOf). A cycle is an invoice, so nothing counts down: a renewal re-run finds the invoice it wrote, a row not saved after its invoice spends no month, days a gift adds or a late payment that moves the period spend none, and a voided (unserved) period leaves its month for the next. A cycle billed on a plan the coupon does not name, or on a year, is billed at its price and spends one. Label "CODE — N% off first month" for one month, "CODE — N% off, month k of M" for more; invoice metadata coupon, couponPercent, couponCents, couponMonth, couponMonths. The quote (GET /v1/billing/plans?coupon=) and the receipt carry couponMonths. A coupon with a trial waits it out (reserved at the sale, granted from the trial's end). ExtendCoupon(ctx, code, apply, by) grants the coupon's current months to every live subscription already bought with it, from its first paid period (the sale's grant, else the first invoice the coupon discounted); it never takes a month or a plan away, writes by Swap, reports billed and missed (cycles among the granted months invoiced without the coupon, paid or owed), dry-runs with apply false, and records subscription.coupon_extended in the org's namespace.

A coupon for the next renewal (api/billing.SetNextCoupon)

Staff put a percent off ONE renewal invoice on a subject's live subscription (its held paid plan, or a trial of one): SetNextCoupon(ctx, org, subject, code, percent) writes Subscription.NextCouponCode/NextCouponPercent (percent 0 clears; 1..100 needs a promo.CouponCode code; an external row is refused, nothing renews it; no live row is IsSubscriptionNotFound). It is not a plan coupon: no window, limit or redemption. It stays a separate fact because it is a separate promise: the NEXT renewal whichever period it bills, late or not, on a monthly or a yearly row, while a plan coupon is dated months; and one row holds both at once, so a win-back inside a sale's months stacks (the renewal coupon off what the plan coupon left) instead of erasing them. engine.RenewSubscription hands it to buildPeriodInvoice as Coupon{Renewal: true} (after the promo and the plan coupon, plan fee only; label "CODE — N% off this renewal", invoice metadata renewalCoupon, renewalCouponPercent, renewalCouponCents), clears it on the row the caller persists, and restores it when no invoice was built. settle clears it too when the owed period's invoice already carries it, so a renewal whose row was not saved after its invoice cannot hand it to a second period. Cloud reaches it as the plane op billing_subscription_coupon.

One way to give plan time — billing.Gift (api/billing/gift.go)

Days of a plan given, never sold: a referral's free week. A subject on a paid plan has the period it holds run the days longer (the next renewal moves with it, no invoice). A subject on none is given the entry plan (dev, sold as Pro) on a manual_gift row canceled at the end of the days, so it lapses to Free on its own. One key gives once (idempotency guard plus the key written in the row's gifts metadata, which a re-claimed guard finds). A sandbox gift (Test) lands only on test rows. A gift row cannot be reactivated.

engine.Over(sub, now): a row canceled at the end of its period confers nothing once that period is over, swept or not. rowTier, holdsPlan and servedSubscription ask it, because the renewal sweep (billing_renew) is off in production and ended only runs inside it.

billing.PaymentsOf(org, subject) is what a referral reads: the earliest subscription period a card paid in full (PaymentMethod == "card"; credit, balance and mixed are not a card), with its refunds, dispute and the paying card's fingerprint, plus every fingerprint the subject keeps on file. Both are platform-only plane ops in cloud (billing_gift, billing_payments).

Cross-Compilation (2026-02-23)

Colima QEMU crashes Go's HTTP/2 and module loader on ARM Mac. Use zig for cross-compilation:

go mod vendor
CC="zig cc -target x86_64-linux-musl" CXX="zig c++ -target x86_64-linux-musl" \
  CGO_ENABLED=1 GOOS=linux GOARCH=amd64 \
  go build -mod=vendor -ldflags="-s -w -extldflags '-static'" -o commerce ./cmd/commerce/main.go

Push to GHCR (Docker Hub credentials not available locally):

docker build --platform linux/amd64 -t ghcr.io/hanzoai/commerce:hotfix .
docker push ghcr.io/hanzoai/commerce:hotfix

App Engine Migration (2026-02-24)

All App Engine dependencies removed. Context handling modernized:

  • middleware.RequestContext() (was AppEngine()) — stores Go request context in Gin
  • middleware.GetContext(c) (was GetAppEngine()) — retrieves it; falls back to c.Request.Context()
  • Gin context key changed from "appengine" → "context" everywhere
  • Legacy aliases AppEngine/GetAppEngine kept as var for backward compat
  • Legacy GAE Python utils moved to .legacy/ (bulkloader, datastore-admin, salesforce-metadata)
  • ORM: all models use mixin.Model[T] with generic CRUD, namespace support, and orm.Register[T]()

OSS Contributor HUSD Payouts (on-chain, 2026-06-27)

25% of cloud revenue is payable to OSS contributors in HUSD (Hanzo USD stablecoin) on the Hanzo EVM. The split + SBOM attribution + payout algorithm live in models/contributor/; the executor is cron/payout/contributor/.

  • Executor: contributor.Payout(ctx, Config) computes period revenue, splits 25% across components by real SBOM weight, and disburses each allocation by PayoutMethod: credits (CreditGrant), stripe (queued transfer), crypto (on-chain HUSD ERC-20 transfer).
  • Crypto path: executeCryptoPayout → util/blockchain.TransferToken (geth-free seam) → thirdparty/ethereum.transferToken (luxfi/geth signing, legacy EIP-155 tx). The impl is registered by a blank import of thirdparty/ethereum in cmd/commerce/main.go — without it, TransferToken returns ErrNoTokenTransfer (crypto payouts skipped, never silently lost). go.work + replace => ./thirdparty/ethereum link it under the GOWORK=off Docker build.
  • Config (KMS → env, HUSDConfig.LoadFromEnv): HUSD_TOKEN_ADDRESS, HUSD_CHAIN_ID, HUSD_RPC_URL, HUSD_TOKEN_DECIMALS, HUSD_TREASURY_KEY. Fails closed: no HUSD_TOKEN_ADDRESS/HUSD_TREASURY_KEY ⇒ ErrHUSDNotConfigured (mainnet has no default token addr, so an unset deploy can't mis-pay). HUSD_TREASURY_KEY is KMS-only, injected into the environment. Non-secret config is plain env.
  • Trigger: POST /v1/contributor/payouts/execute (admin) runs the executor for the org namespace — dry-run by default, ?execute=true to disburse. Mirrors billing auto-recharge/run-all; drive it from a curl CronJob: curl -XPOST -H "Authorization: Bearer $COMMERCE_SERVICE_TOKEN" \ http://commerce.hanzo.svc:8001/v1/contributor/payouts/execute?execute=true. GET /payouts/preview shows the allocation without paying.
  • TESTNET-FIRST: deployed config points at Hanzo testnet (chainId 36962, HUSD = "Lux Dollar" 0xc57b7eCE…4D66, 18 decimals, a testnet hanzod node's /v1/chain/c RPC). Proven on testnet 2026-06-27: tx 0xe5cf03378e2d9dd121dfc5631fa112b2ea03717c9928167d54195ae785866978 (treasury → contributor, 25.50 HUSD, status 0x1, block 9). Live proof: go test -tags onchain -run TestOnChainHUSDPayout_Testnet.
  • MAINNET SWITCH (OFF; requires sign-off + funded mainnet treasury): HUSD_TOKEN_ADDRESS=0xe9e32EF8aaECB68794Da3E1E9191b0a64CeC2c83, HUSD_CHAIN_ID=36963, HUSD_RPC_URL=<a mainnet hanzod node>/v1/chain/c, and re-point HUSD_TREASURY_KEY to a funded mainnet treasury.
  • Gas note: thirdparty/ethereum GasPrice() queries eth_gasPrice (respects the chain's 25 gwei minBaseFee); the old hardcoded 1 wei caused "transaction underpriced" on Hanzo EVM.

Versioning (tag == binary)

commerce.Version is a var (commerce.go), default = the current release. CI injects the immutable image tag at build time so /healthz version always equals the deployed tag:

  • .hanzo/workflows/cicd.yml passes VERSION=<git tag> (build-arg) on v* tag pushes.
  • Dockerfile strips the leading v and applies -ldflags "-X github.com/hanzoai/commerce.Version=<ver>". It is the ONLY Dockerfile that builds the binary — Dockerfile.sqfix, Dockerfile.prebuilt and Dockerfile.sqfix-prebuilt are gone. All three packaged a binary compiled by hand on a laptop (zig cross-compile / hotfix builds), which is not how anything ships, and none was ever referenced by hanzo.yml. One had drifted to build tags that no longer compile, so the only thing they could still do was mislead.
  • Branch builds leave VERSION empty → the in-source default holds.

Cut releases with a v* git tag (git tag -aX vX.Y.Z && git push origin vX.Y.Z) so the build produces ghcr.io/hanzoai/commerce:X.Y.Z whose binary reports X.Y.Z. Do NOT ship named tags (:X.Y.Z-foo) for prod — they re-introduce the tag/binary drift (the live 1.42.31-testmode image reported binary 1.42.5 because Version was a hardcoded const nobody bumped).

commerce-site edge (no nginx)

commerce.hanzo.ai serves the marketing SPA via ghcr.io/hanzoai/static (Dockerfile.site, --spa, :3000) — NEVER nginx. The host's routing lives in the hanzoai/ingress file provider (universe infra/k8s/ingress/routes.yaml): a high-priority commerce-hanzo-ai-api router sends /v1, /api, /healthz to the commerce API (commerce.hanzo.svc:8001); the low-priority commerce-hanzo-ai router catches the rest → the SPA. The operator commerce-site Service CR has ingress.enabled: false (a plain Ingress can't express Traefik priority). The from-source Dockerfile.site Next.js build is currently broken (unbuilt workspace dists, a phantom @kapaai/react-sdk import, a @/providers alias); 0.2.0 re-serves the proven 0.0.1 assets — fix the app build before bumping content.

SBOM-driven OSS-developer payout (2026-06-25)

Every deploy emits an SBOM; Hanzo pays upstream OSS developers up to 25% of all Hanzo cloud spend, attributed by the dependencies actually deployed. One pipeline, decomplected into orthogonal pieces.

The pipeline (one way, all images)

arcd build  --syft-->  CycloneDX SBOM  --normalize-->  POST /v1/billing/sbom
  (deploy)             (per image digest)               (ZAP 0x20 OR HTTP)
                                                              |
org usage  -->  RecordUsage  --go-->  engine.AccrueOSSPayout  v
(charge)        (existing)            (mirrors TrackRevenueShare)
                                          |
                  ossattr.Attribute(spend, SBOM pkgs, policy)   <= 25% pool
                  ossfunding.Resolve(purl) -> maintainer target
                                          |
                                          v
                  OSSAccrual ledger lines (per package, idempotent)
                                          |
                  GET /v1/billing/oss-payout/summary  (disbursement view)

Components

Concern Where Notes
Attribution math (the heart) ossattr/ (leaf, stdlib-only) Pure Attribute(spendCents, []Package, Policy) -> Result. 25% cap is ossattr.MaxPoolFraction (code, not config). Pro-rata by weight; largest-remainder apportionment conserves the pool to the cent. Deterministic + sorted.
Policy config/oss-payout.json + config/oss_payout.go poolFraction (clamped <=0.25), directWeight (1.0), transitiveWeight (0.25). Embedded JSON, sync.Once (mirrors referral-program.json).
Maintainer resolution ossfunding/ (leaf) FromPURL (pure, offline): github.com/ PURLs -> github_sponsors:<owner>; golang.org/x/* -> golang; npm/pypi w/o VCS -> Unresolved -> held. Network FUNDING.yml/registry enrichment behind Resolver (out of hot path).
SBOM storage models/sbomrecord/ Per image digest; Ingest() is the ONE upsert (idempotent on digest), shared by HTTP + ZAP. Components are a noindex JSON blob. Global "system" namespace.
Accrual ledger models/ossaccrual/ Append-only per-package line; idempotency key = ossattr.AccrualID(org, txn, purl). Status: pending|held|queued|settled. "system" namespace (Hanzo->OSS liability, aggregated across orgs).
Accrual engine billing/engine/osspayout.go AccrueOSSPayout(...) — fire-and-forget hook in RecordUsage, structural twin of TrackRevenueShare. Unions SBOM components, runs ossattr.Attribute, resolves funding, writes lines.
HTTP surface api/billing/osspayout.go POST /sbom, GET /sbom, GET /oss-accruals, GET /oss-payout/summary (admin-token).
ZAP-native path infra/zap_osspayout.go OpSBOMIngest = 0x20 on the existing ZAP node; RegisterSBOMIngest(store) wired in commerce.go to the SAME sbomrecord.Ingest.
SBOM emit (build side) arc/cmd/arcd/sbom.go dockerBuilder.emitSBOM after docker push: syft <digest> -o cyclonedx-json -> normalizeSBOM (pure; direct/transitive from the dep graph) -> POST. Env: SBOM_INGEST_URL/TOKEN/ORG. Best-effort (never fails the build).

The 25% math

pool = floor(spend * min(poolFraction, 0.25)). Each package weight = base(scope) * criticality (direct base 1.0, transitive 0.25, criticality default 1.0). share_i = pool * weight_i / sum(weight_j), apportioned to integer cents so sum(shares) == pool exactly. Same PURL across SBOMs is de-duped (highest weight wins) so each package is paid once. No packages / zero weight -> whole pool held, never invented or lost.

Proven (tests, all green)

  • ossattr: cap enforced (incl. over-cap clamp), cent conservation over 5000 spends, direct=4x transitive, criticality, dedup, held-pool, determinism, big spend exactness.
  • ossfunding: PURL->target resolution incl. held cases.
  • billing/engine (real SQLite): deploy SBOM -> $100 spend -> 2500c (25%) across 4 pkgs; gin direct=1000c->github_sponsors:gin-gonic(pending), react transitive=250c->held; idempotent (2 fires -> 1 line); no-SBOM no-op.
  • api/billing (HTTP, real router): ingest -> summary rollup; cap + held + resolution proven over the wire.
  • arc/cmd/arcd: normalizeSBOM CycloneDX->ingest with direct/transitive graph tagging + PURL-less drop.

Money-out (designed, creds-gated — NO fake payouts)

Accrual is real + provable now. Disbursement is a separate job that reads GET /oss-payout/summary (status=pending, resolved targets), batches by funding kind, pays, and flips lines pending->queued->settled (referencing a Payout). Rails are integration/creds-gated:

  • GitHub Sponsors — no programmatic payout API; needs the Sponsors GraphQL + an org Sponsors account funding source. Per-maintainer sponsorship.
  • Open Collective — API for expenses/payouts; needs an OC account + API key.
  • Stripe Connect — Transfer/Payout to connected accounts; needs the maintainer to onboard a connected account (KYC). Best general rail. All require: a funded Hanzo source account, per-rail API credentials in KMS (never plaintext), maintainer onboarding/KYC, tax handling (1099/W-8). The held pool waits until a target is resolved + onboarded.

Nobody is exempt — every subject pays, every call is a row

  • Our orgs resolve like customers. hanzo, admin, lux, zoo and every other org get a tier only from a real recorded subscription (deriveTier), a balance only from the ledger. No org list, no balance floor, no minted operating credit, no unlimited roster. SuperAdmins pay like everyone.
  • The tier is derived, never named. ReadTier(ctx, org, user) derives it; X-Tier / ?tier= are ignored from every caller. A nil org is Free (no store).
  • A wallet spends its own grants. getActiveGrants reads the subject's rows only; hanzo/alice never reaches hanzo's grants.
  • Every usage call writes a row (writeUsage, behind POST /v1/billing/usage and ZAP billing.recordUsage): a $0 or sub-cent call included, carrying model, tokens, scope, costMicros and paidBy (plan|prepaid|credits|line|hanzo). The debit is whole cents, exact over time: each call adds its micros to an atomic per-subject running total (db.Counter.Add, one upsert on _sequences) and debits the cent boundaries its span crosses, so sum(amount) == floor(sum(amountMicros)/10_000) under any concurrency. A store that is not a db.Counter refuses (503).
  • A plan given away is a comp, recorded as one. RecordSubscription with processor comp: priceCents 0 for any plan (enterprise, which publishes no price, included), the approval in reference.order, why in terms. The row is external, so it never renews or charges, and its plan is held at price 0, so it adds nothing to MRR; it confers the plan's tier like any recorded plan (settled external:comp). A comp extends only a comp and a payment only a paid plan; either takes the other over by naming it in Replaces. Every other processor still needs a price equal to the catalog's.
  • Privileged money acts name who took them: /credit, /deposit, /credits (+ void), customer-balance adjustments, bank-transfer matches and comp subscriptions stamp minter(c) on the row they write.
  • commerce keeps no metering client. Every product meters through cloud's one package (hanzo-inc/cloud/metering), which has no off switch: a balance or spend cap it cannot read refuses the call (503 balance_unavailable / spend_cap_unavailable). The metering/ module that was here — its own copy of that client, the meter-proxy sidecar and the deploy renderer — is deleted. No deployment ran the sidecar, and its tagged releases (metering/v0.1.x) stay resolvable for the one module that still pins one.

Three money buckets — credits vs prepaid vs card (billing/bucket)

The ledger was CONFLATED: starter-credit/included-credit: (non-cash grants) and topup/deposits (real money) were all transaction.Deposit rolled into ONE balance. They are now split HONESTLY from the real tags — no fabricated numbers.

  • billing/bucket (pure, no I/O, unit-tested) is the ONE classifier + spend projection:
    • DepositKind(tags) → Credit (tag starter-credit, prefix included-credit:, credit:, grant:) vs Prepaid (everything else — topup/husd/bare deposit = real money; UNKNOWN→Prepaid is fail-closed: never mint spendable-on-GPU value from a mystery deposit).
    • IsGPUWithdrawal(tags) → gpu (gpu/gpu-*/gpu:*).
    • Compute(transs, id, now) → Split{CreditsGranted, CreditsRemaining, PrepaidBalance, PrepaidAvailable, Balance, Holds, Available}.
  • Spend policy (documented, enforced): non-GPU usage draws CREDITS FIRST then prepaid; GPU usage draws ONLY prepaid. The projection reconciles to the cent: CreditsRemaining + PrepaidBalance == Balance (the same balance the gateway gate debits — bucket.Compute is a faithful superset of util.TallyTransactions).
  • Read: GET /v1/billing/balance + /me/balance now ALSO return creditsGranted, creditsRemaining, prepaidBalance, prepaidAvailable, and card{onFile,brand,last4,isDefault} (additive — the pre-split {balance,holds,available} is unchanged, so the cloud proxy + console BFF forward the split verbatim; console shows credits vs prepaid distinctly). util.GetRawByCurrency fetches the raw (un-expiry-filtered) ledger the split needs.
  • GPU rule (the bucket projection, billing/bucket): a gpu-tagged Withdraw draws PREPAID only — IsGPUWithdrawal classifies it and Compute never lets it reach a credit grant. It is a property of the LEDGER, so it holds for every writer rather than for one endpoint. There is no bespoke GPU charge endpoint here: a GPU is a metered resource like any other and is billed through the general metered path (a machine is launched via cloud /v1/machines, which fronts the compute provider's resell endpoint where the balance gate and the per-hour meter live). A GPU charge NEVER calls BurnCredits.
  • CreditGrant (models/creditgrant, BurnCredits) is a SEPARATE grant ledger used only for subscription/invoice collection (cycle/invoices/subscriptions) — NOT the pay-as-you-go wallet the balance endpoint reads. Left as-is.

Chain-Backed Credit Ledger — HUSD-on-Hanzo-EVM (steps 1-6 BUILT + testnet-proven)

Kills the recurring C1 money-mint vuln class by construction: a credit exists ONLY as HUSD (Hanzo USD, ERC-20 on the Hanzo EVM) minted by the KMS treasury key — which commerce does NOT hold. Commerce can only REQUEST a mint; it becomes a read-only indexer of on-chain balances. No commerce code path can create money.

Foundation landed (branch feat/chain-backed-credit-ledger / chain-ledger-wip, 7-step plan — steps 1-3 done, 4-7 staged):

  1. treasury.DeriveAccount(masterSeed, orgID) (treasury/derive.go) — per-org on-chain address, BIP-32-style HMAC-SHA512 → secp256k1 (luxfi/crypto, no geth). Deterministic across restarts, collision-safe for string org ids, golden-vector pinned. Private key (settlement-signing, step 5) in memory only.
  2. treasury.Mint(ctx, MintRequest) (treasury/treasury.go) — the ONE mint path: treasury-signed on-chain HUSD transfer to the org address. TWO locks: mintauth.Require(ctx) (gated HTTP caller needs proven authority; crons ungated) + KMS key custody (husd.Config.Configured() else ErrNotConfigured, no DB fallback). Idempotent by key (deterministic IssuanceID; replay → stored receipt, no new tx; concurrent same-key → exactly ONE on-chain mint). Bucket (credit|prepaid) recorded off-chain → ledger tag (credit:husd|husd). Production store: models/husdissuance (kind 281) + treasury/datastorestore (deterministic-id ON CONFLICT, mirrors models/idempotencykey).
  3. billing/husdindex — chain is source of truth. client.go = CGO-free JSON-RPC read client (BlockNumber/BalanceOf/TransfersTo). index.go = Sync projector: scan Transfer events → project idempotent bucket-tagged credits (dedup on txHash:logIndex), reconcile indexed == balanceOf to the cent.
  • util/husd = the ONE HUSD config + cent↔wei source (contributor payout aliases it now — DRY). mintauth.Require = the shared "gated & unauthorized ⇒ refuse" policy (sink Enforce + mint service both call it).
  • Decomplected for CGO-free proofs: core logic (derive/mint/index) has no sqlite/geth dependency, so unit tests + the live proof run CGO_ENABLED=0 locally (the luxfi/accel cgo link + mattn/go-sqlite3 both fail on a laptop; datastore adapters are CI-tested). geth core/types/rlp are avoided in the live signer too — their transitive luxfi/[email protected] pull trips the upstream force-moved-tag go.sum mismatch; the proof signs EIP-155 with luxfi/crypto + hand-rolled RLP (production uses the geth thirdparty/ethereum signer via the same util/blockchain.TransferToken seam).

PROVEN on testnet (chainId 36962), end-to-end, independently verified via cast:

  • Provisioned a fresh HUSD test token 0xe7f1725e7734ce288f8367e1bb143e90bb3f0512 (the ephemeral testnet had reset — no token, unfunded treasury; funded treasury 0xe6da…a51a from the genesis hardhat account 0xf39F…2266, minted 1M HUSD to treasury). Node serves the new /v1/chain/c surface (post /ext→/v1 cutover).
  • Live mint tx 0xd01bc1c1733e83c93a5143552af1ba7e3ac045b2433c2fecff07a29685cf976a (status 0x1, block 25): treasury.Mint($12.34, credit) → treasury → derived org addr 0x3560…9950. Org balanceOf == 1234c; treasury balance dropped by EXACTLY 12.34 (value from real supply, not fabricated); indexer Sync projected a credit:husd 1234c credit == on-chain; replay of the idem key sent NO second tx. Run: HUSD_RPC_URL=… HUSD_TREASURY_KEY=… HUSD_TOKEN_ADDRESS=… CGO_ENABLED=0 go test -tags onchain -run TestOnChain_ChainBackedLedger ./treasury.

Headline: on testnet, minting is treasury-only (commerce holds no key, cannot mint) and idempotent, and the indexer reflects on-chain balance — proven live.

Steps 4-6 (BUILT + testnet-proven, branch chain-ledger-wip)

  1. Mint-path rewire (billing/husdledger + api/billing/husd_mint.go). husdledger.Service is the ONE mint entrypoint (MintCredit): treasury on-chain mint → bounded synchronous ProjectTx so GET /v1/billing/*/balance reflects it on return; a background/CronJob SyncOnce backfills. Adapters: ledgerStore (projects a bucket-tagged transaction.Deposit — the SAME row the balance read tallies — keyed to the mint's Subject in its Test partition, idempotent on txHash:logIndex), addressBook (derive every org address from the org list + seed), cursorStore (models/husdcursor kind 282), and datastorestore.ByTxHash (the issuance store IS the IssuanceLookup, system-ns). MintRequest/Issuance/Credit gained Subject (per-org address, per-subject ledger) + Test. Handlers Deposit/GrantStarterCredit/PostMyWelcome/GrantStarter/topup branch to the chain mint when enabled (else the DB path, unchanged). Wired at Bootstrap from husd.Config + HUSD_ORG_DERIVATION_SEED. The seed is the ledger's sole intent signal (husdledger.ValidateConfig, PR #69): ENABLED only with a seed AND token+key; token+key WITHOUT a seed is the INERT state (Enabled()==false, DB credit path) — NOT a boot error — because the shared util/husd token+key is ALSO the OSS contributor-payout config, which runs with no ledger seed. Boot refuses ONLY on an incoherent LEDGER config (seed set but token/key missing). This decouple is why v1.46.39 boots inert in prod (the CR carries HUSD token+key for OSS payouts, no seed) instead of crash-looping; the mainnet cutover enables the ledger by provisioning the seed. Proven: live TestOnChain_Step4_ProjectTx (mint 0x3324a75a… → ProjectTx from the REAL receipt → one credit subject/bucket/test) + CI ledgerStore sqlite test (ProjectTx → real Deposit → bucket balance, idempotent).
  2. Settlement (husdledger/settlement.go, husd.SettlementDrift, kind 283). drift = balanceOf(orgAddr) − max(0, ledgerSpendable); sweep org→treasury, signed by the ORG's derived key, when drift ≥ threshold. Self-correcting + idempotent. POST /v1/billing/husd/settle (platform-only). Proven: live TestOnChain_Step5_Settlement (mint $50 → gas-fund org → org-key sweep 0xf323bf25… → org drops EXACTLY $20 == spendable, treasury +$20) + CI SettleOrg sqlite test (real ledger $50−$20 → sweep $20, reconcile, idempotent).
  3. Migration + reconcile (husdledger/migration.go). Snapshot each subject's bucket.Split → treasury-mint the equivalent per bucket → neutralize legacy deposits with two idempotent offset withdraws (non-GPU cancels credit credits-first; GPU-tagged cancels prepaid prepaid-only) so the split is preserved → assert chain balanceOf == pre-migration DB balance == post DB, exact to the cent, zero settlement drift. POST /v1/billing/husd/migrate (dry-run default, platform-only). Proven: CI MigrateOrg sqlite+fakeChain test (alice 2500c/2000p + bob 1000p → reconcile 5500==5500==5500, buckets preserved, idempotent).

Step 7 — DB-mint deletion (the mainnet cutover GATE, NOT done this session)

The named money-in paths (deposit/welcome/topup) route through the chain when enabled; the DB write stays as belt-and-suspenders (bypassed on the chain path, still guarded by #65's mintauth sink) until prod migration is 100%. Before the deletion, these REMAINING credit-mint surfaces MUST also route through chainMintCredit (else they DB-mint credit outside the chain when enabled): billing/allotment (monthly included-credit), billing/trial, billing/credit.GrantIfEligibleNow (shared auto-grant helper), api/billing/refund.go, api/billing/topup_token.go, api/billing/webhooks.go (settled-payment deposits), api/transaction/create.go (raw /v1/transaction). (api/checkout/util/capture.go credits an ORDER wallet, not the gateway-spendable iam-user wallet — out of scope.) THEN delete the DB deposit writes; the mintauth sink stays as the final backstop.

Mainnet cutover sign-off gate (separate; do NOT touch mainnet HUSD): (a) route the remaining credit-mint surfaces above; (b) provision the mainnet HUSD token + a FUNDED mainnet treasury (HUSD_TOKEN_ADDRESS, HUSD_CHAIN_ID=36963, HUSD_RPC_URL (mainnet), HUSD_TREASURY_KEY + HUSD_ORG_DERIVATION_SEED in KMS) with gas float per org address (settlement needs it); (c) run /husd/migrate?execute=true per org, assert reconcile; (d) enable the chain path; (e) run /husd/sync + /husd/settle on a CronJob; (f) after migration is 100%, land step 7's deletion. Testnet-first is complete; mainnet is the sign-off.

Config: HUSD_TOKEN_ADDRESS, HUSD_CHAIN_ID, HUSD_RPC_URL, HUSD_TOKEN_DECIMALS, HUSD_TREASURY_KEY, HUSD_ORG_DERIVATION_SEED (all KMS), HUSD_INDEX_CONFIRMATIONS (default 1), HUSD_SETTLE_THRESHOLD_CENTS (default 1). Testnet proof env: token 0xe7f1725e…0512, chainId 36962, RPC …/v1/chain/c, treasury 0xe6dad4…a51a.

Crypto deposit watcher — the half of the crypto rail that credits (2026-08)

POST /v1/billing/crypto/deposit mints a per-payer MPC custody address and records a CryptoPaymentIntent as Pending. Until now NOTHING advanced it: no component observed DepositAddress, MarkConfirming/MarkSucceeded had zero production callers, and money sent to a minted address was received and never credited. That is why cryptoDepositsCanBeCredited (api/billing/crypto_deposit.go) is false and the handler 503s before any keygen.

The watcher is now built. The gate is still false — see "before flipping".

Two packages, decomplected exactly like husdindex/husdledger:

  • billing/depositwatch — pure policy over small interfaces (no datastore, no HTTP): the scan window, the confirmation rule, the amount arithmetic, the address→intent match. Unit-tested against fakes with no chain and no DB.
  • billing/depositledger — the I/O half: per-org intent store, the idempotent ledger write, the persisted cursor (models/depositcursor), and the schedule. Integration-tested against a real datastore (util/test/ae).

Nothing in depositwatch is EVM-shaped. "Block" is whatever position a chain counts monotonically (a block, or a Solana slot) and the only property relied on is that it increases. Adding a chain adds a Reader; the policy does not move.

Live read-path probes (billing/depositledger/livechain*_test.go) are opt-in, skipped unless CRYPTO_DEPOSIT_RPC_* is set, so CI never depends on a public RPC. ⚠ -count=1 is not optional: Go caches a PASS even though every input came from the environment, so a second run against a DIFFERENT contract replays the first one's result and reports the old address as live evidence.

Chain reads come from ONE read client per chain family, both satisfying depositwatch.Reader:

  • EVM — husdindex.Client, this repo's one ERC-20 JSON-RPC read client, parameterized by (rpcURL, tokenAddr) and named for its first caller, not for a restriction. Decimals() / Symbol() selectors are pinned against luxcrypto.Keccak256 in client_test.go. Adapted to the watcher's event type by depositledger.evmReader (2 lines) because husdindex.Transfer is an ERC-20 log and predates this rail.
  • Solana — billing/solanarpc.Client (2026-08), which implements Reader directly. See "Solana" below.
  • TON — billing/tonrpc.Client (2026-08), over the TON Index (toncenter v3) HTTP API rather than a node: TON's node RPC has no per-account transaction index. See "TON" below.
  • XRPL — billing/xrplrpc.Client (2026-08), over XRPL JSON-RPC. See "XRPL" below.

depositledger.newReader is the ONE place a chain becomes a client, dispatched on Asset.Family() (FamilyEVM|FamilySolana|FamilyTON|FamilyXRPL). It is exhaustive, not defaulted: AssetsFromEnv refuses any chain absent from depositwatch.chainFamily, so an unknown chain can never be silently handed the EVM client. Family() replaced IsSolana() in 2026-08 — one enum beats N IsX() predicates, and it keeps the chain→family table the single source.

Exactly-once = a KEY, not coordination

A credit's storage id is sha256("crypto-deposit\0" + chain:txHash:eventIndex) (depositledger.creditKey). Both backends upsert on (id, kind, namespace) (db/sqlite.go, db/postgres.go), and a balance is a SUM over rows, so every attempt to credit the same transfer lands on the same row: re-scan, crash retry, and N replicas ticking simultaneously all yield one credit. No leader election, no lease, no lock — adding replicas cannot change the answer.

All three parts of the key are load-bearing:

  • chain — a pre-EIP-155 tx can be replayed across EVM chains with an identical hash, so without it two genuine deposits collapse into one credit.
  • txHash — the chain's own transaction id, folded the way that chain's identifiers compare (Asset.Fold; see "Per-chain facts" below).
  • eventIndex — WHICH value movement inside that transaction. It was logIndex until 2026-08 and is now a per-chain concept (depositwatch.Transfer.EventIndex): an ERC-20 log index on the EVM, the index of the token-BALANCE record on Solana, and it would be the vout on Bitcoin. It is 0 on TON and XRPL, and there it is a FACT rather than a placeholder — a TON transaction has at most one inbound message (in_msg is singular), and an XRPL Payment delivers to exactly one destination, so the transaction id names the event by itself. Admitting a second kind of value delivery on either (XRPL CheckCash, EscrowFinish) would require a real index. One transaction can otherwise credit the same address twice or two watched addresses at once; collapsing them swallows the second as a duplicate. The ledger metadata key is eventIndex, not logIndex — a Solana account index stored under a field named logIndex is a lie a later reader cannot detect.

The routing TAG is deliberately NOT in the key: the key names the on-chain event, and which intent it was routed to is a separate question answered by Asset.Identity.

⚠ It must be a function of the EVENT and of nothing else — never a counter over the scan's results. A key that renumbers when the window moves is a double credit. (Mutation-tested: numbering by scan position is caught.)

Ordering is load-bearing: ledger row FIRST, intent second. The reverse would let a succeeded intent survive a failed ledger write, and a later pass would read it as done — money received, never credited.

Scan ahead, commit behind

Scan [cursor+1 … head] every pass (so a customer sees "confirming" within a block); commit only head − requiredConfirmations. The last required blocks are therefore re-read from the canonical chain every pass, which is what makes reorgs harmless: a sighting that vanishes is un-sighted back to Pending (CryptoPaymentIntent.ClearSighting), a sighting that MOVES block is re-read at its new height. Un-sighting only fires for blocks actually inside the scanned range — silence about a block we never read is not evidence.

Cold start (empty cursor) begins at head − required, so no block is ever committed unscanned and no public chain is scanned from genesis.

Confirmation depth comes from cryptopaymentintent.RequiredConfirmationsForChain — the ONE policy, now naming every supported chain (eth 12, rollups+polygon 20, bsc 15, avalanche/lux/zoo 12, solana 32).

Decimals are READ, tokens are VERIFIED and PEGGED

Getting decimals wrong by 12 credits 10^12 too much, so decimals are never configured: verify() asks the CHAIN, once per asset, refuses if the reported symbol disagrees with the configured token, and refuses < 2 or > 36 decimals. A transient RPC failure is not cached. What each answer is worth differs by chain and the difference is stated, not blurred:

EVM Solana
decimals decimals() on the ERC-20 byte 44 of the SPL mint account
symbol symbol() on the ERC-20 — mandated by the standard Metaplex Token Metadata, a SEPARATE account owned by a third-party program and mutable by its update authority

So on Solana the symbol check catches a mistyped mint, which is the failure that actually happens; it cannot prove an adversary's mint is not USDC. The decimals check is the strong one and it holds on both chains. A mint with no metadata account is REFUSED rather than accepted unnamed.

⚠ An SPL token account is 165 bytes and owned by the same program as an 82-byte mint, so parseMint pins the length EXACTLY (== 82, or Token-2022 tagged at offset 165). "Owned by the Token program and ≥ 82 bytes" would accept a token account and read a byte of somebody's pubkey as the decimals.

Config (KMS-injected, NO defaults — mirrors util/husd's "token address has no default" rule):

CRYPTO_DEPOSIT_RPC_<CHAIN>            e.g. CRYPTO_DEPOSIT_RPC_BASE
CRYPTO_DEPOSIT_TOKEN_<CHAIN>_<TOKEN>  e.g. CRYPTO_DEPOSIT_TOKEN_BASE_USDC=0x8335…
                                      or  CRYPTO_DEPOSIT_TOKEN_SOLANA_USDC=EPjFWdd5…
CRYPTO_DEPOSIT_ADDRESS_<CHAIN>        POOLED chains only (XRPL). The one custody
                                      account every payer sends to, e.g.
                                      CRYPTO_DEPOSIT_ADDRESS_XRPL=rMxCKbED…

Unset ⇒ watches nothing (inert). Incoherent (token with no endpoint, unpegged token, unreadable chain, address written for the wrong chain) ⇒ Bootstrap refuses to start: an operator who configured a deposit rail must not get a process watching less than they asked for.

CRYPTO_DEPOSIT_ADDRESS_* is required on a pooled chain and refused on every other one — a shared address on a per-payer chain would hand every payer the same destination on a chain where the address is the only thing that says whose money it is. The invariant the mint path is built on: a pooled asset that exists has an account to be paid to, so pooledAddressFor returning "" means "nothing watches this token", never "watched but unconfigured".

depositwatch.pegCents IS the creditable set (usdc, usdt, rlusd @ 100c; rlusd added 2026-08 with XRPL). There is no price oracle in commerce and inventing one to value a customer's money would be guessing, so only dollar-pegged tokens are creditable — which is also why Solana was the right chain to extend to first: SPL USDC needs no oracle, exactly like ERC-20 USDC. So do jetton USDT on TON and issued USDC/RLUSD on XRPL; only BTC and native coins genuinely need a price feed. Do not add one. Native coins (ETH/MATIC/AVAX/BNB/SOL) are out on price alone, and on the EVM doubly so: a native transfer emits no log, so eth_getLogs cannot see it. Known accepted risk: a depegged stablecoin is credited above market.

Per-chain facts live in exactly one place each

Adding a chain must not scatter if chain == … through the policy. There are exactly five per-chain facts, each with one home:

fact home
how it is READ depositledger.newReader → a Reader implementation
how deep is deep enough cryptopaymentintent.RequiredConfirmationsForChain
how an address is WRITTEN (hex vs base58 vs base64url vs <CUR>.<ISSUER>) depositwatch.chainFamily + Asset.validateContract
whether its identifiers are case-significant depositwatch.Asset.Fold
what IDENTIFIES the intent (address, or address + tag) depositwatch.Asset.Identity + Asset.Pooled

Fold is ONE function for addresses and transaction ids alike, because it encodes one fact. EVM hex folds to lowercase (custody returns EIP-55, logs are lowercase — comparing literally misses every deposit). base58/base64url do NOT fold: aB… and Ab… are different Solana accounts and different signatures, and the same holds for TON and XRPL. That is only safe because those READERS canonicalise before returning — tonrpc and xrplrpc both render a transaction id as lowercase hex whatever the endpoint answered, and both echo back the CALLER's spelling of a watched address rather than the node's, so the dedup key never depends on an endpoint's rendering. An unclassified chain gets the EVM fold, which is the safe default direction — folding too much makes two addresses ambiguous and fails the pass CLOSED, while folding too little makes a real deposit invisible.

The fifth fact was added in 2026-08 for XRPL and is the only one that changed the policy half's SHAPE. indexByAddress became indexWatched, which returns two different sets — identity→intent for matching, and the distinct ADDRESSES to ask the chain about. On every per-payer chain those sets coincide; on XRPL ten thousand identities share one address.

Solana (billing/solanarpc, 2026-08)

Three Solana facts have no EVM equivalent, and each is a way to lose money silently:

  1. An SPL transfer never lands at the address the customer is shown. It lands in the Associated Token Account derived from (owner, tokenProgram, mint). The owner is NOT in the account list of a plain transfer, so getSignaturesForAddress(owner) returns nothing forever while the money sits one derivation away. The client derives and watches the ATA and reports the OWNER back, so the watcher matches the address it minted and never learns an ATA exists. The derivation is find_program_address — a real off-curve bump search, verified live at bumps 255/254/253. ⚠ tokenProgram is a SEED: a Token-2022 mint derives elsewhere, so the mint's owning program is read before anything is derived.
  2. Finality is a commitment level, not a block depth. Every read is at finalized — the point past which reverting means breaking the protocol's safety assumption, versus confirmed, which is one vote round short and can still be dropped. So a transaction the client can SEE is already final, and the 32-slot confirmation rule the watcher applies on top is pure margin inside already-final territory. That keeps ONE confirmation policy for every chain instead of a Solana-shaped exception, and makes "credited then reorged away" unreachable rather than unlikely. (Slots also skip, so slot depth ≥ block depth — conservative in the safe direction.)
  3. Amounts come from the transaction's own token BALANCE records (post − pre), not from decoding a transfer instruction. The delta is what actually arrived: it survives two transfers into one account in one transaction, a transfer in and out again, and a Token-2022 transfer fee that makes the amount received smaller than the amount sent. An account absent from preTokenBalances started at ZERO — the normal shape of a FIRST deposit, where the sender creates the ATA in the same transaction.

Three cross-checks guard every account before it is credited: the balance record must name our mint, be owned by the watched address, and report the same decimals as the mint account. An accountIndex past the end of the account list is an ERROR, never a skip — guessing there is how a deposit becomes invisible.

⚠ isOnCurve does NOT reduce y before decoding, and that is a CHECKED assumption. Solana's own check goes through curve25519-dalek, which reduces; edwards25519 happens to accept the same non-canonical encodings, so the two agree. TestIsOnCurveAcceptsNonCanonicalYAsSolanaDoes pins it (12 of the 19 y ≥ p encodings decode). If that ever changes, the bump search diverges from Solana's for ~1 candidate in 2^250 and derives an address no deposit can reach. Defensive reduction code was written first and then DELETED: no test could distinguish it from its absence, which is exactly what a surviving mutant means.

Known limitation: only the CANONICAL ATA is watched. Funds sent to a non-canonical token account owned by the deposit address are not credited (they are still recoverable — the MPC key owns them). No wallet does this.

TON (billing/tonrpc, 2026-08)

  1. There is no global block height. The basechain is SHARDED and splits and merges under load, so a shard seqno is neither global nor comparable across shards; logical time (lt) is a Lamport clock whose differences measure causality, not age, so head − lt is not a depth. The scan window is the masterchain seqno: a single global sequence that every transaction enters exactly once and never leaves (TON finalises a shard block by REFERENCING it from a masterchain block), which is exactly the property Reader documents. mc_block_seqno is per-transaction on /api/v3/transactions. Masterchain blocks are FAST — measured at 0.39 s — hence 80 confirmations ≈ 30 s.
  2. The index's jetton-transfer view is the SENDER's side. A /api/v3/jetton/transfers record's transaction_hash is a transaction on the source wallet (opcode 0x0f8a7ea5, transfer) — verified by resolving one and finding account == source_wallet. It means tokens were SENT, not that they arrived; a jetton transfer can still fail at the destination and bounce. This client credits the RECEIVING side only: a transaction on OUR jetton wallet whose in_msg.opcode is 0x178d4519 (internal_transfer) and whose compute+action phases succeeded. The jetton wallet contract validates its sender before crediting, so a forged credit FAILS on chain — the success check is what carries that consensus-enforced validation into this process.
  3. Several query parameters are SILENTLY IGNORED, not rejected. Measured: mc_seqno on /jetton/transfers (an ancient seqno returns live rows) and jetton_master on /jetton/wallets (spelled jetton_address there). They return HTTP 200 with UNFILTERED data, which reads as success. So this client verifies every response against what it asked for — account, owner, jetton — rather than trusting any filter. Never adopt a new parameter here without a bogus-value test first.
  4. Decimals are read from the master's ON-CHAIN TEP-64 content dict (USDT: 6). A jetton publishing decimals only in the off-chain JSON its uri points at is REFUSED: the scale of a credit must not come from a web server the issuer controls.
  5. 🔴 USDT on TON cannot currently be armed, deliberately. Its on-chain content carries only decimals and uri — no symbol — and the off-chain document calls it USD₮ (U+20AE), which is not usdt under any honest comparison. Fetching the URI would make token identity an assertion of the issuer's web server; a homoglyph table would be a rule with no principle. Symbol() therefore refuses, and the whole watcher refuses with it. Arming TON needs either a jetton that states its symbol on chain, or a second deliberately-pinned on-chain fact — not a weakening of the identity check.

Addresses: one TON account has SEVERAL correct spellings (EQ…/UQ…/raw), so ParseAddress decodes all of them to (workchain, hash) and verifies the user-friendly form's CRC-16 (a typo fails at boot, not at the first deposit). The workchain byte is SIGNED — read unsigned, masterchain (−1) becomes 255. Testnet-flagged addresses are refused. Two watched addresses that decode to one account are refused as ambiguous.

XRPL (billing/xrplrpc, 2026-08)

  1. 🔴 The deposit model is POOLED, and it inverts address→payer matching. XRPL charges a NON-REFUNDABLE base reserve per funded account, so a fresh address per payer strands that reserve on every deposit forever. The production model — every exchange's — is ONE custody account plus a per-deposit destination tag. Hence Asset.Pooled/Asset.Identity, Watched.Tag/Transfer.Tag, and CryptoPaymentIntent.AddressTag. An empty tag means the payment carried NONE and is NOT the tag "0" — 0 is a legal tag somebody may hold, and collapsing the two credits a stranger's payment to its holder. 1a. 🔴 The WRITE side: the tag is allocated BY THE DATABASE, and that is the only reason it is unique (2026-08, billing/depositledger.NextTag + api/billing.CreateCryptoDeposit's pooled branch). A duplicate (address, tag) does not mis-credit — indexWatched refuses an identity claimed by two intents — it HALTS the asset for every customer, so uniqueness had to be structural rather than likely.

    What guarantees it: db.Sequencer.NextSequence is ONE statement, INSERT … ON CONFLICT DO UPDATE … RETURNING, against one row. No value is ever read into Go and written back, so there is no window in which two callers see N. Postgres row-locks on conflict and re-evaluates DO UPDATE against the newest committed version; SQLite applies the upsert indivisibly. Holds across goroutines, requests and replicas (they share the row), with no lock, lease or leader election — the same "property of the storage, not of coordination" argument that makes creditKey idempotent.

    ⚠ datastore.RunInTransaction IS A NO-OP (datastore/datastore.go — "For now, just run the function directly"). It opens no transaction, ignores opts, and several call sites carry comments claiming protection it does not provide. db.DB.RunInTransaction is real but opens BeginTx(ctx, nil) = READ COMMITTED, where read-modify-write on a counter lets two transactions both read N and both commit N+1. Measured, not assumed: the mutant that wraps a read-modify-write in an explicit transaction still produced duplicate tags against a real Postgres. Put is a blind upsert, so a losing writer OVERWRITES rather than being refused. There is no other atomic allocator reachable from the model layer — every other "unique" number in this repo is a deterministic hash (which collapses duplicates, which is right for dedup and useless for allocation) or a check-then-write. orm.CreateIfAbsent exists in the interface and models/mixin/orm_adapter.go explicitly refuses it for this reason.

    The tag is COMMERCE's, not custody's. It is a routing fact — which intent does this payment belong to — not a key fact. GenerateAddress mints a FRESH MPC wallet per call by design, which on XRPL strands a reserve on every deposit, and the fleet derives no XRPL key at all (its chain switch falls through to the EVM address, so a "minted" XRPL address would be an 0x string nobody can pay). So the pooled account comes from config and the mint path calls neither cryptoProcessor nor GenerateAddress — and offeredFrom offers a pooled chain on its CONFIGURATION rather than on SupportedChains(), which answers "no" for xrpl and would otherwise hide a chain the rail can both receive and credit.

    Tags are dense from 0 and per-CHAIN (crypto-deposit-tag:<chain>), not per-asset: one account holds every currency sent to it. Density is the requirement, not an implementation detail — 32 bits is 77k tags to a coin-flip collision if drawn randomly, and 4.29e9 if counted. Exhaustion REFUSES rather than wraps: intents are watched forever (Watched filters on the asset, never on status), so a wrapped tag would collide with a live one.

  2. An untagged or unknown-tagged payment is RECORDED, not credited and not dropped. Store.RecordUnattributed → models/unattributeddeposit (system namespace, id unattributed:<chain>:<txHash>:<eventIndex>, idempotent). Crediting it means guessing with someone's money; dropping it loses a real deposit the moment the window moves. It is deliberately NOT fatal to the pass — the pooled address is published, so anyone can send one drop with no tag, and wedging on that would be a denial of service against every other customer. A failed record IS fatal, because advancing past it destroys the only evidence.

  3. Ledgers are FINAL on validation, not buried under N ledgers — closer to Solana's finalized than to EVM depth. This client reads only ledger_index: "validated" and asserts validated: true per entry, so what it can SEE is already final; the 8-confirmation depth (~30 s at a measured 3.81 s/ledger) is pure margin kept so there is one confirmation policy across chains.

  4. 🔴 Amounts are DECIMAL STRINGS, not integer base units — an issued currency has no base unit, so there is NO decimals to read. The scale is fixed at xrplrpc.Scale = 15 (XRPL's own significant-digit limit) and the safety property is different in kind but not weaker: the same function that parses the ledger's number reports the scale it parsed at, so no second source can drift from the parse. TestParseValue_RoundTripsAtDecimals pins units / 10^Scale == the stated value using big.Rat (a big.Float comparison fails on a correct parser and can pass on a wrong one). Truncation is toward zero always.

  5. 🔴 meta.delivered_amount, NEVER tx.Amount. With tfPartialPayment, Amount is the maximum the sender offered and the ledger may deliver an arbitrarily smaller fraction — the classic exploit that has drained exchanges. "unavailable" is refused, not treated as zero.

  6. The token is the PAIR (currency, issuer). A currency code is a name anybody may issue, so both halves are compared on every delivery. Identity comes from gateway_balances on the issuer — a pair the issuer does not actually issue is refused. Only Payment is ever a deposit (it is what carries a tag and delivers to exactly one destination); XRP is skipped as unpriceable.

  7. Errors ride INSIDE result ({"result":{"status":"error", …}}) with HTTP 200 and no top-level error member. A client checking only HTTP status reads a failed account_tx as an empty one — "nothing arrived" when the truth is "we could not look". api_version: 1 is pinned so the response shape cannot silently become v2's tx_json.

It is SCHEDULED, in process

depositledger.Service.Start() from Bootstrap, stopped in Shutdown before the DB closes. Interval is a constant (30s), not config. This is the FIRST in-process scheduler in this repo, and deliberately so: every other periodic job here is an endpoint waiting for a CronJob defined in another repository, which is exactly why /v1/billing/husd/sync has never once run. Safe on every replica because a pass is idempotent (above).

Scaling note

Watched() returns every minted address, in every org, with NO status filter — money sent to an expired or already-settled intent is still the customer's money. Address count therefore grows with total deposits ever taken. If an expiry sweeper is ever added, "stop handing the address out" and "stop watching it" are separate decisions and only the first is safe.

Cost per pass is NOT the same shape on both chains, and this is a real operational difference:

  • EVM — eth_getLogs filters server-side over an address SET, chunked at 100 addresses / 2000 blocks. Cost is O(addresses / 100).
  • Solana — there is no eth_getLogs. Deposits are found with getSignaturesForAddress, which takes ONE address, so cost is O(addresses) plus one getTransaction per transaction found. Deposit addresses see a handful of transactions in their lifetime so this is affordable, but it needs a DEDICATED endpoint: the live probe hit HTTP 429 on api.mainnet-beta.solana.com merely by scanning a busy wallet over 300 slots.

A history longer than 25 pages × 1000 signatures for ONE address is an ERROR, not a truncation: truncating drops the oldest signatures in the window and the cursor then advances past them, losing a deposit permanently and silently.

Before flipping cryptoDepositsCanBeCredited

  1. Provision CRYPTO_DEPOSIT_RPC_* + CRYPTO_DEPOSIT_TOKEN_* in KMS and confirm the boot log says crypto deposit watcher ENABLED (…).

  2. Restrict the offered token set. CLOSED in v1.50.19. GetCryptoOptions used to serve the MPC processor's list (btc, eth, matic, avax, lux, zoo, arb, op, base…) — which answers "can I DERIVE an address here?", never "can this be credited?" — so it walked buyers toward assets the watcher credits none of. It now projects the watcher's configured assets through offeredFrom, so the menu cannot name an asset nothing is watching, and an unconfigured watcher offers nothing rather than everything. Nothing to remember at flip time: the list narrows and widens with the config by construction.

    ⚠ offeredFrom is a PURE function and tested as one, deliberately. The first test drove the handler end-to-end and passed against a mutant serving a hardcoded list — the custody-signer probe fails in a test environment, so the handler answered 503 and the assertion never ran. Mutation-test anything that guards money; a green endpoint test can be measuring nothing at all.

  3. GenerateAddress DISCARDS the keygen wallet_id. CLOSED in v1.50.20. It returned a bare string, so the id was parsed off the keygen response and dropped — commerce minted custody addresses it held no handle to, and a credited deposit could not be SWEPT without reconciling against the node's own records. GenerateAddress returns a processor.Wallet{Address, ID} and CryptoPaymentIntent.WalletID records it at mint time, the only moment the signer offers it. The existing method was WIDENED rather than joined by a second one: two ways to mint an address is how the halves drift apart again.

  4. Still open — the creditable set is dollar-pegged tokens only. Native coins (ETH/LUX/ZOO/MATIC/SOL…) cannot be priced without an oracle, and on the EVM they emit NO log on a bare transfer so eth_getLogs cannot see them either. Same for XRP, and additionally XRPL's 10 XRP base reserve is non-refundable, so it wants ONE pooled r-address with per-intent destination tags rather than an address per payer. ⚠ "needs no new cryptography" describes address derivation and says nothing about whether a deposit can be credited.

  5. Still open — SOL/SPL and TON have no signable key. THIS is what blocks Solana, not the watcher. As of 2026-08 the watcher CAN credit SPL USDC: billing/solanarpc reads it, and the read path is proven against mainnet. The minting half is not there. frost.KeygenTaproot is BIP-340 secp256k1; its 32-byte output passes mpcd's len(EDDSAPubKey)==32 gate and base58-encodes into a real-looking Solana address the ring can never sign for. The empty eddsa_pub_key is the system failing closed — leave it until a genuine RFC 8032 ciphersuite exists in luxfi/threshold.

    The two halves are gated INDEPENDENTLY and both must be satisfied: thirdparty/mpc.SupportedChains() omits solana (so CreateCryptoDeposit 400s), and depositwatch.chainFamily now includes it (so the watcher can credit one). Configuring CRYPTO_DEPOSIT_TOKEN_SOLANA_USDC today is inert and safe: nothing can mint a Solana address to send to.

    ⚠ That independence created a NEW way to build a dead end, closed in the same change: GetCryptoOptions derived its menu from the WATCHER alone, so configuring Solana would have put a Solana button in front of a buyer that 400s after they chose an amount — the endpoint's original defect wearing the opposite mask. offeredFrom(assets, mintable) now intersects both, so an asset is offered only when it can be RECEIVED and CREDITED.

zip / ZAP native — what is real, and why billing is NOT typed yet

zip is the ZAP-native framework: a typed op (zip.Get/Post[In,Out]) is ONE value with THREE projections — REST, OpenAPI, MCP. Only typed ops populate app.ops; untyped app.Get(path, func(c *zip.Ctx) error) projects to REST and nothing else. Commerce has exactly one zip route (/_/commerce/healthz) and it is untyped.

zip is pinned at v1.8.3 (was v1.2.0). The climb v1.2.0 → v1.8.3 has exactly ONE breaking change: v1.6.0 removed App.Mount. It was a one-liner (a.All(prefix+"/*", AdaptNetHTTP(h))), so mount.go now calls that directly — same route set. v1.3.0 swapped gofiber/fiber/v3 → zap-proto/fiber/v3 (transparent; commerce never imports fiber). v1.8.3 alone adds App.Authorize and App.Prepare — v1.8.2 and earlier have NEITHER.

Why the /v1/billing surface is still gin (four independent blockers)

  1. Typed ops bind the BODY ONLY. registerTyped passes c.Body() and nothing else; for GET/HEAD it does not even read that, so In is the zero value. There is no path-param, query-param or header binding at ANY version through v1.8.3. GET /billing/invoices/:id, ?customerId=, and Deposit's X-Idempotency-Key (which makes the mint exactly-once) are all unexpressible without a wire change. Changing the money wire to satisfy a framework is backwards.
  2. The MCP projection bypasses route middleware. mcpCall invokes the op core directly, so middleware.PlatformOnly never runs. On v1.8.2 (what cloud pins) there is NO authorizer, so this is ungateable: a typed billing.deposit would be an unauthenticated mint tool. App.Authorize (v1.8.3) closes it and gates REST + MCP with one decision — but it is a single app-level SETTER, so on cloud's shared App the last subsystem to call it silently wins.
  3. The mint gate is keyed on the CONCRETE TYPE *gin.Context. Organization.Namespaced type-switches: a *gin.Context gets mintauth.WithGate; anything else is treated as an internal cron and is NOT gated (mintauth.Require returns nil when !Gated). A typed op holds a plain context.Context, so moving a ledger handler off gin silently disarms the mint invariant — no compile error, no test failure. Pinned by models/organization/namespaced_ctx_test.go:TestNamespaced_InternalCallerUngated.
  4. The tenant key is only in the gin context. GetOrganization(c) is c.MustGet("organization"). A typed op has no org ⇒ no tenant scoping.

Blockers 3 and 4 are the real work, and it is NOT a billing change: identity + datastore ctx must flow through context.Context (set once at the boundary) before ANY handler can be typed. That is the actual phase 1.

mount_test.go:TestMount_NoUngatedTypedOpSurface is the tripwire: it asserts /mcp 404s after Mount+Prepare (commerce registers zero typed ops), and fails the moment one appears. Verified to fire.

/v1/billing is DEAD through the cloud mount

mount.go calls api.Route(apiGroup) on the /v1 gin group, registering ~140 /v1/billing/* routes — but the zip mount only exposes /v1/commerce/* and /_/commerce/*. Measured through Mount: /v1/billing/balance and /v1/billing/deposit return 404 with fiber's body ({"status":404,...}, not gin's NoRoute {"error":"not found"}) — the request never enters the gin engine. The comment at mount.go claiming it wires "the full Commerce API surface (/v1/billing, …)" is false for every prefix except /v1/commerce. In cloud, /v1/billing/* is served by cloud's own clients/billing + clients/account wildcard, NOT by commerce. Do not "fix" the prefix casually: exposing 140 routes into cloud's tree collides with those, and a collision is a BOOT PANIC.

ONE ADDRESS, ONE DECLARATION. This paragraph used to say byte-identical patterns "MERGE silently, first wins". Measured 2026-08-06, they do not — zip refuses the whole program and names every conflicting pair:

zip: this program does not compose, so it has no projection:
zip: GET /v1/commerce/collection: declared by "/collection" at rest/rest.go:133
  (via root → /v1/commerce → /collection) and by "/collection" at rest/rest.go:133

Where it lands depends on the zip in the graph, and a check that only REGISTERS measures nothing: this repo pins v1.24.2, which refuses inside rest.Route(); cloud resolves v1.25.1, which accepts the second call and refuses at COMPOSITION (GetRoutes(), serve). Ask for the composition.

The live consequence: api/resources is also called by api.Route (api.go:125), so a kind added to that leaf while api.go still registers it panics this binary at boot — and the change that caused it may have been made in cloud. Moving a kind onto the leaf is a move, never an add.

schema/commerce.zap — dialect and the real compiler

schema/commerce.zap was proto-ish fiction (// comments, service/message, = 1 field numbers) that no tool could read. It now COMPILES, and is the only .zap in the fleet that does — ai, kms, vfs, iam, o11y and api/billing/billing.zap all fail with expected 'package' declaration.

  • The compiler is zapgen (hanzo/zap/go/cmd/zapgen, Go-only: -out/-single/-type-suffix), NOT zapc. zapc is the Cap'n-Proto-lineage RUST codegen PLUGIN that reads a code_generator_request on stdin; it has no generate --lang go|ts|py|rust verb. That command, documented in every .zap header in the fleet, does not exist.
  • Dialect: # comments; package REQUIRED first; only struct / interface / type (no enum); lowercase primitives (text i64 i32 bool bytes list<T>); methods are name(param: Type) returns (out: Type). The braceless whitespace form is sugar — Desugar adds the braces and auto-assigns each field's @byteOffset.
  • Verify: GOWORK=off go build ./cmd/zapgen && zapgen -out /tmp/gen schema/commerce.zap emits a BillingClient + BillingHandler + DispatchBilling ordinal router that builds clean.
  • The generated client is not committed: nothing consumes it, and the hand-rolled ZapDispatch (api/billing/zap.go) already serves that surface. Committing it would create a second, competing billing client. interface Billing mirrors exactly the five methods ZapDispatch really routes — method ordinals are wire identity, so APPEND only.

Gotchas

  • New ORM kinds MUST be registered in util/hashid/kind.go (monotonic, never reorder) or Create() panics "Unknown kind" — sbom-record=262, oss-accrual=263, B2B company=264/employee=265/quote=266/quote-message=267/approval=268, gift-card=269/gift-card-redemption=270, exchange=271, idempotency-key=272, husd-issuance=281 (chain-backed credit ledger).

Medusa parity (native Go /v1 — 1.42.41)

The /v1 model bundle (api/api.go, mounted at /v1/* by cmd/commerce

  • mount.go; the /v1/commerce/* prefix is the SEPARATE checkout/tenant surface) covers Medusa v2's admin domains natively — no Medusa/Node fork. Reference: ~/work/medusa/medusa/packages/medusa/src/api/admin/* + packages/modules/*.

Newly wired (models existed, routes were orphaned → 404 in prod)

api/api.go now calls these 5 previously-unwired sub-routers:

  • fulfillment — fulfillmentset, servicezone, geozone, shippingoption, shippingoptionrule, shippingprofile, fulfillmentprovider, POST /fulfillment/:id/ship|cancel
  • tax — taxregion, taxrate, taxraterule, taxprovider, POST /tax/calculate
  • customergroup — customergroup + /:id/members, customergroupmembership
  • apikey — publishableapikey, role, apipermission
  • notification — notification

Net-new native domains

  • Gift cards (models/giftcard, models/giftcardredemption, api/giftcard): gift-card code+initial balance; balance is a PROJECTION initial − Σ redemptions (no mutable counter to race). POST /giftcard/:id/redeem|void| + GET /balance|redemptions. Admin-gated. Redeem is idempotent (see money design below).
  • B2B (models/company|employee|quote|approval, api/b2b): company → employees (spending limits, RemainingSpendCents/WithinLimit pure fns) → quotes (RFQ, POST /:id/accept|reject, message thread) → approvals (approval.NextStatus pure transition, POST /:id/approve|reject).
  • Order exchange (models/exchange, api/exchange): return + replacement, DifferenceDueCents, POST /exchange/:id/confirm|cancel.
  • Idempotency guard (models/idempotencykey): reusable Begin/Complete for money-moving requests; wired into api/checkout Refund via X-Idempotency-Key (replay returns stored response; in-flight → 409).

Platform product catalog — commerce is the CMS SOT (1.42.41)

models/catalogentry (catalog-entry, kind 273) is the source-of-truth for the platform product catalog (Models, Vector, KMS, …) — the list docs. + the console sidebar + pricing all derive from. commerce owns the DATA (source + seed + edits); @hanzo/products (hanzoai/ui/pkgs/products) owns the SCHEMA + the iconKey→component and brandColor→css code-maps. Conform to its CatalogEntry exactly — the shape is NOT ours to change.

  • GET /v1/commerce/catalog?brand=<b> (public, no auth, on the commerce public group so it serves that exact path) → {brand, categories[], products[]}; products is the @hanzo/products CatalogEntry[] (the client reads .products, tolerating a bare array or {catalog|data|items} too).
  • CatalogEntry contract fields (models/catalogentry/projection.go Item): id (== slug, unique), name, category (EXACTLY one of the 13 canonical labels), brandColor (a swatch KEY like "violet" — NOT hex; client maps key→css), iconKey (a lucide export NAME like "Brain" — NOT a component), slug, route ("/"), docsUrl ("https://docs.hanzo.ai/docs/services/"), apiPath (/v1-prefixed), pricingId (plans/.json key OR null — emit *string, nil→null). brands is a category-DERIVED convenience, never a hand-authored filter. Extra fields (gcp/repo/admin/status/priceCents/order/ productId) are additive — the client ignores unknowns.
  • Brand scope is by CATEGORY (categoriesForBrand, mirroring @hanzo/products catalogForBrand/BRAND_CATEGORIES): hanzo=all 13; lux/zoo/pars=5 (Web3/Network/Security/Dev/Settings). There is NO per-entry brand filter — the same system-namespace store serves every brand; an entry shows iff its category is in the requested brand's set. Slug is globally unique.
  • Admin CRUD /v1/catalog/entries (+ /seed) gates on auth.IAMClaims.IsSuperAdmin() — the catalog is cross-tenant PLATFORM data in the system namespace, so an org-level admin must NOT edit it. Keyed by slug.
  • Seed: the embedded seed/hanzo-catalog.json is the @hanzo/products snapshot (hanzoai/ui/pkgs/products/snapshot/catalog.json) VERBATIM — 95 products with correct brandColor/iconKey/route/docsUrl/apiPath/pricingId. SeedIfEmpty auto-runs on first boot (count-gated no-op once populated; COMMERCE_CATALOG_SEED=false to skip). Idempotent + non-destructive — never clobbers CMS edits. Re-baseline by re-copying the @hanzo/products snapshot.
  • Namespace: platform-global (system ns), namespaced via CONTEXT (nscontext.WithNamespace) — struct SetNamespace is a no-op for queries (they read ns from d.Context), a latent gotcha api/billing/osspayout systemDB also trips (works only because its reads+writes are symmetric in default ns).

Enso's prices are the model API's, read back on every boot (catalogentry.Reprice)

Each paid Enso SKU sells at ONE fixed price set in hanzoai/ai controllers/rates.go (ensoPrices, owner 2026-10-15): enso-flash 0.33 / 1.65, enso-pro 4.40 / 22.00, enso-ultra 5.50 / 27.50 a million tokens, cache reads and writes in the same multiple. enso is the router over the three (ai v1.833.349), billed at the tier that answers and listed at a request's that states no depth, enso-flash's; enso-free is the one free model. ai lists and bills those numbers whatever a catalog row says, so an Enso price is not an admin's merchandising choice. seed/enso-models.json carries it (seed_test.go billedRetail is the second witness, moved in the same commit), and runEnsoCatalogSeed calls Reprice after the first-boot seed: each Enso row the seed names takes the seed's price, rate by rate (a missing class is added), compared as decimals so twice is once. Cost (a sync's), name, order, published, a row the seed does not name (enso-free) and a deleted row are untouched. A new SKU in the seed (enso-pro) is created only on a store the seed has never run on; production's admin adds it, or the price list (cloud apps/pricing/enso.go) appends it from ai's answer.

Model catalog — the OpenRouter cost sync (models/catalogentry/{openrouter,sync}.go)

A model IS a catalogentry (model.go): same kind, same system store, same super-admin CRUD, same margin projection. Price is a Rate VECTOR (rate.go) — cost is what WE pay (synced), price is what the CUSTOMER pays (set in admin), margin is derived. UpsertModels is the ONE write seam and enforces costOnly. This section is the PULL half: reading a live upstream on a schedule.

  • FetchOpenRouter / DecodeOpenRouter — split on purpose: the fetch is the only I/O, so every rule about what a payload MEANS is testable against testdata/openrouter-models.json (captured from live) with no network. Decode FAILS on a malformed or empty body rather than returning a partial read: a half-decoded catalog looks exactly like an upstream that dropped 200 models, and the sync's answer to that is to withdraw them, so the two must be told apart before that decision is reached.
  • Per-token → per-MTok is EXACT: dec(s).Shift(6), decimal all the way, no float. 0.000000003625 → 0.003625. A component the upstream prices at ZERO is omitted, never stored as cost "0" (which would read as 100% margin); a fully free model keeps its row with no rates — it is still a model and this catalog lists every model.
  • Vendor is the PARTY, not party+product — "Meta", never "Meta Llama". resolveVendors settles ONE name per NAMESPACE across the whole payload (per-model derivation gives "OpenAI" and "Openai" for one vendor). It votes on the upstream's OWN display prefix, so a vendor added tomorrow is named right with no edit; vendorNames corrects only the ~5 namespaces publishing no prefix. ~vendor/x-latest aliases are listed and marked floating in metadata — the id does not pin a version, so its cost can move under a stable id.

Refresh(db, serves, rows) wraps UpsertModels with the two things a scheduled PULL needs that a push of decided rows does not:

  1. Withdrawal sweep. Upsert only sees what the upstream still publishes, so on its own a withdrawn model stays listed and routable forever. Refresh withdraws the difference: Spec.Enabled=false + Spec.Unavailable = WithdrawnUpstream. MARKED, NEVER DELETED — usage rows, invoices and old conversations reference it by slug. It stays published and priced. A later run that finds it restores it, but ONLY if WE withdrew it: an operator who disabled a model by hand keeps that decision.
  2. Fail-safe. ErrEmptyUpstream — zero models is indistinguishable from an outage, and believing it would withdraw the whole catalog in one run. A failed sync writes nothing and the last good catalog stands.

The sweep is scoped by Spec.Serves, so an OpenRouter outage can never withdraw an Enso or Zen model.

freezePrice — why a new row is priced explicitly. RetailPrice falls back to cost × markup when no price is stored, which spares a 341-row long tail from hand-pricing. But a price COMPUTED FROM A LIVE COST moves whenever the upstream moves — raise their cost and every bill rises, decided by nobody. Refresh therefore writes the retail price out explicitly at CREATION only: the row arrives at cost × markup (so landing it changes no bill and nothing needs hand-pricing), and from then on the price is a stored number only a human changes, so the next upstream move surfaces as a MARGIN THAT MOVED in admin.hanzo.ai. Creation is the only legitimate moment — the row is new, so there is no customer relying on a price to disturb. NOTE: a row created by hand without a price still tracks cost through the fallback; only synced rows are frozen.

  • Route: POST /v1/catalog/models/refresh (super-admin). POST /v1/catalog/models remains the PUSH of decided rows; both funnel through UpsertModels, so the cost/price rule holds whichever endpoint a row came through. Scheduled by a CronJob curling it with COMMERCE_SERVICE_TOKEN, the same shape the contributor payout uses. It deliberately does NOT run at boot — a boot-time call to a third party is a boot hazard.
  • /catalog/entries/* is a WILDCARD, not :slug. A model's slug IS its callable id and those contain a slash; a segment param stops at the slash, so every model row would be listed and un-editable. Measured on this router: fiber's + param does not bind, * does, and it matches a single-segment slug identically (infra-tier edits unchanged). Pinned by TestAdminRoute_AddressesASlashBearingSlug, which drives the real AdminRoute.
  • Proven against live OpenRouter: 341 models created; a second run against the same payload creates, withdraws and restores nothing; halving the upstream withdrew 171 and deleted ZERO (all 341 rows still present); recovery restored 171. claude-opus-5 lands at cost 5/25 → price 6/30 per MTok, margin 16.67% on each rate; the public projection carries price and no cost. Refresh is stable on STATE — UpsertModels still rewrites every row each run, so the write count is not yet a no-op.

Money-correctness design (idempotency WITHOUT working transactions)

  • datastore.RunInTransaction (datastore/datastore.go) is a NO-OP — it runs fn(New(ctx)) with no tx/lock/isolation. db.SQLiteDB.RunInTransaction IS real (writeMu.Lock+BeginTx) but mixin.Model[T] routes to the no-op. So model-level read-modify-write is NOT atomic.
  • A read-modify-write swaps, it does not put (db.Swapper, datastore.Swap, mixin.Model.Swap): the write lands only while the stored entity's updatedAt is still the one read (insert-if-absent, a deleted row counting as absent, when it was never stored); otherwise it writes nothing and the caller reads again, at most mixin.Swaps times. One statement on both backends, so READ COMMITTED is enough and two processes over one store cannot lose each other's change. Used by refund and dispute recording (engine.RecordRefund/RecordDispute, paymentorg.RecordDispute) and by idempotencykey.Begin, whose first sighting and stale re-claim are swaps, so a guarded money move runs once across replicas; the in-process returns and lockPeriod mutexes are gone. api/transaction's lockFunds stays in-process: its routes mount only standalone (single writer), never in cloud. Pinned by TestReturnsAcrossTwoReplicasAreNeitherLostNorDoubled, TestASwapWritesOnlyOverWhatWasRead and TestAKeySightedAtOnceOnTwoReplicasRunsOnce (two store handles on one file).
  • The money-safe primitive used here: deterministic-id ledger records. A redemption/guard's STORAGE id = sha256(scope‖key) via orm.WithStringKey. Concurrent duplicate submits collapse onto ONE row via the storage ON CONFLICT(id,kind,namespace) upsert — proven (giftcard 25-goroutine same-key test → 1 debit). Balance is a projection so distinct concurrent redemptions are additive, never lost. Refund guard replays the stored response. The narrow concurrent-first-submit window is closed at the gateway (Square/Stripe honor the same idempotency key on the refund call).
  • GOTCHA — a struct field named Key SHADOWS Model[T].Key() and silently breaks mixin.Entity (.Query() returns nil entity → nil panic in Get). Name idempotency fields IdemKey, not Key. Compiler catches it only via an explicit var _ mixin.Entity = (*T)(nil) assertion.
  • GOTCHA — namespace scoping keys off CONTEXT, not the datastore struct. db.SetNamespace(ns) sets d.namespace but the SQL layer reads nscontext.GetNamespace(ctx) (db/query.go getNamespace). Production is correct (org.Namespaced(c) → nscontext.WithNamespace); tests MUST build the datastore from a namespaced CONTEXT (datastore.New(nscontext.WithNamespace(ctx, ns))), not SetNamespace, or cross-tenant isolation appears broken in tests only.
  • datastore/query/query.go ById switch: all hashid-only Model[T] kinds (no slug/name secondary lookup) MUST be in the "return not-found" case, else a non-hashid id hits default: → HTTP 500 instead of 404. 43 such kinds were missing and are now listed.

Endpoint contract note (console → /v1/commerce)

The model bundle serves /v1/product etc. (bare /v1). console2 today reaches it via a same-origin /commerce BFF proxy (app/commerce/[...path]/route.ts → commerce.hanzo.svc:8001/v1/*). To honor the "console calls /v1/commerce/*, nothing before /v1/" law without breaking the live /v1/billing/* money path, add an ingress rewrite on the console host (console.hanzo.ai/v1/commerce/* → commerce/v1/* strip) rather than re-prefixing the backend bundle.

  • Healthcheck: use curl -f not wget --spider (Gin only handles GET)
  • go-sqlite3 must be pinned to v1.14.x (replace in go.mod) — the transitive v2.0.3+incompatible fails to compile on musl/Alpine (pread64/pwrite64)
  • Meilisearch v0.35.1 changed AddDocuments/DeleteDocuments signatures
  • Production Dockerfile uses CGO_ENABLED=1 for SQLite (not CGO_ENABLED=0)
  • Global entities (Organization, User, Token) use DefaultNamespace = true by design
  • PVC is ReadWriteOnce — use Recreate deployment strategy, not RollingUpdate
  • Filter field names are PascalCase (Go struct) — auto-converted to camelCase JSON
  • Boolean false with omitempty may be omitted from JSON — COALESCE handles NULL=false

Un-fork: THIS repo is canonical; cloud imports it (v1.47.0)

Cloud's inlined fork (hanzoai/cloud clients/commerce/, no go.mod) is being retired: cloud import github.com/hanzoai/commerce + a thin subsystems adapter. v1.47.0 = content parity with the fork's last cloud-side deltas:

  • product.created/updated events (events/, api/product_events.go) — storefront loop
  • middleware/accesstoken.go: service-token checked BEFORE the IAM branch (S2S metering dispatch must never hit the per-user scope gate) + ensureIAMOrg (org from gateway X-Org-Id via svcorg.Resolve; IAM is the one org authority)
  • sessionless webhook fixes: resolveWebhookOrg uses GetOrganizationOK (no panic on provider deliveries); square provider seeds from env at init so webhook validation works with no tenant resolver in front
  • Embedded.brand (set by the mounter; surfaced in the in-process OrgConfig)
  • v1.47.1: api/api flattened to api (package api at api/; the old GAE-era api/main.go binary moved to cmd/api) + the service-token-precedence regression tests live HERE with the accesstoken fix they pin NOT ported (deliberate): the fork's GCP-style BillingAccount (Default/LimitCents/projectbinding budget cluster) — the money-of-record BillingAccount + Binding + resolveBilling chain HERE is the one design; the fork's spend-cap/budget features must be re-expressed on Binding (#70), not imported as a competing model. Also not ported: thirdparty/ethereum (dropped deliberately, LGPL), and the cloud bridges (finance-coupled metering client, in-process entitlement client, Deps adapter) — those live cloud-side.

v1.48.0 — NATIVE zip (zap-proto/fiber): gin is GONE

Whole repo runs on zap-proto/zip v1.7.5 (zap-proto/fiber underneath). No gin, no gofiber, no net/http adaptation in the serving path. Handlers are func(*zip.Ctx) error; returning the render IS the abort (no gin Abort()).

  • ONE request context: middleware.RequestContext() does c.SetContext(mintauth.WithGate(...)) at the boundary; everything reads c.Context(). GetContext/Locals("context") are DEAD. middleware.AuthorizeMint SetContexts the authorized ctx — the ledger sink reads it there.
  • Route chains are gin-order (middleware…, handler LAST) — zip ≥v1.7.4 executes them in that order (v1.7.4 fixed an inversion; chain_order_test pins it).
  • Empty leaf = group root (zip ≥v1.7.5 normPath); fiber param names are dash-free (rest.ParamId squashes dashes: kind "product-option" → param "productoptionid").
  • Co-residence contract: EmbedConfig.App *zip.App / Config.SharedApp — commerce registers routes ON the host's app (one router, one specificity space); standalone-only surfaces (healthz, legacy /admin SPA, checkout SPA catch-all, Listen/TLS) are skipped when embedded. Embedded.HTTPHandler/HTTPAddr DELETED → Embedded.Zip().
  • SPA serving: zip.Static(+WithIndex/WithFallback) for admin (ui.FS); checkout SPAHandler is a native zip handler (same security headers).
  • Test idioms: zipclient (ex ginclient) drives via the fiber adaptor with SERVER-style requests; PostForm encodes the body (fasthttp parses bytes, not req.PostForm). zip TestCtx for direct handler calls.
  • Pre-existing debt: thirdparty/reamaze verifyHMAC has an inversion bug (flagged, not fixed here).

Licensing — MIT OR Apache-2.0 (HIP-0137)

The repo is dual-licensed per HIP-0137 "One License" (hanzoai/hips, HIPs/hip-0137-one-license.md). Three root files, one declaration each, no restating:

  • LICENSE — the dual offer + SPDX id + inbound=outbound contribution clause. Points at the two texts and at NOTICE. Copyright 2014-2026 Hanzo AI, Inc. (the real span: the code dates from 2014-09-29, when it was kaching).
  • LICENSE-MIT — canonical MIT, Hanzo copyright line.
  • LICENSE-APACHE — canonical Apache-2.0, BYTE-IDENTICAL to apache.org/licenses/LICENSE-2.0.txt. sha256 cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30, 11358 bytes, git blob d645695673349e3947e8e5ae42332d0ac3164cd7. Verify with sha256sum / cmp / git hash-object — never diff (it is a shell function on the dev box and has reported "differs" on byte-identical files). Any alteration to operative text means the file is no longer that licence.

Every package.json in the tree (15 of them, all under app/) declares "license": "MIT OR Apache-2.0". There is no Cargo.toml or pyproject.toml here, and go.mod has no licence field, so the manifests are the whole surface.

NOTICE is the third-party attribution file — it scopes vendored dependencies, not this repo's own terms. Its btcd/btcec (ISC, replace/btcec) and Go stdlib (BSD-3) sections are untouched. Appended: Medusa (medusajs/medusa, MIT, Copyright (c) 2021 Medusajs), which was vendored but unattributed. The whole TypeScript frontend under app/ is rebranded Medusa — app/admin is @medusajs/dashboard (its CHANGELOG.md still carries that header), app/site is the Medusa Book (upstream Cloudinary "Medusa Book" asset URLs are still inline), app/packages/* are the corresponding @medusajs packages, and app/store derives from medusajs/nextjs-starter-medusa (MIT, Copyright (c) 2022 Medusa; its yarn.lock still resolves medusa-next@workspace:.). MIT grants sublicensing, so offering the whole under MIT OR Apache-2.0 is sound provided the upstream copyright notice is retained — that retention is what the NOTICE append is for. Upstream ships no NOTICE file, so nothing further propagates.

The Go backend is not a Medusa fork — it is an independent native implementation that reaches parity with Medusa v2's admin domains (see "Medusa parity" above). Only app/ carries upstream lineage.

That file was the upstream starter's MIT licence with line 3 changed from Copyright (c) 2022 Medusa to Copyright (c) 2024 Hanzo AI. Every other byte matched upstream. It is restored: the file is now byte-identical to medusajs/nextjs-starter-medusa's LICENSE — sha256 770a61c897d73cdeb53ff2a367d537a80eccf57b31f81e4a5fe951fecc40c632, git blob 94def62e7e4e5016c3c4af42bc426beb3ec483e8, which is the blob id GitHub reports for the upstream file itself.

The rule is "never edit an upstream LICENSE," and restoring this is not an exception to it — it is the rule being enforced. The rule exists to protect upstream attribution. This was not an intact upstream licence being altered; it was an already-altered one, altered in the single way MIT forbids: MIT conditions the grant on "the above copyright notice ... shall be included in all copies or substantial portions of the Software." A swapped holder line is a continuing breach for as long as it ships. Restoring is repair, not edit. The test to apply next time: does this change move the file toward or away from what upstream published? Toward = repair. Away = the thing the rule forbids.

This is the second confirmed instance of this failure class, after the Papermark incident (a branding pass retitled Papermark's commercial licence and swapped its copyright holder over 100% upstream code). Both were produced by rebrand-everything sweeps that treated a licence file as branding surface. Licence and NOTICE files are not branding surface.

Swept the rest of the repo for the same swap. Tracked licence files repo-wide are exactly five: the three root ones, NOTICE, and app/store/LICENSE — no others, nested or otherwise (the node_modules/ and .next/ hits a filesystem grep returns are untracked build output, 0 tracked). app/store/LICENSE was also the only Hanzo copyright claim anywhere in the vendored app/ tree: no Medusa source file was stamped with a Hanzo header. The vendored btcd/btcec under replace/ keeps its ISC notices in-file (22 of 44 files carry the header; the rest are go.mod/go.sum/README/testdata).

Per-file Go headers — all 72 now agree with the root

Every Hanzo-authored Go file carries the same two-line header, and it says the same thing the root LICENSE says:

// Copyright (c) 2014-present Hanzo AI, Inc.
// Licensed under MIT OR Apache-2.0. See LICENSE-MIT and LICENSE-APACHE.

That replaced two stale forms: // Copyright © 2026 Hanzo AI. MIT License. (71 files, understated the dual offer) and, in cmd/commerce/telemetry.go, // Copyright 2023-2026 Hanzo AI Inc. All Rights Reserved. (1 file, a proprietary reservation left over from before the relicense). Headers are what licence scanners actually read, so a header contradicting the root is the contradiction that gets found first.

The span is 2014-present — one string, everywhere, matching LICENSE, LICENSE-MIT and NOTICE. It is the true span: the code dates from 2014-09-29, when this was kaching. The old 2026 was wrong on its own terms, post-dating files that had existed for a decade. Do not reintroduce a per-file year convention — nobody maintains one, and it drifts from the root within a release.

Six of the 72 have a //go:build constraint below the header. The rewrite replaced line 1 in place rather than inserting, so the constraint keeps its position behind blank-lines-and-line-comments only — the one way this breaks silently. go vet ./... is the gate that proves it (its buildtag analyzer flags a misplaced constraint); it is green, as is CGO_ENABLED=0 go build ./....

Nothing vendored was restamped: the only other Go files carrying a copyright line are the 33 under replace/btcec, which keep their upstream btcsuite and Decred ISC notices verbatim. If a file's header is upstream's, it is not ours to restamp — git ls-files the path before touching it, and note that app/node_modules/ and app/store/.next/ are untracked build output, not repo content.

Build note unrelated to licensing: plain go build ./... fails under cgo with undefined: cgoBuildNeedsSQLiteMathFunctions from hanzoai/base. That is a deliberate compile-time guard in the dependency (//go:build cgo && !sqlite_math_functions) telling you to use the pure-Go backend or add the tag. It reproduces identically on a clean tree, and CGO_ENABLED=0 — what the Dockerfile ships — is green. Not caused by the headers.

Public repository, one root

hanzoai/commerce is public and starts at a single root commit. The history before it carried third-party credentials, customer data and old customer orgs, so it was not published: every branch, archive ref and tag the repo used to carry lives privately in hanzo-inc/commerce under refs/archive/public/*.

Test data is fictional and stays that way: the Such Tees store in models/fixtures, example.com and acme.test addresses. Never a customer org, a real person, a real address or a captured provider payload.

Custody sweep — the half of the crypto rail that SPENDS (2026-08)

The deposit watcher made money arrive and be credited. Nothing could move it out. What stood in for a spend path was five calls from thirdparty/mpc to POST {mpcEndpoint}/api/v1/transactions, asking the signer to build, sign and broadcast a chain transaction.

mpcd serves no such route. Verified against a running mpcd node: /api/v1/transactions → 404, /sign → 401, /keygen → 401. Its internal API (:9800) has exactly five routes — /healthz, /keys, /backup, /keygen, /sign. The similarly-spelled POST /v1/transactions on the node-0 dashboard server is not a builder either: it records a row and, if policy approves, broadcasts a raw_tx the CALLER already built.

Those calls are deleted, not repointed. A threshold signer that knew about chains, tokens and amounts would be a wallet; custody is safe because the component holding the key shares only ever sees a digest.

Shape

  • billing/custody — the vocabulary. Chain drafts and broadcasts; Signer turns a digest into a signature; Sweep is the twenty lines between. Draft.Digests is PLURAL from the start so Bitcoin needs no special case. Draft.Seal proves every signature belongs to the address being spent before it will assemble anything — that is where a wrong wallet id, a reshared key and a wrong-curve answer all land.
  • billing/custody/{evm,bitcoin,solana} — one package per chain family.
  • thirdparty/mpc.Sign — the mpcd client, implementing custody.Signer.
  • Transport lives with the existing readers: billing/husdindex (EVM), billing/bitcoinrpc (Esplora), billing/solanarpc. Each grew the calls a spend needs; none grew a second JSON-RPC client.

mpcd's /sign contract — the parts that bite

  • Body is {org_id, wallet_id, network, payload_hash, idempotency_key}. All five are required except wallet_id (which fails later, at share lookup).
  • There is no curve field and must not be one. The daemon derives the curve from network deliberately. key_type — which luxfi/mpc's own stale integration test and lux/wallet's custody adapter both still send — decodes into nothing and yields 400 unknown network: (empty).
  • A signing FAILURE arrives as HTTP 200 with status:"rejected". 503 is reserved for no-quorum. Trusting the status code reads an empty signature as a good one.
  • secp256k1 answers 65 bytes r‖s‖v (v ∈ {0,1}, low-S normalised per EIP-2, which is also Bitcoin's BIP-62 rule). Ed25519 answers 64 raw bytes.
  • payload_hash is hex, signed UNTOUCHED — no hashing anywhere in the daemon. secp256k1 requires exactly 32 bytes; Ed25519 takes any length, because it hashes internally and a pre-hashed payload would be a digest of a digest.
  • The idempotency key is derived from the content (sha256(tag‖org‖wallet‖network‖digest)), so a resumed sweep collects the signature it already paid for and two transfers can never collide. A random key would strand a signature after a timeout and sign a second transaction over the same coins.

Chain facts worth not rediscovering

  • avalanche and zoo can be minted and never signed for. Both are in thirdparty/mpc's mintChains; neither is in mpcd's alias table (400 unknown network). custody.curves refuses them locally.
  • BTC addresses are legacy P2PKH, and mpcd's docs are wrong. pubKeyToBtcAddress is base58check(0x00 ‖ hash160(compressed pubkey)); docs/content/docs/bitcoin.mdx shows NewAddressWitnessPubKeyHash, NewAddressTaproot and CalcWitnessSigHash. Building from the docs produces a segwit input for a legacy output. That derivation has NO test in luxfi/mpc — the only address there derived by hand and the only one with no vector — though it is correct on the live path (proved here against a mined tx).
  • btc_address is unconditionally mainnet. Version byte 0x00 is a literal; a testnet deployment still publishes 1…. custody/bitcoin.New therefore takes *chaincfg.Params explicitly.
  • XRPL is secp256k1 on this rail, not Ed25519.
  • A Bitcoin sweep has NO change output — it takes everything to one address, so a partial amount is refused rather than quietly given change.
  • Solana's address IS the Ed25519 public key, so Seal verifies directly rather than recovering.

What is proven, and how

Each chain is anchored on a signature this code did not produce:

  • EVM — digest and assembled transaction byte-identical to eth-account's for a signed USDC transfer; EIP-155's published vector recovers to its published address; live, the deployed mainnet USDC contract accepts our transfer calldata (EVM_LIVE=1).
  • Bitcoin — our signature hash verifies the signature in mainnet block 961911; our scriptSig rebuilds that block's bytes exactly; a real testnet sweep was built, signed and broadcast by this code (21bf975dd58f84b89d16f2a63d17a1379e45949fd6891ec6cbcdb4fa638368e6).
  • Solana — a finalised mainnet message round-trips through our builder byte for byte, and its signature verifies over the raw message and NOT over a hash of it; live, the mainnet runtime executed our drafted transfer successfully and, with sigVerify on, accepted our signature before failing on AccountNotFound (SOL_LIVE=1).

Not built

TON cannot be swept, and XRPL cannot be swept. TON needs cell/BOC construction (a new dependency or a hand-rolled serializer), a seqno get-call, and — on a wallet's FIRST outgoing message — the StateInit carried inline. Neither testnet funds nor an unthrottled toncenter key were available to prove it to the standard the other three meet, so it was not shipped. CRYPTO_DEPOSIT_RPC_TON

  • _TOKEN_TON_TON are armed in production; until a TON sweep exists and is proven, disarming them is the only way to stop that hole growing.

Nothing CALLS custody.Sweep yet: there is no treasury destination in config, no schedule, and no durable record of a sweep. That is deliberate — rollout is a human decision.

The Circle Gateway top-up rail is retired (2026-10-30)

billing/gateway, payment/providers/circle, models/remittance and /v1/billing/crypto/gateway (terms, submit, read) are deleted, with the gateway networks in /v1/billing/crypto/options. The rail reimplemented Circle's batched-payment protocol in Go. Paying in USDC through Circle Gateway moves to Circle's own SDK (@circle-fin/x402-batching) in Hanzo's x402 gateway service in front of cloud: a wallet deposits into Gateway once and pays per request with x402. Crypto top-ups by authorization pause until that gateway takes them. The deposit-address rail (/v1/billing/crypto/deposit) is unchanged. History: the rail is in git up to v1.50.183; hashid kinds 292 and 293 stay taken.

Commerce - LLM Context

Overview

Multi-tenant e-commerce platform. Standalone Go binary with embedded SQLite, migrated from Google App Engine. Zero GAE dependencies remain.

Live at: https://commerce.hanzo.ai | Version: v1.33.0

Architecture

Commerce App (Cobra CLI + Gin HTTP + Hooks + Events)
  |
  +-- User SQLite (data/users/{userID}/data.db) + sqlite-vec
  +-- Org SQLite (data/orgs/{orgID}/data.db) + sqlite-vec
  +-- PostgreSQL (alternative, pgvector)
  +-- Datastore via hanzo/datastore-go (analytics)

Commerce Admin

There is ONE Commerce admin: the Next static export in app/admin (@hanzo/commerce-dashboard), embedded in the Go binary (ui/dist, //go:embed) and served by it at /admin. It renders @hanzo/ui/product — the same component set the cloud console renders (DataTable, PageHeader, StatusTag, EmptyState, MetricCard, BackendStateCard, CommerceResource) — on @hanzo/gui. There is no commerce-local design system for the admin, and no second shell.

One artifact, one URL contract. next.config.ts sets basePath: '/admin' (from src/lib/basepath.ts), so chunks are /admin/_next/* and the client router resolves routes under the same prefix. A basePath-less export is built for / and CANNOT be served from a sub-path: its root-absolute /_next/* escapes the mount and its router 404s on its own pathname. That is why the separate commerce-admin static-container image is gone — it served the same bundle at a second origin's ROOT, i.e. one artifact with two incompatible URL contracts, and keeping it is what let the embedded copy rot.

The other three spellings are gone: the Vite SPA at frontend/ (deleted — its @hanzogui/admin file: target no longer exists), hanzoai/admin apps/admin-commerce (retired), and the dead Medusa React-Router tree under app/admin/src/routes (deleted along with the vendored Tailwind surface).

  • The catalog is data. src/lib/resources.ts is a plain list of rows — slug, backend noun, copy, column specs. src/lib/columns.tsx turns a column spec into a rendered column; src/app/(dashboard)/[resource]/page.tsx emits one static page per row via generateStaticParams, and the sidebar reads the same list. Adding a merchant surface is adding a ROW, never a directory. The module is deliberately component-free: generateStaticParams runs it in Node at build time.
  • One client. src/lib/commerce.ts calls the bare /v1/<kind> resources (/v1/product, /v1/order, /v1/c/user, /v1/collection, /v1/stocklocation, /v1/store/current) on NEXT_PUBLIC_COMMERCE_API_URL with the IAM bearer and X-Org-Id. Failures THROW with their status, so classifyBackend renders the honest state (402 → add credits, 403 → not enabled, 404 → not mounted) instead of a fabricated empty table.
  • Effects are injected. @hanzo/ui/product never imports a router or an auth module; src/app/providers.tsx supplies signIn/addCredits once through HostProvider.
  • IAM is native PKCE through @hanzo/iam; the active organization scopes every read and is the sole tenant selector.
  • @hanzo/ui comes from the registry at the range the console pins (^8.0.12) so the admin and the console cannot drift, and so a developer's build and an image build resolve the identical package. It was a link: to a sibling checkout that every image rewrote to the registry range before installing — an image built what nobody had built locally, which is how a missing @hanzogui/next-theme (imported by @hanzo/ui/product's ThemeToggle, declared by @hanzo/ui only as a devDependency) reached a release and failed next build there and only there.
  • @hanzo/ui declares none of the singletons it imports, so webpack must be told. Each consumer installs its own @hanzo/gui / @hanzogui/* and resolution is free to pick a transitive copy. The bundle then holds two @hanzogui/webs — two ThemeStateContexts — GuiProvider publishes to one, every shared component reads the other, and the admin dies on load with Missing theme. tsconfig paths already pinned this for the TYPE layer; next.config.ts's SINGLETONS alias now pins the RUNTIME layer to match. Symptom of a regression here: a blank admin with "Application error: a client-side exception".
  • The build is a real gate: turbo run typecheck build --filter=@hanzo/commerce-dashboard. next.config.ts no longer sets ignoreBuildErrors, and hanzo.yml runs the typecheck in CI.
  • @hanzo/commerce-ui (app/packages/ui, the vendored Medusa design system) is now a STOREFRONT-only dependency — store/ still renders it. The admin is fully off it.
  • The embedded copy in the Go binary is the SAME export: scripts/sync-admin-ui.sh copies app/admin/out into ui/dist for //go:embed, served at the /admin/* mount. ui/dist is BUILD OUTPUT — gitignored but for .gitkeep, and that sync is its ONE producer. It was tracked content until it shipped a retired Vite SPA: a committed bundle is a bundle nobody rebuilds. Run the producer before any go build, which is exactly what Dockerfile does.
  • A sub-path mount is not a constraint on the export, it is a PARAMETER of it: an export can be served from /admin precisely because it was BUILT for /admin (basePath). What cannot be served there is an export built for / — its root-absolute /_next/* escapes the mount. TestEmbedAdminServesItsOwnAssets asserts this, on content type: the static fallback answers every miss with index.html and a 200, so status alone cannot see the break.
  • Dockerfile.store builds the storefront and its workspace packages through the root pnpm/Turbo graph, then publishes the standalone server. Do not copy a partial package set or introduce a second package manager.

Three credentials, three distinct jobs — do NOT collapse them

An audit will read commerce as having "three competing auth systems". It has one identity system and two machine credentials, and they are different VALUES, not three ways to spell one:

  • IAM / OIDC — human identity. The only source of user identity. A validated principal is proven by X-User-Id, which is minted solely from a verified credential and never restored from client input.
  • COMMERCE_SERVICE_TOKEN — ONE global platform machine identity (the cloud→commerce S2S dispatch). Checked FIRST in TokenRequired, ahead of the IAM branch, so an S2S call never meets the per-user scope gate.
  • Published storefront token — a PER-ORG machine credential, minted by an org admin via POST /v1/store/token (org.AddToken("storefront", permission.Published)) and consumed as HANZO_COMMERCE_STOREFRONT_TOKEN by the storefront BFF. Org-bound, scoped, revocable by re-minting.

That third one rides the legacy per-org access-token path in middleware/accesstoken.go — GetWithAccessToken → tok.Verify(org.SecretKey) → tok.Permissions → org.Live. It LOOKS like dead duplication and is not: delete it and every storefront's checkout stops authenticating. IAM does not currently issue per-org, org-admin-mintable, revocable scoped keys, so folding this into IAM is a migration that needs that capability first — not a cleanup.

Multi-Tenancy

  • Namespace-based: Organization.Name IS the namespace
  • middleware.Namespace() sets context namespace for downstream datastore
  • rest.New() auto-applies namespace middleware unless DefaultNamespace = true
  • Dual auth: legacy access token (org-bound) + IAM JWT (OIDC/JWKS via hanzo.id)
  • Every org (incl. "platform") is STRICTLY scoped to its own name. The legacy "platform" -> "" (cross-org) namespace bypass was REMOVED (1.42.40, Red M1): it keyed cross-org datastore on an org-NAME string, not real platform-admin identity. Cross-org access gates on auth.IAMClaims.IsSuperAdmin() ONLY.

Gateway Trust Headers (2026-04-27)

commerced does NOT validate JWTs in-binary. The trust boundary is hanzoai/gateway. Only the gateway-minted X-* identity headers are trusted. Headers are stripped unconditionally on ingress to prevent client spoofing (see gateway/auth_middleware.go stripIdentityHeaders).

Directly-exposed edge (COMMERCE_EDGE_AUTH=true) — boundary mount point: commerce-api is reachable in-cluster at commerce.hanzo.svc:8001 WITHOUT the gateway, so it strips + re-mints identity itself via middleware.EdgeAuth. That boundary MUST be installed before any route group: gin applies engine.Use() only to routes registered AFTER the call. It is now installed in Bootstrap (server.go installIdentityBoundary, called before setupRoutes), so it covers /_/commerce/*, /v1/commerce/* AND the post-Bootstrap /v1 api.Route() bundle. The boundary NEVER 401s opaque service tokens (not JWTs), so the cloud-api → commerce per-org billing money path is untouched (require=false; COMMERCED_REQUIRE_IDENTITY is incompatible with the service-token path). Every identity header is stripped UNCONDITIONALLY. A client's X-Org-Id survives only as a SELECTION that tenancy grants or refuses against a verified JWT's signed membership set (below); an opaque bearer selects nothing. Regression: middleware/edgeauth_test.go, middleware/edgeauth_tenancy_test.go.

Header Written by the edge from Use
X-Org-Id the org the request acts in Org slug; namespace + scope
X-User-Id JWT sub User identity
X-User-Email JWT email Email; audit + notifications
X-User-IsAdmin authz.Claims.Sudo "true" iff a SuperAdmin — the ONE verdict
X-User-IsOrgAdmin authz.Claims.OrgAdmin(acting org) "true" iff admin of the org acted in
X-User-Owner authz.Claims.Home (orgs[0]) HOME org — identity and billing anchor
X-User-Permissions edge-derived bit.Field as base-10 int; 0 fails closed

The edge in front of commerce — cloud's SanitizeIdentity in production, authz/edge behind a gateway, middleware.EdgeAuth at a standalone one — strips every one of these from the request and writes them back from a verified token alone. X-User-IsGlobalAdmin and X-Roles are RETIRED and still stripped.

SuperAdmin is the edge's verdict, read in one place. auth.IAMClaims.Sudo is X-User-IsAdmin == "true" (iammiddleware.GetIAMClaims), and IAMClaims.IsSuperAdmin() returns it and nothing else. No org comparison happens in commerce. It used to: IsSuperAdmin was strings.EqualFold(homeOrg(), "admin"), with homeOrg falling back to the owner claim (the MINTING APPLICATION's org), and the money bits (Admin|Live) were granted on X-User-Owner == "admin". Cloud writes X-User-Owner for an application (its own org) and an API key (its holder's org) as well as a person, so an admin-org application or an admin-org person's sk- key could mint money and edit the catalog here while being nobody in the rest of the same process, and an org spelled Admin would have passed the fold (IAM refuses to store one: schema.OrgName). No machine is SuperAdmin (owner decision D1, until answered: no). middleware.PlatformOnly, IsPlatform, MayMintMoney, MayReadPlatform and every catalog/plan/rate/promo/invite gate read the verdict. The standalone EdgeAuth decides it with the one predicate, IAMClaims.Authz().Sudo() over the verified token's signed orgs/type, and writes X-User-Owner from Home().

Org-admin vs SuperAdmin (anti-conflation): IAMClaims.IsAdmin is the org-level admin of the org the request acts in (X-User-IsOrgAdmin), for org-scoped RBAC only. middleware.RequireAdmin is the LOOSER org-money gate — IsAdmin || IsSuperAdmin() — and is never the cross-tenant one. Never gate platform work on IsAdmin or on an org-mintable role NAME like "superadmin". Tests: auth/superadmin_test.go, auth/sudo_test.go, middleware/iammiddleware/sudo_test.go, middleware/platformonly_test.go, middleware/edgeauth_test.go.

EdgeAuth tenancy (middleware/edgeauth.go)

At the standalone edge (COMMERCE_EDGE_AUTH=true) tenancy resolves a verified token to two orgs by the rule cloud's SanitizeIdentity applies, so one token is one tenant whichever edge commerce sits behind:

  • X-User-Owner, the HOME org: a person's orgs[0] (authz.Claims.Home).
  • X-Org-Id, the org the request ACTS in: the client's selection (X-Org-Id, or ?org=<slug> on a /billing/ path, consumed either way) when the signed set admits it, else the default (authz.Claims.EffectiveOrg). A SuperAdmin may select any org; anyone else an org the set opens, never the admin org, a brand's directory, or a platform org held as a plain member.
  • A program (type: application) is its application: both are the owner claim, and it selects nothing. A person's token without orgs acts in no org.

The owner claim of a person's token is the org of the application they signed in through (the brand app every customer uses), and decides nothing: it used to set X-Org-Id and the billing subject, so every self-serve user signing in through the brand app acted in and billed to the brand's org. /billing/ requests pin user/userId/customerId (query and JSON write body) to the acting org. Tests: middleware/edgeauth_tenancy_test.go.

Call sites read identity via:

  • pkg/auth.OrgID(ctx) / UserID(ctx) / UserEmail(ctx) (preferred)
  • iammiddleware.GetIAMClaims(c) returns a real *auth.IAMClaims populated from headers (Owner=X-Org-Id, Subject=X-User-Id, Name=X-User-Id, Email=X-User-Email, IsAdmin=X-User-IsAdmin=="true", Roles split from X-Roles). Never nil; do not add nil-guards downstream.

Webhooks — the platform's, not commerce's

An org's webhooks are cloud's /v1/webhook (apps/webhook: HMAC-signed, retried, every platform event). Commerce's own per-org webhooks are being retired:

  • No route creates or edits a row (the /v1/commerce/webhook CRUD is gone).
  • The legacy sender (models/webhook/tasks) posts {"data": ...} and nothing else — the hook's AccessToken used to ride in every body — through hanzoai/outbound, so a subscriber URL is never a private address.
  • What it reaches, measured (TestEmitReadsOnlyTheSystemStore): rows in the SYSTEM store, which are sent every org's events; never a row in an org's own store, where the CRUD wrote every row (it set a namespace on a datastore whose queries ignore one). So CRUD-made rows have received nothing.
  • models/webhook/census reads both stores without creating anything, and Embed logs it once per boot: one commerce: legacy webhook subscriber line per row (store, org, id, host, a digest of the URL, events, enabled, live — never a credential) and a commerce: legacy webhook census total. Read it with kubectl logs before the sender, the model and the census are deleted.

Key Directories

commerce/
  cmd/commerce/    CLI entry point
  commerce.go      Main app framework
  db/              SQLite, Postgres, Datastore backends
  hooks/           Hook system (Base-compatible): Hook[T], TaggedHook[T], Resolver
  events/          Unified event forwarding to Datastore/Insights/Analytics
  insights/        Hanzo Insights integration + Gin middleware
  api/             HTTP handlers (store, cart, analytics, namespace, etc.)
  models/          Data models
  middleware/      HTTP middleware (auth, namespace, IAM)
  infra/           Infrastructure clients (Redis, Meilisearch, etc.)

Running

go run cmd/commerce/main.go serve --dev     # Development
./commerce serve 0.0.0.0:80                 # Production

Environment Variables

Variable Default Notes
COMMERCE_DIR ./commerce_data Data directory
COMMERCE_SECRET change-me-in-production Encryption secret
COMMERCE_HTTP 127.0.0.1:8090 Listen address
REDIS_URL - redis://[:pass@]host:port[/db] (priority over VALKEY_URL)
DATASTORE_URL - Datastore DSN
INSIGHTS_ENABLED false Hanzo Insights product analytics
ANALYTICS_ENABLED false Umami-like web analytics
KMS_URL - Standalone only: Hanzo KMS server; unset = no secret reader
KMS_ORG - Standalone only: the org the KMS identity belongs to (required with KMS_URL)
KMS_CLIENT_ID / KMS_CLIENT_SECRET - Standalone only: the KMS machine identity, exchanged at HANZO_IAM_URL
SQUARE_WEBHOOK_SIGNATURE_KEY / SQUARE_WEBHOOK_URL - The deployment's Square webhook subscription key and its exact notification URL

Analytics Endpoints

Endpoint Method Purpose
/v1/analytics/event POST Single event
/v1/analytics/events POST Batch events
/v1/analytics/identify POST User identification
/v1/analytics/ast POST astley.js page AST (JSON-LD)
/v1/analytics/pixel.gif GET Pixel tracking
/v1/analytics/ai/message POST AI message event
/v1/analytics/ai/completion POST AI completion event

SaaS Metrics God-View (api/metrics, 2026-07)

GET /v1/commerce/metrics/saas — the whole-business SaaS operations snapshot for admin.hanzo.ai's "SaaS Metrics" board. ONE cross-org walk (proven all-orgs pattern from api/billing.RunBillingCycleAllOrgs), cached 45s, folding every org's subscription ledger (models/subscription) + api-usage transaction ledger into: revenue (MRR/ARR, new/churned MRR, MRR by plan category from the embedded @hanzo/plans catalog), subscription mix (per-plan active/trialing/seats + a recent create/cancel events feed), metered pay-as-you-go revenue totals, and top customers. Params: window (7d/30d/90d/mtd/all), limit (customers cap). Bundle-child subs (ProviderType "bundle") are excluded from counts/MRR. Upgrades/downgrades are null (plan-change events are not written to models/billingevent).

Gate: middleware.RequirePlatformAdmin — the SINGLE cross-org read gate (global admin OR trusted service token; org-admin refused). api/costs.requireCostsAdmin delegates to it, so the two god-views share ONE gate definition. The route-level TokenRequired(permission.Admin) is a no-op on the IAM path — the in-handler gate is the boundary. The console reaches it through its OWN SuperAdmin-gated proxy (app/admin/saas) forwarding COMMERCE_SERVICE_TOKEN.

Deliberately NOT here (owned by the fleet o11y god-view GET /v1/admin/o11y): per-model LLM tokens/latency/error and the per-org AI-spend ranking. Commerce names those in the response gaps[] array, never fabricates. Per-model latency is captured nowhere in the stack; per-model error rate is in Datastore cloud_usage.status but not yet surfaced by the o11y topModels SQL.

Dependencies

Core: cobra, go-sqlite3, gin, hanzoai/datastore-go Infra: go-redis/v9, hanzos3/go (the house S3 client — package minio, republishes storage-go under the s3-go import path), meilisearch-go, nats.go, temporal SDK Vector: Qdrant via REST/HTTP (port 6333). No gRPC, no vector-go SDK -- plain net/http + encoding/json.

Secrets: the host's KMS, one reader (secrets)

Commerce holds no KMS client. It reads through secrets.Reader, which the host installs at embed time (EmbedConfig.Secrets): the cloud binary passes its own KMS handle (deps.KMS, KMSPeer over the internal plane), and the standalone binary (cmd/commerce) passes hanzoai/kms/sdk/go as its machine identity (KMS_URL, KMS_ORG, KMS_CLIENT_ID/SECRET, HANZO_IAM_URL). thirdparty/kms is deleted: it read /v1/kms/orgs/<org>/secrets/..., which cloud stopped serving, so per-org payment credentials never loaded — and it was off in production anyway (KMS_ENABLED unset).

One address. An org's own material is orgs/<org>/<provider>/<NAME> in the default env — cloud's OrgPath and the convention every in-process reader uses. secrets.Org(ctx, org, name) is the only way to read it, and it asks AS org: cloud's KMS (apps/kms admit) serves a tenant ref only to a call acting for that tenant, and a request's caller is the PAYER when a customer pays a brand, so the read re-points that caller with zip.ActingAs (the call names who is acting; the KMS writes no audit row of it). A raw handler passes c.Forward(), which carries the request's caller; its c.Context() carries none. A call with no caller reads nothing — cloud's KMS answers an unstated call as its own brand. org must already be a namespace.Sanitize-clean slug. The ref carries no leading slash, so the tenancy check runs. /tenants/<org>/<provider> was the old client's layout and nothing reads it now.

Who loads which credentials. payment.Hydrate(ctx, org, providers...) fills an org's integration fields from those refs. It is called where the org RECEIVES the money: storefront checkout (api/checkout), storefront subscriptions (api/subscription), storefront wire instructions, and the billing wire rail, where the receiving org is the brand that owns the host (orgs/hanzo/wire/* for pay.hanzo.ai). Hanzo's own billing (api/billing top-ups, plans, invoices, renewals, auto-recharge, saved cards) never hydrates the PAYING org: its Square account is not where that money goes. That path charges the deployment's own account through ProcessorsForOrg's environment fallback (SQUARE_*, KMS /commerce-secrets → KMSSecret → env). Moving it onto the brand org's own credential is an open owner decision.

Nothing is cached. The store is the process next door; a rotation lands on the next read. commerce seed no longer copies env credentials into KMS — a second copy of a secret is a second place for it to go stale.

Square webhooks: one route, one verifier

  • Route: POST /v1/commerce/webhooks/:provider, declared once in setupRoutes on the public /v1/commerce group, so standalone and embedded serve the same address from one declaration (cloud no longer declares it). /v1/billing/webhooks is gone. The URL is part of Square's signature: it moves only with the Square subscription (payment/orgsetup.go defaultSquareWebhookURL).
  • Verifier: payment/providers/square ValidateWebhook, reached through processor.Global(). thirdparty/square is the API client only — no HMAC, no init registration, not a PaymentProcessor — so no second Square type can be registered. Every registry holds *square.Provider (ProcessorsForOrg too).
  • The path names the processor, and it must hold its secret. Only the processor registered under :provider verifies a delivery, and only if it is a processor.WebhookReceiver that ReceivesWebhooks() — it holds the secret its deliveries are signed with (Square: key and URL; MPC: MPC_WEBHOOK_SECRET). Every provider package registers itself configured or not, and some verifiers need no secret (Solana Pay's chain lookup), so asking one of them would let a caller author an event; an unknown name or a secretless processor is refused. The receiver set is pinned to {mpc, square} by TestTheWebhookReceiversAreTheSecretHolders. payment/router, a second dispatcher that tried every processor, is deleted.
  • A clawback stays in the verifying account's org. When the event names an org (a per-org account) and the payment's receipt is another org's, the refund or dispute moves nothing (RECONCILE).
  • No freshness window; the event id is the replay guard. Square keeps an event's created_at across every retry, so the old 5-minute window refused every retry of an event first sent during a deploy, forever. Each verified event is recorded once for the whole deployment — idempotencykey in the system namespace, scope webhook, key <processor>:<event id>, an insert-if-absent swap — so a repeat is acked duplicate:true (or 409 while the first is still being applied) whatever path or org it is presented under. Every money posting is also keyed on the payment, refund or dispute id. A signed body with no event_id is refused.
  • An effect that did not land leaves the delivery open. The settlement, clawback and subscription steps return an error when a store or the ledger did not answer; the handler then answers 503 without completing the guard, so the provider redelivers and the guard is re-claimed after StartedTTL. Every effect is safe to apply twice. Unattributable money, and a receipt naming an invoice these books do not hold, are not errors: RECONCILE, refused, acknowledged. The org's billing_event audit row is written once, by the delivery that applied the event.
  • The books are the verifying account's. bookedOrg reads processor.WebhookEvent.Org (the org the Square account belongs to; the deployment's env-seeded account names none) and falls back to checkout.DefaultBrandSlug(). No header or session picks the org: a provider sends no identity, and one on a replay is the replayer's.
  • Accounted for, wherever: settlementDestination reads the payment index (models/paymentorg) first. An in-session charge writes its receipt in the PAYER's org and indexes it; the delivery resolves the processor account's org, so without the index every such payment raised a false RECONCILE.
  • Production (2026-10-19): one Square subscription, wbhk_44c9077bfece4c0c96a463ac9fe56e0f, created 2026-09-23, notification URL https://pay.hanzo.ai/v1/commerce/webhooks/square; its key is KMS org hanzo /commerce-secrets/SQUARE_WEBHOOK_SIGNATURE_KEY@prod (measured equal to Square's). Square's Events API holds nothing to replay: event search was never enabled on the account.

Checkout Sessions

POST /v1/checkout/sessions — AUTHENTICATED hosted-checkout mint (Square Payment Link). Same publishedRequired gate as every sibling checkout route (TokenRequired(Admin, Published)): a service token, a per-org Published storefront token, or an IAM principal. There is no anonymous path — an unauthenticated POST is 401 BEFORE any org resolution or Square call.

An opaque (non-JWT) bearer never selects an org. EdgeAuth (middleware/edgeauth.go) strips a client X-Org-Id and puts nothing back. TokenRequired's IAM branch ENFORCES the requested masks (len(masks)==0 || hasScope(c, permissions)), and IAMTokenRequired requires a validated principal (X-User-Id), not merely an X-Org-Id selector — the same predicate as cloud's clients/principal.Validated, needing no env flag. Regression: middleware/edgeauth_test.go, middleware/accesstoken_test.go, middleware/iammiddleware/authz_test.go.

Request: { company, providerHint, currency, customer, items, successUrl, cancelUrl, couponCode? } Response: { checkoutUrl, sessionId }

Security invariants (do not regress):

  • One org source: the org is SOLELY the authenticated principal's (authedOrg(c) = middleware.GetOrganization + KMS hydrate). There is NO request-body org/tenant field and NO ?org/X-Org-Id/X-IAM-Org override on the mint path — a caller can only ever mint for its OWN tenant.
  • Fail-closed creds: an org without its own per-tenant KMS Square creds cannot mint live — squareCheckoutClientForOrg has NO production env/platform fallback (isPlatformOrg removed). Only SANDBOX keeps the env fallback (no money at risk), behind auth, so the per-org sandbox storefront still mints.
  • Redirect allowlist: successUrl/cancelUrl are bounded to the org's own domains via checkout.AllowedCheckoutRedirect (org Websites + brand first-party hosts + brand registrable domains) — closes the open-redirect / phishing pivot on the minted link. Enforced for authed callers too.
  • Item prices are server-authoritative from the org's per-org catalog (catalogPrice); the client unitPrice is ignored. Emits checkout.started.

The legitimate storefront BFF (server-side, holds the service token or the per-org Published storefront token — mint via POST /v1/store/token) is the ONE reachable checkout entry. A browser must never call this endpoint directly with a client-chosen org.

SQLite Query Engine (2026-02-23)

Critical: PascalCase→camelCase conversion. The SQLite query builder converts Go struct field names (PascalCase) to JSON field names (camelCase) automatically via toJSONFieldName(). This is needed because json.Marshal() uses json struct tags (camelCase), but callers use Go field names in Filter()/Order() calls (legacy from Cloud Datastore migration).

  • Filter("Test=", false) → json_extract(data, '$.test') = 0 (not $.Test)
  • Filter("DestinationKind=", kind) → json_extract(data, '$.destinationKind')
  • Nested paths handled: Filter("Account.TransactionHash=", h) → $.account.transactionHash
  • Boolean false/0 handled via COALESCE: COALESCE(json_extract(...), 0) = 0 (handles NULL from omitempty)

Data directory: {COMMERCE_DIR}/orgs/{orgName}/data.db per org. System data in orgs/system/data.db.

Tenant store lifecycle lives in orm/db.Registry, not here. db.Manager used to hold a userDBs and an orgDBs map, opening a store per tenant ever touched and closing them only at shutdown — descriptors and memory grew with the tenant count, with no bound and no replication. It now delegates to ormdb.Registry[db.DB]: open on demand, bounded by Config.MaxOpenTenants (256), idle stores closed after Config.TenantIdleTTL (5m), coldest evicted first, never one that is mid-operation.

Registry.Do(ctx, tenant, fn) scopes a handle to a callback, on purpose — a handle you can keep is one someone must remember to give back. Manager.User/ Org still return a db.DB because every call site keeps one (Store.DB, the namespaced datastore resolver, the user service), but what they return holds no handle: db/tenant.go is a VALUE that borrows the tenant's store for each operation and gives it back. So a held db.DB costs four words, never a file descriptor, and two consecutive operations on it may run on different open handles — nothing needs otherwise, since anything atomic across statements is already inside RunInTransaction (one borrow).

Registry's Materialize (restore a tenant file from object storage on a local miss) and OnOpen/OnClose (bracket a handle's life) are where hanzoai/replicate attaches. Both are unset today: commerce tenant files are still local-only, and turning them on needs an object-store config, not more code here.

PVC: commerce-data (10Gi, do-block-storage) — deployment uses Recreate strategy (not RollingUpdate) because the PVC is ReadWriteOnce.

At-Rest Encryption (SQLCipher via hanzoai/cek)

Per-tenant SQLite files hold money data (balances, transactions, usage), so they can be encrypted at rest with github.com/hanzoai/cek on top of github.com/hanzoai/sqlite v0.5.0 (SQLCipher AES-256) — the SAME derived-key scheme cloud and IAM use, so there is one at-rest model platform-wide. Code: db/encryption.go.

  • Master key from KMS, ONE source: COMMERCE_KMS_MASTER_KEY (64 hex), read ONLY by ResolveMasterKey(). Injected from KMS into the environment, the same mechanism as HUSD_TREASURY_KEY. Never in git/code.
  • Derived, not wrapped: a tenant's file key is cek.DeriveKey(master, ns, "commerce") — HKDF over the master, the tenant's hanzoai/namespace name and the subsystem. It is a pure function of the tenant's name, so there is NO DEK, no sidecar, no unwrap step, no rewrap and no per-file key material to lose. A file reopens after a restart with nothing persisted beside it. Losing the master loses the data, which is the property at-rest encryption is for.
  • Why DeriveKey and not cek.Open: commerce opens a DUAL pool — a concurrent read pool and a serialized single-connection writer — against ONE file under ONE key. cek.Open hands back a single *sql.DB. The key is the shared part; the pool is commerce's.
  • Posture decided once: unset key → unencrypted (dev/CI). Set + codec linked → encrypted. Set + cgo-but-no-libsqlcipher → refuse (CodecLinked() probe), never silent plaintext — commerce's dual pool cannot use the single-writer codec envelope a non-libsqlcipher build falls back to.
  • Layout: an ORG's store is <DataDir>/orgs/<slug>/commerce.db, placed by namespace.Path, so the file's key and the file's path are two renderings of one name. This is the layout every Hanzo service shares.
  • Per-USER stores are UNENCRYPTED, and this is a real limit: namespace.Key refuses KindUser — the shared layout has no place for a user — so a per-user store cannot be named, and therefore cannot be keyed, in this scheme. An encrypted open of a user tenant fails closed with an error naming the limitation (TestEncryptedUserTenantIsRefused); the unencrypted path is unaffected. Per-user files stay at <UserDataDir>/<id>/data.db. Giving hanzoai/namespace a user layout is the prerequisite for closing this.

No migration. Databases written under the previous DEK-sidecar scheme do not open under this derivation, by design — there is no fallback, dual-read, version probe or config flag, because a second way to key a file is how two of them end up disagreeing. Wipe and recreate.

Rollout: build the image with libsqlcipher linked — add libsqlite3 to the build tags, CGO_CFLAGS=-DSQLITE_HAS_CODEC -DSQLITE_USE_URI=1 -I/usr/include/sqlcipher, CGO_LDFLAGS=-lsqlcipher, apk add sqlcipher-dev (builder) + sqlcipher-libs (runtime), and a test stage with SQLITE_REQUIRE_CODEC=1 so a mis-link fails CI — then supply a 32-byte key.

Where the key comes from: an EMBEDDER passes EmbedConfig.MasterKey (cloud hands over the KEK it already resolves, so one process has one key, not two); a STANDALONE reads COMMERCE_KMS_MASTER_KEY. One source per deployment shape, never both.

Billing API (2026-02-23)

Endpoint Method Purpose
/v1/billing/balance GET Balance by user+currency (cents)
/v1/billing/balance/all GET All currency balances
/v1/billing/usage POST Record API usage (withdraw)
/v1/billing/deposit POST Create deposit transaction (per-subject money-in / settlement)
/v1/billing/credit POST THE ONE mint-gated, org-keyed credit grant (see below)
/v1/billing/zap POST Clear balance

All require permission.Admin token (org live/test JWT). Cloud-api connects via commerceEndpoint + commerceToken env vars.

The invoice PDF (api/billing/invoice_pdf.go, pdf.go)

InvoicePDF(ctx, org, id, brand) renders one invoice; brand is the slug of the brand that bills (hanzo, lux, zoo, pars; anything else is hanzo). Cloud passes the caller's brand; the standalone route reads it from the request host (checkout.BrandSlugForHost). The page is US Letter in Zen Regular and SemiBold (embedded TrueType subsets in fonts/, OFL, credited in NOTICE) and two inks: the brand's mark (vector paths from @hanzo/logo, @hanzo/brand and @zooai/logo) and name top left, the number and a status pill top right (PAID is the one solid pill), billed to / issued / due or paid / period, the amount in words, a ruled line-item table, totals right-aligned, and the brand's legal name and address at the foot (Hanzo Industries Inc., 995 Market St, San Francisco, CA 94103; Lux and Zoo state none and close with their name and site). Long invoices continue onto numbered pages. Text is Windows-1252 (accents kept, other scripts read "?") and uncompressed, so the number and amounts stay greppable; the output is a pure function of the row. Tests: invoice_pdf_test.go.

One way to price a plan — the published catalog, applied by a DEPLOY

The plan catalog is @hanzo/plans (npm, public). scripts/fetch-plans.sh vendors it into api/billing/plans/*.json at build time, go:embed bakes it in, and plan.Seed reconciles the live models/plan rows to it on EVERY boot. So changing a price is: edit the package → publish → re-vendor + bump PinnedPlansVersion → bump commerce → bump cloud's go.mod → deploy. Every step versioned, reviewed and revertible.

There is no second way. cloud/scripts/seed-plans.sh (a human curling production with a SuperAdmin bearer, with the ladder retyped in bash) is DELETED — it was a second statement of the ladder kept in step by hand, which is exactly how the pricing page came to publish Pro at $49 while billing charged $20.

The boot seed CAN change what a plan charges. An older comment claimed the opposite ("seed values EQUAL the embed, so it changes NO charge"); that stopped being true when the seed gained the ability to correct its own prior output.

Why the seed could not correct itself (and what fixed it)

Managed was set by BOTH plan.Seed and api/plan's Create/Update, so one bit carried two facts — "authoritative" and "who decided it" — and if existing.Managed { continue } skipped every row the seed had ever written. Publishing a new catalog would therefore CREATE the plans that were missing and LEAVE every plan that had changed at its old price: half-new, half-stale.

AdminEdited is the discriminator that was missing. ONLY api/plan sets it. Seed reconciles anything else to the catalog — the rule everyone already assumed held: the package decides, an admin edit overrides. Proven end to end by api/billing/plans_converge_test.go, which plants the real production catalog (Managed rows, retired tiers, old prices) and boots it against the published one.

A draft catalog row is created unlisted (plans 1.8.17)

The boot seed makes a catalog price chargeable the moment a deploy carries it, so the catalog can publish a rung as "status": "draft" (max-50x $500 and max-100x $1,000 are). plan.Seed reads the catalog's status in BOTH directions: a draft row is created unlisted, a row already where the catalog says is left alone, active makes a row sellable and draft takes a seed-written row back off sale. An admin edit still wins (AdminEdited). A draft still resolves (lookupPlan, resolveSubscriptionPlan) so a renewal or invoice could price itself; it is on sale nowhere: planAuthorityRows, the embed fallback (listedCatalog), StaticPlans and the three purchase entrypoints all go through Listed() / the draft filter. Activating a rung is one edit in @hanzo/plans (delete the status line), a release, a re-vendor and a deploy, and it needs the owner's word because it makes a live price chargeable.

Lifecycle: archive, never delete (the Shopify product model)

plan.Status is active | draft | archived, and empty means active — every row written before the field existed has no value for it, and a zero-value meaning "draft" would blank the catalog on deploy. plan.Listed() is the ONE predicate, gating both halves of "on sale":

  • the public catalog (planAuthorityRows skips unlisted rows), and
  • the three purchase entrypoints — CreateBillingSubscription, the PATCH plan-change, and the card subscribe — which 404 an unlisted plan. Hiding a tier from the page is worth nothing if the API still sells it, and the card path refuses BEFORE the charge.

resolveSubscriptionPlan is deliberately NOT gated: it answers "which row is this" and must keep answering for an archived plan, or a renewal on a retired tier could not price itself. Retiring stops new sales; it never strands a subscriber. Seed archives what the catalog stopped publishing; an admin-created plan is exempt (it is not "missing from the catalog", it is theirs).

The wire shape IS the model shape

A plan's display envelope (features/limits/bundles/includedIn) used to live only as a packed JSON string in Metadata, with the packer private to api/billing — somewhere the admin handler could not reach. So PUT /v1/plans/entries/:slug carrying features set the price and SILENTLY DISCARDED the copy: a tier could be repriced but never re-described. Those fields are now first-class on plan.Plan (JSON-visible, datastore:"-", still persisted packed under the same one key), so whatever GET /v1/billing/plans emits, the admin CRUD accepts. planLimits is an ALIAS of plan.Limits, not a second declaration.

Gotchas this cost

  • A row from GetAll is a VALUE with no datastore binding. Update() on it writes nowhere AND RETURNS NO ERROR. Re-load through the bound point query (New(db) + Query().Filter("Slug=",…).Get()) before writing.
  • Save() must pack into Metadata, not Metadata_. The ORM stores a row as JSON of the struct and Metadata_ is json:"-", so a value written only there does not survive. Measured, not assumed.
  • paidTier counts a contact-sales plan as paid. It stores Price=0 because its price is null, not free; reading price alone made a negotiated tier self-serve, so a catalog row with a real allotment could be minted with no payment. That never fired before only because the tier with the large allotment also carried a large price — luck, not a gate.
  • A cgo build needs -tags sqlite_math_functions (hanzoai/base asserts it at compile time). Without it v1.49.26–v1.49.37 each pushed a release tag and published NO image — twelve dead releases.

The seed is only as good as what it compares

planEqual/limitsEqual decide whether the seed writes at all, so a field copied but not compared is invisible: the stored row and the published catalog disagree, the seed reports success, and nothing logs. copyInto's comment has always said the two lists must match; nothing enforced it.

TestEveryLimitCopiedIsAlsoCompared walks the struct by reflection, so the next field added to Limits fails immediately instead of two deploys later.

The catalog decides whether a plan is sold, in BOTH directions. The seed archives a slug the catalog stops publishing; it now un-archives one the catalog names again. Without the second half a retired tier could never come back — Status is carried by neither copyInto nor planEqual, so a republished row reconciled to the new values and stayed invisible. That is what happened to dev: retired at plans 1.4.16, republished at 1.4.20, still absent from the live catalog. An admin's own unlisting still wins, because AdminEdited is checked first — the package is the source, a human edit is an override.

OPEN: the list and the enforcement read different sources

ListPlans/GetPlan read planAuthorityRows — the DB, where an admin edit lands. Everything that ENFORCES reads lookupPlan, which walks the Go catalog embed: tier resolution, trials, DNS usage, the plan-move score and the rest of plans.go. Many call sites, one source, and it is not the one the admin writes to.

So an operator editing a plan's limits at admin.hanzo.ai changes what a customer is SHOWN and not what they GET. Which of the two should win is a real design question — this file's own heading says a plan is "applied by a DEPLOY", which argues for the embed; the AdminEdited override argues the other way — and it should be answered deliberately rather than by whichever call site a reader happens to open.

A team mixes seat types on ONE subscription (plans 1.8.20)

Team is sold in two seat types one team can mix: team (standard, $25 or $240 a year) and team-premium ($100 or $960). The catalog's group ("team" on both) says they are seat types of one team, and its minSeats (2) and maxMembers (100) count every seat: 1 standard + 1 premium is a team, 1 alone is not.

The model is one row, not one per seat type: Plan/Quantity are the first seat type and subscription.Lines the others (Lines, each with its slug, price for the row's period, quantity and pending quantity). So one charge pays the period, one invoice carries one plan line per seat type (buildPeriodInvoice), one renewal moves them together (pending quantities included), and "one paid subscription per subject" (billingSubscription) and the tier resolver still see the team as one row. Two rows would have needed a second notion of which subscription governs the subject, two charges that can half-fail, and a minimum that a single cancel breaks.

  • Buy: POST /v1/billing/subscribe/card {"planId":"team","quantity":1, "lines":[{"planId":"team-premium","quantity":1}],"interval":"month"|"year"}. linesAt refuses a line outside the plan's group, the plan itself, a type named twice, zero seats, or another currency, before any charge; teamSeats is the floor and ceiling over every seat. A different mix is a different idempotency fact.
  • Change seats: PATCH lines, or ChangeSeats (cloud's plane op), pending to the renewal as quantity is. A seat type of the group the row does not hold yet is ADDED (nextLines, priced by linesAt at the row's period): it holds 0 seats now and the renewal that opens the next period charges it. So a team bought with one seat type, before premium existed or from a checkout that sent no lines, mixes later; a second sale stays refused (one paid subscription). An added seat type is Line.Full: billed outside the promotion the row was sold under (buildPeriodInvoice discounts only the rest), since the promotion was sold for what was bought. One not yet charged is taken back by asking for 0; a charged one keeps a seat. A comp's seats do not change (compSeats): a record names them. Over HTTP a member billed from the org's pool changes nothing (mayChangeWallet, the one wallet rule), and members are <org>/<name>. The renewal still writes its row back after the charge, so a change of seats made during the charge is lost to it, as a change of quantity always was; making the renewal a Swap is its own change to the engine. engine.ChangePlan refuses a team with lines: a move prices one plan's seats against another's.
  • Record: RecordIn.Lines opens a recorded team with both seat types, its price checked against every seat; extending one checks the price of every seat it holds for the period, and a payment naming other lines is refused. Each recorded period keeps its seats per seat type (periods[].lines), which periodPaid reads back as a charged invoice's plan lines, so cloud serves a recorded premium seat as premium.
  • Offers: a promotion or coupon takes its percent off a team only when it covers every seat type bought (everyLine), since the charge and the invoice discount every plan line together.
  • Read: the view's lines carry each type's seats paid for (periodPaid keys plan lines by PlanId: the row's plan id for the first type, the slug for a line); seats and chargedCents count all of them. MRR includes lines.
  • Assign: PUT /v1/billing/subscriptions/:id/seats {"assigned":[{"member", "plan"}]} (AssignSeats, admins only over HTTP: mayActForOrg) replaces who holds which seat type, never more members on a type than its seats paid now. A member is a person of the org as accounts name one, <org>/<name> folded (member); anything else is refused. It writes by Swap, so it never puts back what a renewal or a change of seats wrote meanwhile. It moves no money and bounds no usage: cloud's limiter pools a team's seats. The older members → bundle-child rows (provisionMembers) is standalone-only and never removes a row; it is not the assignment.

A plan states no AI usage

What a plan allows of AI — per model class, per session and day, per model — is cloud's usage policy (hanzo-inc/cloud apps/ai/limits), config in cloud's flags store, edited by a SuperAdmin at admin.hanzo.ai and read by customers at /v1/ai/limits. Commerce holds no second statement of it:

  • plan.Limits has no requestsPer*/tokensPerMinute field, so an admin PUT carrying one stores nothing and the seed has nothing to reconcile.
  • /v1/billing/tier and /v1/billing/usage/rollup carry no windows.
  • The public answer (ReadPlans → PlanView, which cloud forwards as GET /v1/billing/plans) narrows limits to Capacity: agents, bots, guests, members, seats and the DNS quotas. The stored row is never served verbatim.
  • A plan move (PATCH) is scored by listCents: the embed's monthly list price, per seat on a per-seat plan, resolved slug, contact-sales as the most there is. A move that raises it needs MayMintMoney; lateral and downgrade are self-serve.

One way to price a UNIT — the rate authority, edited at admin.hanzo.ai

A plan says what a SUBSCRIPTION costs. A rate says what one UNIT costs — a model per million tokens, a tool per call, storage per GB-month. They are different questions with different lifecycles, and models/rate is the second one.

Rates used to live in hanzoai/pricing as an embedded 150KB JSON: 506 priced items, where changing one number meant editing a module, cutting a tag, bumping commerce, bumping cloud and deploying. That is the right lifecycle for a plan ladder (see the section above — versioned, reviewed, revertible) and the wrong one for 506 upstream prices that move when a vendor moves theirs.

So the rate authority has NO EMBED, deliberately. There is nothing to vendor and no PinnedRatesVersion. POST /v1/commerce/rates/import loads a whole document and RECONCILES it: a matching row is left alone, a drifted one corrected, an operator-edited one skipped. Importing the same document twice is a no-op.

AdminEdited means the same thing it means for plans, and matters more here: an operator's price outranks the document it came from. Without it you would change a rate in admin.hanzo.ai, watch it apply, and find it silently reverted by the next import — which would make an editable catalog WORSE than a hardcoded one.

Identity is (product, meter), derived into one indexed Slug by Bind(). The same model name can be metered by two products at two rates, so keying on the name alone lets one product's price overwrite another's.

Rates are NANO-dollars. Cents cannot hold them: a cheap model is fractions of a cent per million tokens, and rounding to a cent prices it at zero or at 100x. The usage ledger already carries cost_nano/billed_nano/margin_nano.

models/rate is NOT models/meter. A meter is an event AGGREGATION rule (EventName, AggregationType sum/count/last, Dimensions) — what to count. A rate is what one of those units costs. They shared a name for one commit and billing/engine broke immediately.

One way to grant credit — POST /v1/billing/credit (2026-07-16)

POST /v1/billing/credit is the ONLY way credit enters an org ledger. It is mint-gated (registered through middleware.Mint, which applies middleware.PlatformOnly): only the internal service token OR a SuperAdmin (owner=="admin") reaches it — every self-service / org-owner / no-auth caller is 403/401. A client-supplied mint amount must never be self-service; that is the money-critical core (a user cannot credit itself). Body: {org, amountCents, reason, tag?, currency?, expiresAt?, idempotencyKey?}, org-keyed (the org POOL account, not a per-human subject), idempotent on idempotencyKey.

The mint gate declares itself. A mint route is registered through middleware.Mint(api) — mint.Post("/deposit", Deposit) — which BOTH applies PlatformOnly() and records the route's method + full path in the package-level registry. Registration is the single declaration, so the gate and the record cannot drift. middleware.MintRoutes() []MintRoute exports that surface (after mounting) for cross-service checks: cloud's /v1/billing bridge forwards with the admin COMMERCE_SERVICE_TOKEN, which satisfies MayMintMoney, so its billingForwardable allowlist must stay DISJOINT from MintRoutes() — an assertion cloud can now make against this API instead of hand-copying the list. Routes registered through util/rest's deferred table (api/affiliate) carry the same PlatformOnly chain explicitly but are not self-declaring.

Consolidated three prior handlers into this one: deleted self-service GrantStarterCredit (old /credit), PostMyWelcome (/me/welcome), and GrantStarter (/grant-starter). "Starter credit" is now a PARAMETERIZED call (tag=starter-credit, amountCents=500, expiry=+365d) driven by the billing/credit constants (reduced to data-only). The mint-surface guard (api/billing/mint_surface_test.go) proves /credit is the sole gated credit entry point.

Backing = injected double-entry ledger (billing/creditledger). Commerce runs embedded in the cloud binary, where the AI spend-gate reads cloud's native the ledger. Commerce must NOT import cloud, so the host injects a creditledger.CreditLedger via EmbedConfig.Ledger (embed.go → creditledger.Set). When set, Credit and GetBalance route to it — a credit lands in the SAME per-org account the gate reads (one ledger, no split). When nil (standalone), both fall back to commerce's own datastore (tagged Deposit). The interface is compiler-enforced; cloud implements a ~40-line ledger adapter.

type CreditLedger interface {
    Credit(ctx, CreditInput) (txID string, balanceCents int64, err error)
    Balance(ctx, org, currency string) (availableCents int64, err error)
}
type CreditInput struct { Org, Currency, Reason, Tag, IdempotencyKey string; AmountCents int64; ExpiresAt *time.Time }

A plan coupon runs for months (engine.Grant)

A plan coupon (api/promo, models/coupon with Filter == "plan") is a percent off the first Months billing cycles of one purchase (Months 0 reads as 1, the first month; 1..36). The sale grants the subscription the coupon (engine.Grant: CouponCode/Percent/Months/Plans, CouponFrom = the period the sale holds, which on a trial is the first paid period after it). An invoice, the sale's first included, takes the percent while fewer than Months cycles were invoiced in [CouponFrom, its period) (engine.Billed: distinct plan-fee periods, voided ones aside), the row bills monthly (engine.BillsMonthly; the sale refuses a coupon on anything else), and the coupon names the row's plan and every seat type sold (engine.couponOf). A cycle is an invoice, so nothing counts down: a renewal re-run finds the invoice it wrote, a row not saved after its invoice spends no month, days a gift adds or a late payment that moves the period spend none, and a voided (unserved) period leaves its month for the next. A cycle billed on a plan the coupon does not name, or on a year, is billed at its price and spends one. Label "CODE — N% off first month" for one month, "CODE — N% off, month k of M" for more; invoice metadata coupon, couponPercent, couponCents, couponMonth, couponMonths. The quote (GET /v1/billing/plans?coupon=) and the receipt carry couponMonths. A coupon with a trial waits it out (reserved at the sale, granted from the trial's end). ExtendCoupon(ctx, code, apply, by) grants the coupon's current months to every live subscription already bought with it, from its first paid period (the sale's grant, else the first invoice the coupon discounted); it never takes a month or a plan away, writes by Swap, reports billed and missed (cycles among the granted months invoiced without the coupon, paid or owed), dry-runs with apply false, and records subscription.coupon_extended in the org's namespace.

A coupon for the next renewal (api/billing.SetNextCoupon)

Staff put a percent off ONE renewal invoice on a subject's live subscription (its held paid plan, or a trial of one): SetNextCoupon(ctx, org, subject, code, percent) writes Subscription.NextCouponCode/NextCouponPercent (percent 0 clears; 1..100 needs a promo.CouponCode code; an external row is refused, nothing renews it; no live row is IsSubscriptionNotFound). It is not a plan coupon: no window, limit or redemption. It stays a separate fact because it is a separate promise: the NEXT renewal whichever period it bills, late or not, on a monthly or a yearly row, while a plan coupon is dated months; and one row holds both at once, so a win-back inside a sale's months stacks (the renewal coupon off what the plan coupon left) instead of erasing them. engine.RenewSubscription hands it to buildPeriodInvoice as Coupon{Renewal: true} (after the promo and the plan coupon, plan fee only; label "CODE — N% off this renewal", invoice metadata renewalCoupon, renewalCouponPercent, renewalCouponCents), clears it on the row the caller persists, and restores it when no invoice was built. settle clears it too when the owed period's invoice already carries it, so a renewal whose row was not saved after its invoice cannot hand it to a second period. Cloud reaches it as the plane op billing_subscription_coupon.

One way to give plan time — billing.Gift (api/billing/gift.go)

Days of a plan given, never sold: a referral's free week. A subject on a paid plan has the period it holds run the days longer (the next renewal moves with it, no invoice). A subject on none is given the entry plan (dev, sold as Pro) on a manual_gift row canceled at the end of the days, so it lapses to Free on its own. One key gives once (idempotency guard plus the key written in the row's gifts metadata, which a re-claimed guard finds). A sandbox gift (Test) lands only on test rows. A gift row cannot be reactivated.

engine.Over(sub, now): a row canceled at the end of its period confers nothing once that period is over, swept or not. rowTier, holdsPlan and servedSubscription ask it, because the renewal sweep (billing_renew) is off in production and ended only runs inside it.

billing.PaymentsOf(org, subject) is what a referral reads: the earliest subscription period a card paid in full (PaymentMethod == "card"; credit, balance and mixed are not a card), with its refunds, dispute and the paying card's fingerprint, plus every fingerprint the subject keeps on file. Both are platform-only plane ops in cloud (billing_gift, billing_payments).

Cross-Compilation (2026-02-23)

Colima QEMU crashes Go's HTTP/2 and module loader on ARM Mac. Use zig for cross-compilation:

go mod vendor
CC="zig cc -target x86_64-linux-musl" CXX="zig c++ -target x86_64-linux-musl" \
  CGO_ENABLED=1 GOOS=linux GOARCH=amd64 \
  go build -mod=vendor -ldflags="-s -w -extldflags '-static'" -o commerce ./cmd/commerce/main.go

Push to GHCR (Docker Hub credentials not available locally):

docker build --platform linux/amd64 -t ghcr.io/hanzoai/commerce:hotfix .
docker push ghcr.io/hanzoai/commerce:hotfix

App Engine Migration (2026-02-24)

All App Engine dependencies removed. Context handling modernized:

  • middleware.RequestContext() (was AppEngine()) — stores Go request context in Gin
  • middleware.GetContext(c) (was GetAppEngine()) — retrieves it; falls back to c.Request.Context()
  • Gin context key changed from "appengine" → "context" everywhere
  • Legacy aliases AppEngine/GetAppEngine kept as var for backward compat
  • Legacy GAE Python utils moved to .legacy/ (bulkloader, datastore-admin, salesforce-metadata)
  • ORM: all models use mixin.Model[T] with generic CRUD, namespace support, and orm.Register[T]()

OSS Contributor HUSD Payouts (on-chain, 2026-06-27)

25% of cloud revenue is payable to OSS contributors in HUSD (Hanzo USD stablecoin) on the Hanzo EVM. The split + SBOM attribution + payout algorithm live in models/contributor/; the executor is cron/payout/contributor/.

  • Executor: contributor.Payout(ctx, Config) computes period revenue, splits 25% across components by real SBOM weight, and disburses each allocation by PayoutMethod: credits (CreditGrant), stripe (queued transfer), crypto (on-chain HUSD ERC-20 transfer).
  • Crypto path: executeCryptoPayout → util/blockchain.TransferToken (geth-free seam) → thirdparty/ethereum.transferToken (luxfi/geth signing, legacy EIP-155 tx). The impl is registered by a blank import of thirdparty/ethereum in cmd/commerce/main.go — without it, TransferToken returns ErrNoTokenTransfer (crypto payouts skipped, never silently lost). go.work + replace => ./thirdparty/ethereum link it under the GOWORK=off Docker build.
  • Config (KMS → env, HUSDConfig.LoadFromEnv): HUSD_TOKEN_ADDRESS, HUSD_CHAIN_ID, HUSD_RPC_URL, HUSD_TOKEN_DECIMALS, HUSD_TREASURY_KEY. Fails closed: no HUSD_TOKEN_ADDRESS/HUSD_TREASURY_KEY ⇒ ErrHUSDNotConfigured (mainnet has no default token addr, so an unset deploy can't mis-pay). HUSD_TREASURY_KEY is KMS-only, injected into the environment. Non-secret config is plain env.
  • Trigger: POST /v1/contributor/payouts/execute (admin) runs the executor for the org namespace — dry-run by default, ?execute=true to disburse. Mirrors billing auto-recharge/run-all; drive it from a curl CronJob: curl -XPOST -H "Authorization: Bearer $COMMERCE_SERVICE_TOKEN" \ http://commerce.hanzo.svc:8001/v1/contributor/payouts/execute?execute=true. GET /payouts/preview shows the allocation without paying.
  • TESTNET-FIRST: deployed config points at Hanzo testnet (chainId 36962, HUSD = "Lux Dollar" 0xc57b7eCE…4D66, 18 decimals, a testnet hanzod node's /v1/chain/c RPC). Proven on testnet 2026-06-27: tx 0xe5cf03378e2d9dd121dfc5631fa112b2ea03717c9928167d54195ae785866978 (treasury → contributor, 25.50 HUSD, status 0x1, block 9). Live proof: go test -tags onchain -run TestOnChainHUSDPayout_Testnet.
  • MAINNET SWITCH (OFF; requires sign-off + funded mainnet treasury): HUSD_TOKEN_ADDRESS=0xe9e32EF8aaECB68794Da3E1E9191b0a64CeC2c83, HUSD_CHAIN_ID=36963, HUSD_RPC_URL=<a mainnet hanzod node>/v1/chain/c, and re-point HUSD_TREASURY_KEY to a funded mainnet treasury.
  • Gas note: thirdparty/ethereum GasPrice() queries eth_gasPrice (respects the chain's 25 gwei minBaseFee); the old hardcoded 1 wei caused "transaction underpriced" on Hanzo EVM.

Versioning (tag == binary)

commerce.Version is a var (commerce.go), default = the current release. CI injects the immutable image tag at build time so /healthz version always equals the deployed tag:

  • .hanzo/workflows/cicd.yml passes VERSION=<git tag> (build-arg) on v* tag pushes.
  • Dockerfile strips the leading v and applies -ldflags "-X github.com/hanzoai/commerce.Version=<ver>". It is the ONLY Dockerfile that builds the binary — Dockerfile.sqfix, Dockerfile.prebuilt and Dockerfile.sqfix-prebuilt are gone. All three packaged a binary compiled by hand on a laptop (zig cross-compile / hotfix builds), which is not how anything ships, and none was ever referenced by hanzo.yml. One had drifted to build tags that no longer compile, so the only thing they could still do was mislead.
  • Branch builds leave VERSION empty → the in-source default holds.

Cut releases with a v* git tag (git tag -aX vX.Y.Z && git push origin vX.Y.Z) so the build produces ghcr.io/hanzoai/commerce:X.Y.Z whose binary reports X.Y.Z. Do NOT ship named tags (:X.Y.Z-foo) for prod — they re-introduce the tag/binary drift (the live 1.42.31-testmode image reported binary 1.42.5 because Version was a hardcoded const nobody bumped).

commerce-site edge (no nginx)

commerce.hanzo.ai serves the marketing SPA via ghcr.io/hanzoai/static (Dockerfile.site, --spa, :3000) — NEVER nginx. The host's routing lives in the hanzoai/ingress file provider (universe infra/k8s/ingress/routes.yaml): a high-priority commerce-hanzo-ai-api router sends /v1, /api, /healthz to the commerce API (commerce.hanzo.svc:8001); the low-priority commerce-hanzo-ai router catches the rest → the SPA. The operator commerce-site Service CR has ingress.enabled: false (a plain Ingress can't express Traefik priority). The from-source Dockerfile.site Next.js build is currently broken (unbuilt workspace dists, a phantom @kapaai/react-sdk import, a @/providers alias); 0.2.0 re-serves the proven 0.0.1 assets — fix the app build before bumping content.

SBOM-driven OSS-developer payout (2026-06-25)

Every deploy emits an SBOM; Hanzo pays upstream OSS developers up to 25% of all Hanzo cloud spend, attributed by the dependencies actually deployed. One pipeline, decomplected into orthogonal pieces.

The pipeline (one way, all images)

arcd build  --syft-->  CycloneDX SBOM  --normalize-->  POST /v1/billing/sbom
  (deploy)             (per image digest)               (ZAP 0x20 OR HTTP)
                                                              |
org usage  -->  RecordUsage  --go-->  engine.AccrueOSSPayout  v
(charge)        (existing)            (mirrors TrackRevenueShare)
                                          |
                  ossattr.Attribute(spend, SBOM pkgs, policy)   <= 25% pool
                  ossfunding.Resolve(purl) -> maintainer target
                                          |
                                          v
                  OSSAccrual ledger lines (per package, idempotent)
                                          |
                  GET /v1/billing/oss-payout/summary  (disbursement view)

Components

Concern Where Notes
Attribution math (the heart) ossattr/ (leaf, stdlib-only) Pure Attribute(spendCents, []Package, Policy) -> Result. 25% cap is ossattr.MaxPoolFraction (code, not config). Pro-rata by weight; largest-remainder apportionment conserves the pool to the cent. Deterministic + sorted.
Policy config/oss-payout.json + config/oss_payout.go poolFraction (clamped <=0.25), directWeight (1.0), transitiveWeight (0.25). Embedded JSON, sync.Once (mirrors referral-program.json).
Maintainer resolution ossfunding/ (leaf) FromPURL (pure, offline): github.com/ PURLs -> github_sponsors:<owner>; golang.org/x/* -> golang; npm/pypi w/o VCS -> Unresolved -> held. Network FUNDING.yml/registry enrichment behind Resolver (out of hot path).
SBOM storage models/sbomrecord/ Per image digest; Ingest() is the ONE upsert (idempotent on digest), shared by HTTP + ZAP. Components are a noindex JSON blob. Global "system" namespace.
Accrual ledger models/ossaccrual/ Append-only per-package line; idempotency key = ossattr.AccrualID(org, txn, purl). Status: pending|held|queued|settled. "system" namespace (Hanzo->OSS liability, aggregated across orgs).
Accrual engine billing/engine/osspayout.go AccrueOSSPayout(...) — fire-and-forget hook in RecordUsage, structural twin of TrackRevenueShare. Unions SBOM components, runs ossattr.Attribute, resolves funding, writes lines.
HTTP surface api/billing/osspayout.go POST /sbom, GET /sbom, GET /oss-accruals, GET /oss-payout/summary (admin-token).
ZAP-native path infra/zap_osspayout.go OpSBOMIngest = 0x20 on the existing ZAP node; RegisterSBOMIngest(store) wired in commerce.go to the SAME sbomrecord.Ingest.
SBOM emit (build side) arc/cmd/arcd/sbom.go dockerBuilder.emitSBOM after docker push: syft <digest> -o cyclonedx-json -> normalizeSBOM (pure; direct/transitive from the dep graph) -> POST. Env: SBOM_INGEST_URL/TOKEN/ORG. Best-effort (never fails the build).

The 25% math

pool = floor(spend * min(poolFraction, 0.25)). Each package weight = base(scope) * criticality (direct base 1.0, transitive 0.25, criticality default 1.0). share_i = pool * weight_i / sum(weight_j), apportioned to integer cents so sum(shares) == pool exactly. Same PURL across SBOMs is de-duped (highest weight wins) so each package is paid once. No packages / zero weight -> whole pool held, never invented or lost.

Proven (tests, all green)

  • ossattr: cap enforced (incl. over-cap clamp), cent conservation over 5000 spends, direct=4x transitive, criticality, dedup, held-pool, determinism, big spend exactness.
  • ossfunding: PURL->target resolution incl. held cases.
  • billing/engine (real SQLite): deploy SBOM -> $100 spend -> 2500c (25%) across 4 pkgs; gin direct=1000c->github_sponsors:gin-gonic(pending), react transitive=250c->held; idempotent (2 fires -> 1 line); no-SBOM no-op.
  • api/billing (HTTP, real router): ingest -> summary rollup; cap + held + resolution proven over the wire.
  • arc/cmd/arcd: normalizeSBOM CycloneDX->ingest with direct/transitive graph tagging + PURL-less drop.

Money-out (designed, creds-gated — NO fake payouts)

Accrual is real + provable now. Disbursement is a separate job that reads GET /oss-payout/summary (status=pending, resolved targets), batches by funding kind, pays, and flips lines pending->queued->settled (referencing a Payout). Rails are integration/creds-gated:

  • GitHub Sponsors — no programmatic payout API; needs the Sponsors GraphQL + an org Sponsors account funding source. Per-maintainer sponsorship.
  • Open Collective — API for expenses/payouts; needs an OC account + API key.
  • Stripe Connect — Transfer/Payout to connected accounts; needs the maintainer to onboard a connected account (KYC). Best general rail. All require: a funded Hanzo source account, per-rail API credentials in KMS (never plaintext), maintainer onboarding/KYC, tax handling (1099/W-8). The held pool waits until a target is resolved + onboarded.

Nobody is exempt — every subject pays, every call is a row

  • Our orgs resolve like customers. hanzo, admin, lux, zoo and every other org get a tier only from a real recorded subscription (deriveTier), a balance only from the ledger. No org list, no balance floor, no minted operating credit, no unlimited roster. SuperAdmins pay like everyone.
  • The tier is derived, never named. ReadTier(ctx, org, user) derives it; X-Tier / ?tier= are ignored from every caller. A nil org is Free (no store).
  • A wallet spends its own grants. getActiveGrants reads the subject's rows only; hanzo/alice never reaches hanzo's grants.
  • Every usage call writes a row (writeUsage, behind POST /v1/billing/usage and ZAP billing.recordUsage): a $0 or sub-cent call included, carrying model, tokens, scope, costMicros and paidBy (plan|prepaid|credits|line|hanzo). The debit is whole cents, exact over time: each call adds its micros to an atomic per-subject running total (db.Counter.Add, one upsert on _sequences) and debits the cent boundaries its span crosses, so sum(amount) == floor(sum(amountMicros)/10_000) under any concurrency. A store that is not a db.Counter refuses (503).
  • A plan given away is a comp, recorded as one. RecordSubscription with processor comp: priceCents 0 for any plan (enterprise, which publishes no price, included), the approval in reference.order, why in terms. The row is external, so it never renews or charges, and its plan is held at price 0, so it adds nothing to MRR; it confers the plan's tier like any recorded plan (settled external:comp). A comp extends only a comp and a payment only a paid plan; either takes the other over by naming it in Replaces. Every other processor still needs a price equal to the catalog's.
  • Privileged money acts name who took them: /credit, /deposit, /credits (+ void), customer-balance adjustments, bank-transfer matches and comp subscriptions stamp minter(c) on the row they write.
  • commerce keeps no metering client. Every product meters through cloud's one package (hanzo-inc/cloud/metering), which has no off switch: a balance or spend cap it cannot read refuses the call (503 balance_unavailable / spend_cap_unavailable). The metering/ module that was here — its own copy of that client, the meter-proxy sidecar and the deploy renderer — is deleted. No deployment ran the sidecar, and its tagged releases (metering/v0.1.x) stay resolvable for the one module that still pins one.

Three money buckets — credits vs prepaid vs card (billing/bucket)

The ledger was CONFLATED: starter-credit/included-credit: (non-cash grants) and topup/deposits (real money) were all transaction.Deposit rolled into ONE balance. They are now split HONESTLY from the real tags — no fabricated numbers.

  • billing/bucket (pure, no I/O, unit-tested) is the ONE classifier + spend projection:
    • DepositKind(tags) → Credit (tag starter-credit, prefix included-credit:, credit:, grant:) vs Prepaid (everything else — topup/husd/bare deposit = real money; UNKNOWN→Prepaid is fail-closed: never mint spendable-on-GPU value from a mystery deposit).
    • IsGPUWithdrawal(tags) → gpu (gpu/gpu-*/gpu:*).
    • Compute(transs, id, now) → Split{CreditsGranted, CreditsRemaining, PrepaidBalance, PrepaidAvailable, Balance, Holds, Available}.
  • Spend policy (documented, enforced): non-GPU usage draws CREDITS FIRST then prepaid; GPU usage draws ONLY prepaid. The projection reconciles to the cent: CreditsRemaining + PrepaidBalance == Balance (the same balance the gateway gate debits — bucket.Compute is a faithful superset of util.TallyTransactions).
  • Read: GET /v1/billing/balance + /me/balance now ALSO return creditsGranted, creditsRemaining, prepaidBalance, prepaidAvailable, and card{onFile,brand,last4,isDefault} (additive — the pre-split {balance,holds,available} is unchanged, so the cloud proxy + console BFF forward the split verbatim; console shows credits vs prepaid distinctly). util.GetRawByCurrency fetches the raw (un-expiry-filtered) ledger the split needs.
  • GPU rule (the bucket projection, billing/bucket): a gpu-tagged Withdraw draws PREPAID only — IsGPUWithdrawal classifies it and Compute never lets it reach a credit grant. It is a property of the LEDGER, so it holds for every writer rather than for one endpoint. There is no bespoke GPU charge endpoint here: a GPU is a metered resource like any other and is billed through the general metered path (a machine is launched via cloud /v1/machines, which fronts the compute provider's resell endpoint where the balance gate and the per-hour meter live). A GPU charge NEVER calls BurnCredits.
  • CreditGrant (models/creditgrant, BurnCredits) is a SEPARATE grant ledger used only for subscription/invoice collection (cycle/invoices/subscriptions) — NOT the pay-as-you-go wallet the balance endpoint reads. Left as-is.

Chain-Backed Credit Ledger — HUSD-on-Hanzo-EVM (steps 1-6 BUILT + testnet-proven)

Kills the recurring C1 money-mint vuln class by construction: a credit exists ONLY as HUSD (Hanzo USD, ERC-20 on the Hanzo EVM) minted by the KMS treasury key — which commerce does NOT hold. Commerce can only REQUEST a mint; it becomes a read-only indexer of on-chain balances. No commerce code path can create money.

Foundation landed (branch feat/chain-backed-credit-ledger / chain-ledger-wip, 7-step plan — steps 1-3 done, 4-7 staged):

  1. treasury.DeriveAccount(masterSeed, orgID) (treasury/derive.go) — per-org on-chain address, BIP-32-style HMAC-SHA512 → secp256k1 (luxfi/crypto, no geth). Deterministic across restarts, collision-safe for string org ids, golden-vector pinned. Private key (settlement-signing, step 5) in memory only.
  2. treasury.Mint(ctx, MintRequest) (treasury/treasury.go) — the ONE mint path: treasury-signed on-chain HUSD transfer to the org address. TWO locks: mintauth.Require(ctx) (gated HTTP caller needs proven authority; crons ungated) + KMS key custody (husd.Config.Configured() else ErrNotConfigured, no DB fallback). Idempotent by key (deterministic IssuanceID; replay → stored receipt, no new tx; concurrent same-key → exactly ONE on-chain mint). Bucket (credit|prepaid) recorded off-chain → ledger tag (credit:husd|husd). Production store: models/husdissuance (kind 281) + treasury/datastorestore (deterministic-id ON CONFLICT, mirrors models/idempotencykey).
  3. billing/husdindex — chain is source of truth. client.go = CGO-free JSON-RPC read client (BlockNumber/BalanceOf/TransfersTo). index.go = Sync projector: scan Transfer events → project idempotent bucket-tagged credits (dedup on txHash:logIndex), reconcile indexed == balanceOf to the cent.
  • util/husd = the ONE HUSD config + cent↔wei source (contributor payout aliases it now — DRY). mintauth.Require = the shared "gated & unauthorized ⇒ refuse" policy (sink Enforce + mint service both call it).
  • Decomplected for CGO-free proofs: core logic (derive/mint/index) has no sqlite/geth dependency, so unit tests + the live proof run CGO_ENABLED=0 locally (the luxfi/accel cgo link + mattn/go-sqlite3 both fail on a laptop; datastore adapters are CI-tested). geth core/types/rlp are avoided in the live signer too — their transitive luxfi/[email protected] pull trips the upstream force-moved-tag go.sum mismatch; the proof signs EIP-155 with luxfi/crypto + hand-rolled RLP (production uses the geth thirdparty/ethereum signer via the same util/blockchain.TransferToken seam).

PROVEN on testnet (chainId 36962), end-to-end, independently verified via cast:

  • Provisioned a fresh HUSD test token 0xe7f1725e7734ce288f8367e1bb143e90bb3f0512 (the ephemeral testnet had reset — no token, unfunded treasury; funded treasury 0xe6da…a51a from the genesis hardhat account 0xf39F…2266, minted 1M HUSD to treasury). Node serves the new /v1/chain/c surface (post /ext→/v1 cutover).
  • Live mint tx 0xd01bc1c1733e83c93a5143552af1ba7e3ac045b2433c2fecff07a29685cf976a (status 0x1, block 25): treasury.Mint($12.34, credit) → treasury → derived org addr 0x3560…9950. Org balanceOf == 1234c; treasury balance dropped by EXACTLY 12.34 (value from real supply, not fabricated); indexer Sync projected a credit:husd 1234c credit == on-chain; replay of the idem key sent NO second tx. Run: HUSD_RPC_URL=… HUSD_TREASURY_KEY=… HUSD_TOKEN_ADDRESS=… CGO_ENABLED=0 go test -tags onchain -run TestOnChain_ChainBackedLedger ./treasury.

Headline: on testnet, minting is treasury-only (commerce holds no key, cannot mint) and idempotent, and the indexer reflects on-chain balance — proven live.

Steps 4-6 (BUILT + testnet-proven, branch chain-ledger-wip)

  1. Mint-path rewire (billing/husdledger + api/billing/husd_mint.go). husdledger.Service is the ONE mint entrypoint (MintCredit): treasury on-chain mint → bounded synchronous ProjectTx so GET /v1/billing/*/balance reflects it on return; a background/CronJob SyncOnce backfills. Adapters: ledgerStore (projects a bucket-tagged transaction.Deposit — the SAME row the balance read tallies — keyed to the mint's Subject in its Test partition, idempotent on txHash:logIndex), addressBook (derive every org address from the org list + seed), cursorStore (models/husdcursor kind 282), and datastorestore.ByTxHash (the issuance store IS the IssuanceLookup, system-ns). MintRequest/Issuance/Credit gained Subject (per-org address, per-subject ledger) + Test. Handlers Deposit/GrantStarterCredit/PostMyWelcome/GrantStarter/topup branch to the chain mint when enabled (else the DB path, unchanged). Wired at Bootstrap from husd.Config + HUSD_ORG_DERIVATION_SEED. The seed is the ledger's sole intent signal (husdledger.ValidateConfig, PR #69): ENABLED only with a seed AND token+key; token+key WITHOUT a seed is the INERT state (Enabled()==false, DB credit path) — NOT a boot error — because the shared util/husd token+key is ALSO the OSS contributor-payout config, which runs with no ledger seed. Boot refuses ONLY on an incoherent LEDGER config (seed set but token/key missing). This decouple is why v1.46.39 boots inert in prod (the CR carries HUSD token+key for OSS payouts, no seed) instead of crash-looping; the mainnet cutover enables the ledger by provisioning the seed. Proven: live TestOnChain_Step4_ProjectTx (mint 0x3324a75a… → ProjectTx from the REAL receipt → one credit subject/bucket/test) + CI ledgerStore sqlite test (ProjectTx → real Deposit → bucket balance, idempotent).
  2. Settlement (husdledger/settlement.go, husd.SettlementDrift, kind 283). drift = balanceOf(orgAddr) − max(0, ledgerSpendable); sweep org→treasury, signed by the ORG's derived key, when drift ≥ threshold. Self-correcting + idempotent. POST /v1/billing/husd/settle (platform-only). Proven: live TestOnChain_Step5_Settlement (mint $50 → gas-fund org → org-key sweep 0xf323bf25… → org drops EXACTLY $20 == spendable, treasury +$20) + CI SettleOrg sqlite test (real ledger $50−$20 → sweep $20, reconcile, idempotent).
  3. Migration + reconcile (husdledger/migration.go). Snapshot each subject's bucket.Split → treasury-mint the equivalent per bucket → neutralize legacy deposits with two idempotent offset withdraws (non-GPU cancels credit credits-first; GPU-tagged cancels prepaid prepaid-only) so the split is preserved → assert chain balanceOf == pre-migration DB balance == post DB, exact to the cent, zero settlement drift. POST /v1/billing/husd/migrate (dry-run default, platform-only). Proven: CI MigrateOrg sqlite+fakeChain test (alice 2500c/2000p + bob 1000p → reconcile 5500==5500==5500, buckets preserved, idempotent).

Step 7 — DB-mint deletion (the mainnet cutover GATE, NOT done this session)

The named money-in paths (deposit/welcome/topup) route through the chain when enabled; the DB write stays as belt-and-suspenders (bypassed on the chain path, still guarded by #65's mintauth sink) until prod migration is 100%. Before the deletion, these REMAINING credit-mint surfaces MUST also route through chainMintCredit (else they DB-mint credit outside the chain when enabled): billing/allotment (monthly included-credit), billing/trial, billing/credit.GrantIfEligibleNow (shared auto-grant helper), api/billing/refund.go, api/billing/topup_token.go, api/billing/webhooks.go (settled-payment deposits), api/transaction/create.go (raw /v1/transaction). (api/checkout/util/capture.go credits an ORDER wallet, not the gateway-spendable iam-user wallet — out of scope.) THEN delete the DB deposit writes; the mintauth sink stays as the final backstop.

Mainnet cutover sign-off gate (separate; do NOT touch mainnet HUSD): (a) route the remaining credit-mint surfaces above; (b) provision the mainnet HUSD token + a FUNDED mainnet treasury (HUSD_TOKEN_ADDRESS, HUSD_CHAIN_ID=36963, HUSD_RPC_URL (mainnet), HUSD_TREASURY_KEY + HUSD_ORG_DERIVATION_SEED in KMS) with gas float per org address (settlement needs it); (c) run /husd/migrate?execute=true per org, assert reconcile; (d) enable the chain path; (e) run /husd/sync + /husd/settle on a CronJob; (f) after migration is 100%, land step 7's deletion. Testnet-first is complete; mainnet is the sign-off.

Config: HUSD_TOKEN_ADDRESS, HUSD_CHAIN_ID, HUSD_RPC_URL, HUSD_TOKEN_DECIMALS, HUSD_TREASURY_KEY, HUSD_ORG_DERIVATION_SEED (all KMS), HUSD_INDEX_CONFIRMATIONS (default 1), HUSD_SETTLE_THRESHOLD_CENTS (default 1). Testnet proof env: token 0xe7f1725e…0512, chainId 36962, RPC …/v1/chain/c, treasury 0xe6dad4…a51a.

Crypto deposit watcher — the half of the crypto rail that credits (2026-08)

POST /v1/billing/crypto/deposit mints a per-payer MPC custody address and records a CryptoPaymentIntent as Pending. Until now NOTHING advanced it: no component observed DepositAddress, MarkConfirming/MarkSucceeded had zero production callers, and money sent to a minted address was received and never credited. That is why cryptoDepositsCanBeCredited (api/billing/crypto_deposit.go) is false and the handler 503s before any keygen.

The watcher is now built. The gate is still false — see "before flipping".

Two packages, decomplected exactly like husdindex/husdledger:

  • billing/depositwatch — pure policy over small interfaces (no datastore, no HTTP): the scan window, the confirmation rule, the amount arithmetic, the address→intent match. Unit-tested against fakes with no chain and no DB.
  • billing/depositledger — the I/O half: per-org intent store, the idempotent ledger write, the persisted cursor (models/depositcursor), and the schedule. Integration-tested against a real datastore (util/test/ae).

Nothing in depositwatch is EVM-shaped. "Block" is whatever position a chain counts monotonically (a block, or a Solana slot) and the only property relied on is that it increases. Adding a chain adds a Reader; the policy does not move.

Live read-path probes (billing/depositledger/livechain*_test.go) are opt-in, skipped unless CRYPTO_DEPOSIT_RPC_* is set, so CI never depends on a public RPC. ⚠ -count=1 is not optional: Go caches a PASS even though every input came from the environment, so a second run against a DIFFERENT contract replays the first one's result and reports the old address as live evidence.

Chain reads come from ONE read client per chain family, both satisfying depositwatch.Reader:

  • EVM — husdindex.Client, this repo's one ERC-20 JSON-RPC read client, parameterized by (rpcURL, tokenAddr) and named for its first caller, not for a restriction. Decimals() / Symbol() selectors are pinned against luxcrypto.Keccak256 in client_test.go. Adapted to the watcher's event type by depositledger.evmReader (2 lines) because husdindex.Transfer is an ERC-20 log and predates this rail.
  • Solana — billing/solanarpc.Client (2026-08), which implements Reader directly. See "Solana" below.
  • TON — billing/tonrpc.Client (2026-08), over the TON Index (toncenter v3) HTTP API rather than a node: TON's node RPC has no per-account transaction index. See "TON" below.
  • XRPL — billing/xrplrpc.Client (2026-08), over XRPL JSON-RPC. See "XRPL" below.

depositledger.newReader is the ONE place a chain becomes a client, dispatched on Asset.Family() (FamilyEVM|FamilySolana|FamilyTON|FamilyXRPL). It is exhaustive, not defaulted: AssetsFromEnv refuses any chain absent from depositwatch.chainFamily, so an unknown chain can never be silently handed the EVM client. Family() replaced IsSolana() in 2026-08 — one enum beats N IsX() predicates, and it keeps the chain→family table the single source.

Exactly-once = a KEY, not coordination

A credit's storage id is sha256("crypto-deposit\0" + chain:txHash:eventIndex) (depositledger.creditKey). Both backends upsert on (id, kind, namespace) (db/sqlite.go, db/postgres.go), and a balance is a SUM over rows, so every attempt to credit the same transfer lands on the same row: re-scan, crash retry, and N replicas ticking simultaneously all yield one credit. No leader election, no lease, no lock — adding replicas cannot change the answer.

All three parts of the key are load-bearing:

  • chain — a pre-EIP-155 tx can be replayed across EVM chains with an identical hash, so without it two genuine deposits collapse into one credit.
  • txHash — the chain's own transaction id, folded the way that chain's identifiers compare (Asset.Fold; see "Per-chain facts" below).
  • eventIndex — WHICH value movement inside that transaction. It was logIndex until 2026-08 and is now a per-chain concept (depositwatch.Transfer.EventIndex): an ERC-20 log index on the EVM, the index of the token-BALANCE record on Solana, and it would be the vout on Bitcoin. It is 0 on TON and XRPL, and there it is a FACT rather than a placeholder — a TON transaction has at most one inbound message (in_msg is singular), and an XRPL Payment delivers to exactly one destination, so the transaction id names the event by itself. Admitting a second kind of value delivery on either (XRPL CheckCash, EscrowFinish) would require a real index. One transaction can otherwise credit the same address twice or two watched addresses at once; collapsing them swallows the second as a duplicate. The ledger metadata key is eventIndex, not logIndex — a Solana account index stored under a field named logIndex is a lie a later reader cannot detect.

The routing TAG is deliberately NOT in the key: the key names the on-chain event, and which intent it was routed to is a separate question answered by Asset.Identity.

⚠ It must be a function of the EVENT and of nothing else — never a counter over the scan's results. A key that renumbers when the window moves is a double credit. (Mutation-tested: numbering by scan position is caught.)

Ordering is load-bearing: ledger row FIRST, intent second. The reverse would let a succeeded intent survive a failed ledger write, and a later pass would read it as done — money received, never credited.

Scan ahead, commit behind

Scan [cursor+1 … head] every pass (so a customer sees "confirming" within a block); commit only head − requiredConfirmations. The last required blocks are therefore re-read from the canonical chain every pass, which is what makes reorgs harmless: a sighting that vanishes is un-sighted back to Pending (CryptoPaymentIntent.ClearSighting), a sighting that MOVES block is re-read at its new height. Un-sighting only fires for blocks actually inside the scanned range — silence about a block we never read is not evidence.

Cold start (empty cursor) begins at head − required, so no block is ever committed unscanned and no public chain is scanned from genesis.

Confirmation depth comes from cryptopaymentintent.RequiredConfirmationsForChain — the ONE policy, now naming every supported chain (eth 12, rollups+polygon 20, bsc 15, avalanche/lux/zoo 12, solana 32).

Decimals are READ, tokens are VERIFIED and PEGGED

Getting decimals wrong by 12 credits 10^12 too much, so decimals are never configured: verify() asks the CHAIN, once per asset, refuses if the reported symbol disagrees with the configured token, and refuses < 2 or > 36 decimals. A transient RPC failure is not cached. What each answer is worth differs by chain and the difference is stated, not blurred:

EVM Solana
decimals decimals() on the ERC-20 byte 44 of the SPL mint account
symbol symbol() on the ERC-20 — mandated by the standard Metaplex Token Metadata, a SEPARATE account owned by a third-party program and mutable by its update authority

So on Solana the symbol check catches a mistyped mint, which is the failure that actually happens; it cannot prove an adversary's mint is not USDC. The decimals check is the strong one and it holds on both chains. A mint with no metadata account is REFUSED rather than accepted unnamed.

⚠ An SPL token account is 165 bytes and owned by the same program as an 82-byte mint, so parseMint pins the length EXACTLY (== 82, or Token-2022 tagged at offset 165). "Owned by the Token program and ≥ 82 bytes" would accept a token account and read a byte of somebody's pubkey as the decimals.

Config (KMS-injected, NO defaults — mirrors util/husd's "token address has no default" rule):

CRYPTO_DEPOSIT_RPC_<CHAIN>            e.g. CRYPTO_DEPOSIT_RPC_BASE
CRYPTO_DEPOSIT_TOKEN_<CHAIN>_<TOKEN>  e.g. CRYPTO_DEPOSIT_TOKEN_BASE_USDC=0x8335…
                                      or  CRYPTO_DEPOSIT_TOKEN_SOLANA_USDC=EPjFWdd5…
CRYPTO_DEPOSIT_ADDRESS_<CHAIN>        POOLED chains only (XRPL). The one custody
                                      account every payer sends to, e.g.
                                      CRYPTO_DEPOSIT_ADDRESS_XRPL=rMxCKbED…

Unset ⇒ watches nothing (inert). Incoherent (token with no endpoint, unpegged token, unreadable chain, address written for the wrong chain) ⇒ Bootstrap refuses to start: an operator who configured a deposit rail must not get a process watching less than they asked for.

CRYPTO_DEPOSIT_ADDRESS_* is required on a pooled chain and refused on every other one — a shared address on a per-payer chain would hand every payer the same destination on a chain where the address is the only thing that says whose money it is. The invariant the mint path is built on: a pooled asset that exists has an account to be paid to, so pooledAddressFor returning "" means "nothing watches this token", never "watched but unconfigured".

depositwatch.pegCents IS the creditable set (usdc, usdt, rlusd @ 100c; rlusd added 2026-08 with XRPL). There is no price oracle in commerce and inventing one to value a customer's money would be guessing, so only dollar-pegged tokens are creditable — which is also why Solana was the right chain to extend to first: SPL USDC needs no oracle, exactly like ERC-20 USDC. So do jetton USDT on TON and issued USDC/RLUSD on XRPL; only BTC and native coins genuinely need a price feed. Do not add one. Native coins (ETH/MATIC/AVAX/BNB/SOL) are out on price alone, and on the EVM doubly so: a native transfer emits no log, so eth_getLogs cannot see it. Known accepted risk: a depegged stablecoin is credited above market.

Per-chain facts live in exactly one place each

Adding a chain must not scatter if chain == … through the policy. There are exactly five per-chain facts, each with one home:

fact home
how it is READ depositledger.newReader → a Reader implementation
how deep is deep enough cryptopaymentintent.RequiredConfirmationsForChain
how an address is WRITTEN (hex vs base58 vs base64url vs <CUR>.<ISSUER>) depositwatch.chainFamily + Asset.validateContract
whether its identifiers are case-significant depositwatch.Asset.Fold
what IDENTIFIES the intent (address, or address + tag) depositwatch.Asset.Identity + Asset.Pooled

Fold is ONE function for addresses and transaction ids alike, because it encodes one fact. EVM hex folds to lowercase (custody returns EIP-55, logs are lowercase — comparing literally misses every deposit). base58/base64url do NOT fold: aB… and Ab… are different Solana accounts and different signatures, and the same holds for TON and XRPL. That is only safe because those READERS canonicalise before returning — tonrpc and xrplrpc both render a transaction id as lowercase hex whatever the endpoint answered, and both echo back the CALLER's spelling of a watched address rather than the node's, so the dedup key never depends on an endpoint's rendering. An unclassified chain gets the EVM fold, which is the safe default direction — folding too much makes two addresses ambiguous and fails the pass CLOSED, while folding too little makes a real deposit invisible.

The fifth fact was added in 2026-08 for XRPL and is the only one that changed the policy half's SHAPE. indexByAddress became indexWatched, which returns two different sets — identity→intent for matching, and the distinct ADDRESSES to ask the chain about. On every per-payer chain those sets coincide; on XRPL ten thousand identities share one address.

Solana (billing/solanarpc, 2026-08)

Three Solana facts have no EVM equivalent, and each is a way to lose money silently:

  1. An SPL transfer never lands at the address the customer is shown. It lands in the Associated Token Account derived from (owner, tokenProgram, mint). The owner is NOT in the account list of a plain transfer, so getSignaturesForAddress(owner) returns nothing forever while the money sits one derivation away. The client derives and watches the ATA and reports the OWNER back, so the watcher matches the address it minted and never learns an ATA exists. The derivation is find_program_address — a real off-curve bump search, verified live at bumps 255/254/253. ⚠ tokenProgram is a SEED: a Token-2022 mint derives elsewhere, so the mint's owning program is read before anything is derived.
  2. Finality is a commitment level, not a block depth. Every read is at finalized — the point past which reverting means breaking the protocol's safety assumption, versus confirmed, which is one vote round short and can still be dropped. So a transaction the client can SEE is already final, and the 32-slot confirmation rule the watcher applies on top is pure margin inside already-final territory. That keeps ONE confirmation policy for every chain instead of a Solana-shaped exception, and makes "credited then reorged away" unreachable rather than unlikely. (Slots also skip, so slot depth ≥ block depth — conservative in the safe direction.)
  3. Amounts come from the transaction's own token BALANCE records (post − pre), not from decoding a transfer instruction. The delta is what actually arrived: it survives two transfers into one account in one transaction, a transfer in and out again, and a Token-2022 transfer fee that makes the amount received smaller than the amount sent. An account absent from preTokenBalances started at ZERO — the normal shape of a FIRST deposit, where the sender creates the ATA in the same transaction.

Three cross-checks guard every account before it is credited: the balance record must name our mint, be owned by the watched address, and report the same decimals as the mint account. An accountIndex past the end of the account list is an ERROR, never a skip — guessing there is how a deposit becomes invisible.

⚠ isOnCurve does NOT reduce y before decoding, and that is a CHECKED assumption. Solana's own check goes through curve25519-dalek, which reduces; edwards25519 happens to accept the same non-canonical encodings, so the two agree. TestIsOnCurveAcceptsNonCanonicalYAsSolanaDoes pins it (12 of the 19 y ≥ p encodings decode). If that ever changes, the bump search diverges from Solana's for ~1 candidate in 2^250 and derives an address no deposit can reach. Defensive reduction code was written first and then DELETED: no test could distinguish it from its absence, which is exactly what a surviving mutant means.

Known limitation: only the CANONICAL ATA is watched. Funds sent to a non-canonical token account owned by the deposit address are not credited (they are still recoverable — the MPC key owns them). No wallet does this.

TON (billing/tonrpc, 2026-08)

  1. There is no global block height. The basechain is SHARDED and splits and merges under load, so a shard seqno is neither global nor comparable across shards; logical time (lt) is a Lamport clock whose differences measure causality, not age, so head − lt is not a depth. The scan window is the masterchain seqno: a single global sequence that every transaction enters exactly once and never leaves (TON finalises a shard block by REFERENCING it from a masterchain block), which is exactly the property Reader documents. mc_block_seqno is per-transaction on /api/v3/transactions. Masterchain blocks are FAST — measured at 0.39 s — hence 80 confirmations ≈ 30 s.
  2. The index's jetton-transfer view is the SENDER's side. A /api/v3/jetton/transfers record's transaction_hash is a transaction on the source wallet (opcode 0x0f8a7ea5, transfer) — verified by resolving one and finding account == source_wallet. It means tokens were SENT, not that they arrived; a jetton transfer can still fail at the destination and bounce. This client credits the RECEIVING side only: a transaction on OUR jetton wallet whose in_msg.opcode is 0x178d4519 (internal_transfer) and whose compute+action phases succeeded. The jetton wallet contract validates its sender before crediting, so a forged credit FAILS on chain — the success check is what carries that consensus-enforced validation into this process.
  3. Several query parameters are SILENTLY IGNORED, not rejected. Measured: mc_seqno on /jetton/transfers (an ancient seqno returns live rows) and jetton_master on /jetton/wallets (spelled jetton_address there). They return HTTP 200 with UNFILTERED data, which reads as success. So this client verifies every response against what it asked for — account, owner, jetton — rather than trusting any filter. Never adopt a new parameter here without a bogus-value test first.
  4. Decimals are read from the master's ON-CHAIN TEP-64 content dict (USDT: 6). A jetton publishing decimals only in the off-chain JSON its uri points at is REFUSED: the scale of a credit must not come from a web server the issuer controls.
  5. 🔴 USDT on TON cannot currently be armed, deliberately. Its on-chain content carries only decimals and uri — no symbol — and the off-chain document calls it USD₮ (U+20AE), which is not usdt under any honest comparison. Fetching the URI would make token identity an assertion of the issuer's web server; a homoglyph table would be a rule with no principle. Symbol() therefore refuses, and the whole watcher refuses with it. Arming TON needs either a jetton that states its symbol on chain, or a second deliberately-pinned on-chain fact — not a weakening of the identity check.

Addresses: one TON account has SEVERAL correct spellings (EQ…/UQ…/raw), so ParseAddress decodes all of them to (workchain, hash) and verifies the user-friendly form's CRC-16 (a typo fails at boot, not at the first deposit). The workchain byte is SIGNED — read unsigned, masterchain (−1) becomes 255. Testnet-flagged addresses are refused. Two watched addresses that decode to one account are refused as ambiguous.

XRPL (billing/xrplrpc, 2026-08)

  1. 🔴 The deposit model is POOLED, and it inverts address→payer matching. XRPL charges a NON-REFUNDABLE base reserve per funded account, so a fresh address per payer strands that reserve on every deposit forever. The production model — every exchange's — is ONE custody account plus a per-deposit destination tag. Hence Asset.Pooled/Asset.Identity, Watched.Tag/Transfer.Tag, and CryptoPaymentIntent.AddressTag. An empty tag means the payment carried NONE and is NOT the tag "0" — 0 is a legal tag somebody may hold, and collapsing the two credits a stranger's payment to its holder. 1a. 🔴 The WRITE side: the tag is allocated BY THE DATABASE, and that is the only reason it is unique (2026-08, billing/depositledger.NextTag + api/billing.CreateCryptoDeposit's pooled branch). A duplicate (address, tag) does not mis-credit — indexWatched refuses an identity claimed by two intents — it HALTS the asset for every customer, so uniqueness had to be structural rather than likely.

    What guarantees it: db.Sequencer.NextSequence is ONE statement, INSERT … ON CONFLICT DO UPDATE … RETURNING, against one row. No value is ever read into Go and written back, so there is no window in which two callers see N. Postgres row-locks on conflict and re-evaluates DO UPDATE against the newest committed version; SQLite applies the upsert indivisibly. Holds across goroutines, requests and replicas (they share the row), with no lock, lease or leader election — the same "property of the storage, not of coordination" argument that makes creditKey idempotent.

    ⚠ datastore.RunInTransaction IS A NO-OP (datastore/datastore.go — "For now, just run the function directly"). It opens no transaction, ignores opts, and several call sites carry comments claiming protection it does not provide. db.DB.RunInTransaction is real but opens BeginTx(ctx, nil) = READ COMMITTED, where read-modify-write on a counter lets two transactions both read N and both commit N+1. Measured, not assumed: the mutant that wraps a read-modify-write in an explicit transaction still produced duplicate tags against a real Postgres. Put is a blind upsert, so a losing writer OVERWRITES rather than being refused. There is no other atomic allocator reachable from the model layer — every other "unique" number in this repo is a deterministic hash (which collapses duplicates, which is right for dedup and useless for allocation) or a check-then-write. orm.CreateIfAbsent exists in the interface and models/mixin/orm_adapter.go explicitly refuses it for this reason.

    The tag is COMMERCE's, not custody's. It is a routing fact — which intent does this payment belong to — not a key fact. GenerateAddress mints a FRESH MPC wallet per call by design, which on XRPL strands a reserve on every deposit, and the fleet derives no XRPL key at all (its chain switch falls through to the EVM address, so a "minted" XRPL address would be an 0x string nobody can pay). So the pooled account comes from config and the mint path calls neither cryptoProcessor nor GenerateAddress — and offeredFrom offers a pooled chain on its CONFIGURATION rather than on SupportedChains(), which answers "no" for xrpl and would otherwise hide a chain the rail can both receive and credit.

    Tags are dense from 0 and per-CHAIN (crypto-deposit-tag:<chain>), not per-asset: one account holds every currency sent to it. Density is the requirement, not an implementation detail — 32 bits is 77k tags to a coin-flip collision if drawn randomly, and 4.29e9 if counted. Exhaustion REFUSES rather than wraps: intents are watched forever (Watched filters on the asset, never on status), so a wrapped tag would collide with a live one.

  2. An untagged or unknown-tagged payment is RECORDED, not credited and not dropped. Store.RecordUnattributed → models/unattributeddeposit (system namespace, id unattributed:<chain>:<txHash>:<eventIndex>, idempotent). Crediting it means guessing with someone's money; dropping it loses a real deposit the moment the window moves. It is deliberately NOT fatal to the pass — the pooled address is published, so anyone can send one drop with no tag, and wedging on that would be a denial of service against every other customer. A failed record IS fatal, because advancing past it destroys the only evidence.

  3. Ledgers are FINAL on validation, not buried under N ledgers — closer to Solana's finalized than to EVM depth. This client reads only ledger_index: "validated" and asserts validated: true per entry, so what it can SEE is already final; the 8-confirmation depth (~30 s at a measured 3.81 s/ledger) is pure margin kept so there is one confirmation policy across chains.

  4. 🔴 Amounts are DECIMAL STRINGS, not integer base units — an issued currency has no base unit, so there is NO decimals to read. The scale is fixed at xrplrpc.Scale = 15 (XRPL's own significant-digit limit) and the safety property is different in kind but not weaker: the same function that parses the ledger's number reports the scale it parsed at, so no second source can drift from the parse. TestParseValue_RoundTripsAtDecimals pins units / 10^Scale == the stated value using big.Rat (a big.Float comparison fails on a correct parser and can pass on a wrong one). Truncation is toward zero always.

  5. 🔴 meta.delivered_amount, NEVER tx.Amount. With tfPartialPayment, Amount is the maximum the sender offered and the ledger may deliver an arbitrarily smaller fraction — the classic exploit that has drained exchanges. "unavailable" is refused, not treated as zero.

  6. The token is the PAIR (currency, issuer). A currency code is a name anybody may issue, so both halves are compared on every delivery. Identity comes from gateway_balances on the issuer — a pair the issuer does not actually issue is refused. Only Payment is ever a deposit (it is what carries a tag and delivers to exactly one destination); XRP is skipped as unpriceable.

  7. Errors ride INSIDE result ({"result":{"status":"error", …}}) with HTTP 200 and no top-level error member. A client checking only HTTP status reads a failed account_tx as an empty one — "nothing arrived" when the truth is "we could not look". api_version: 1 is pinned so the response shape cannot silently become v2's tx_json.

It is SCHEDULED, in process

depositledger.Service.Start() from Bootstrap, stopped in Shutdown before the DB closes. Interval is a constant (30s), not config. This is the FIRST in-process scheduler in this repo, and deliberately so: every other periodic job here is an endpoint waiting for a CronJob defined in another repository, which is exactly why /v1/billing/husd/sync has never once run. Safe on every replica because a pass is idempotent (above).

Scaling note

Watched() returns every minted address, in every org, with NO status filter — money sent to an expired or already-settled intent is still the customer's money. Address count therefore grows with total deposits ever taken. If an expiry sweeper is ever added, "stop handing the address out" and "stop watching it" are separate decisions and only the first is safe.

Cost per pass is NOT the same shape on both chains, and this is a real operational difference:

  • EVM — eth_getLogs filters server-side over an address SET, chunked at 100 addresses / 2000 blocks. Cost is O(addresses / 100).
  • Solana — there is no eth_getLogs. Deposits are found with getSignaturesForAddress, which takes ONE address, so cost is O(addresses) plus one getTransaction per transaction found. Deposit addresses see a handful of transactions in their lifetime so this is affordable, but it needs a DEDICATED endpoint: the live probe hit HTTP 429 on api.mainnet-beta.solana.com merely by scanning a busy wallet over 300 slots.

A history longer than 25 pages × 1000 signatures for ONE address is an ERROR, not a truncation: truncating drops the oldest signatures in the window and the cursor then advances past them, losing a deposit permanently and silently.

Before flipping cryptoDepositsCanBeCredited

  1. Provision CRYPTO_DEPOSIT_RPC_* + CRYPTO_DEPOSIT_TOKEN_* in KMS and confirm the boot log says crypto deposit watcher ENABLED (…).

  2. Restrict the offered token set. CLOSED in v1.50.19. GetCryptoOptions used to serve the MPC processor's list (btc, eth, matic, avax, lux, zoo, arb, op, base…) — which answers "can I DERIVE an address here?", never "can this be credited?" — so it walked buyers toward assets the watcher credits none of. It now projects the watcher's configured assets through offeredFrom, so the menu cannot name an asset nothing is watching, and an unconfigured watcher offers nothing rather than everything. Nothing to remember at flip time: the list narrows and widens with the config by construction.

    ⚠ offeredFrom is a PURE function and tested as one, deliberately. The first test drove the handler end-to-end and passed against a mutant serving a hardcoded list — the custody-signer probe fails in a test environment, so the handler answered 503 and the assertion never ran. Mutation-test anything that guards money; a green endpoint test can be measuring nothing at all.

  3. GenerateAddress DISCARDS the keygen wallet_id. CLOSED in v1.50.20. It returned a bare string, so the id was parsed off the keygen response and dropped — commerce minted custody addresses it held no handle to, and a credited deposit could not be SWEPT without reconciling against the node's own records. GenerateAddress returns a processor.Wallet{Address, ID} and CryptoPaymentIntent.WalletID records it at mint time, the only moment the signer offers it. The existing method was WIDENED rather than joined by a second one: two ways to mint an address is how the halves drift apart again.

  4. Still open — the creditable set is dollar-pegged tokens only. Native coins (ETH/LUX/ZOO/MATIC/SOL…) cannot be priced without an oracle, and on the EVM they emit NO log on a bare transfer so eth_getLogs cannot see them either. Same for XRP, and additionally XRPL's 10 XRP base reserve is non-refundable, so it wants ONE pooled r-address with per-intent destination tags rather than an address per payer. ⚠ "needs no new cryptography" describes address derivation and says nothing about whether a deposit can be credited.

  5. Still open — SOL/SPL and TON have no signable key. THIS is what blocks Solana, not the watcher. As of 2026-08 the watcher CAN credit SPL USDC: billing/solanarpc reads it, and the read path is proven against mainnet. The minting half is not there. frost.KeygenTaproot is BIP-340 secp256k1; its 32-byte output passes mpcd's len(EDDSAPubKey)==32 gate and base58-encodes into a real-looking Solana address the ring can never sign for. The empty eddsa_pub_key is the system failing closed — leave it until a genuine RFC 8032 ciphersuite exists in luxfi/threshold.

    The two halves are gated INDEPENDENTLY and both must be satisfied: thirdparty/mpc.SupportedChains() omits solana (so CreateCryptoDeposit 400s), and depositwatch.chainFamily now includes it (so the watcher can credit one). Configuring CRYPTO_DEPOSIT_TOKEN_SOLANA_USDC today is inert and safe: nothing can mint a Solana address to send to.

    ⚠ That independence created a NEW way to build a dead end, closed in the same change: GetCryptoOptions derived its menu from the WATCHER alone, so configuring Solana would have put a Solana button in front of a buyer that 400s after they chose an amount — the endpoint's original defect wearing the opposite mask. offeredFrom(assets, mintable) now intersects both, so an asset is offered only when it can be RECEIVED and CREDITED.

zip / ZAP native — what is real, and why billing is NOT typed yet

zip is the ZAP-native framework: a typed op (zip.Get/Post[In,Out]) is ONE value with THREE projections — REST, OpenAPI, MCP. Only typed ops populate app.ops; untyped app.Get(path, func(c *zip.Ctx) error) projects to REST and nothing else. Commerce has exactly one zip route (/_/commerce/healthz) and it is untyped.

zip is pinned at v1.8.3 (was v1.2.0). The climb v1.2.0 → v1.8.3 has exactly ONE breaking change: v1.6.0 removed App.Mount. It was a one-liner (a.All(prefix+"/*", AdaptNetHTTP(h))), so mount.go now calls that directly — same route set. v1.3.0 swapped gofiber/fiber/v3 → zap-proto/fiber/v3 (transparent; commerce never imports fiber). v1.8.3 alone adds App.Authorize and App.Prepare — v1.8.2 and earlier have NEITHER.

Why the /v1/billing surface is still gin (four independent blockers)

  1. Typed ops bind the BODY ONLY. registerTyped passes c.Body() and nothing else; for GET/HEAD it does not even read that, so In is the zero value. There is no path-param, query-param or header binding at ANY version through v1.8.3. GET /billing/invoices/:id, ?customerId=, and Deposit's X-Idempotency-Key (which makes the mint exactly-once) are all unexpressible without a wire change. Changing the money wire to satisfy a framework is backwards.
  2. The MCP projection bypasses route middleware. mcpCall invokes the op core directly, so middleware.PlatformOnly never runs. On v1.8.2 (what cloud pins) there is NO authorizer, so this is ungateable: a typed billing.deposit would be an unauthenticated mint tool. App.Authorize (v1.8.3) closes it and gates REST + MCP with one decision — but it is a single app-level SETTER, so on cloud's shared App the last subsystem to call it silently wins.
  3. The mint gate is keyed on the CONCRETE TYPE *gin.Context. Organization.Namespaced type-switches: a *gin.Context gets mintauth.WithGate; anything else is treated as an internal cron and is NOT gated (mintauth.Require returns nil when !Gated). A typed op holds a plain context.Context, so moving a ledger handler off gin silently disarms the mint invariant — no compile error, no test failure. Pinned by models/organization/namespaced_ctx_test.go:TestNamespaced_InternalCallerUngated.
  4. The tenant key is only in the gin context. GetOrganization(c) is c.MustGet("organization"). A typed op has no org ⇒ no tenant scoping.

Blockers 3 and 4 are the real work, and it is NOT a billing change: identity + datastore ctx must flow through context.Context (set once at the boundary) before ANY handler can be typed. That is the actual phase 1.

mount_test.go:TestMount_NoUngatedTypedOpSurface is the tripwire: it asserts /mcp 404s after Mount+Prepare (commerce registers zero typed ops), and fails the moment one appears. Verified to fire.

/v1/billing is DEAD through the cloud mount

mount.go calls api.Route(apiGroup) on the /v1 gin group, registering ~140 /v1/billing/* routes — but the zip mount only exposes /v1/commerce/* and /_/commerce/*. Measured through Mount: /v1/billing/balance and /v1/billing/deposit return 404 with fiber's body ({"status":404,...}, not gin's NoRoute {"error":"not found"}) — the request never enters the gin engine. The comment at mount.go claiming it wires "the full Commerce API surface (/v1/billing, …)" is false for every prefix except /v1/commerce. In cloud, /v1/billing/* is served by cloud's own clients/billing + clients/account wildcard, NOT by commerce. Do not "fix" the prefix casually: exposing 140 routes into cloud's tree collides with those, and a collision is a BOOT PANIC.

ONE ADDRESS, ONE DECLARATION. This paragraph used to say byte-identical patterns "MERGE silently, first wins". Measured 2026-08-06, they do not — zip refuses the whole program and names every conflicting pair:

zip: this program does not compose, so it has no projection:
zip: GET /v1/commerce/collection: declared by "/collection" at rest/rest.go:133
  (via root → /v1/commerce → /collection) and by "/collection" at rest/rest.go:133

Where it lands depends on the zip in the graph, and a check that only REGISTERS measures nothing: this repo pins v1.24.2, which refuses inside rest.Route(); cloud resolves v1.25.1, which accepts the second call and refuses at COMPOSITION (GetRoutes(), serve). Ask for the composition.

The live consequence: api/resources is also called by api.Route (api.go:125), so a kind added to that leaf while api.go still registers it panics this binary at boot — and the change that caused it may have been made in cloud. Moving a kind onto the leaf is a move, never an add.

schema/commerce.zap — dialect and the real compiler

schema/commerce.zap was proto-ish fiction (// comments, service/message, = 1 field numbers) that no tool could read. It now COMPILES, and is the only .zap in the fleet that does — ai, kms, vfs, iam, o11y and api/billing/billing.zap all fail with expected 'package' declaration.

  • The compiler is zapgen (hanzo/zap/go/cmd/zapgen, Go-only: -out/-single/-type-suffix), NOT zapc. zapc is the Cap'n-Proto-lineage RUST codegen PLUGIN that reads a code_generator_request on stdin; it has no generate --lang go|ts|py|rust verb. That command, documented in every .zap header in the fleet, does not exist.
  • Dialect: # comments; package REQUIRED first; only struct / interface / type (no enum); lowercase primitives (text i64 i32 bool bytes list<T>); methods are name(param: Type) returns (out: Type). The braceless whitespace form is sugar — Desugar adds the braces and auto-assigns each field's @byteOffset.
  • Verify: GOWORK=off go build ./cmd/zapgen && zapgen -out /tmp/gen schema/commerce.zap emits a BillingClient + BillingHandler + DispatchBilling ordinal router that builds clean.
  • The generated client is not committed: nothing consumes it, and the hand-rolled ZapDispatch (api/billing/zap.go) already serves that surface. Committing it would create a second, competing billing client. interface Billing mirrors exactly the five methods ZapDispatch really routes — method ordinals are wire identity, so APPEND only.

Gotchas

  • New ORM kinds MUST be registered in util/hashid/kind.go (monotonic, never reorder) or Create() panics "Unknown kind" — sbom-record=262, oss-accrual=263, B2B company=264/employee=265/quote=266/quote-message=267/approval=268, gift-card=269/gift-card-redemption=270, exchange=271, idempotency-key=272, husd-issuance=281 (chain-backed credit ledger).

Medusa parity (native Go /v1 — 1.42.41)

The /v1 model bundle (api/api.go, mounted at /v1/* by cmd/commerce

  • mount.go; the /v1/commerce/* prefix is the SEPARATE checkout/tenant surface) covers Medusa v2's admin domains natively — no Medusa/Node fork. Reference: ~/work/medusa/medusa/packages/medusa/src/api/admin/* + packages/modules/*.

Newly wired (models existed, routes were orphaned → 404 in prod)

api/api.go now calls these 5 previously-unwired sub-routers:

  • fulfillment — fulfillmentset, servicezone, geozone, shippingoption, shippingoptionrule, shippingprofile, fulfillmentprovider, POST /fulfillment/:id/ship|cancel
  • tax — taxregion, taxrate, taxraterule, taxprovider, POST /tax/calculate
  • customergroup — customergroup + /:id/members, customergroupmembership
  • apikey — publishableapikey, role, apipermission
  • notification — notification

Net-new native domains

  • Gift cards (models/giftcard, models/giftcardredemption, api/giftcard): gift-card code+initial balance; balance is a PROJECTION initial − Σ redemptions (no mutable counter to race). POST /giftcard/:id/redeem|void| + GET /balance|redemptions. Admin-gated. Redeem is idempotent (see money design below).
  • B2B (models/company|employee|quote|approval, api/b2b): company → employees (spending limits, RemainingSpendCents/WithinLimit pure fns) → quotes (RFQ, POST /:id/accept|reject, message thread) → approvals (approval.NextStatus pure transition, POST /:id/approve|reject).
  • Order exchange (models/exchange, api/exchange): return + replacement, DifferenceDueCents, POST /exchange/:id/confirm|cancel.
  • Idempotency guard (models/idempotencykey): reusable Begin/Complete for money-moving requests; wired into api/checkout Refund via X-Idempotency-Key (replay returns stored response; in-flight → 409).

Platform product catalog — commerce is the CMS SOT (1.42.41)

models/catalogentry (catalog-entry, kind 273) is the source-of-truth for the platform product catalog (Models, Vector, KMS, …) — the list docs. + the console sidebar + pricing all derive from. commerce owns the DATA (source + seed + edits); @hanzo/products (hanzoai/ui/pkgs/products) owns the SCHEMA + the iconKey→component and brandColor→css code-maps. Conform to its CatalogEntry exactly — the shape is NOT ours to change.

  • GET /v1/commerce/catalog?brand=<b> (public, no auth, on the commerce public group so it serves that exact path) → {brand, categories[], products[]}; products is the @hanzo/products CatalogEntry[] (the client reads .products, tolerating a bare array or {catalog|data|items} too).
  • CatalogEntry contract fields (models/catalogentry/projection.go Item): id (== slug, unique), name, category (EXACTLY one of the 13 canonical labels), brandColor (a swatch KEY like "violet" — NOT hex; client maps key→css), iconKey (a lucide export NAME like "Brain" — NOT a component), slug, route ("/"), docsUrl ("https://docs.hanzo.ai/docs/services/"), apiPath (/v1-prefixed), pricingId (plans/.json key OR null — emit *string, nil→null). brands is a category-DERIVED convenience, never a hand-authored filter. Extra fields (gcp/repo/admin/status/priceCents/order/ productId) are additive — the client ignores unknowns.
  • Brand scope is by CATEGORY (categoriesForBrand, mirroring @hanzo/products catalogForBrand/BRAND_CATEGORIES): hanzo=all 13; lux/zoo/pars=5 (Web3/Network/Security/Dev/Settings). There is NO per-entry brand filter — the same system-namespace store serves every brand; an entry shows iff its category is in the requested brand's set. Slug is globally unique.
  • Admin CRUD /v1/catalog/entries (+ /seed) gates on auth.IAMClaims.IsSuperAdmin() — the catalog is cross-tenant PLATFORM data in the system namespace, so an org-level admin must NOT edit it. Keyed by slug.
  • Seed: the embedded seed/hanzo-catalog.json is the @hanzo/products snapshot (hanzoai/ui/pkgs/products/snapshot/catalog.json) VERBATIM — 95 products with correct brandColor/iconKey/route/docsUrl/apiPath/pricingId. SeedIfEmpty auto-runs on first boot (count-gated no-op once populated; COMMERCE_CATALOG_SEED=false to skip). Idempotent + non-destructive — never clobbers CMS edits. Re-baseline by re-copying the @hanzo/products snapshot.
  • Namespace: platform-global (system ns), namespaced via CONTEXT (nscontext.WithNamespace) — struct SetNamespace is a no-op for queries (they read ns from d.Context), a latent gotcha api/billing/osspayout systemDB also trips (works only because its reads+writes are symmetric in default ns).

Enso's prices are the model API's, read back on every boot (catalogentry.Reprice)

Each paid Enso SKU sells at ONE fixed price set in hanzoai/ai controllers/rates.go (ensoPrices, owner 2026-10-15): enso-flash 0.33 / 1.65, enso-pro 4.40 / 22.00, enso-ultra 5.50 / 27.50 a million tokens, cache reads and writes in the same multiple. enso is the router over the three (ai v1.833.349), billed at the tier that answers and listed at a request's that states no depth, enso-flash's; enso-free is the one free model. ai lists and bills those numbers whatever a catalog row says, so an Enso price is not an admin's merchandising choice. seed/enso-models.json carries it (seed_test.go billedRetail is the second witness, moved in the same commit), and runEnsoCatalogSeed calls Reprice after the first-boot seed: each Enso row the seed names takes the seed's price, rate by rate (a missing class is added), compared as decimals so twice is once. Cost (a sync's), name, order, published, a row the seed does not name (enso-free) and a deleted row are untouched. A new SKU in the seed (enso-pro) is created only on a store the seed has never run on; production's admin adds it, or the price list (cloud apps/pricing/enso.go) appends it from ai's answer.

Model catalog — the OpenRouter cost sync (models/catalogentry/{openrouter,sync}.go)

A model IS a catalogentry (model.go): same kind, same system store, same super-admin CRUD, same margin projection. Price is a Rate VECTOR (rate.go) — cost is what WE pay (synced), price is what the CUSTOMER pays (set in admin), margin is derived. UpsertModels is the ONE write seam and enforces costOnly. This section is the PULL half: reading a live upstream on a schedule.

  • FetchOpenRouter / DecodeOpenRouter — split on purpose: the fetch is the only I/O, so every rule about what a payload MEANS is testable against testdata/openrouter-models.json (captured from live) with no network. Decode FAILS on a malformed or empty body rather than returning a partial read: a half-decoded catalog looks exactly like an upstream that dropped 200 models, and the sync's answer to that is to withdraw them, so the two must be told apart before that decision is reached.
  • Per-token → per-MTok is EXACT: dec(s).Shift(6), decimal all the way, no float. 0.000000003625 → 0.003625. A component the upstream prices at ZERO is omitted, never stored as cost "0" (which would read as 100% margin); a fully free model keeps its row with no rates — it is still a model and this catalog lists every model.
  • Vendor is the PARTY, not party+product — "Meta", never "Meta Llama". resolveVendors settles ONE name per NAMESPACE across the whole payload (per-model derivation gives "OpenAI" and "Openai" for one vendor). It votes on the upstream's OWN display prefix, so a vendor added tomorrow is named right with no edit; vendorNames corrects only the ~5 namespaces publishing no prefix. ~vendor/x-latest aliases are listed and marked floating in metadata — the id does not pin a version, so its cost can move under a stable id.

Refresh(db, serves, rows) wraps UpsertModels with the two things a scheduled PULL needs that a push of decided rows does not:

  1. Withdrawal sweep. Upsert only sees what the upstream still publishes, so on its own a withdrawn model stays listed and routable forever. Refresh withdraws the difference: Spec.Enabled=false + Spec.Unavailable = WithdrawnUpstream. MARKED, NEVER DELETED — usage rows, invoices and old conversations reference it by slug. It stays published and priced. A later run that finds it restores it, but ONLY if WE withdrew it: an operator who disabled a model by hand keeps that decision.
  2. Fail-safe. ErrEmptyUpstream — zero models is indistinguishable from an outage, and believing it would withdraw the whole catalog in one run. A failed sync writes nothing and the last good catalog stands.

The sweep is scoped by Spec.Serves, so an OpenRouter outage can never withdraw an Enso or Zen model.

freezePrice — why a new row is priced explicitly. RetailPrice falls back to cost × markup when no price is stored, which spares a 341-row long tail from hand-pricing. But a price COMPUTED FROM A LIVE COST moves whenever the upstream moves — raise their cost and every bill rises, decided by nobody. Refresh therefore writes the retail price out explicitly at CREATION only: the row arrives at cost × markup (so landing it changes no bill and nothing needs hand-pricing), and from then on the price is a stored number only a human changes, so the next upstream move surfaces as a MARGIN THAT MOVED in admin.hanzo.ai. Creation is the only legitimate moment — the row is new, so there is no customer relying on a price to disturb. NOTE: a row created by hand without a price still tracks cost through the fallback; only synced rows are frozen.

  • Route: POST /v1/catalog/models/refresh (super-admin). POST /v1/catalog/models remains the PUSH of decided rows; both funnel through UpsertModels, so the cost/price rule holds whichever endpoint a row came through. Scheduled by a CronJob curling it with COMMERCE_SERVICE_TOKEN, the same shape the contributor payout uses. It deliberately does NOT run at boot — a boot-time call to a third party is a boot hazard.
  • /catalog/entries/* is a WILDCARD, not :slug. A model's slug IS its callable id and those contain a slash; a segment param stops at the slash, so every model row would be listed and un-editable. Measured on this router: fiber's + param does not bind, * does, and it matches a single-segment slug identically (infra-tier edits unchanged). Pinned by TestAdminRoute_AddressesASlashBearingSlug, which drives the real AdminRoute.
  • Proven against live OpenRouter: 341 models created; a second run against the same payload creates, withdraws and restores nothing; halving the upstream withdrew 171 and deleted ZERO (all 341 rows still present); recovery restored 171. claude-opus-5 lands at cost 5/25 → price 6/30 per MTok, margin 16.67% on each rate; the public projection carries price and no cost. Refresh is stable on STATE — UpsertModels still rewrites every row each run, so the write count is not yet a no-op.

Money-correctness design (idempotency WITHOUT working transactions)

  • datastore.RunInTransaction (datastore/datastore.go) is a NO-OP — it runs fn(New(ctx)) with no tx/lock/isolation. db.SQLiteDB.RunInTransaction IS real (writeMu.Lock+BeginTx) but mixin.Model[T] routes to the no-op. So model-level read-modify-write is NOT atomic.
  • A read-modify-write swaps, it does not put (db.Swapper, datastore.Swap, mixin.Model.Swap): the write lands only while the stored entity's updatedAt is still the one read (insert-if-absent, a deleted row counting as absent, when it was never stored); otherwise it writes nothing and the caller reads again, at most mixin.Swaps times. One statement on both backends, so READ COMMITTED is enough and two processes over one store cannot lose each other's change. Used by refund and dispute recording (engine.RecordRefund/RecordDispute, paymentorg.RecordDispute) and by idempotencykey.Begin, whose first sighting and stale re-claim are swaps, so a guarded money move runs once across replicas; the in-process returns and lockPeriod mutexes are gone. api/transaction's lockFunds stays in-process: its routes mount only standalone (single writer), never in cloud. Pinned by TestReturnsAcrossTwoReplicasAreNeitherLostNorDoubled, TestASwapWritesOnlyOverWhatWasRead and TestAKeySightedAtOnceOnTwoReplicasRunsOnce (two store handles on one file).
  • The money-safe primitive used here: deterministic-id ledger records. A redemption/guard's STORAGE id = sha256(scope‖key) via orm.WithStringKey. Concurrent duplicate submits collapse onto ONE row via the storage ON CONFLICT(id,kind,namespace) upsert — proven (giftcard 25-goroutine same-key test → 1 debit). Balance is a projection so distinct concurrent redemptions are additive, never lost. Refund guard replays the stored response. The narrow concurrent-first-submit window is closed at the gateway (Square/Stripe honor the same idempotency key on the refund call).
  • GOTCHA — a struct field named Key SHADOWS Model[T].Key() and silently breaks mixin.Entity (.Query() returns nil entity → nil panic in Get). Name idempotency fields IdemKey, not Key. Compiler catches it only via an explicit var _ mixin.Entity = (*T)(nil) assertion.
  • GOTCHA — namespace scoping keys off CONTEXT, not the datastore struct. db.SetNamespace(ns) sets d.namespace but the SQL layer reads nscontext.GetNamespace(ctx) (db/query.go getNamespace). Production is correct (org.Namespaced(c) → nscontext.WithNamespace); tests MUST build the datastore from a namespaced CONTEXT (datastore.New(nscontext.WithNamespace(ctx, ns))), not SetNamespace, or cross-tenant isolation appears broken in tests only.
  • datastore/query/query.go ById switch: all hashid-only Model[T] kinds (no slug/name secondary lookup) MUST be in the "return not-found" case, else a non-hashid id hits default: → HTTP 500 instead of 404. 43 such kinds were missing and are now listed.

Endpoint contract note (console → /v1/commerce)

The model bundle serves /v1/product etc. (bare /v1). console2 today reaches it via a same-origin /commerce BFF proxy (app/commerce/[...path]/route.ts → commerce.hanzo.svc:8001/v1/*). To honor the "console calls /v1/commerce/*, nothing before /v1/" law without breaking the live /v1/billing/* money path, add an ingress rewrite on the console host (console.hanzo.ai/v1/commerce/* → commerce/v1/* strip) rather than re-prefixing the backend bundle.

  • Healthcheck: use curl -f not wget --spider (Gin only handles GET)
  • go-sqlite3 must be pinned to v1.14.x (replace in go.mod) — the transitive v2.0.3+incompatible fails to compile on musl/Alpine (pread64/pwrite64)
  • Meilisearch v0.35.1 changed AddDocuments/DeleteDocuments signatures
  • Production Dockerfile uses CGO_ENABLED=1 for SQLite (not CGO_ENABLED=0)
  • Global entities (Organization, User, Token) use DefaultNamespace = true by design
  • PVC is ReadWriteOnce — use Recreate deployment strategy, not RollingUpdate
  • Filter field names are PascalCase (Go struct) — auto-converted to camelCase JSON
  • Boolean false with omitempty may be omitted from JSON — COALESCE handles NULL=false

Un-fork: THIS repo is canonical; cloud imports it (v1.47.0)

Cloud's inlined fork (hanzoai/cloud clients/commerce/, no go.mod) is being retired: cloud import github.com/hanzoai/commerce + a thin subsystems adapter. v1.47.0 = content parity with the fork's last cloud-side deltas:

  • product.created/updated events (events/, api/product_events.go) — storefront loop
  • middleware/accesstoken.go: service-token checked BEFORE the IAM branch (S2S metering dispatch must never hit the per-user scope gate) + ensureIAMOrg (org from gateway X-Org-Id via svcorg.Resolve; IAM is the one org authority)
  • sessionless webhook fixes: resolveWebhookOrg uses GetOrganizationOK (no panic on provider deliveries); square provider seeds from env at init so webhook validation works with no tenant resolver in front
  • Embedded.brand (set by the mounter; surfaced in the in-process OrgConfig)
  • v1.47.1: api/api flattened to api (package api at api/; the old GAE-era api/main.go binary moved to cmd/api) + the service-token-precedence regression tests live HERE with the accesstoken fix they pin NOT ported (deliberate): the fork's GCP-style BillingAccount (Default/LimitCents/projectbinding budget cluster) — the money-of-record BillingAccount + Binding + resolveBilling chain HERE is the one design; the fork's spend-cap/budget features must be re-expressed on Binding (#70), not imported as a competing model. Also not ported: thirdparty/ethereum (dropped deliberately, LGPL), and the cloud bridges (finance-coupled metering client, in-process entitlement client, Deps adapter) — those live cloud-side.

v1.48.0 — NATIVE zip (zap-proto/fiber): gin is GONE

Whole repo runs on zap-proto/zip v1.7.5 (zap-proto/fiber underneath). No gin, no gofiber, no net/http adaptation in the serving path. Handlers are func(*zip.Ctx) error; returning the render IS the abort (no gin Abort()).

  • ONE request context: middleware.RequestContext() does c.SetContext(mintauth.WithGate(...)) at the boundary; everything reads c.Context(). GetContext/Locals("context") are DEAD. middleware.AuthorizeMint SetContexts the authorized ctx — the ledger sink reads it there.
  • Route chains are gin-order (middleware…, handler LAST) — zip ≥v1.7.4 executes them in that order (v1.7.4 fixed an inversion; chain_order_test pins it).
  • Empty leaf = group root (zip ≥v1.7.5 normPath); fiber param names are dash-free (rest.ParamId squashes dashes: kind "product-option" → param "productoptionid").
  • Co-residence contract: EmbedConfig.App *zip.App / Config.SharedApp — commerce registers routes ON the host's app (one router, one specificity space); standalone-only surfaces (healthz, legacy /admin SPA, checkout SPA catch-all, Listen/TLS) are skipped when embedded. Embedded.HTTPHandler/HTTPAddr DELETED → Embedded.Zip().
  • SPA serving: zip.Static(+WithIndex/WithFallback) for admin (ui.FS); checkout SPAHandler is a native zip handler (same security headers).
  • Test idioms: zipclient (ex ginclient) drives via the fiber adaptor with SERVER-style requests; PostForm encodes the body (fasthttp parses bytes, not req.PostForm). zip TestCtx for direct handler calls.
  • Pre-existing debt: thirdparty/reamaze verifyHMAC has an inversion bug (flagged, not fixed here).

Licensing — MIT OR Apache-2.0 (HIP-0137)

The repo is dual-licensed per HIP-0137 "One License" (hanzoai/hips, HIPs/hip-0137-one-license.md). Three root files, one declaration each, no restating:

  • LICENSE — the dual offer + SPDX id + inbound=outbound contribution clause. Points at the two texts and at NOTICE. Copyright 2014-2026 Hanzo AI, Inc. (the real span: the code dates from 2014-09-29, when it was kaching).
  • LICENSE-MIT — canonical MIT, Hanzo copyright line.
  • LICENSE-APACHE — canonical Apache-2.0, BYTE-IDENTICAL to apache.org/licenses/LICENSE-2.0.txt. sha256 cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30, 11358 bytes, git blob d645695673349e3947e8e5ae42332d0ac3164cd7. Verify with sha256sum / cmp / git hash-object — never diff (it is a shell function on the dev box and has reported "differs" on byte-identical files). Any alteration to operative text means the file is no longer that licence.

Every package.json in the tree (15 of them, all under app/) declares "license": "MIT OR Apache-2.0". There is no Cargo.toml or pyproject.toml here, and go.mod has no licence field, so the manifests are the whole surface.

NOTICE is the third-party attribution file — it scopes vendored dependencies, not this repo's own terms. Its btcd/btcec (ISC, replace/btcec) and Go stdlib (BSD-3) sections are untouched. Appended: Medusa (medusajs/medusa, MIT, Copyright (c) 2021 Medusajs), which was vendored but unattributed. The whole TypeScript frontend under app/ is rebranded Medusa — app/admin is @medusajs/dashboard (its CHANGELOG.md still carries that header), app/site is the Medusa Book (upstream Cloudinary "Medusa Book" asset URLs are still inline), app/packages/* are the corresponding @medusajs packages, and app/store derives from medusajs/nextjs-starter-medusa (MIT, Copyright (c) 2022 Medusa; its yarn.lock still resolves medusa-next@workspace:.). MIT grants sublicensing, so offering the whole under MIT OR Apache-2.0 is sound provided the upstream copyright notice is retained — that retention is what the NOTICE append is for. Upstream ships no NOTICE file, so nothing further propagates.

The Go backend is not a Medusa fork — it is an independent native implementation that reaches parity with Medusa v2's admin domains (see "Medusa parity" above). Only app/ carries upstream lineage.

That file was the upstream starter's MIT licence with line 3 changed from Copyright (c) 2022 Medusa to Copyright (c) 2024 Hanzo AI. Every other byte matched upstream. It is restored: the file is now byte-identical to medusajs/nextjs-starter-medusa's LICENSE — sha256 770a61c897d73cdeb53ff2a367d537a80eccf57b31f81e4a5fe951fecc40c632, git blob 94def62e7e4e5016c3c4af42bc426beb3ec483e8, which is the blob id GitHub reports for the upstream file itself.

The rule is "never edit an upstream LICENSE," and restoring this is not an exception to it — it is the rule being enforced. The rule exists to protect upstream attribution. This was not an intact upstream licence being altered; it was an already-altered one, altered in the single way MIT forbids: MIT conditions the grant on "the above copyright notice ... shall be included in all copies or substantial portions of the Software." A swapped holder line is a continuing breach for as long as it ships. Restoring is repair, not edit. The test to apply next time: does this change move the file toward or away from what upstream published? Toward = repair. Away = the thing the rule forbids.

This is the second confirmed instance of this failure class, after the Papermark incident (a branding pass retitled Papermark's commercial licence and swapped its copyright holder over 100% upstream code). Both were produced by rebrand-everything sweeps that treated a licence file as branding surface. Licence and NOTICE files are not branding surface.

Swept the rest of the repo for the same swap. Tracked licence files repo-wide are exactly five: the three root ones, NOTICE, and app/store/LICENSE — no others, nested or otherwise (the node_modules/ and .next/ hits a filesystem grep returns are untracked build output, 0 tracked). app/store/LICENSE was also the only Hanzo copyright claim anywhere in the vendored app/ tree: no Medusa source file was stamped with a Hanzo header. The vendored btcd/btcec under replace/ keeps its ISC notices in-file (22 of 44 files carry the header; the rest are go.mod/go.sum/README/testdata).

Per-file Go headers — all 72 now agree with the root

Every Hanzo-authored Go file carries the same two-line header, and it says the same thing the root LICENSE says:

// Copyright (c) 2014-present Hanzo AI, Inc.
// Licensed under MIT OR Apache-2.0. See LICENSE-MIT and LICENSE-APACHE.

That replaced two stale forms: // Copyright © 2026 Hanzo AI. MIT License. (71 files, understated the dual offer) and, in cmd/commerce/telemetry.go, // Copyright 2023-2026 Hanzo AI Inc. All Rights Reserved. (1 file, a proprietary reservation left over from before the relicense). Headers are what licence scanners actually read, so a header contradicting the root is the contradiction that gets found first.

The span is 2014-present — one string, everywhere, matching LICENSE, LICENSE-MIT and NOTICE. It is the true span: the code dates from 2014-09-29, when this was kaching. The old 2026 was wrong on its own terms, post-dating files that had existed for a decade. Do not reintroduce a per-file year convention — nobody maintains one, and it drifts from the root within a release.

Six of the 72 have a //go:build constraint below the header. The rewrite replaced line 1 in place rather than inserting, so the constraint keeps its position behind blank-lines-and-line-comments only — the one way this breaks silently. go vet ./... is the gate that proves it (its buildtag analyzer flags a misplaced constraint); it is green, as is CGO_ENABLED=0 go build ./....

Nothing vendored was restamped: the only other Go files carrying a copyright line are the 33 under replace/btcec, which keep their upstream btcsuite and Decred ISC notices verbatim. If a file's header is upstream's, it is not ours to restamp — git ls-files the path before touching it, and note that app/node_modules/ and app/store/.next/ are untracked build output, not repo content.

Build note unrelated to licensing: plain go build ./... fails under cgo with undefined: cgoBuildNeedsSQLiteMathFunctions from hanzoai/base. That is a deliberate compile-time guard in the dependency (//go:build cgo && !sqlite_math_functions) telling you to use the pure-Go backend or add the tag. It reproduces identically on a clean tree, and CGO_ENABLED=0 — what the Dockerfile ships — is green. Not caused by the headers.

Public repository, one root

hanzoai/commerce is public and starts at a single root commit. The history before it carried third-party credentials, customer data and old customer orgs, so it was not published: every branch, archive ref and tag the repo used to carry lives privately in hanzo-inc/commerce under refs/archive/public/*.

Test data is fictional and stays that way: the Such Tees store in models/fixtures, example.com and acme.test addresses. Never a customer org, a real person, a real address or a captured provider payload.

Custody sweep — the half of the crypto rail that SPENDS (2026-08)

The deposit watcher made money arrive and be credited. Nothing could move it out. What stood in for a spend path was five calls from thirdparty/mpc to POST {mpcEndpoint}/api/v1/transactions, asking the signer to build, sign and broadcast a chain transaction.

mpcd serves no such route. Verified against a running mpcd node: /api/v1/transactions → 404, /sign → 401, /keygen → 401. Its internal API (:9800) has exactly five routes — /healthz, /keys, /backup, /keygen, /sign. The similarly-spelled POST /v1/transactions on the node-0 dashboard server is not a builder either: it records a row and, if policy approves, broadcasts a raw_tx the CALLER already built.

Those calls are deleted, not repointed. A threshold signer that knew about chains, tokens and amounts would be a wallet; custody is safe because the component holding the key shares only ever sees a digest.

Shape

  • billing/custody — the vocabulary. Chain drafts and broadcasts; Signer turns a digest into a signature; Sweep is the twenty lines between. Draft.Digests is PLURAL from the start so Bitcoin needs no special case. Draft.Seal proves every signature belongs to the address being spent before it will assemble anything — that is where a wrong wallet id, a reshared key and a wrong-curve answer all land.
  • billing/custody/{evm,bitcoin,solana} — one package per chain family.
  • thirdparty/mpc.Sign — the mpcd client, implementing custody.Signer.
  • Transport lives with the existing readers: billing/husdindex (EVM), billing/bitcoinrpc (Esplora), billing/solanarpc. Each grew the calls a spend needs; none grew a second JSON-RPC client.

mpcd's /sign contract — the parts that bite

  • Body is {org_id, wallet_id, network, payload_hash, idempotency_key}. All five are required except wallet_id (which fails later, at share lookup).
  • There is no curve field and must not be one. The daemon derives the curve from network deliberately. key_type — which luxfi/mpc's own stale integration test and lux/wallet's custody adapter both still send — decodes into nothing and yields 400 unknown network: (empty).
  • A signing FAILURE arrives as HTTP 200 with status:"rejected". 503 is reserved for no-quorum. Trusting the status code reads an empty signature as a good one.
  • secp256k1 answers 65 bytes r‖s‖v (v ∈ {0,1}, low-S normalised per EIP-2, which is also Bitcoin's BIP-62 rule). Ed25519 answers 64 raw bytes.
  • payload_hash is hex, signed UNTOUCHED — no hashing anywhere in the daemon. secp256k1 requires exactly 32 bytes; Ed25519 takes any length, because it hashes internally and a pre-hashed payload would be a digest of a digest.
  • The idempotency key is derived from the content (sha256(tag‖org‖wallet‖network‖digest)), so a resumed sweep collects the signature it already paid for and two transfers can never collide. A random key would strand a signature after a timeout and sign a second transaction over the same coins.

Chain facts worth not rediscovering

  • avalanche and zoo can be minted and never signed for. Both are in thirdparty/mpc's mintChains; neither is in mpcd's alias table (400 unknown network). custody.curves refuses them locally.
  • BTC addresses are legacy P2PKH, and mpcd's docs are wrong. pubKeyToBtcAddress is base58check(0x00 ‖ hash160(compressed pubkey)); docs/content/docs/bitcoin.mdx shows NewAddressWitnessPubKeyHash, NewAddressTaproot and CalcWitnessSigHash. Building from the docs produces a segwit input for a legacy output. That derivation has NO test in luxfi/mpc — the only address there derived by hand and the only one with no vector — though it is correct on the live path (proved here against a mined tx).
  • btc_address is unconditionally mainnet. Version byte 0x00 is a literal; a testnet deployment still publishes 1…. custody/bitcoin.New therefore takes *chaincfg.Params explicitly.
  • XRPL is secp256k1 on this rail, not Ed25519.
  • A Bitcoin sweep has NO change output — it takes everything to one address, so a partial amount is refused rather than quietly given change.
  • Solana's address IS the Ed25519 public key, so Seal verifies directly rather than recovering.

What is proven, and how

Each chain is anchored on a signature this code did not produce:

  • EVM — digest and assembled transaction byte-identical to eth-account's for a signed USDC transfer; EIP-155's published vector recovers to its published address; live, the deployed mainnet USDC contract accepts our transfer calldata (EVM_LIVE=1).
  • Bitcoin — our signature hash verifies the signature in mainnet block 961911; our scriptSig rebuilds that block's bytes exactly; a real testnet sweep was built, signed and broadcast by this code (21bf975dd58f84b89d16f2a63d17a1379e45949fd6891ec6cbcdb4fa638368e6).
  • Solana — a finalised mainnet message round-trips through our builder byte for byte, and its signature verifies over the raw message and NOT over a hash of it; live, the mainnet runtime executed our drafted transfer successfully and, with sigVerify on, accepted our signature before failing on AccountNotFound (SOL_LIVE=1).

Not built

TON cannot be swept, and XRPL cannot be swept. TON needs cell/BOC construction (a new dependency or a hand-rolled serializer), a seqno get-call, and — on a wallet's FIRST outgoing message — the StateInit carried inline. Neither testnet funds nor an unthrottled toncenter key were available to prove it to the standard the other three meet, so it was not shipped. CRYPTO_DEPOSIT_RPC_TON

  • _TOKEN_TON_TON are armed in production; until a TON sweep exists and is proven, disarming them is the only way to stop that hole growing.

Nothing CALLS custody.Sweep yet: there is no treasury destination in config, no schedule, and no durable record of a sweep. That is deliberate — rollout is a human decision.