Files
cloud/resource_billing.go
Hanzo Dev 92441c6a98 billing: resolve the caller before the money plane is asked to price a spend
ResourceMeter.Gate is the one pre-create gate all 23 priced surfaces call, and
handed an empty org it asked the money plane to price a spend for a nameless
subject. The money plane then answered a question about IDENTITY in the
vocabulary of MONEY, in two measured shapes:

  co-resident ledger — metering refuses an empty org fail-closed, which is not a
    4xx, so the wire's fallback renders 503 "Billing temporarily unavailable":
    the caller is told the biller is broken.
  peer ledger — gatePeer ships AuthorizeIn{Subject:""}, commerce's own
    validate:"required" rejects it, and because that refusal IS a 4xx the wire
    preserves it verbatim: 400 `field "subject" is required` — a field that
    appears in no published request schema on any of these surfaces, so no
    caller can ever satisfy it.

apps/risk already carried this guard at its own door; the same disagreement is
reachable wherever a handler's tenant check and principal.Ledger differ.
provisioning is the live instance: create() admits an admin with no org, and an
org over MaxOrgLen, through tenant() — and Ledger answers "" for both. Its seven
routes serve on api.hanzo.ai today.

An empty org is now refused as ErrNoLedger before either branch runs, and denial
— the one decision both renderings read — renders it as the tenant gate's own
403 rather than as a fault of the biller. This changes no allow/deny outcome on
a named caller: an empty org already failed on both branches. It does close a
worse case than the wrong status, proven by the mutation below: under fail-OPEN
with the biller down, an unidentified caller was allowed, so the surface was
free rather than gated.

Mutation-proven via scripts/mutate.py:

  gate: ask the money plane to price a spend for a nameless subject   KILLED
  gate: render an identity refusal as a fault of the biller           KILLED
  risk: widen the empty-ledger guard until it refuses every caller    KILLED

Two rows are deliberately absent and say so in place: "guard below fail-open"
is a semantic no-op (fail-open lives inside Authorize/gatePeer, so no placement
in Gate can follow it), and "delete ops.gate's guard" now SURVIVES — with Gate
refusing, the risk-level guard is no longer load-bearing for the status or the
sentence, only for the envelope.

Co-authored-by: Hanzo Dev <dev@hanzo.ai>
2026-08-03 22:44:23 -07:00

395 lines
19 KiB
Go

package cloud
// Resource billing — the ONE per-org gate+meter every non-LLM "create a
// resource" handler uses, so provisioned infrastructure (databases, buckets,
// caches, vector collections, ML models, training jobs) is paid for, in EVERY
// environment, by the resource's OWN org.
//
// It is the in-handler analogue of BillingGate (the request-edge LLM gate): both
// wrap the SAME canonical metering client (github.com/hanzoai/cloud/apps/metering,
// Deps.Metering) and the SAME commerce ledger. The edge gate prices by PATH and
// runs as middleware; resource billing runs INSIDE a create handler, where the
// caller's org has already been resolved to the exact slug that namespaces the
// backend resource — so the balance check and the debit are guaranteed to target
// that one org, never a default, never another org.
//
// Multitenancy contract (the whole point):
// - org is the caller's resolved org slug (the value that namespaces the
// resource). It is sent to commerce as BOTH the user identity AND the
// X-IAM-Org-Id org header, OVERRIDING the metering client's default org —
// so an unfunded org can never provision on another org's balance, and
// a funded org's spend lands only on its own ledger.
// - Fail-closed: a non-positive balance is 402; a commerce that cannot be
// reached is 503 (unless the client was built fail-open). No free
// provisioning on a billing outage for a priced resource.
//
// Env awareness: the real-money-vs-sandbox decision is structural — each
// deployment (mainnet/testnet/devnet, the 3-env split) points at its OWN
// commerce/Square, so test and dev STILL meter and bill, just against that env's
// sandbox ledger. This gate fires identically in every env; it never branches on
// env to bypass billing. Env is recorded for attribution only.
import (
"context"
"errors"
"net/http"
"os"
"strconv"
"strings"
"github.com/hanzoai/cloud/apps/metering"
luxlog "github.com/luxfi/log"
"github.com/zap-proto/zip"
)
// DefaultResourceFeeCents is the fallback flat provision/create fee (in cents)
// when no operator override is set — $1.00, so "every service costs money" holds
// out of the box. It is a configurable POLICY default, not a fabricated market
// price: ops sets the real number per deployment/kind via ResourceFeeCents's env
// knobs. Set a kind to 0 to make it free (and therefore un-gated, mirroring the
// edge gate's price==0 pass-through).
const DefaultResourceFeeCents int64 = 100
// ResourceMeter gates and meters per-org spend for non-LLM resource creation,
// reusing Deps.Metering (the single commerce billing client). Build it with
// NewResourceMeter. A nil meter, or one whose commerce URL is unset, makes Gate
// allow and Meter a no-op — so an unconfigured deployment is never blocked,
// exactly like BillingGate.
type ResourceMeter struct {
m *metering.Client
provider string // commerce "provider" label for attribution (e.g. "provisioning", "compute").
env string // deployment env (mainnet|testnet|devnet); attribution only — never a billing bypass.
log luxlog.Logger
}
// NewResourceMeter builds a ResourceMeter from the shared deps. provider labels
// the recorded usage so spend is attributable to the surface that metered it.
func NewResourceMeter(deps Deps, provider string) *ResourceMeter {
return &ResourceMeter{
m: deps.Metering,
provider: provider,
env: deps.Env,
log: deps.Logger,
}
}
// Enabled reports whether THIS process holds the ledger — a commerce URL is
// configured here. False no longer means "nothing bills": once apps are their
// own binaries the ledger usually lives one socket away, and Gate asks it.
func (rm *ResourceMeter) Enabled() bool { return rm != nil && rm.m != nil && rm.m.Enabled() }
// Gate is the pre-create balance gate. It returns:
//
// nil -> allow (balance positive, OR not priced,
// OR billing not configured).
// metering.ErrInsufficientBalance -> deny, out of funds (render 402).
// other error -> balance unknown; fail-closed denies
// (render 503). Fail-open returns nil.
//
// costCents<=0 means the kind is free → no gate (mirrors BillingGate's price==0
// short-circuit). org MUST be the caller's resolved slug; it is sent as the
// commerce user AND X-Org-Id so the CALLER's ledger is checked, overriding
// the client default org — the anti-cross-org property. The balance check
// honors ctx (a client disconnect/timeout cancels it).
//
// costCents is forwarded as AuthInput.AmountCents so the gate enforces
// available >= costCents, not merely available > 0 — otherwise a 1-cent balance
// would authorize an arbitrarily expensive charge (the debit still lands, taking
// the ledger negative). This mirrors what a prepaid gate must do: refuse a
// request the balance cannot cover BEFORE the work runs.
// project + projectValidated are the caller's org SUB-SCOPE and whether it is
// bound to a VALIDATED identity claim — principal.ValidatedProject(c), the SAME
// signal the edge BillingGate threads. When validated, a project-scoped spend cap
// (issue #70) on (project, provider) HARD-enforces (402) on resource creation
// exactly as on the request edge; when not, it DEGRADES to soft (org- and
// service-scoped caps stay hard), so a forgeable X-Project-Id can neither
// hard-stop nor be evaded. service is intrinsically this meter's provider. Pass
// ("", false) on a background/no-principal path (org- and service-scoped caps only).
func (rm *ResourceMeter) Gate(ctx context.Context, org, project string, projectValidated bool, kind string, costCents int64) error {
if costCents <= 0 {
return nil
}
// An empty org is an IDENTITY refusal, and it is answered here rather than by
// the money plane, because every way of asking the money plane to price a spend
// for a nameless subject answers in the vocabulary of money: the co-resident
// meter refuses an empty org fail-closed and renders 503 "Billing temporarily
// unavailable", and gatePeer ships AuthorizeIn{Subject:""} whose far end refuses
// with 400 `field "subject" is required` — a field that appears in no published
// request schema on any of these surfaces, so no caller can satisfy it.
//
// It sits ABOVE both branches so a fail-OPEN money policy cannot make an
// unidentified caller free: fail-open decides what to do when the ledger is
// unreachable, never who the caller is. See [ErrNoLedger] and [denial].
if org == "" {
return ErrNoLedger
}
// NEVER CONSTRUCTED is not the same as NO LOCAL LEDGER, and only the second
// is answerable by asking a peer:
//
// rm == nil, or rm.m == nil -> defensive. Serve always builds a meter with
// a client, so this is a construction defect,
// not a deployment shape. Allow, and above all
// never panic — the peer path dereferences rm.
// rm.m != nil, !Enabled() -> real: this process holds no ledger because
// the ledger lives with commerce. Ask it.
if rm == nil || rm.m == nil {
return nil
}
if !rm.Enabled() {
// No meter in THIS process, which is the normal case once apps are their own
// binaries: the ledger has one writer and it lives with commerce. Ask it.
//
// Allowing here — which is what "billing not configured" used to mean — turns
// every priced act free the moment an app is split out, and does it silently.
// The distinction that matters is "nobody bills in this deployment" versus
// "the biller is one socket away", and only the second is answerable.
return rm.gatePeer(ctx, org, project, projectValidated, costCents)
}
return rm.m.Authorize(ctx, metering.AuthInput{
User: org, Org: org, AmountCents: costCents,
// Service (=provider) is server-set → always validated. Project hardens iff
// it is claim-bound (projectValidated) — the anti project-spoof gate.
Project: project, ProjectValidated: projectValidated, Service: rm.provider,
})
}
// Meter records a successful charge to the caller's org ledger. It is the ONE
// metering entry point for BOTH the one-time create fee AND any recurring
// footprint charge (storage GB-month, GPU-hour): the caller supplies the amount,
// so a future recurring meter reuses this same method with a usage-derived
// amount. No-op when billing is not configured or amountCents<=0.
//
// The debit is fire-and-forget on a background context: the resource already
// exists, so the charge must never block or corrupt the response the caller
// received, and a request-context cancellation must not cancel the debit (mirror
// of BillingGate). A debit failure is logged for reconciliation, not swallowed.
func (rm *ResourceMeter) Meter(org, project, kind string, amountCents int64, requestID, clientIP string) {
rm.MeterUsage(org, kind, metering.Usage{
Model: kind, // the billed unit within the product (e.g. "sql", "invoke", "op") — per-item ledger attribution.
AmountCents: amountCents,
Project: project, // scope attribution → the per-scope cap sums over it.
RequestID: requestID,
ClientIP: clientIP,
})
}
// MeterUsage is the general-purpose per-org debit: it records the caller-built
// usage event after forcing the per-org billing invariants that make the debit
// land on the CALLER's ledger and never another org's:
//
// - u.User and u.Org are OVERWRITTEN to the caller's org slug (the per-org
// prepaid billing key + the X-Org-Id namespace) — a caller can never bill
// someone else, and a surface can't accidentally leave them unset (which
// would debit the client-default org).
// - Provider defaults to the meter's provider; Status defaults to "success";
// Currency defaults to "usd".
//
// Everything else the caller supplies (AmountCents, Model, Actor, RequestID,
// token counts, ClientIP) flows through so a metered surface can attribute spend
// richly. Like Meter it is fire-and-forget on a background context and a no-op
// when billing is unconfigured or AmountCents<=0. kind is for the failure log.
func (rm *ResourceMeter) MeterUsage(org, kind string, u metering.Usage) {
rm.meterUsage(org, kind, u, nil)
}
// meterUsage is MeterUsage with the one thing a RESERVATION needs and a
// fire-and-forget caller does not: posted, run once the debit has reached the
// ledger (or failed to).
//
// A hold cannot be released when the call returns — the debit is still in flight
// then, so the next gate would read a balance that still contains money already
// spent, which is the very window the reservation exists to close. The only
// moment that is true is inside the recording goroutine, so that is where the
// release is handed. posted runs on EVERY exit, including the ones that record
// nothing: a hold released late is a customer locked out of their own balance.
func (rm *ResourceMeter) meterUsage(org, kind string, u metering.Usage, posted func()) {
if posted == nil {
posted = func() {}
}
// Same defensive line as Gate: never-constructed records nothing and never
// panics; no-local-ledger asks the peer.
if rm == nil || rm.m == nil {
posted()
return
}
// Ask the VALUE what it is worth, never the wire fields. Usage carries three
// amount sources with a documented precedence (typed Amount, then micro-USD,
// then cents) and Usage.Money resolves them; reading two of the three here
// dropped every usage priced ONLY as a typed Amount — the shape a per-token
// 18-dp caller sends — before Record could bill it. Silently: no error, no log,
// no row. Record itself has always guarded on the resolved value, so this line
// was the one place the fleet disagreed with itself about what money is.
if amt := u.Money(); amt.IsZero() || amt.IsNeg() {
posted()
return
}
// OWN THE STRINGS BEFORE THEY OUTLIVE THE REQUEST. Both paths below hand this
// value to a background goroutine, and a Usage built in a handler carries
// zero-copy views into the server's reused request arena (c.User(),
// c.RequestID(), the forwarded IP). Without the clone the debit eventually
// marshals the NEXT request's bytes onto this caller's row — and connections
// are reused across tenants, so the row it corrupts belongs to someone else.
// One clone here, rather than every caller of a fire-and-forget meter having
// to remember. See [metering.Usage.Clone].
u = u.Clone()
if !rm.Enabled() {
rm.meterPeer(org, kind, u, posted)
return
}
u.User = org // per-ORG billing: ledger keyed on the org slug.
u.Org = org // X-Org-Id -> caller's namespace (overrides client default).
if u.Provider == "" {
u.Provider = rm.provider
}
if u.Service == "" {
u.Service = rm.provider // scope service axis = this meter's provider.
}
if u.Status == "" {
u.Status = "success"
}
if u.Currency == "" {
u.Currency = "usd"
}
m, log, env := rm.m, rm.log, rm.env
go func() {
defer posted() // the hold ends where the money lands, not where the call returned.
if _, err := m.Record(context.Background(), u); err != nil && log != nil {
log.Error("resource debit failed (resource created, not billed)",
"org", org, "kind", kind, "provider", u.Provider,
"cents", u.AmountCents, "env", env, "err", err)
}
}()
}
// denial is the ONE decision behind a refused Gate: which status the money wire
// answers with, and the code+message it carries. Both renderings below read it,
// so an untyped handler and a typed op can never describe the same refusal two
// different ways.
// ErrNoLedger is a priced act with no ledger to charge: there is nobody to bill
// because there is nobody. It is the ONE value [ResourceMeter.Gate] answers with
// when it is handed an empty org, so all of its callers refuse identically
// without any of them re-deciding it — [denial] renders it as the tenant gate's
// own 403 rather than as a fault of the biller.
var ErrNoLedger = errors.New("no validated principal")
func denial(err error) (int, string, string) {
// Identity before money: this refusal is about WHO is asking, so it never
// takes a money status. Checked first because it is the more specific fact.
if errors.Is(err, ErrNoLedger) {
return http.StatusForbidden, "forbidden", ErrNoLedger.Error()
}
// The ONE classifier (errmap.go), so the money wire and the app's error
// renderer answer one refusal identically. It keeps the DISTINCT 402s — a
// per-scope cap (issue #70) is never the out-of-funds shape, which would be
// the wrong code plus a retry storm against a ceiling that will not clear
// until the period rolls over — and it keeps an UPSTREAM'S status, so a 402
// commerce already decided is not re-decided into 503 here.
if he, ok := refused(err); ok {
return he.Status, he.Code, he.Msg
}
// The gate could not decide. Unknown is not free, and it is not a refusal the
// caller can act on either.
return http.StatusServiceUnavailable, "balance_unavailable", "Billing temporarily unavailable"
}
// denyBody is the money wire's body: the NESTED {"error":{"code","message"}} the
// edge gate emits, so every Hanzo surface refuses in one contract.
func denyBody(code, msg string) map[string]any {
return map[string]any{"error": map[string]string{"code": code, "message": msg}}
}
// DenyResource renders the Gate denial outcomes as the SAME JSON the edge gate
// returns (denyBilling), so every Hanzo surface emits one error contract:
// 402 insufficient_balance / 503 balance_unavailable.
//
// It WRITES the response, which is what an untyped handler wants and what a
// typed op cannot use: zip stamps the op's own status over a hand-written one
// after the handler returns, so a typed op refuses with Denied instead.
func DenyResource(c *zip.Ctx, err error) error {
status, code, msg := denial(err)
return c.JSON(status, denyBody(code, msg))
}
// deniedErr carries a refused Gate as an ERROR, which is the only refusal
// channel a typed op has. It holds the money wire's own status and body, and
// DenyEnvelope writes them back untouched — so a typed op refuses with exactly
// the bytes DenyResource writes beside it, rather than reshaping a 402 the whole
// fleet reads by `error.code` into zip's flat {status,code,error}.
//
// It is not an escape from typing. The op still declares its In and its Out, so
// the document, the MCP tool, the CLI command and the SDK method all exist; what
// it declines to do is invent a SECOND vocabulary for a refusal the platform
// already has words for.
type deniedErr struct {
status int
code string
msg string
}
func (e *deniedErr) Error() string { return e.msg }
// Unwrap gives the refusal a status and a message OFF the HTTP path, where there
// is no response to write bytes into: an MCP tools/call and an in-process CLI
// invoke run the op without passing through DenyEnvelope, so zip's own error
// handler renders this instead — the same status and sentence in zip's envelope,
// rather than a blanket 500 that loses both.
func (e *deniedErr) Unwrap() error {
return &zip.HTTPError{Status: e.status, Code: e.code, Msg: e.msg}
}
// Denied turns a refused Gate into the error a TYPED op returns. Pair it with
// DenyEnvelope on the routes that can refuse; without the envelope the refusal
// still carries the right status and sentence (Unwrap), just in zip's shape.
func Denied(err error) error {
status, code, msg := denial(err)
return &deniedErr{status: status, code: code, msg: msg}
}
// DenyEnvelope writes a Denied refusal back as the money wire's own bytes.
// Install it on the group whose typed ops gate on balance — and BEFORE those
// routes, since fiber runs middleware in registration order — so the REST
// projection answers the 402/503 contract it always has. Anything else passes
// through untouched.
func DenyEnvelope() zip.Handler {
return func(c *zip.Ctx) error {
err := c.Continue()
var d *deniedErr
if errors.As(err, &d) {
return c.JSON(d.status, denyBody(d.code, d.msg))
}
return err
}
}
// ResourceFeeCents resolves the flat create fee (in cents) for kind from
// operator config, most specific first:
//
// <envPrefix>_<KIND> e.g. CLOUD_PROVISION_FEE_CENTS_SQL=500
// <envPrefix> e.g. CLOUD_PROVISION_FEE_CENTS=100 (all kinds)
// DefaultResourceFeeCents ($1.00)
//
// A clearly-named, configurable policy knob — never a fabricated price. A value
// of 0 makes the kind free (and un-gated). Negative/invalid values are ignored
// (fall through), so a typo can never make a paid resource free by accident.
func ResourceFeeCents(envPrefix, kind string) int64 {
if v, ok := parseNonNegCents(os.Getenv(envPrefix + "_" + strings.ToUpper(kind))); ok {
return v
}
if v, ok := parseNonNegCents(os.Getenv(envPrefix)); ok {
return v
}
return DefaultResourceFeeCents
}
func parseNonNegCents(s string) (int64, bool) {
s = strings.TrimSpace(s)
if s == "" {
return 0, false
}
n, err := strconv.ParseInt(s, 10, 64)
if err != nil || n < 0 {
return 0, false
}
return n, true
}