THE billing fix (B): split 'who pays' (HOME org, X-User-Owner) from 'whose data'
(EFFECTIVE org, X-Org-Id), which the old code conflated onto one org. A platform
SuperAdmin masquerading into another org now spends from the admin ledger; data
scope stays on the acted-on org.
- clients/principal: Owner(c) (X-User-Owner, bounded+cloned), BillingOrg(c) (home w/
effective fallback, Validated-gated), Payer(c) (bare-string for the in-handler meters).
- middleware_identity: mint X-User-Owner=home for every validated principal (before the
admin org-switch) + strip it on ingress (authorityHeaders) so a client can't forge who pays.
- middleware_billing: identityFromCtx keys User+Org (balance check AND debit) on BillingOrg.
- 11 resource-meter live sites (visor/s3/security/platform-run/functions/ml/provisioning/
bots/tracker/automations): Gate+Meter+MeterUsage bill principal.Payer(c) (home); Org stays
effective for the namespace. Background/reconcile paths (a.Org/r.Org/b.Org, meterRun) and
the studio_render 'org' param bill the RESOURCE's own org — FLAGGED (see Red handoff).
- metered_ai + types: ChatRequest/EmbedRequest gain BillingOrg; meteredAI gate+record key on
billedOrg(BillingOrg,Org); threaded principal.Payer(c) through the code engine (Synthesize/
Embed) so internal RAG bills home too. Inner AI call keeps req.Org (data scope).
Part A (embedded commerce, clients/commerce mirror of the commerce repo): drop the spoofable
IsSuperAdmin boolean; auth.IAMClaims.IsSuperAdmin() gates homeOrg()=="admin" (HomeOrg from
X-User-Owner ?: Owner); iammiddleware reads X-User-Owner->HomeOrg (Owner stays effective);
EdgeAuth strips+mints X-User-Owner before the ?org override, stops minting X-User-IsSuperAdmin;
all .SuperAdmin() call sites -> .IsSuperAdmin(). Org-scoped IsAdmin untouched.
Tests (green): TestIdentityFromCtx_AdminMasqueradeBillsHomeOrg (owner=admin+effective=victim ->
debit on admin, data on victim), TestBilledOrg, TestMeteredAI_AdminMasqueradeBillsHomeOrg
(end-to-end debit user=admin), vendored TestIAMClaims_IsSuperAdmin masquerade+anti-escalation,
EdgeAuth strips forged X-User-Owner. go build ./...=0, go vet=0.
Rollout: pre-gateway (no X-User-Owner) billing falls back to effective (home==effective for a
normal caller; masquerade fails CLOSED). Deploy gateway (mints X-User-Owner) first.
Cloud was the odd one out: it minted SuperAdmin from TWO signals
(claims.IsAdmin && owner == adminOrg) while IAM's canonical User.IsSuperAdmin() is
just user.Owner == conf.AdminOrg, and IsOrgAdmin folded SuperAdmin into itself. Two
predicates that each meant one-and-a-half things.
Decomplected — two orthogonal facts, one predicate each:
IsSuperAdmin = owner == adminOrg (platform sudo; the SAME equality IAM uses)
IsOrgAdmin = the IAM isAdmin bit (admin of one's OWN org; implies nothing about super)
A gate admitting either now writes IsSuperAdmin(c) || IsOrgAdmin(c) explicitly, so the
superset is visible AT the gate, not hidden inside a predicate. GuardScoped already did.
The admin org holds ONLY SuperAdmins (provisioned in, never promoted), so membership IS
the fact — the isAdmin bit is the orthogonal org scope, never a super gate. The KMS
machine-principal exclusion STAYS (a real guard: an admin-org machine token must never
be super). Test renamed + strengthened to lock the one predicate. Build ./... clean,
identity/admin/principal suites all green, KMS-machine exclusion tests pass.
Advances the in-process IAM embed (clients/iam, HIP-0106) from the
v1.31.20 pre-release pin (31a798fb, #117) to the published v1.31.22 tag
(387c4a60). Brings the project claim mint + default-project seed (#120)
into cloud's embedded IAM so console admin surfaces match standalone
hanzo.id, already on v1.31.22.
API-compatible: iamserver.InitEmbed and object.Project CRUD unchanged
(iam go.mod hash identical across versions), no clients/iam adaptation.
CGO_ENABLED=0 go build ./... green; clients/iam + clients/platform tests pass.
zip v1.6.0 deleted the deprecated/redundant surface (App.Mount/Route/ModuleFn/
UseFiber). cloud is off App.Mount (#257); licensing v0.1.4 migrated its one
App.Mount call. Build 0, framework tests green.
Co-authored-by: zeekay <ai@hanzo.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
WS4 audit flagged projects.Project and tracker.Project as duplicate types.
They are not: projects.Project is the Slug-keyed deployable site (one global
projects.db, org column); tracker.Project is a KEY-prefixed Linear-style issue
team living in a per-(org,project) tracker.db WITHIN one of those sites. Only
the universal storage-row skeleton (id/org/name/description/timestamps) is
shared; the natural keys carry different semantics (DNS label vs issue-ID
prefix) and the field sets are not subsets. Record the distinction on each type
so the finding is not re-litigated. Comment-only; no API or behavior change.
Co-authored-by: Hanzo <dev@hanzo.ai>
principal.ValidatedProject(c) now flows through ResourceMeter.Gate into
metering.AuthInput.ProjectValidated, so resource-creation caps harden
consistently with the edge BillingGate. Removes the hardcoded
ProjectValidated:false at resource_billing.go.
- Gate signature gains projectValidated bool (adjacent to project, mirroring
principal.ValidatedProject's return + AuthInput's field order).
- Every *zip.Ctx caller passes principal.ValidatedProject(c); the three
no-principal/client-body paths (metered_ai LLM decorator, background agent
run, content studio in.Project) pass false — unvalidated stays soft, never
fabricated true.
- Meter/Record path unchanged: Usage carries no ProjectValidated; cap
enforcement is Gate-only, so threading it there would be dead code.
Stays SOFT in prod today: ValidatedProject is true only for a validated NAMED
project, and IAM seeds none yet, so no org has a project claim. No behavior
change now; named-project caps auto-harden per-org as IAM seeds them.
Co-authored-by: Hanzo <dev@hanzo.ai>
The build path (buildFrontendCmd) already chose its BuildKit frontend on
dockerfile presence alone: an explicit Dockerfile → dockerfile.v0, otherwise
hanzoai/pack via gateway.v0. buildType never gated the build — it was stored
metadata whose closed set still advertised the retired nixpacks/buildpacks/
static strategies and defaulted git apps to nixpacks.
Collapse it to the one true surface:
- closed set is {pack, dockerfile}; git source defaults to pack, not nixpacks
- image source always yields buildType image (no build), forced by source, so a
client can no longer stamp a git strategy on a prebuilt-image app
- Goa contract (design.go + generated openapi3.json/yaml) enum, description and
examples track the same {pack, dockerfile, image} set
Adds TestBuildTypeSurface pinning the default (pack), the dockerfile escape
hatch, rejection of every retired strategy, and image-source forcing.
Co-authored-by: Hanzo <dev@hanzo.ai>
Six identity reads re-derived org/project inline instead of going through
principal — the ONE accessor. Route them all through it so the trust
decision lives in exactly one place and cannot drift.
- analytics tenant(): delegate to principal.Org — fixes a retained-buffer
bug (org keyed the cloud_usage ledger past request end as a zero-copy
fasthttp view; principal.Org clones). Gate unchanged.
- git/security projectScope(): read via principal.Project; absent header
and literal "default" both map to the un-suffixed default scope via
principal.IsDefaultProject, keeping today's (org,project) key shape.
- admin core.ResolveScope / core.GuardScoped / me / core.EmitAudit: read
org via principal.Org (validated principal + org), composed with the new
principal.IsOrgAdmin predicate. Admin-org bucket kept local.
Co-authored-by: Hanzo <dev@hanzo.ai>
Pin hanzoai/ai v1.805.7 (exempt concept removed: no BALANCE_EXEMPT_USERS/KEYS, no
fail-open; balance-unverifiable now DENIES). BuildDeps sets aiobject.CommerceTransport
= commerceinproc.Transport() and self-configures the ai module's commerceEndpoint to the
in-process placeholder when commerce is co-resident, so the embedded ai router's balance
read + usage debit hit the raw commerce gin (service-token, no socket) — the SAME one
ledger the metering client bills — not the customer proxy that 401s a service token.
Every external inference is now prepaid-gated with zero exempt principals.
hanzo/z was a normal customer, never admin; exempting it leaked money. Closed.
Projects are now owned by Hanzo IAM (hanzo.id) as the org-scoped
(owner,name) resource. Platform REFERENCES that store instead of owning
one: apps still live under a project, but create/list/get/delete/exists
of the bare project delegate to IAM in-process.
- New ProjectStore port (projects.go) with an in-process iamProjects
adapter over github.com/hanzoai/iam/object — no HTTP hop, and IAM's
canonical *object.Project is used verbatim (no platform-local clone).
- Delete platform's project ownership: the Project struct, the
platform_projects table, and Create/Get/GetByID/List/Update/Delete
project store methods. DeleteProject is replaced by DeleteProjectApps
(cascade of the app tree only; the project row is IAM's).
- Apps key on the IAM project NAME (platform_apps.project_id holds it);
the internal proj_ id indirection is gone, so reconcile/preview/domains
use app.ProjectID directly and drop their project lookups.
- run.go get-or-creates the default project via the ProjectStore, using
principal.DefaultProject as the single source of truth for "default".
- tenant() gates on principal.Validated(c) (canonical), not a raw
c.User()=="" whitespace-blind check.
- Tests: fake ProjectStore + TestProjectLifecycleDelegatesToIAM proving
create/delete route through IAM and delete cascades platform's apps.
Co-authored-by: Hanzo <dev@hanzo.ai>
SanitizeIdentity now mints X-Project-Id from the validated JWT `project` claim
(idClaims.mintedProject), exactly like X-Org-Id from `owner` — the raw client
X-Project-Id is never a source. Still checked non-foreign to the effective org, so
a global admin viewing another org drops their own-org project. Mirrors the edge
(iamauth.Claims.MintedProject), so the in-binary path binds the same header.
principal.ValidatedProject flips CONDITIONALLY: (project, true) only for a
validated, NON-default project claim — the signal a project-scoped spend cap uses
to HARD-enforce. The default project stays soft (IAM does not seed default
projects yet; a blanket true would wrongly hard-402 every org). Named project caps
auto-harden one org at a time as projects are seeded.
Tests: claim-sourced project binding + forged/override client X-Project-Id ignored
+ cross-org claim refused on admin org-switch + X-Org-Id forgery still blocked;
ValidatedProject claim-backed→(project,true), absent/default→soft, unvalidated→soft.
Follow-up (WS3, out of scope): resource_billing.go Gate hardcodes
ProjectValidated:false — resource-creation caps stay soft until it threads
principal.ValidatedProject.
Co-authored-by: Hanzo <dev@hanzo.ai>
Eliminate "global admin" terminology across the cloud repo in favor of the
standard SuperAdmin term. Pure naming refactor; behavior preserved.
Identifiers (commerce auth + middleware + root):
- (*auth.IAMClaims).GlobalAdmin() -> SuperAdmin() + all call sites
- IAMClaims.IsGlobalAdmin field -> IsSuperAdmin (JSON tag isGlobalAdmin
-> isSuperAdmin; SuperAdmin() still honors owner=="admin", the canonical
predicate, so behavior is preserved)
- edgeauth isGlobalAdmin() helper -> isSuperAdmin(); requireGlobalAdmin ->
requireSuperAdmin; test-local globalAdmin vars -> superAdmin
- const HeaderUserIsGlobalAdmin -> HeaderUserIsSuperAdmin, value
"X-User-IsGlobalAdmin" -> "X-User-IsSuperAdmin" (commerce-internal header:
minted+stripped in edgeauth, read in iammiddleware; no external sender)
- auth/globaladmin_test.go -> auth/superadmin_test.go
clients/admin: the deprecated isGlobalAdmin JSON alias (sibling of the
canonical isSuperAdmin, same value) is REMOVED rather than renamed — a rename
would collide with the existing IsSuperAdmin field, and the alias was a
transitional back-compat scaffold for this very migration. Tautological
alias-equality assertions dropped; isSuperAdmin checks retained.
Prose: "global admin" / "global-admin" / "GLOBAL-ADMIN" / "GlobalAdmin" in
comments and docs (LLM.md, k8s manifests) rewritten to SuperAdmin, case-aware.
Left untouched: cloud gateway header X-User-IsAdmin; principal.IsSuperAdmin /
IsOrgAdmin; svcorg's DefaultNamespace (global) kind.
build ./... clean; vet clean; SuperAdmin behavior-locking tests pass
(commerce auth/edgeauth/iammiddleware/catalog/costs/checkout, admin
scope/money, principal, root identity). gofmt clean. Zero GlobalAdmin /
global-admin remaining in .go/.md/.yaml.
Two SuperAdmin endpoints for multi-provider credit-management (the console renders
them; contract is authoritative):
GET /v1/admin/providers/credit -> [{provider, grant_cents, burn_cents,
remaining_cents, runway_days, has_credit, is_paid_only}]
GET /v1/admin/usage/funding?from&to -> [{provider, model, funding, tokens,
cost_cents, requests}], funding in {credit,paid,paid_only,byo}
DRY: reuses the admin auth guard (core.Guard), finance.go's DO billing read (real
k grant, live remaining/burn/runway), and the ONE cloud_usage warehouse (no new
store). DO is the seeded real row; others land as keys arrive. Funding is
provider-level-derived in v1; the per-call split lands when the ai metering write
stamps a funding column (next commit). Envelope = core.OK {status,msg,data:[...]}.
Two-ledger model kept distinct: UPSTREAM provider credits here vs DOWNSTREAM Hanzo
customer billing (commerce). Pure logic TDD'd.
Per the SuperAdmin convention (SOC2/FedRAMP; standard term SuperAdmin, NEVER
'global admin'): name the two admin scopes explicitly.
- principal.IsSuperAdmin(c) = c.IsAdmin() — platform sudo; SuperAdmin ⟺ owner==admin org
(X-User-IsAdmin minted only for that identity).
- principal.IsOrgAdmin(c) = IsSuperAdmin(c) || X-User-IsOrgAdmin — admin of one's own org.
GuardScoped now reads IsSuperAdmin (fast-path) + IsOrgAdmin (scoped) instead of bare
c.IsAdmin()/OrgAdmin. Pure rename — no behavior change, identity suite green.
v1.805.6 makes credit/quota exhaustion + agreement gates (402/403/insufficient_quota)
cascade to the next provider in the fallback chain — Phase 2 of multi-provider
credit-management + fixes the do-ai->anthropic opus fallback.
GuardScoped admitted ANY validated org member to their own org's admin
panels (/v1/admin/{overview,orgs,users,usage,analytics,me,bases}), not
just org-admins: SanitizeIdentity discarded the JWT org-level isAdmin bit
for non-admin-org principals (it only minted the GLOBAL X-User-IsAdmin
when owner==adminOrg), so GuardScoped's fallback (validated User+Org) let
a plain member through.
Mint a new, unforgeable X-User-IsOrgAdmin signal and require it:
- middleware_identity.go: add X-User-IsOrgAdmin to authorityHeaders
(stripped on every ingress request, so a client can never forge it),
and mint it for any validated principal with claims.IsAdmin &&
!isKMSMachinePrincipal — covering both a global admin and an org-admin,
owner-scoped and machine-excluded.
- clients/principal: add OrgAdmin(c) = IsAdmin() OR X-User-IsOrgAdmin.
- clients/admin/core.GuardScoped: require principal.OrgAdmin(c) in
addition to validated User+Org; a validated non-admin member now gets
the same 403 as an unvalidated caller. Guard (global-only) untouched.
Tests: orgAdminHdr now carries the bit; add TestScope_MemberWithoutOrg
AdminDenied (member without the bit is 403 on every scoped panel) and
TestSanitizeIdentity_OrgAdminHeader (org-admin gets the bit not global,
member/machine get neither, forged bit stripped on bearer + anon paths).
Finding 3 (grant durability — ARCHITECT: FAIL-CLOSED). EmitAudit is a silent
no-op when State.AuditStore == nil, so on a no-audit-store deployment a
SuperAdmin grant moved REAL money with NO durable cloud-side before/after record.
Decision: FAIL-CLOSED. Grants are a real-money op and the audit store is always
present in any real deployment — audit_serve.go REQUIRES a persistent data dir
for the trail (hard boot error otherwise), and a nil store arises ONLY from the
explicit CLOUD_AUDIT_DISABLED dev opt-out. So ApplyGrant now refuses the grant
with 503 BEFORE any money moves when AuditStore == nil: no unaudited money move.
The check sits after org validation and before the deposit, so the existing
validation errors (amount/cap/unknown-org) are unaffected.
Residual (documented, not regressed): a post-deposit Append failure (store
present but the write errors) keeps the money-moved success response — the money
DID land, and failing it would both misreport the grant and, absent idempotency,
invite a double-credit retry. It stays a loud log, backstopped by the
request-level AuditTrail middleware which independently records the request and
fails the request CLOSED on its own write error (serve.go / audit_middleware.go).
Finding 4 (deposit double-credit on retry — WIRED). commerce.Deposit posted
/v1/billing/deposit with no idempotency key, so a commit-then-timeout (cloud's
15s client) could drive an operator double-credit on retry. The commerce backend
DOES enforce idempotency: POST /v1/billing/deposit reads X-Idempotency-Key,
scoped billing-deposit:<subject> — a completed key REPLAYS the receipt, an
in-flight key 409s, an absent key is legitimately additive (never dedup by
amount). Verified in ~/work/hanzo/commerce/api/billing/deposit.go.
Fix: thread a DETERMINISTIC key from the grant. Deposit/post gained an
idempotencyKey arg sent as X-Idempotency-Key. ApplyGrant derives it as
sha256(org|amountCents|currency|source|nonce) where nonce is the operator-supplied
Idempotency-Key — so a retried grant carrying the SAME nonce dedupes at commerce
while two DISTINCT grants (even same org+amount) never collide, and a nonce reused
for a DIFFERENT amount still lands (dedup can never silently DROP a real grant).
No nonce => no key => additive default preserved; we do NOT fabricate a
content-only key (it would wrongly dedupe two legitimate identical comps).
Residual (cross-service, frontend): effective end-to-end once the operator
console sends an Idempotency-Key per grant attempt, reused verbatim on retry.
Finding 2 (GuardScoped over-visibility — NEEDS CROSS-SERVICE, not fixed here).
GuardScoped admits any validated principal with non-empty c.User()+c.Org().
SanitizeIdentity (middleware_identity.go) mints X-User-Id for EVERY validated org
member and mints the admin signal X-User-IsAdmin ONLY for a GLOBAL admin
(owner==adminOrg) — it DISCARDS the JWT org-level isAdmin bit for non-admin-org
principals. So a regular member's sanitized identity is byte-identical to an
org-admin's, and GuardScoped admits regular members to their own org's admin
panels (same-tenant over-visibility of financials + user directory; NOT
cross-tenant — ResolveScope pins to c.Org()). The finding is REAL, but the
prescribed fix (require org-admin) CANNOT be done in clients/admin/*: the
org-admin signal is not minted, and middleware_identity.go is read-only in this
scope. Requiring the only admin signal we have (c.IsAdmin(), GLOBAL) would break
the deliberate, test-locked org-scoped tier (scope_test.go admits org-admins).
Correct cross-service fix: in SanitizeIdentity's `case owner != ""` branch, mint
X-User-IsOrgAdmin=true when claims.IsAdmin (owner safe, non-KMS), add it to
authorityHeaders (stripped on ingress so unforgeable), then GuardScoped requires
c.IsAdmin() || X-User-IsOrgAdmin=="true". Left for the middleware owner.
Tests: TestGrantCredit_NilAuditStoreFailsClosed (503, no deposit, balance
unchanged); TestGrantCredit_IdempotencyKeyForwarded (same nonce=>same key,
distinct nonce=>distinct key, no nonce=>no key). Existing money must-pass tests
(TestGrantCredit_DepositLandsAndAudited, TestSuspendReactivate_*, TestAdminAudit_*)
still pass unchanged.
Finding 1 (undercount masked as healthy). core.OrgMoney swallowed the
Commerce.Spend/Credits errors to a silent zero, and /overview derived the
"commerce" source freshness from a SINGLE probe org — so if commerce was down
for 40 of 41 orgs the fleet spend/credits totals were an undercount while the
source still read "healthy". revenue.go / finance.go already report this
correctly via a `partial` flag + the core.ErrPartialRevenue sentinel.
Fix (decomplect to ONE partial pattern, not a second one):
- OrgMoney now returns (spend, credits int64, ok bool); ok is false when the
spend OR credits read failed — the SAME (row, ok) contract revenue.revenueOf
uses. An unwired commerce is NOT a failure (Spend/Credits return (0,nil) when
unconfigured), so ok stays true and Ready() still distinguishes not-configured.
- overview folds ANY per-org OrgMoney failure into the commerce source as
degraded, reusing core.ErrPartialRevenue, and derives freshness from the SAME
per-org reads the totals fold instead of a single probe org.
orgs is a per-ROW panel (OrgRow[] via OKList; NO sources[] channel): a failed
read degrades THAT org's row to an honest zero — there is no aggregate total to
mislabel, so no change beyond the signature. The customer per-row/detail call
sites are best-effort and unchanged in behavior.
Test: TestOverview_CommercePartialOnPerOrgError — commerce 500s for one org of
two; the commerce source reports not-ok with an error while the healthy org's
spend still contributes (honest partial total, not a hard panel fail).
The subpackage contained only replication_test.go referencing an undefined Producer
(no NewProducer/BackupOnce/etc. anywhere), so it never compiled and failed go vet.
Zero importers. Pike: the best code is no code.
Rewrite the top-level admin package as the Mount + aggregator only. state ->
core.State (exported fields), and every top-level handler + helper is retyped to
*cloud.Service[core.State] and calls core.* for the shared kernel:
- routes() registers the org-scoped panels (me/overview/orgs/users/usage/
analytics/bases) behind core.GuardScoped and the platform reads
(roles/applications/products/compute/o11y/sync + flags/waitlist) behind
core.Guard, then delegates to audit/customer/revenue/finance Routes().
- analytics.go keeps only the analytics-specific derivation (growth/retention/
churn/active/LTV) folding over the core activity model + spend series.
- o11y/compute/bases/waitlist/flags/types retyped to core.State + core.*.
Delete the now-moved audit.go/customers.go/grants.go/revenue.go/finance.go/
scope.go (their handlers live in the domain packages; their shared helpers in
core). doTokenFromEnv moves to admin config; iamAuditQuery moves to the audit
package.
Move tests with their code: audit store tests -> audit package; grantTag test ->
core; finance pure-math tests -> finance package. The full-mount integration
tests (admin/scope/cockpit/finance-aggregation) stay in the admin package and
now drive the real routes() so the harness mirrors Mount exactly. Behavior,
routes, tenant scoping and every assertion unchanged.
Introduce clients/admin/core as the subsystem's shared kernel — the State
struct (upstream clients + adminOrg + audit store) and the one-copy business
primitives every admin surface composes: the two-tier gate (Guard/GuardScoped),
the /v1 envelope writers (OK/OKList/OKRaw/Fail), CallerCreds, the tenant-scope
predicate (TenantScope/ResolveScope/ScopedOrgs/Descendants), the IAM fan-in
(ListOrgs/OrgMoney/FindOrg/Display/SrcOf/SourceStatus), the ONE credit-write path
(ApplyGrant + EmitAudit + grantTag/grantNote/CreditRequest) and the fleet
activity/time-series model shared by analytics and revenue
(CustActivity/TxnPoint/SeriesPoint/FleetActivity/SpendSeries + bucket helpers).
Carve one package per handler domain over that kernel:
- audit -> /v1/admin/audit{,/verify} (store-backed + IAM fallback)
- customer -> /v1/admin/customers* + /v1/admin/grants (list/detail/credit/
suspend/reactivate/grants ledger)
- revenue -> /v1/admin/revenue
- finance -> /v1/admin/finance (+ the pure ComputeFinance derivation)
Each domain imports core for the kernel and shared logic; no business logic is
duplicated. Routes are registered per-domain via <domain>.Routes(app, s).
Completes the bots surface (launch -> launch+list+stop). Both are thin,
org-scoped proxies onto the in-cluster bot-gateway (BOT_GATEWAY_URL, the same
server-side knob clients/bot uses), carrying the caller's validated tenant
context (X-Org-Id pinned to principal.Org, never a request param).
- GET /v1/bots normalizes the gateway's session rows into
{runId,task,surface,status,sessionUrl,startedAt}, deriving sessionUrl here
(the one place a session URL is built). Honest-empty {"bots":[]} on an
unconfigured/unreachable gateway, a non-2xx, or an undecodable body -- never 5xx.
- POST /v1/bots/:runId/stop returns {runId,status:stopped}; a run the caller's
org does not own is 404; an unreachable gateway is a clean 502.
Both require a validated principal so org-scoping can't ride a forged X-Org-Id.
Hermetic list+stop tests against a fake gateway assert normalization, sessionUrl
derivation, caller-org scoping, honest-empty, and 404/502 paths.
v1.805.5 adds the controller-side balance-gate fail-open (BALANCE_GATE_FAIL_OPEN_ON_ERROR)
on top of v1.805.4's nil-guard, so a broken/misconfigured Commerce billing backend
degrades to allowed-but-ungated instead of 500-ing every authenticated chat for
non-exempt users. Together they let the CR drop the commerce.hanzo.svc:8001 bridge
and keep commerceEndpoint unset per the in-process design.
ai v1.805.4 guards the nil balanceGate deref in resolveBillingKey that returned a
bare HTTP 500 on EVERY authenticated /v1 request (models, chat/completions,
messages) whenever commerceEndpoint is unconfigured (balance enforcement disabled).
RateLimitFilter calls resolveBillingKey before BalanceGateFilter's own nil guard,
so the embedded AI subsystem crashed every authed request while anonymous requests
(no token) 401'd correctly. Fail-open: a disabled billing subsystem resolves no
billing subject and never crashes a request.
Cloud edge for Hanzo Sentry (the /v1/sentry product face served by the embedded
o11y runtime):
- mountSentry registers the /v1/sentry/* wildcard, forwarding to the SAME gated
runtime handler the /v1/o11y wildcard uses (one runtime, two path families; no
path rewrite — the Sentry routes are literal /v1/sentry/... in the runtime).
- gate() now exempts the DSN-authenticated Sentry ingest writes (isSentryIngestPath:
POST /v1/sentry/{project}/envelope|store/, tight method+prefix+suffix match) from
the principal gate — the runtime authenticates the DSN key, not a Hanzo session —
while EVERY Sentry read/write API stays principal-gated (no cross-tenant leak).
Uses only the existing o11y runtime-handler API, so it builds against the current
pinned o11y (v1.5.12) and is inert (404) until the o11y dep is bumped to a build
that carries the /v1/sentry routes.
FOLLOW-ONS (coordinated separately):
- Bump the hanzoai/o11y dep to the tag containing the /v1/sentry routes to activate.
- Gateway needs a byte-identical isErrorIngestPath sibling for POST
/v1/sentry/{project}/envelope|store/ so the tokenless DSN ingest is not 403'd at
the edge (do NOT touch ~/work/hanzo/gateway here).
Test: TestGateExemptsSentryIngestButGatesReads (ingest exempt, reads/writes gated).
The Account.Token comment claimed the store is SQLCipher-encrypted and 'keeps
it encrypted at rest'. False: openStore uses cek.Open, which today runs the
no-key plaintext fallback (real WithRawKey-from-KMS lands with the connect
flow). Token column is empty today, so no plaintext secret ships.
Finishes the batch-F rework (the extraction landed in 850d4c0; this is the second,
first-principles half that the earlier cherry-pick stopped short of):
- clients/admin/money — type Cents int64; the unit lives in the type, so
ConsumedCents/MRRCents/CreditsCents collapse to Consumed/MRR/Credits. int64 casts
only at the operator-contract boundary (wire unchanged, JSON tags kept).
- clients/admin/digitalocean — do.go extracted from the inline client; Client.{Ready,
Balance,History}. Named digitalocean (not do) so it never collides with the local do var.
- clients/admin/commerce — reworked to the money.Cents unit + collapsed the always-equal
(org,user) into one subject; dropped the MRRCents + duplicate-rollup shims.
Applied cleanly (0 conflicts) atop the extraction + tenant->org main. Build ./... green,
vet clean, admin+commerce+digitalocean tests pass (pure-Go cek gate).
Extracts admin's inline upstream reader clients into self-contained, package-namespaced
units and gives billing a single value type. Decomplect + package-as-namespace, per the
Hickey/DRY bar:
- clients/admin/money — type Cents int64; the unit lives in the type (ConsumedCents/
MRRCents/… collapse to Consumed/MRR). int64 casts only at the operator-contract edge.
- clients/admin/commerce — Client.{Ready,Spend,Credits,Plan,Ledger,Costs,Deposit}
(Deposit = the one write). Collapses the always-equal (org,user) into one subject;
the bare-slug X-Org-Id+user invariant is baked into the client. Drops MRRCents +
duplicate rollup-balance shims.
- clients/admin/digitalocean — Client.{Ready,Balance,History}
- clients/admin/health — Client.{Ready,Up}
Wire unchanged (JSON tags kept, DTOs stay int64 cents). Integrated onto current main:
preserves the commerceinproc.BaseURL(...) in-process routing and the tenant->org
vocabulary. Build ./... green, vet clean, admin+commerce+integrations tests pass
(pure-Go, cek gate). Completes the batch-F rework the user authorized.
Fold the live social stack's publish + schedule onto the native /v1/social domain:
- Publish edge (publish.go): the ONE publishPost path — claim (at-most-once across
the HTTP handler and a scheduler tick), fan out to the channel's connected accounts
through the injectable Publisher seam, record the honest outcome (published + external
id, or failed + reason) on the post. POST /v1/social/posts/:id/publish + on-create
fanout (scheduled-for-now-or-earlier) both call it.
- Scheduler (scheduler.go): an in-process periodic sweep (the native equivalent of the
live stack's Temporal timer + hourly missing-post poller) that advances every org's
scheduled -> published when the time arrives; idempotent, per-org, clean shutdown.
Mirrors clients/commerce/sweep.go.
- Provider seam: fail-closed default (notConfiguredPublisher) that reports EXACTLY which
OAuth-app credentials are missing (the live orchestrator's env var names) and NEVER
fakes success. No Hanzo deployment carries provider creds today, so this is prod's
honest state. GET /v1/social/providers reports per-network publish-readiness.
- Store: token on accounts (SQLCipher-encrypted at rest; live stack stores it plaintext),
external_id/account_id/error on posts, ClaimForPublish/MarkPublished/MarkFailed,
DueScheduled (the ONE deliberate cross-org system read), RecoverStuckPublishing.
Tenant isolation preserved: every publish is org-scoped; the scheduler's cross-org sweep
only reads (org,id) to dispatch into the org-scoped path. 12 tests (9 new) prove publish
success/not-configured/no-account, idempotency, per-org isolation, the scheduler sweep,
crash recovery, and live capabilities. go build -tags 'cloud cloud_mount' ./... green;
frozen wire-order guard green.
Rebased blue's RED-reviewed encrypt-at-rest onto the CURRENT live-prod commit
(v1.786.185). The release train added stores blue's base never saw, all opening
PLAINTEXT — closed every one through the same cek.Open seam:
- tenantdb.go (the SOLE per-tenant opener → code/git/functions/tracker)
- gojabase, gatewaypolicy (gateway.db), dataroom (link_index.db)
Result: ZERO plaintext sql.Open("sqlite") left in the cloud data plane
(commerce self-encrypts under its own KMS key; cek internals excepted).
Flattened internal/cek → top-level one-word package cek. Trimmed the AI-slop
import comments to one line.
Verified CGO=1+SQLCipher: go build ./... green, cek tests green, and blue's
cek.Open shipping path pre-proven on ALL 47 real prod DBs (ciphertext + plaintext
shredded + exact row-count parity, 0 fail).
The frozen-fixture test ran in Go CI but not inside the image under the pinned
Alpine libsqlcipher, so a sqlcipher-dev pin/base bump that changes the on-disk
format could go green in CI while bricking prod. Add the frozen-format gate to
the Docker RED gate (beside TestEncryptionProof), and make requireCipher honor
SQLITE_REQUIRE_CODEC=1 (a would-be skip becomes a FAILURE) so the in-image gate
is airtight: any format change now fails the IMAGE build, not just Go CI.
Verified: SQLITE_REQUIRE_CODEC=1 go test -run TestFrozenFixtureOpens ./internal/cek = ok.
RED re-review closers:
1. Cross-version brick guard (vector b): the version-freeze was soft (unpinned
apk sqlcipher-dev resolves from the live Alpine repo). Now:
- Dockerfile pins sqlcipher-dev=4.6.1-r0 → a repo bump fails the build LOUDLY
(never a silent prod brick).
- Commit a FROZEN encrypted fixture (internal/cek/testdata/frozen/store.db
+ .dek, written under cipher_compatibility 4) + TestFrozenFixtureOpens that
copies it to temp, opens via cek, and reads a known canary row. A future
libsqlcipher format change makes the FROZEN store fail to open → red CI,
which build-time SQLITE_REQUIRE_CODEC (fresh-db only) cannot catch.
TestGenerateFrozenFixture (CEK_GEN_FIXTURE=1) regenerates it on an
intentional format rev.
2. Scope doc (vector c): cek package doc now states it provides CONFIDENTIALITY
at rest, NOT integrity/authenticity/anti-rollback vs a PV-write
(node-compromise) adversary — out of the stated read-only model. The
logical-id+epoch binding is deliberately NOT added (out-of-model complexity).
9/9 cek tests green on real SQLCipher (cgo) + nocgo; gofmt/vet clean. Migration
still deferred; live cutover gated on the user's supervised go.
RED review fixes on the encrypt-at-rest codec:
1. [HIGH] Shred the pre-migration <db>.plain.bak after the verified keyed reopen
(overwrite+remove, single call site in openEncrypted). Nothing reaped it
before, so every migrated store left a COMPLETE plaintext replica on the
volume forever — the exact threat the codec exists to kill. Test asserts the
backup is gone post-migration.
2. [MED] KEK no longer binds to the CLOUD_DATA_DIR-relative path (brittle: a
dir change bricked the plane). A random per-file id is stored in the sidecar
head (fileID(16) || wrapped-DEK) and the KEK derives from it — intrinsic to
the file, config-independent. Test moves a store to a new dir + wrong
CLOUD_DATA_DIR and it still opens.
3. [MED] Missing key on an encryption-capable build is now FATAL (fail-closed,
same posture as KMS), not a silent plaintext downgrade. Encrypting() is wired
into the boot log (serve.go). No new gate/env — keyed off the existing
CLOUD_KMS_MASTER_KEY_REF + build capability.
4. [MED] Parity gate gains a rowid-independent per-table content hash (commutative
multiset sum of per-row hashes over user columns), catching value mutations
that count+schema+integrity_check miss while ignoring benign rowid renumbering.
Dropped the inaccurate 'byte-faithful' wording. Cross-version cipher stability
is enforced by FREEZING libsqlcipher in the build (an at-open cipher_compat pin
is infeasible with mattn's URI-key requirement — the URI param is ignored and a
post-open pragma runs after mattn reads the header; proven); any mismatch fails
closed (test: corrupt/missing sidecar refuses).
5. [LOW] recoverInterrupted scrubs stale -wal/-shm; migration removes them before
the swap so a plaintext WAL never sits beside an encrypted db. No statement
logging on keyed conns; the key never rides a logged DSN/error.
8/8 tests green on real SQLCipher 3.53.1 (cgo) + nocgo fatal-key path.
The CGO+libsqlcipher cloud binary shipped encryption DORMANT: all 29 stores
opened via bare sql.Open("sqlite", path), so /var/lib/cloud/*.db (crm PII,
treasury ledger, wallets, audit, team/entitlements) were plaintext despite
CLOUD_KMS_MASTER_KEY_REF being present. The SQLCipher primitives were linked
(hanzoai/sqlite cek.go) but never USED.
internal/cek.Open is the ONE encryption-at-rest seam every store now routes
through. When the master key is configured it transparently, under a per-file
flock: mints a per-DB random DEK (SQLCipher page key), wraps it AES-256-GCM
under KEK=HKDF-SHA256(master, principal) in a <db>.dek sidecar, and — for an
existing PLAINTEXT file — migrates it to SQLCipher via sqlcipher_export, then
verifies schema + per-table row-count parity + integrity_check by re-opening
the encrypted copy EXACTLY as the app will, BEFORE an atomic swap. Fail-secure:
key set on a non-encrypting build is a hard error; unverifiable migration leaves
plaintext intact; wrong master fails closed; crash mid-swap is recovered.
Proven with real SQLCipher (TDD): plaintext→ciphertext header, row parity,
.dek unwrap, .plain.bak preserved, idempotent reopen, wrong-key rejected,
dev-mode gated, nocgo fail-closed.
Migration is deferred to a maintenance-window cutover (this image built via
arcd) — NEVER encrypt against the current binary, which cannot open SQLCipher.
Rewires 29 open-sites; base/o11y external-module DBs are a follow-on.
zip #5 deprecated (*App).Mount(prefix, h): it is exactly
app.All(prefix+"/*", zip.AdaptNetHTTP(h))
kept only as a behaviour-identical alias. Move every foreign-http.Handler
mount onto the explicit primitive so there is ONE way to put a route on the
app, and so the cloud money binary keeps building once zip deletes App.Mount.
Migrated all THREE route-mount call sites (the task scoped two; commerce is a
third — its exclusion note referred to the commerce.Mount *function*, not the
app.Mount(p, handler) inside it):
- clients/plugin/plugin.go:125 app.Mount(p.Prefix, h) → app.All(p.Prefix+"/*", zip.AdaptNetHTTP(h))
- clients/iam/iam.go:159 app.Mount(p, handler) → app.All(p+"/*", zip.AdaptNetHTTP(handler))
- clients/commerce/mount.go:147 app.Mount(p, handler) → app.All(p+"/*", zip.AdaptNetHTTP(handler))
Behaviour-identical (Mount IS this composition). Also updated the four doc
comments that named the deprecated zip.App.Mount so no dangling reference to a
soon-deleted method remains. grep-confirmed ZERO app.Mount( route-mounts left.
go.sum: removed the two inert zap-proto/zip v1.3.0 hash lines (nothing in the
module graph requires v1.3.0 — orphan cruft a working `go mod tidy` would drop).
Full `go mod tidy` is blocked by a PRE-EXISTING, unrelated force-moved tag on
luxfi/keys@v1.2.2 (server serves h1:nuD+y5…; committed go.sum records
h1:XH5mRm…), so the two dead lines were removed surgically instead — luxfi/keys
and every other entry left byte-identical to origin/main. No GONOSUMCHECK hack.
Co-authored-by: zeekay <ai@hanzo.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Per the one-authority directive: commerce must not require its OWN Organization
record; IAM is the sole org/user/auth authority. The service-token path already
auto-projects the org from X-Org-Id via the cached GetOrCreate resolver, but the
IAM-principal path c.Next()'d without resolving the org — it depended on
iammiddleware running upstream, and when it hadn't, GetOrganization MustGet-
panicked (500) / the org was absent, so an IAM org with no pre-existing commerce
row could not view or create billing. ensureIAMOrg now resolves the validated
X-Org-Id through the SAME cached GetOrCreate resolver on the IAM path too
(idempotent), so any IAM org "just works" and a thin billing record is
auto-projected on first use — commerce derives the org from IAM, never its own
table. This is also the root cause behind the "hanzo has no commerce org record"
wall on the live 2-org proof.
The billing-account CRUD (List/Create/Get/Update/Delete + project bindings +
members + loadOwnedAccount) called middleware.GetOrganization, which MustGet-
panics (→ recovered 500) when the request's org has no commerce Organization
record — the same class as the already-deployed spend-alert fix. Switch all to
GetOrganizationOK with a safe default (reads → empty, mutations → 400/404), so
an IAM principal whose org lacks a commerce record gets a clean response, not a
500. Completes the metering-path panic hardening.
Bump github.com/zap-proto/zip v1.3.0 → v1.5.0 (verified drop-in) and move
subsystem teardown onto zip's OnShutdown hook, deleting the hand-rolled
ShutdownAll reverse-loop.
Before: serve.go called ShutdownAll(specs) — a reverse-mount teardown loop —
BEFORE app.ShutdownWithContext. Subsystems were torn down while the listener
was still accepting and in-flight requests were still draining: a latent race
(a request could use a store that teardown had just closed).
After: MountAll registers each enabled spec's ShutdownFunc via app.OnShutdown
right after the subsystem mounts. zip drains those hooks LIFO — AFTER the
listeners stop accepting and in-flight requests drain — so registration-at-mount
reproduces the exact reverse-mount teardown order ShutdownAll gave, minus the
race. app.ShutdownWithContext now owns the whole teardown.
Scope is deliberately narrow: MountSpec, MountAll, Wire(), Typed, and the
OwnsHealth health loop all STAY (the imperative composition-root flatten was
proven unsound — a generic /v1/:name/health route is shadowed by 3-segment
subsystem routes). Only ShutdownAll and its serve.go call site are deleted;
audit/gateway-policy/telemetry teardown keep their positions.
Test: build_onshutdown_test.go drives a real in-flight request over a loopback
listener, shuts down mid-request, and proves (1) Shutdown blocks until the
request drains and (2) the MountAll-registered hooks then run LIFO = reverse
mount order. A second test locks the enablement axis + nil-Shutdown guard.
Co-authored-by: zeekay <ai@hanzo.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The Sentry error-ingest wire endpoints (POST /v1/o11y/api/<project>/envelope|store/)
authenticate with a DSN public key downstream in the o11y handler, not a Hanzo
principal. cloud's o11y gate() 403'd them for lacking X-User-Id, so the tokenless
ingest that the gateway allowlists (isErrorIngestPath) was blocked one layer deeper
— error-tracking could never ingest end-to-end.
Add the cloud-side counterpart: gate() now also exempts isErrorIngestPath
(byte-for-byte the gateway's matcher — method + /v1/o11y/api/ prefix + envelope|store
suffix, never a bare prefix). Reads under /v1/o11y/api/vN/... and the Issues
list/detail/update stay principal-gated; the exempted ingest still fails closed on a
bad/absent DSN key (401/503). Tested: TestGateExemptsErrorIngestButGatesReads.
The live signed-webhook e2e (#118) got past HMAC verification and then
500'd: resolveWebhookOrg called middleware.GetOrganization — gin MustGet
— but webhook ingress runs OUTSIDE the auth-token group, so no
middleware ever set "organization" and every signature-VALID provider
delivery panicked ("key organization does not exist"). Switch to the
GetOrganizationOK variant that exists precisely for signature-verified
sessionless ingress; the header/env/default fallback chain below it now
actually runs.
Regression test drives resolveWebhookOrg on a bare sessionless gin
context and asserts no panic.
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Every inbound Square webhook 401'd 'square processor not configured':
payment/providers/square registers an EMPTY Provider whose Configure()
only runs from BD's per-tenant charge resolver — but tryValidateWebhook
(the sessionless inbound path) reaches the registry slot directly. The
env-registering thirdparty/square init could never win the slot either
(import-order clobber / defer-to-existing), so webhook validation had no
configured processor in ANY deployment — 100% of live Square deliveries
were rejected.
init() now seeds Configure() from the deployment env (same vars +
SQUARE_ENVIRONMENT sandbox switch thirdparty reads; charge creds required,
webhook fields optional with ValidateWebhook's own fail-closed contract).
BD's per-tenant Configure still overrides per request for outbound money.
Regression guard drives env -> configFromEnv -> Configure ->
ValidateWebhook over a real Square-spec HMAC (url+body), plus tamper and
sandbox-switch cases.
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Standalone commerce was swept by a k8s CronJob (curl image + cluster hop +
token secret) POSTing /v1/billing/auto-recharge/run-all every 15m at
commerce.hanzo.svc:8001. With commerce folded into the cloud binary that
loop is redundant moving parts: the binary now dispatches the SAME request
loopback through its own embedded gin handler — full middleware chain
(TokenRequired service-token branch -> PlatformOnly mint gate -> datastore/
KMS context), byte-identical to the wire path (guarded by
TestSweepOnceWireContract).
- interval: COMMERCE_AUTORECHARGE_INTERVAL (default 15m; 0/off disables;
garbage disables fail-safe — a card-charging loop must never guess)
- token: COMMERCE_SERVICE_TOKEN (absent -> disabled with a loud warn,
not a 403-every-tick loop)
- first fire after one FULL interval (never front-run the outgoing CronJob
during rollout)
- runs only where commerce mounts (single-writer pod; reader role never
mounts commerce) -> exactly one sweeper, same guarantee the single
CronJob schedule gave
- stops with Embedded.Stop
Unblocks deleting the commerce-auto-recharge CronJob + the standalone
commerce/commerce-sandbox pods (#118).
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Turns the agentic-marketing content loop from scaffolding into a working
edge pair, then closes every finding from the adversarial review.
Edges (wired at Mount, fail-closed until configured):
- Generator: zen5 copy via deps.AI.ChatCompletion (metered Bill.Gate/MeterUsage)
+ studio assets via the ComfyUI Qwen-Image-Edit-2511 graph.
- Distributor: hanzoai/social Public API fan-out, per-brand key custodied in KMS.
errNotConfigured until a brand connects a key — no key, no fan-out.
Hardening:
- Publish TOCTOU interlock: a per-item, store-backed lease (framework fw_locks,
the ONE cross-process coordination primitive) serializes concurrent publishes
of the same item across drivers and pods; non-idempotent fan-out runs once.
- source_media SSRF gate: the URL validator fails closed on unparseable/backslash
input.
- external_ids is server-managed: the before_save guard rejects a client write;
Publish records it under the lease as a trusted server write.
Adversarial regression suite pins each guarantee from the outside (raw framework
PUT, direct ops, concurrency): red_adversarial / red_rereview / red_final /
red_lease_final, plus blue_hardening and per-file unit tests.
One open item recorded as a HARD GATE in clients/content/LLM.md: lease
TTL-preemption re-opens the double-post window for a brand with ~15+ slow
channels (fan-out (N+1)x20s > 5m TTL, external_ids recorded only at fan-out end).
ZERO exposure until a brand connects a social key; the fix (lease heartbeat/renew
preferred) MUST land in the same change that connects the first real key.
Claude-Session: https://claude.ai/code/session_013jh8aka8q8RvhhVQ1psMeW
Two INFO items from RED's #250 re-review:
- TestFrameworkContentModulesLinked asserted only the module registry. erp's
ledger-posting HOOKS register in a SEPARATE init() step, so a future split of
registerHooks() out of erp's module init() could drop the hooks while the guard
stayed green. Add framework.RegisteredHookCount() and assert it > 0 so the guard
fails if erp's hooks (computeJournalTotals, journalEntry/paymentEntry
submit+cancel, …) are ever unlinked from the binary.
- gojabase Config.DataDir comment said "{tenantSlug}.db" (the pre-C1 name); the
on-disk segment is TenantSegment (injective, traversal-safe base32 of raw org
bytes). Comment now matches the code.
go build + go test ./subsystems/... ./clients/framework/... green.
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Net-new per-org ad-campaign domain on the ONE cloud framework (zip/Fiber +
cloud.Deps + per-org SQLite), the same shape clients/crm uses — twin of crm.
Registered in subsystems.Wire() right after crm; frozen wire sequence updated.
/v1/ads surface (all org-scoped, tenant-isolated on the bearer owner claim):
GET /v1/ads/health (auto, serve.go liveness)
GET /v1/ads/summary per-org roll-up (total/active/budget/spend)
GET /v1/ads/campaigns list (?status=)
POST /v1/ads/campaigns create
GET /v1/ads/campaigns/:id detail
PUT /v1/ads/campaigns/:id update
DELETE /v1/ads/campaigns/:id delete
Campaign = root of the ad hierarchy (campaign -> ad sets -> ads; ad-set/ad legs
hang off this seam): Platform (meta/google/tiktok/x), Status (draft/active/
paused/completed), Objective, Budget/Spend (cents).
Tests: per-org isolation + full CRUD + summary round-trip, both green.
In-process fold of github.com/hanzoai/social (social-backend/frontend/
orchestrator, a Postiz-style scheduler) onto the ONE cloud framework
(zip/Fiber + per-org SQLite), twin of clients/crm + sibling of the
marketing fold. NOT a proxy to the standalone social pods.
Two entities faithful to the live Public API (clients/content publish.go
already talks to it): Account = a connected channel (the stack's
integration), Post = content published/scheduled to a channel. Scheduling
is a Post with status=scheduled + a future scheduleAt, not a third entity.
Every query filters WHERE org=? on principal.Org (validated bearer owner,
HIP-0026); one tenant can never read/mutate another's rows. Registered in
subsystems Wire() after crm with a ctxShutdown that closes the DB.
Surface (org-scoped, /v1 only): summary + accounts CRUD + posts CRUD.
Generic liveness serves GET /v1/social/health.
Build: go build -tags 'cloud cloud_mount' ./... GREEN; 3 store tests pass
(per-org isolation across accounts+posts, post CRUD+summary, account CRUD).
Mount github.com/hanzoai/marketing in-process on the ONE cloud framework
(zip/Fiber + cloud.Deps + per-org SQLite), the same shape clients/crm uses —
NOT a proxy to a standalone marketing pod. Registered in subsystems.Wire()
right after its twin crm; frozen wire sequence updated to match.
/v1/marketing surface (all org-scoped, tenant-isolated on the bearer owner claim):
GET /v1/marketing/health (auto, serve.go liveness)
GET /v1/marketing/summary per-org roll-up (total/active/budget/spend)
GET /v1/marketing/campaigns list (?status=)
POST /v1/marketing/campaigns create
GET /v1/marketing/campaigns/:id detail
PUT /v1/marketing/campaigns/:id update
DELETE /v1/marketing/campaigns/:id delete
Campaign faithful to the repo domain: Channel (email/sms/social/meta/google/
tiktok), Status (draft/active/paused/completed), Objective, Budget/Spend (cents).
Genetic-optimizer / ML-forecasting / ad-platform integrations not folded yet.
Tests: per-org isolation + full CRUD + summary round-trip, both green.
Bumps github.com/hanzoai/ai v1.805.2 -> v1.805.3. v1.805.3 pins the GenAI
tracer at adopt time so the embedded o11y/SigNoz runtime reassigning the
process-global OTel tracer provider can no longer redirect gen_ai spans off
the o11y sink — they reach o11y_traces. No cloud source change; go.mod/go.sum only.
Add clients/content — the ONE Go-native replacement for the bespoke karma
Python pipeline. A framework app-lane (module "marketing": Campaign, SocialPost,
Asset) + a thin /v1/content/* control-plane. Store-less: content IS framework
documents; the subsystem is a stateless orchestrator over the framework store +
the zen5/studio/social edges.
- lifecycle.go: ONE state machine (draft→in_review→approved→queued→published,
+archived), a pure value read by the before_save hook (enforces edge legality
at the storage boundary) and the transition endpoint (same check + fan-out).
- hooks.go: before_save gate on every publishable DocType.
- content.go: board (cross-DocType aggregate), lifecycle, transition (+best-effort
distribution), generate/publish/channels. Org-scoped via principal.Org; never a
5xx from a foreseeable condition (fail-closed 503 / honest 4xx).
- generate.go/publish.go: Generator + Distributor seams with fail-closed defaults;
real zen5 (deps.AI) + studio (ComfyUI) + hanzoai/social wiring documented for the
follow-up. Exported Generate/Publish/Transition are the ONE impl the HTTP surface
AND the automations connector call.
- framework: add Get (read-one), exported ErrNotFound/ErrConflict/ErrBadRef +
IsValidationError so in-process callers classify errors without string-matching.
- automations: connector_content.go exposes content_generate/transition/publish as
flow steps + MCP tools (in-process, org-scoped) so the loop runs autonomously
(cron flow: generate → wait_for_approval → transition → publish).
- subsystems: Wire content after knowledge; frozen wire-order test updated.
Tests: lifecycle table, before_save hook, end-to-end loop over the real framework
store (install→create→board→transition→publish), forge-403, cross-org isolation.
go build ./... + go test (content/framework/automations/subsystems) all pass.
Claude-Session: https://claude.ai/code/session_013jh8aka8q8RvhhVQ1psMeW
#248 (Wire() composition root) rewrote subsystems.go and dropped the blank
imports for cms/erp/help. Those three are NOT mount subsystems — no HTTP surface,
never in Wire(). Each registers DocType fixtures and, for erp, ledger-posting
lifecycle hooks (computeJournalTotals, journalEntry submit/cancel, paymentEntry
submit/cancel, …) into the always-on clients/framework engine from a package
init() (framework.RegisterModule). Dropping the imports left them out of the
binary: /v1/framework/* carried no erp/cms/help and the erp ledger hooks were
silently gone. No mount test caught it (frozen[]/Wire() cover only mount specs).
Fix (RED-verified): re-add the three blank imports under an explicit "framework
content modules" comment; add framework.RegisteredModules() + a guard test
(TestFrameworkContentModulesLinked) asserting the engine carries each lane so the
drop cannot silently recur. Also relax TestDepGatedSubsystemsFailClosed to accept
any non-2xx — a dep-disabled subsystem denying 403 is fail-closed; >=500 was
wrongly strict (o11y denies 403, ai returns 5xx).
go list -deps ./cmd/cloud shows cms/erp/help; go build ./... green;
go test ./subsystems/... ./cmd/cloud/... -race green.
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The S2S metering path (ScopeRules → ListSpendAlerts; AuthorizeVerdict →
AuthorizeSpendCap) can reach these handlers with no "organization" context key
when X-Org-Id does not resolve to a commerce org (e.g. the agents scheduler
probing an unprovisioned org). middleware.GetOrganization MustGet-panicked
there → a recovered 500 storm on the money path (observed live: "key
organization does not exist", ~every 5-15s). Switch the metering-verdict
handlers + billingSubject to GetOrganizationOK with a safe default — no org
means no org-scoped caps, so ListSpendAlerts returns empty and
AuthorizeSpendCap allows — instead of panicking. Regression test proves neither
handler panics with no org.
types.Claims gains Project + BillingAccount; principal.BillingAccount(c) reads
the gateway-minted X-Billing-Account-Id, added to subScopeHeaders so the raw
client copy is stripped on ingress and re-injected only for a validated
principal (mirrors X-Project-Id). It is an ATTRIBUTION hint only — the debited
account is always resolved server-side by commerce from the org's
ProjectBinding, never trusted from the header — so a mislabelled account can
only ever misattribute the caller's OWN spend within its own org, never
redirect spend to another tenant. Inert until IAM emits the billing_account
claim + the gateway mints the header (slices 63/64).
Also fixes a stale test left by de2a97a3 (/metrics is now a console page, not an
API 404 — dropped from the must-404 list).
Replace the init()-registry (blank imports + cloud.Register(name, order, …) +
magic order-ints) with ONE explicit subsystems.Wire() []cloud.MountSpec. Slice
position IS the mount order — MountAll iterates it as-given, ShutdownAll in reverse.
Core (build.go/serve.go/cmd): delete MountSpec.Order, Register,
RegisterWithShutdown, the HealthOwner option func, and var Registry. MountAll and
ShutdownAll take the []MountSpec slice; Serve threads it in (cloud never imports
subsystems → no cycle). cmd/cloud + cmd/hanzo call subsystems.Wire(). Typed and the
factory hooks (RegisterKMSClientFactory, RegisterCommerceClientFactory,
RegisterOrgScopeResolver, RegisterPushBuilder, RegisterTelemetryInstaller) are
untouched — a different mechanism.
In-repo (~64 clients/*): delete each init() registration; export the mount fn where
unexported (commerce.MountFromDeps, o11y.MountO11y) and the shutdowns. ctxShutdown
bridges the func() error shutdowns in one place.
Externals wired explicitly; the wave-2 tags no longer self-register:
ai v1.805.2 (@150 catch-all), o11y v1.5.12 (@70 wildcard) — origin/main already
pins these (genai-span) but did NOT wire them, so main currently DROPS ai + the
o11y wildcard from its live registry (main's own TestRegistryAssemblesSubsystems
fails on that). This composition root RESTORES both. authz v1.10.7 (@70),
licensing v0.1.3 (@110, Mount is already a MountFunc — wired direct), metrics
v1.110.2 (@40, takes its own metrics.Deps via mountMetrics). vfs unchanged: it
never registered a subsystem, so it is not wired.
Order is proven by subsystems.TestWireOrderMatchesFrozen against the sequence
captured empirically from origin/main @c504d2b (68 self-registering specs) with ai
+ the o11y wildcard restored at their order-int slots. The Service[state] refactor
reshuffled same-order tie positions on main; the frozen sequence adopts main's new
order (functionally inert — tied subsystems own disjoint route prefixes).
Also fixes a pre-existing clean-main build break (#247): clients/sign/sign.go
referenced an undefined `log` — main does not compile without it, so the whole tree
(subsystems → cmd/cloud) failed to build. One-line fix (log → deps.Logger); flag
for the owner to factor out if preferred.
Co-authored-by: zeekay <ai@hanzo.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
#247's VFS-nil health-only guard was authored against the pre-refactor sign.go
(which had a local `log` var); it auto-merged cleanly onto the concurrently-landed
cloud.Service refactor (principal.Org / s.Log) but referenced an undefined `log`,
red-ing the build (clients/sign/sign.go: undefined: log). Point it at deps.Logger
(guaranteed non-nil by the guard above it). No behavior change.
go build ./... + go build -tags "libsqlite3 sqlite_fts5" ./cmd/cloud +
go test -race ./clients/{gojabase,captable,sign,dataroom}/... all GREEN.
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Second wave of RED's consolidation review — the mediums that needed new
hanzoai/captable + hanzoai/sign bundle versions (now merged) plus their cloud-side
wiring and tests.
M5 (captable bundle @119bf9c0) — capTable summed ALL option rows regardless of
status, so EXERCISED/EXPIRED/CANCELLED grants inflated fullyDilutedShares and
every ownership %. Fixed to count OUTSTANDING options only. New
TestDilutiveOptionsExcludeTerminal proves terminal-state grants don't dilute.
M8 — ONE blob seam in gojabase (Config.Blob + a tenant-scoped globalThis.__blob =
{ put(key,b64), get(key) }), keys namespaced {Name}/{TenantSegment} so a bundle
can only reach its own tenant's objects. sign (bundle @315a3fe6) now routes PDF
bytes THROUGH it instead of inlining 32 MiB base64 in document_data.data: create
stores the original blob key, seal writes a sealed key, view/download read via
__blob.get. The sign leaf passes deps.VFS as the seam (health-only without it).
document_data holds only the key now (type BLOB_KEY). Same VFS/S3 plane dataroom
uses — one blob strategy, not two.
M7 (sign bundle @315a3fe6) — SEQUENTIAL signing order was declared but never
enforced. assertTurn gates signField+signComplete so a later signer can't act
until every earlier signer has SIGNED. New TestSequentialSigningOrder proves an
out-of-turn signer is refused (403) and the turn opens once the earlier one signs.
sign tests now inject an in-memory VFS and assert PDF bytes land on the seam
(tenant-scoped key), NOT in the tenant SQLite. All four gate packages pass under
go test -race against the published module versions.
Gate: go build ./... + go build -tags "libsqlite3 sqlite_fts5" ./cmd/cloud +
go test -race ./clients/{gojabase,captable,sign,dataroom}/... all GREEN.
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
Co-authored-by: hanzo-dev <dev@hanzo.ai>
cloud installs the ONE global tracer provider wired to the o11y in-process trace
sink, but in in-process-sink mode it sets no OTLP/ZAP exporter endpoint. The embedded
hanzoai/ai module then found no endpoint and DISABLED its gen_ai span emit, so the
gen_ai plane in o11y_traces was dark (cloud's own request spans flowed; every LLM
call's gen_ai span was gated off). The provider install + adopt also lived ONLY in
cmd/cloud/main.go, leaving every 'hanzo <svc>' entrypoint (which shares cloud.Serve)
telemetry-dark.
Hoist the bootstrap into cloud.Serve — the ONE serve body cmd/cloud AND every
'hanzo <svc>' share — so both install the provider and adopt it into ai identically,
BEFORE MountAll mounts ai (ai's InitTelemetry reads the adopted-ready flag at mount).
cloud.Serve cannot import clients/o11y (clients/o11y imports cloud -> cycle), so the
concrete bootstrap (SetTracerProvider + aiobject.AdoptHostTracerProvider) lives in
clients/o11y and registers via cloud.RegisterTelemetryInstaller — the same cycle-free
inversion as RegisterKMSClientFactory / RegisterPushBuilder. Serve flushes the
provider on shutdown BEFORE ShutdownAll tears the sink down, so buffered spans drain.
Removes cmd/cloud/telemetry.go (moved to clients/o11y/traceprovider.go). Repins ai to
the branch carrying AdoptHostTracerProvider.
Red review (SHIP): MED-1 — this hoist (both entrypoints adopt identically). Tests:
cloud.installTelemetry dispatch/no-op (telemetry_test.go); clients/o11y
installTraceProvider adopt-latching + OTLP-env-retire + disabled-noop
(traceprovider_test.go).
ai: -> v1.805.1-0.20260710213829-b5eb789d3878 (feat/genai-span-host-provider @ b5eb789d)
* cloud: IAM-native vocabulary — org/user/project everywhere, tenant concept removed (TenantDB→OrgDB)
"tenant" was a non-IAM synonym for org. Replace it with the IAM-native nouns
org / user / project / billing account so there is ONE name for the concept.
Root framework primitives (package cloud) — now 100% tenant-free:
TenantDB -> OrgDB (tenantdb.go -> orgdb.go)
TenantStore[T] -> OrgStore[T]
NewTenantStore -> NewOrgStore
tenantDBPath / openTenantDB -> orgDBPath / openOrgDB
TenantScopeResolver -> OrgScopeResolver (tenant_scope.go -> org_scope.go)
RegisterTenantScopeResolver -> RegisterOrgScopeResolver
Dropped the legacy X-Tenant-Id / X-Tenant-ID entries from the identity
strip-list (nothing reads them; X-Org-Id is the live header).
Cross-subsystem seams:
principal.Tenant(c) -> principal.Org(c) (clients/principal; no collision —
Project/User kept, already IAM-native). ~50 call sites + ~70 stale doc refs.
types.TenantConfig -> types.OrgConfig; CommerceClient.GetTenantConfig ->
GetOrgConfig (+ commerce client impl, disabled/rpc/entitlements consumers,
the cloud.OrgConfig alias in deps.go).
Adopters deep-cleaned (identifiers, comments, test names): clients/code, git,
functions, tracker, provisioning, projects. Docs: subsystems.go composition
comments, README.md, and a new LLM.md "Framework doctrine" section.
GATED EXCEPTION (flagged, intentionally unchanged): clients/platform derives
LIVE k8s namespaces, registry image refs, and quota/limit objects from a
"tenant-<org>" string prefix. Renaming it orphans deployed namespaces + built
images, so the literal string is retained behind a // NAMING(gated) note in
clients/platform/k8s.go. The rbac test file + its funcs were renamed
(tenant_rbac_test.go -> org_rbac_test.go).
Out of scope (separate waves; independent domains, touched only for the
principal.Org call-site + stale-ref fix): commerce internal tenant tables
(active commerce-dissolve branch), runner tenantsource, treasury ledger, and
other subsystems' own tenant vocab.
Build: GOWORK=off CGO_ENABLED=0 go build ./... -> exit 0.
Tests: every touched package green. The 38 pre-existing failures (fakeAI
missing Embed; undefined Producer; commerce integration tests; o11y/graph/zt/
kms route-precedence & service tests) fail identically on the base commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* rebase: apply org vocabulary to post-branch main commits (metering/HA/tracing comments); drop unused TenantConfig alias
---------
Co-authored-by: zeekay <ai@hanzo.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Follow-up to the reader-tier commit — align the CLOUD_WRITER_URL/
CLOUD_READER_RETRY_BUDGET/CLOUD_WRITER_LEASE struct-literal keys with gofmt so
the CI fmt gate is clean. No behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RED adversarial review of the one-binary consolidation (gojabase + captable/sign/
dataroom folds). Fixes the two ship-blockers plus the in-repo mediums/slop.
C1 (HIGH) — cross-tenant collapse via non-injective filename encoding.
gojabase.slugify ToLower+folded [^a-z0-9._-]→'_', collapsing DISTINCT owners
(Acme/acme, "a b"/"a_b") onto ONE per-tenant SQLite file — a cross-tenant break,
the exact fold principal.Tenant keeps the org verbatim to avoid. Replaced with
gojabase.TenantSegment: lowercased unpadded base32 of the RAW org bytes —
INJECTIVE (distinct orgs never share a file) and traversal-safe ([a-z2-7], never
"."/".."/"/"). Table test proves injectivity (Acme≠acme≠a b≠a_b) + traversal
containment. IAM owner charset permits case/separator variants, so this was a
real (not theoretical) collapse. Fresh-tenant-only: the folds pre-date any
release, so no on-disk data uses the old names — nothing to migrate.
H2 (HIGH) — code==docs on staging. captable/sign/dataroom were un-staged in
config.go (stagedSubsystems={iam,ingress}) yet every leaf/README claimed
"STAGED / standalone keeps authority". Evidence (do-sfo3-hanzo-k8s): NO
standalone captable or dataroom deployment exists; the esign pod runs but its
SQLite holds ZERO tenant data (User=0, DocumentData=0, Recipient=0, Field=0 —
only BackgroundJob/RateLimit churn). Decision (a): keep un-staged, delete the
false comments (captable.go, sign.go, dataroom.go, subsystems.go, README ×2).
No migration needed (dead/empty apps).
M3 drop dead commercesvc api.Route(/v1/{billing,checkout,store,subscription,
account}) — unreachable in embed mode (only /v1/commerce/* mounted, no
rewrite); money path is clients/billing, unaffected. Removed unused import.
M4 gojabase per-tenant *sql.DB cache is now an LRU (cap CLOUD_GOJABASE_MAX_DBS=256)
with idle eviction (CLOUD_GOJABASE_IDLE_TTL_SEC=300) and dispatch PINNING —
only idle handles evict, never a DB mid-transaction. Bounds fd/memory for N
tenants.
M6 dataroom deny_list is now ENFORCED in view.authenticate and WINS over allow
(checked first); surfaced on linkOut. One shared email matcher (DRY).
M9 sign refuses to boot with a self-signed cert in a production env — requires the
KMS-custodied CLOUD_SIGN_CERT_PEM/KEY_PEM; dev still self-signs.
M10 added a -race concurrent multi-tenant dispatch test (heavy eviction churn +
case-variant tenants) proving pool + cache + eviction are race-free and
isolation holds under concurrency.
Slop/decomplect:
- goja: a response with no explicit valid status now FAILS CLOSED (default 500,
rolls the transaction back) instead of committing at a defaulted 200.
- newID (gojabase) + randKey (dataroom): a crypto/rand failure now returns an
error / fails the dispatch instead of a predictable time/zero fallback.
- subsystems.go: removed 6 duplicate blank imports (pubsub, eval, exec, plan,
plugin, pricing).
- dataroom: unified on ONE tenant encoding — the object-store key prefix now uses
gojabase.TenantSegment, matching the SQLite filename (was raw org).
Gate: go build ./... + go build -tags "libsqlite3 sqlite_fts5" ./cmd/cloud +
go test -race ./clients/{gojabase,captable,sign,dataroom}/... all GREEN.
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The metering decorator (metered_ai.go) wraps deps.AI so every chat + embed
call is authorized against the caller's org balance/budget/freeze BEFORE the
call and debited its billing account AFTER — the single DRY chokepoint through
which no inference runs unattributed. The billing scope (org, project) rides
the request value (ChatRequest/EmbedRequest), so it is compile-time impossible
to call AI without declaring who pays; no bypass, no side-channel key.
Closes the exempt holes — clients/code (index+search+ask), clients/knowledge
(search + index embeds), clients/crm (application screen) — all previously ran
the balance-exempt M2M path unmetered; each now threads org/project to the
chokepoint.
- types.AIClient.Embed takes *EmbedRequest (scope); ChatRequest carries
Org/Project; ChatResponse surfaces token counts for exact metering.
- httpAI stays pure transport but surfaces resp.Usage tokens and stamps
hanzo.org/hanzo.project on its gen_ai spans (feeds per-tenant o11y isolation).
- token-based micro-USD debit (CLOUD_AI_PRICE_UUSD_PER_1K, default $2/1M tok);
metering.Usage gains AmountMicros so a sub-cent call meters exactly instead
of rounding to zero and slipping through unbilled.
- reuses ResourceMeter over Deps.Metering — same per-org invariants; a
transparent pass-through when commerce is unconfigured (dev never blocked).
- fixes a latent slice-1 test break (crm/agents AIClient fakes lacked Embed).
- tests: pricing, scope forwarding, system-call, ModelLister preservation.
The console catch-all 404'd /metrics because it was in apiPrefixes (the Prometheus
scrape-path convention). But the real Prometheus surface is on the SEPARATE ops
listener (:9090, healthMux); on the product API (:8000) /metrics is the console
MetricsModule page. Drop /metrics from apiPrefixes so it falls through to the SPA
shell. One surface per port: :8000 product+SPA, :9090 ops. (ServiceMonitor repointed
to the ops port in the same change.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The cloud writer embeds an exclusive-lock ZapDB KMS store. A new probe
(clients/kms.TestConcurrentOpen_LiveWriterStoreIsNotROShareable) proves that
opening that store READ-ONLY while the writer is live FAILS ("Log truncate
required to run DB") — Badger's RO open replays the live memtable WAL and
refuses to truncate it. So the prior groundwork's assumption that a reader can
open the KMS store RO off the writer's PVC is false for a LIVE writer (only the
sequential close-then-reopen case worked). The audit SQLite store IS
concurrently shareable (audit/shareability_probe_test.go); the KMS store is the
one that is not, and every mutation is audited on the writer anyway.
Reader tier is therefore a transparent, always-ready reverse proxy (opens no
stores) to the single writer:
- reader_proxy.go: CLOUD_ROLE=reader boots serveReaderProxy BEFORE BuildDeps —
forwards every request to CLOUD_WRITER_URL, streams SSE, preserves inbound
Host. Dial-only retry (retryTransport) absorbs the writer's roll gap: it
retries ONLY when the connection was never established (no ready endpoint /
refused), so a non-idempotent POST is never double-executed; bounded by
CLOUD_READER_RETRY_BUDGET (default 25s) then 502.
- The reader Deployment rolls RollingUpdate(maxUnavailable:0), so the edge
Service always has a ready endpoint — this removes the ~30s console blip that
the writer's Recreate/replicas:1 causes today.
Writer zero-gap roll (opt-in, default OFF = byte-identical Recreate):
- writer_lease.go (+_unix/_other): CLOUD_WRITER_LEASE takes an exclusive fcntl
flock on {DataDir}/.writer.lock BEFORE opening the RWO stores and releases it
LAST at shutdown (after every store closes). A surge writer blocks until the
old one releases, so the exclusive ZapDB/audit stores are handed off, never
double-opened. Fail-closed on timeout.
Removes the dead ReaderGuard (the reader no longer runs the full pipeline; it is
the proxy). Unset CLOUD_ROLE + unset CLOUD_WRITER_LEASE ⇒ writer, byte-identical
to today.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
BillingAccount is a funding entity separate from the org: it holds a spend
limit + level + freeze and funds 1..N of the org's projects (dedicated or
shared). A project resolves to its account via ProjectBinding; usage debits
carry Transaction.AccountId so an account's balance and calendar-month spend
sum from the SAME append-only ledger the balance gate + per-scope caps read —
no parallel ledger, no stored total, never drifts.
The authorize verdict layers an account cap + freeze on top of the per-scope
caps (most-restrictive-wins, fail-closed). Resolution is ALWAYS server-side
from the org namespace — the account is never a forgeable client header.
Backward-compatible: (org,"default") folds to the org-wide pool, so an org
with no account behaves byte-for-byte as before.
- models/billingaccount, models/projectbinding (+ hashid kinds 284/285,
query reserved-kind list)
- Transaction.AccountId indexed ledger axis
- resolveAccountId + accountSpentCents + account cap/freeze in AuthorizeSpendCap
- real account + project-binding CRUD replacing the org-wrapper stubs
- tests: account cap, freeze kill-switch, dedicated-binding isolation,
backward-compat pool fold
zip v1.2.1 -> v1.3.0 rides zap-proto/fiber v3.2.1 (gofiber v3.2.0 + native
ServeMux-1.22 specificity precedence: most-specific wins regardless of
registration order, ambiguous overlaps panic at registration, use/mount stay
declaration-ordered barriers). TestIAMKeysBeatsWildcard now passes as a
FRAMEWORK property — subsystem order-ints are no longer load-bearing for
route precedence (their init-ordering role remains; removal rides the
composition-root refactor). Also swaps the 6 direct gofiber test-file imports
to the fork (production code was already 100% behind zip).
Co-authored-by: zeekay <ai@hanzo.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Both semantic subsystems (clients/code, clients/knowledge) embedded via their own
static CLOUD_AI_API_KEY HTTP client — a side-channel credential separate from the
chat path, and the root of vectors:0 (the key was dead + entitlement-gated). Add
Embed() to types.AIClient and route both embedders through deps.AI — the SAME
client chat runs through, which authenticates with the IAM client-credentials
(M2M) token when no static key is set: ONE org/project-aligned identity, one way,
no static key to rotate. Emits gen_ai OTel spans so embeddings are observable end
to end like chat.
This is the ai-module leg of the canonical path
ingress -> gateway -> commerce -> ai module; metering + no-exempt + billing-on land
next on this seam. Supersedes the CLOUD_EMBED_* dedicated-provider detour.
- types.AIClient: + Embed(ctx, model, inputs)
- clients/aihttp: httpAI.Embed (raw POST reusing the static/M2M authed transport)
- clients/disabled, clients/rpc: stub Embed (fail-closed / not-wired)
- clients/code, clients/knowledge: embed through deps.AI; drop the static key path
- knowledge test: exercise the deps.AI seam (inject a deterministic fake embedder)
Decomplect the observability estate to exactly ONE top-level concept.
## 5 -> 1 registration collapse
The plane was FIVE separately-registered subsystems (o11yscope 69,
o11y-runtime 71, o11y-event-ingest 68, o11y-otlp-ingest 72,
o11y-trace-inproc 73) whose names leaked five public concepts (five config
toggles + five /v1/<name>/health routes). The k8s-style ordering was an
internal impl detail. They collapse to ONE
`cloud.RegisterWithShutdown("o11y", 69, mountO11y, shutdownO11y, HealthOwner)`:
mountO11y performs the ordered sub-mounts in-process (mountEventIngest ->
mountScope -> mountRuntime -> mountIngest -> mountTraceSink), so the PUBLIC
concept is a single `o11y`. Behavior is preserved EXACTLY — every specific
/v1/o11y/* route still registers inside the one order-69 mount, hence BEFORE
the upstream hanzoai/o11y wildcard (order 70), so Fiber's in-order match still
gives the specific routes precedence. The upstream module co-owns the `o11y`
name at order 70 (the wildcard); HealthOwner on this entry keeps /v1/o11y/health
registered exactly once (never a duplicate).
## Flat, version-less public surface (one /v1/, no nested /api/vN)
The upstream SigNoz engine version is an internal impl detail resolved inside
the handlers, never leaked into a route:
- /v1/o11y/vm/{query,query_range} (was /v1/o11y/vm/api/v1/*) — VM proxy; the
upstream VM api/v1/* path stays INSIDE the handler (queryRaw). SuperAdmin
gate + {up,sum(up),count(up)} allowlist unchanged.
- /v1/o11y/{query,query_range} (new, query.go) — the flat builder query;
resolves to the v3 engine route SERVER-SIDE (the version-less alias would
float to v5, which 400s the console's v3 composite payload), delegating to
the same gated runtime handler the wildcard uses.
o11y stays EMBEDDED in-process (the everything-binary); OTLP ingest + trace
sink stay opt-in. Registers exactly one `o11y` in clients (grep-verified),
alongside analytics/evals/usage/audit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Absorb the thin clients/commercesvc mount wrapper AND the fail-closed
clients/commerce_local.go stub into the in-tree commerce library, and expose
a real in-process commerce.Client — one package now owns the commerce library,
its /v1/commerce subsystem, and its cloud.CommerceClient, self-registered via
cloud's inversion hooks (the clients/kms pattern; cf. #241 gatewaysvc→gateway).
cloud never imports commerce, so there is no cloud⇄commerce import cycle.
commerce.Client (client.go): a real in-process types.CommerceClient. The former
stub failed CLOSED on every CheckEntitlement (the Phase-2 gap). It now resolves
org→active-subscription→plan-tier→license-features from commerce's OWN models +
the @hanzo/plans vocabulary:
- org → namespace via commerce's canonical org.Resolve (the same binding the
money path uses), so a read can never cross tenants;
- active, unexpired subscriptions via a plain Status filter (matches commerce's
own read idiom, so grant- and payment-created subs are both found);
- plan tier → flat license features via a new plan.LicenseEntitlement seam that
runs @hanzo/plans' toLicenseFeatures (the vocabulary stays in one place, not
re-implemented in Go);
- Active iff the plan's features carry "licensing.product:<id>".
MONEY-SAFETY: Active:true only for a real active sub whose real plan really
licenses the product. Every unresolvable piece (commerce not co-resident, org
unresolvable, subscription query error, plans vocabulary unavailable) returns an
ERROR → the entitlements gate treats it as "cannot verify ⇒ 503"; a clean
"no plan licenses this product" is Active:false (→ 402), never an error and never
a fabricated grant.
Decomplect: commerce.Mount takes a MountConfig VALUE (Brand/Env/DataDir/Domain)
+ a logger — the values it uses, not the whole cloud.Deps bag; mountFromDeps is
the one place Deps is narrowed and carries the PCI Payments/Vault warnings.
Network path preserved: pickCommerceClient still selects the ZAP-RPC/disabled
client when commerce is NOT enabled in-process (CLOUD_COMMERCE_ZAP_ADDR). This
fold does NOT force the live in-process cutover — that stays operator-gated.
Deleted: clients/commercesvc, clients/commerce_local.go(+test), the legacy
//go:build cloud clients/commerce/mount.go(+ its two tests), and the redundant
cmd/commerce --cloud boot path (cloud_boot.go/cloud_stub.go) — one way to serve
commerce inside cloud: the folded subsystem.
Co-authored-by: zeekay <ai@hanzo.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Concurrent RO opener reads all records while the serialized writer appends;
writer chain stays intact and a fresh RW re-open recovers the head with no
fork/gap. Audit-store analog of clients/kms TestReaderReadOnlyRoundTrip.
Locks in the reader/writer shareability invariant the cloud HA carve depends
on. Test-only; zero runtime change (writer path byte-identical).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The code + knowledge embedders read CLOUD_AI_BASE_URL/CLOUD_AI_API_KEY,
which are ALSO the chat/synth AIClient's config (config.go:308). That
braided embeddings onto the IAM-gated api.hanzo.ai path, whose /embeddings
403s customer keys and 401s gateway keys — only a superadmin JWT passes —
so the vector tier indexed vectors:0 platform-wide.
Split embeddings onto dedicated CLOUD_EMBED_BASE_URL / CLOUD_EMBED_API_KEY /
CLOUD_EMBED_MODEL, each falling back to the shared CLOUD_AI_* for back-compat.
Both semantic subsystems (clients/code + clients/knowledge) read the same
three vars — one embed config, orthogonal to chat. Deployment points them at
DigitalOcean GenAI (bge-m3 @1024-dim, first-party, not IAM-gated) via the
DO_AI_API_KEY already in cloud-api-llm-keys; chat stays on CLOUD_AI_*.
Secrets are write-only: toAppView masks a secret value to "" on read, so a client
editing env re-submits kept secrets with an empty value. sealSecretEnv now treats an
empty secret value as "keep the already-sealed value" — it is preserved, not resealed
to empty (which wiped the KMS secret). Only a non-empty value seals (and requires KMS).
Makes the masked-read -> setEnv round-trip a no-op for untouched secrets instead of a
data-loss footgun; unblocks the console env editor's Keep path. Adds a focused test.
The gateway subsystem package carried an "svc" suffix as a naming habit
only: there is no clients/gateway to collide with, and the package imports
no "gateway" library, so it renames cleanly to the plain domain name.
- git mv clients/gatewaysvc → clients/gateway (history preserved)
- gatewaysvc.go / gatewaysvc_test.go → gateway.go / gateway_test.go
- package gatewaysvc → package gateway; Mount() error strings + doc comment
updated to gateway.*
- update the blank import in subsystems/subsystems.go and the doc comments
in deps.go, middleware_edge.go, clients/gatewaypolicy/policy.go
commercesvc is intentionally NOT renamed in this PR. clients/commerce
already exists in-tree — the embedded upstream Hanzo Commerce library
absorbed by #114 (package commerce, ~1583 files) — and commercesvc.go
imports it as github.com/hanzoai/cloud/clients/commerce. Moving
clients/commercesvc → clients/commerce is therefore a directory-level
collision, not an import-alias shadow; re-aliasing does not free the path.
Renaming commerce cleanly needs the operator to decide the target name (or
relocate the embedded library), so it is flagged for follow-up, not forced.
Build: GOWORK=off CGO_ENABLED=0 go build ./... → exit 0
Test: GOWORK=off CGO_ENABLED=0 go test ./clients/gateway/... \
./clients/gatewaypolicy/... . → ok
Co-authored-by: zeekay <ai@hanzo.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The console's /metrics and /status SuperAdmin infra-health board read
VictoriaMetrics `up{}` via the console's Next.js `/telemetry/[...path]` route.
That server route is STRIPPED by the static-export embed (cloud go:embeds
console as output:'export'), so the browser call 404s — the last console error.
Add a same-origin, SuperAdmin-gated VM read proxy on the cloud `/v1` API that
serves the exact three queries the board issues, registered at o11yscope order
69 (before the o11y wildcard at 70):
GET /v1/o11y/vm/api/v1/query?query=up
GET /v1/o11y/vm/api/v1/query_range?query=sum(up)|count(up)&start&end&step
Security follows scope.go's contract ("the client never supplies a raw query"):
- admin(c) gate (X-User-IsAdmin, reserved admin org) — 403 for everyone else.
- The ?query param is ALLOWLISTED to exactly {up, sum(up), count(up)}; anything
else is 400. Range args (start/end/step) validated as positive integers. No
generic PromQL passthrough — this can never become an exfiltration/DoS surface.
- VM's native Prometheus envelope is returned VERBATIM (c.Bytes) so the console's
parseInstant/parseRange work unchanged. Reuses the existing newVMClient()
(O11Y_VM_URL); adds vmClient.queryRaw for the verbatim forward.
Pairs with console 1beb6fca9 (telemetry.ts repoint). Verified: go build + go vet
clean on clients/o11y.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every co-resident subsystem that spoke the commerce billing S2S surface over HTTP
to the standalone pod (commerce.hanzo.svc:8001) now dispatches straight into the
in-tree commerce handler (#114) — no socket, no serialization change. This is what
lets the standalone be retired.
- New seam clients/commerceinproc: commercesvc.Mount publishes the embedded
commerce http.Handler (now carrying /v1/billing via api.Route) once at boot; a
self-routing http.RoundTripper dispatches each S2S request to it in-process when
co-resident, else falls back to plain HTTP (the pre-#111 split-deploy behavior).
Behaviour-preserving: same path, same Bearer+X-Org-Id headers, same body/status
bytes either way — proven by commerceinproc_test (in-process dispatch, HTTP
fallback, base resolution).
- Converted: clients/{billing,account,admin,referrals,authors,affiliates,usage} —
each keeps its OWN request-building + tenant subject-pinning (the security
boundary is untouched); only the transport swaps. account keeps its shared
httpClient for HUSD EVM JSON-RPC and routes ONLY commerce calls in-process.
- The request-edge METERING gate (build.go buildMeteringClient) debits the
in-process handler; base pinned non-empty when co-resident so the gate never
silently no-ops (free-money hole) even after CLOUD_COMMERCE_HTTP_URL is dropped.
GATES: go build ./... + cmd/cloud (sqlite tags) green; commerceinproc + all 8
converted packages' tests pass. LIVE money-parity gate (in-process vs standalone
balance/deposit/usage on shared Postgres + metering debits) precedes retiring the
standalone — that live cutover is gated, not in this commit.
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
Move hanzoai/commerce out of the external-dep seam and INTO the cloud module,
so /v1/commerce (and, next, /v1/billing) is served by one repo, one binary, one
way. No external github.com/hanzoai/commerce* require remains.
Absorbed the EXACT versions cloud already compiled (behavior-preserving — not
main), copied from the module cache so the in-tree tree == what the live binary
built:
- github.com/hanzoai/commerce v1.46.40 -> clients/commerce
- github.com/hanzoai/commerce/metering v0.1.4 -> clients/commerce/metering
- github.com/hanzoai/commerce/thirdparty/ethereum v1.40.0 -> clients/commerce/thirdparty/ethereum
- ONE Go module: removed the three nested go.mod/go.sum; commerce's deps merged
into cloud/go.mod via `go mod tidy` (MVS keeps the highest existing patch — the
single major conflict, luxfi/zap, already resolved to cloud's v1.2.1 in the old
build, so nothing recompiles differently). New direct deps discovered by tidy:
huandu/facebook, square-go-sdk/v3.
- Imports rewritten repo-wide: github.com/hanzoai/commerce* ->
github.com/hanzoai/cloud/clients/commerce* (1103 files: the absorbed tree +
cloud's own build.go/deps.go/middleware_billing.go/clients/{bots,functions}/…
metering importers). commercesvc leaf now imports the in-tree package.
- Excluded the app/ frontend pnpm workspace (59M, NOT referenced by any .go —
the built ui/dist + billing/ui/dist + checkout/ui/dist that Go //go:embed's are
kept). //go:build cloud files kept (excluded from the plain build as before).
GATES (all green):
go build ./... -> ok
go build -tags "libsqlite3 sqlite_fts5" ./cmd/cloud -> ok (633MB binary boots)
go test ./clients/commercesvc/... ./clients/commerce/{,datastore,api/billing} -> ok
go test -run 'Billing|Metering|SpendCap|Commerce' . -> ok
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
cloud pinned beego at a PSEUDO-VERSION
(v2.4.1-0.20260710093857-0ad99bdf8b90). A pseudo-version forces `go` to
resolve the dep via a live VCS fetch-by-commit-hash into the shared ARC
GOMODCACHE cache/vcs. Parallel containment jobs racing that shallow bare
repo hit `fatal: shallow file has changed since we read it` → intermittent
FAIL, which has forced admin/green-locally squash-merges (#235, #236) and
defeated the containment gate.
beego HEAD (0ad99bdf) is exactly one benign commit past tag v2.4.0 (silences
a missing-default-app.conf stderr warning, beego#29). Cut that commit as a
proper patch tag v2.4.1 and pin to it — byte-identical code, but `go` now
fetches the immutable refs/tags/v2.4.1 ref instead of shallow-fetching a
bare commit, so no cache/vcs race. beego is private (no module-proxy), but
the tag ref path is deterministic and does not trip the shallow-file check.
go.sum regenerated via `go mod tidy` (authentic v2.4.1 hash). `go build ./...`
green. iam stays on tag v2.3.10 (already a tag, not a pseudo-version).
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
Co-authored-by: hanzo-dev <dev@hanzo.ai>
cloud installs the ONE global tracer provider wired to the o11y in-process trace
sink, but in in-process-sink mode it sets no OTLP/ZAP exporter endpoint. The
embedded hanzoai/ai module then found no endpoint and DISABLED its gen_ai span
emit, so the gen_ai plane in o11y_traces was dark (cloud's own request spans
flowed; every LLM call's gen_ai span was gated off).
After otel.SetTracerProvider, call aiobject.AdoptHostTracerProvider() so ai emits
every gen_ai span through THIS provider -> the o11y in-process sink -> o11y_traces.
Repins ai to the branch carrying AdoptHostTracerProvider. The OTLP-env unset stays
as orthogonal defense against any other lib forking a competing provider.
ai: github.com/hanzoai/ai v1.804.1 -> v1.804.2-0.20260710172916-723e03f81bca
Simple subsystems use one-line cloud.Mount[S]. Subsystems whose Mount is more
than build+routes (background reconciler, package-global for cross-package hooks,
shutdown cancel — e.g. platform) construct the Service value directly with
cloud.NewBase + &cloud.Service[state]{...} and wire routes via Handle — still the
ONE generic type, one Base derivation.
The ONE server abstraction: a subsystem is cloud.Base (shared deps derived once:
Log/KMS/Bill/Brand/Env/Domain/DataDir) + its own typed State. Handlers are FREE
FUNCTIONS func(*Service[S], *zip.Ctx) error bound with cloud.Handle — so a package
declares NO service/receiver type, only its State (plain data) and its handlers.
cloud.Mount[S] is the one generic entrypoint (build state, wire routes).
Pike minimalism: generics carry the state type, functions carry behaviour. Kills
the 'type svc struct{ …re-plumbed deps… }' shape copied ~40×.
Proven on clients/usage: svc type gone, log via embedded Base (s.Log), commerce
reader in State (s.State.commerce), all handlers/helpers free functions. Build +
vet + test green, routes + tenant isolation unchanged.
* feat(#105): un-stage commerce — serve /v1/commerce in-process from the one binary
Phase 2 final step (tasks #96 → #105). commerce was the last staged
in-process subsystem; drop it from stagedSubsystems so the mount-all default
serves /v1/commerce (+ /_/commerce) from the cloud binary instead of proxying
to the standalone commerce pod. iam + ingress STAY staged (the IAM embed
corrupts its own Beego bootstrap under mount-all).
commerce owns the money path, so the cutover is data-neutral BY CONSTRUCTION:
the authoritative stores stay put and shared — balances/deposits/credits live
in Hanzo SQL (SQL_URL), analytics in DATASTORE_URL, blobs in S3 — and the
in-process commerce.Embed reads the SAME stores via the SAME backend env the
standalone CR carried. Only the small per-org merchant SQLite + tenant `base`
tree migrate into the cloud data dir. No money is copied or split.
Tests: go test -run TestEnabled ./ → ok (staged contract: commerce now mounts
under the empty-Enable default; iam/ingress still gated to explicit CLOUD_ENABLE).
Also gofmt: fixes a pre-existing struct-literal misalignment in LoadConfig.
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
* fix(commercesvc): isolate in-process commerce data under {DataDir}/commerce
In-process commerce writes orgs/ and base/ under its DataDir. cloud sets
deps.DataDir=/var/lib/cloud, where cloud ALSO owns /var/lib/cloud/orgs (its
own per-org subsystem SQLite, HIP-0302) and /var/lib/cloud/base (its Base/IAM
store) — verified live. Sharing the root would open two apps on the same
SQLite files (base/data.db) and corrupt them. Always nest commerce under
{deps.DataDir}/commerce (was only the empty-DataDir fallback), keeping the
commerce ledgers physically separate on the same cloud-api-data PVC. This is
the target path the #105 data migration copies the per-org merchant SQLite +
tenant base into.
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
---------
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The consumer half of the SBOM datastore lane. The registry is the source of
truth: CI produces the CycloneDX SBOM and `cosign attach`es it to the image
digest. Cloud now PULLS that attached artifact and materializes its components
into the global hanzo.sbom_component table — no CI push-ingest.
clients/sbom/pull.go (new): registry pull with go-containerregistry (pure-Go,
no binary deps). Resolves ref→digest, locates the CycloneDX SBOM by the cosign
SBOM tag (sha256-<hex>.sbom) first, OCI 1.1 referrers second, and parses it
through the SAME parseComponents the POST /v1/sbom path uses (one flattener).
Triggers:
- Pull-on-miss: GET /v1/sbom/{ref} with 0 rows and an image ref pulls the
attached SBOM, upserts it, and rereads FINAL — the console goes live with no
console change (the panel already GETs by repository:tag).
- Deploy-time: platform applyLive fires sbom.Prefetch(ref) async when a
deployment goes live (best-effort, idempotent; platform→sbom, one direction).
go.mod: + github.com/google/go-containerregistry v0.21.7 (direct), authentic
go.sum via go mod tidy.
Tests: hermetic end-to-end over a real in-memory OCI registry (cosign tag +
OCI referrers + no-attachment + bare-digest), all green with -race. Plus an
env-gated live end-to-end (pull_live_test.go) proven against registry:2 + real
ClickHouse serving a real cyclonedx-gomod SBOM: GET → pull-on-miss → production
pullSBOM → parse → INSERT → reread → 200 with the real components + cache hit.
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
Co-authored-by: hanzo-dev <dev@hanzo.ai>
A push landed on the embedded git server (clients/git) now fires a build
for every app that tracks that repo+branch — no GitHub, no Actions.
- build.go: GitPushEvent + RegisterPushBuilder/OnGitPush — the same
package-level inversion as kmsClientFactory, so git never imports
platform and there is no git<->platform cycle.
- clients/git/smart_http.go: after a push lands (post-metering),
firePushBuilds fires OnGitPush for each branch ref that advanced
(tags + deletes skipped). Best-effort: a trigger failure is logged,
never fails the push the client already committed.
- clients/platform/push.go: buildFromPush resolves every git-source app
whose RepoURL+branch matches the pushed ref and launches a build via
the ONE shared build-launch core.
- clients/platform/deploy.go: extract startGitBuild (ctx-only) as that
single core; deployGit maps it onto the HTTP deploy, buildFromPush onto
the push trigger — one build-launch path, no duplication.
- clients/platform/validate.go: the cloud's own embedded-git apex
(deps.Domain) is always a trusted build source, so a self-hosted-git
app builds with no env — host all repos on our own git, not GitHub.
Tests: TestPushFiresBuildTrigger (real go-git push -> OnGitPush fires with
org/repo/branch/commit/cloneURL), TestBuildFromPush_{LaunchesMatchingApp,
NoMatchIsNoop,IgnoresImageApp}. Full platform + git suites green.
Jul-5 team-go→cloud migrated workspaces threw "Confirmed social identity is attached to the wrong person" on transactor connect: the confirmed hanzo:<account> SocialIdentity was still attached to a team-go-era Person id, not the deterministic person-<account>. reconcile() now runs remapMigratedSocialIds(uid) to re-point it — migrated-only, idempotent, non-destructive. CI red check (controlplane-containment) is an unrelated shared-runner Go module-cache flake on hanzoai/beego; fix is clients/team-only and green locally (go build ./..., make build, go test/-race/vet/gofmt).
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
Set O11Y_AUTHZ_PROVIDER=local so the in-process o11y authorizes org-scoped,
gateway-authenticated users locally instead of round-tripping to an external IAM
Casbin enforcer the one-binary has no credentials for. That round-trip was 401ing
every /v1/o11y read (provision Grant -> add-policy authz_unavailable), breaking the
console overview-metrics widgets on ~9 secondary product pages. Same enforced
policy; tuples in-process. Bumps o11y 1.5.9 -> 1.5.10.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
v1.5.8 allows digits in TypeRole selectors, unbreaking the built-in o11y-admin role
grant that panicked (→500) on EVERY authenticated embedded o11y data read. Completes
the o11y-telemetry fix (v1.5.6 aliases + v1.5.7 /api passthrough + v1.5.8 grant):
/v1/o11y/{query_range,services,rules,dashboards} now resolve for the console
overview-metrics widgets.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
v1.5.7 routes /api/*-prefixed paths straight to the router in the ExternalPath wrapper,
fixing the double-strip that 404'd EVERY embedded o11y data call. With v1.5.6's
version-less aliases, /v1/o11y/{query_range,services,rules,dashboards}+/metrics now
resolve — fixes the console overview-metrics widgets on studio/gateway/cli/registry/
desktop/console/dashboards/alerts/metrics.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
v1.5.6 registers the version-less /api/<resource> aliases on the app.Server router
the embedded runtime uses (o11y e01015954), so the console's /v1/o11y/{query_range,
services,rules,dashboards} + /metrics resolve instead of 404 — fixes the o11y-telemetry
overview widgets on studio/gateway/cli/registry/desktop/console/dashboards/alerts/metrics.
Also folds in the v1.5.5 C1 cross-tenant llmobs read fix.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The prior fix degraded the orgRouters-error path, but zt's gate() fail-closes with
503 BEFORE that when ZT_CLIENT_ID/SECRET are unset — so /networks + /edge still 503'd
a console error on every load. Short-circuit the two READ handlers to an honest-EMPTY
list (200) when unconfigured, ahead of the gate. WRITES keep the fail-closed 503.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every `hanzo` control command (version, whoami, config, apps, login, …) printed
three lines of server-mode init chatter first:
init global config instance failed ... open conf/app.conf: no such file...
failed to load persistent registry signing key: KMS_SERVICE_TOKEN ... required
generating ephemeral registry signing key for non-production runtime
These come from dependency package init() functions that run before main(), so
main.go's client-vs-server branch cannot gate them, and Go's alphabetical
package-init order (beego < cloud < iam) means no cloud-side os.Stderr swap can
pre-empt them — verified with an init-order probe. Masking the output would also
leave a wasteful KMS fetch + RSA keygen firing on every CLI call. Fixed at the
source instead:
- hanzoai/iam#117: registry signing key resolved lazily (sync.Once) at its use
sites, not in package init(); server token paths keep identical semantics,
the CLI never touches KMS or generates a key.
- hanzoai/beego#29: the benign conf/app.conf probe is silent when the default
file is absent (the CLI case), loud only on a present-but-broken file.
Bump both deps to the fixed versions and document the quiet-output invariant at
the CLIENT MODE gate in cmd/hanzo/main.go.
Result: `hanzo version|whoami|config path|apps --help` emit ZERO stderr; server
subcommands (`hanzo ai --help`, serves) initialize unchanged.
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
Co-authored-by: hanzo-dev <dev@hanzo.ai>
All four are folded in-process and validated on cloud-unified-canary
(v1.786.167): commerce embedded, captable/sign/dataroom "mounted in-process
(goja + per-tenant Base)", /v1/{commerce,captable,sign,dataroom}/health all 200.
Dropping them from stagedSubsystems makes the mount-all default serve them, so
the main cloud (empty CLOUD_ENABLE) runs them natively and their standalone
Postgres/Next pods retire. iam + ingress stay staged (IAM embed corrupts its own
bootstrap under mount-all; iam served by the standalone pod).
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
Co-authored-by: hanzo-dev <dev@hanzo.ai>
console.hanzo.ai /indexer, /oracles, /networks, /edge fired 502/503 console errors
because these list handlers returned the upstream error when the chain indexer /
price-feed oracle / ZT controller is unreachable or unconfigured (ZT_CLIENT_ID
optional). Same graceful fold already shipped for visor clusters/machines/gpus:
log + return an honest-EMPTY list (200). An empty list is honest (you have none),
never fabricated; ZT WRITES stay fail-closed. Clears the console errors on 4
secondary product pages for every org without those backends deployed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
"console" is just our cloud FE name; there must be NO /v1/console/* API domain.
Rename clients/console → clients/account and re-home every route onto its REAL
domain, forwards-only (no /v1/console aliases, no compat shim):
keys GET/POST/DELETE /v1/console/keys → /v1/iam/keys
onboard POST /v1/console/onboard → /v1/iam/onboard
csrf GET /v1/console/csrf → /v1/csrf
embed-status GET /v1/console/embed-status → /v1/embed-status
topup POST /v1/console/topup/wallet → /v1/commerce/topup/wallet
health /v1/console/health → dropped (generic /v1/<name>/health)
waitlist /v1/console/waitlist → dropped (SPA uses clients/base /v1/waitlist)
billing/commerce bridges /v1/billing/*,/v1/commerce/* paths unchanged, relocated
The same handlers, CSRF protection (requireCSRF), per-IP rate limiting, and the
VALIDATED-principal tenancy are preserved — only the PATHS change.
IAM wildcard ordering: keys/onboard live on /v1/iam/*, which clients/iam (order 50)
mounts as a WILDCARD. The package registers TWO subsystems so the specific routes win
Fiber's first-match scan: `account` (order 48) mounts /v1/iam/{keys,onboard} + /v1/csrf
+ /v1/embed-status + /v1/commerce/topup/wallet BEFORE the wildcard (and before the
commerce embed at 100); `account-bridge` (order 122) mounts the /v1/billing/* +
/v1/commerce/* catch-all bridges AFTER clients/billing (121) + the commerce embed (100).
Both share one svc + a process-wide CSRF key so a /v1/csrf token verifies on the bridge
writes. Proven by TestIAMKeysBeatsWildcard (native /v1/iam/keys beats the wildcard;
/v1/iam/oauth/token still falls through) and TestRegisteredOrders (account<50, bridge=122).
waitlist.go/waitlist_test.go deleted; httpClient relocated to topup.go. All 54 tests
pass; go build ./... green.
Co-authored-by: zeekay <ai@hanzo.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Fold hanzoai/dataroom (Papermark fork: Next.js + Prisma + Postgres) FULLY into
the unified cloud binary via the gojahost pattern (HIP-0106, task #101 / epic
Postgres, no Next.js.
REUSE the shared binding, don't build a second. clients/dataroom runs the
self-contained dataroom goja bundle (byte-identical to hanzoai/dataroom/goja/
bundle.js, go:embed) on the REUSABLE clients/gojabase host — the SAME RW-Base
binding captable (#97) pilots and esign (#100) reuses — which injects
__db/__newId/__now, opens one SQLite file per tenant, and runs each dispatch in
one transaction (commits iff status<400). The leaf adds only: the per-tenant
Schema (schema.go), the object-storage seam for document bytes, a bcrypt HostFn
for link passwords, and the public link->org index. Zero domain logic in Go.
gojabase gains ONE generic, domain-free seam — Config.HostFns — the extension
point esign/dataroom both reach for (dataroom injects __bcrypt; the reserved
__db/__newId/__now always win). captable is unaffected (nil HostFns).
Storage: document BYTES go through the cloud object-storage seam (deps.VFS — the
S3/SeaweedFS data plane, a Go storage host-fn in the leaf, NOT local FS); the
bundle persists only the opaque key. View-analytics events (page-by-page
tracking) are Base rows in the tenant DB.
Auth: admin routes require a validated cloud principal (principal.Tenant -> org);
public viewer routes resolve their org from the link index. Passwords hashed with
bcrypt in Go — never plaintext. Registered order 134, HealthOwner, STAGED behind
CLOUD_ENABLE. go.mod unchanged.
Proof (clients/dataroom/flow_test.go, in-process over real per-tenant Base + the
VFS seam): create dataroom -> upload document (bytes to storage) -> attach ->
create email+password-gated share link -> open as public viewer -> authenticate
(wrong password 401, disallowed email 403) -> record per-page views -> analytics:
{"pages":[{"pageNumber":1,"views":2,"totalDuration":6000,"avgDuration":3000},
{"pageNumber":2,"views":1,"totalDuration":900,"avgDuration":900}],
"totalPageViews":3,"totalViews":1}
plus viewer byte download round-trip and cross-org isolation. Live binary boots
with CLOUD_ENABLE=dataroom, mounts in-process, health 200, admin 403 fail-closed.
Companion bundle PR: hanzoai/dataroom#6.
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Fold the esign product (Documenso fork — open-source DocuSign) FULLY into the
unified cloud binary, per HIP-0106 (epic #96, task #100). Cloud serves /v1/sign/*
itself — the TS domain on dop251/goja backed by per-tenant Hanzo Base/SQLite. No
Next.js, no Prisma, no Postgres. Reuses the SAME clients/gojabase RW-Base binding
the captable pilot (#96) established — ONE binding, not a second.
clients/gojabase: add an additive, backward-compatible Config.HostFns passthrough
— extra native host globals injected per dispatch alongside __db/__newId/__now.
esign uses it for __pdf; captable is unchanged (tests green).
clients/sign (leaf on gojabase):
- schema.go — the per-tenant sign.db DDL (gojabase Config.Schema).
- signer.go — THE HARD PART as Go host-functions injected via HostFns as
__pdf = { stamp (pdfcpu renders field values onto the PDF), sign (real
x509/PKCS#7 seal via digitorus/pdfsign) }; signer sourced from KMS PEM,
persisted PEM, or a self-signed dev cert. Signing orchestration stays TS.
- sign.go — Mount + route table; owner routes gated by principal.Tenant,
recipient token routes org-in-path. Registered + STAGED behind CLOUD_ENABLE
(config.stagedSubsystems); retires the standalone esign pod on cutover.
- sign_test.go — end-to-end wire proof: create→recipient→fields→send→sign→
complete seals a REAL signed PDF (/ByteRange,/Type /Sig,PKCS7) + full audit,
per-tenant Base-backed, with cross-tenant isolation.
Bundle: github.com/hanzoai/sign (goja/bundle.js) — the ESM-free domain port on
the gojabase contract (__db/__newId/__now/__pdf, handle{route,params,query,orgId,
body}, one txn per dispatch). go.mod additive; pdfcpu pinned v0.11.0,
hhrutter/pkcs7 v0.2.0 (no bumps).
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Ship the crown-jewel security fix: o11y v1.5.5 (org-scoped llmobs span views — closes
the cross-tenant read exposed by the Observe->/v1/o11y repoint) + ai v1.804.1 (BYO
billed_cost). The leaky observation read is already deleted upstream (#217); this repin
ships the org-scope via the cloud-embedded o11y runtime.
(Coordinator asked for v1.786.162 but that tag + through v1.786.165 were already cut on
origin; this is the next free tag.)
Crown-jewel security repin off the pseudo-versions to the merged release tags:
- o11y v1.5.4 -> v1.5.5: the llmobs span-view SQL is now org-scoped
(gen_ai.hanzo.org_id = <validated tenant>, fail-closed) — closes the cross-tenant
read the Observe->/v1/o11y repoint exposed. Cloud EMBEDS the o11y runtime
(clients/o11y), so this is the ship vehicle for the fix.
- ai <pseudo> -> v1.804.1: gen_ai span emits _o11y.gen_ai.billed_cost alongside
total_cost (BYO invoice reconciliation) + the enriched gen_ai span.
The leaky cloud_usage-as-observations read stays DELETED (upstream via #217); the
metering warehouse (cloud_usage via /v1/usage + the #218 /v1/evals/metrics dashboard)
is untouched. No go mod tidy (luxfi/keys go.sum fragility) — only the two repins.
Build: go build ./... green. Test: clients/eval + clients/o11y green.
Cloud now serves /v1/captable/* ITSELF, per tenant, on Base/SQLite — the PILOT of
epic #96 (fold the Captable,Inc app into the unified binary; drop Next.js/Prisma/
Postgres). Where clients/plan + clients/pricing host a read-ONLY @hanzo catalog in
goja, captable hosts the tRPC business LOGIC (ported to a self-contained goja
bundle in github.com/hanzoai/captable) and gives it PERSISTENCE over per-tenant
Base/SQLite. The bundle carries logic; the Go host carries storage.
REUSABLE Base-goja binding (clients/gojabase) — the deliverable esign (#100) +
dataroom (#101) rebase onto. It is the storage-bearing sibling of clients/goja
(the pure JS engine): given a Bundle + a per-tenant Schema (DDL) + DataDir, it
- opens ONE SQLite file per tenant (lazy, migrated once, cached; slug-contained),
- injects per dispatch a tenant-bound __db bridge (query/exec) + __newId + __now,
- runs globalThis.handle inside ONE transaction that commits iff status<400 and
handle didn't throw (atomic multi-statement mutations for free), and
- carries ZERO domain logic. clients/goja gains DispatchWith (per-call native
globals) as the read-WRITE extension of the read-only plan/pricing path.
clients/captable leaf: go:embed'd bundle (hanzoai/captable.Bundle) + the per-
tenant schema (Prisma model → SQLite DDL) + a company seed (OnOpen) + the
/v1/captable/* zip routes. Org resolves from the VALIDATED principal
(principal.Tenant), never a client header; that org selects the DB file AND
scopes every row. Registered order 133; STAGED behind CLOUD_ENABLE (joins iam/
ingress/commerce in config.stagedSubsystems) so main stays shippable and the
standalone captable service keeps authority until the phase-2 cutover.
Full fold over Base: stakeholders, share classes, equity plans, securities
issuance (shares + options), share transfers (full + partial, atomic), SAFEs +
convertible notes, rounds + investments (a priced round issues shares and
dilutes), and a computed cap table (fully-diluted ownership, per-class
authorized-vs-issued, convertibles + rounds summary).
Proven:
- clients/gojabase: RW round-trip, per-request rollback (caught-500 + raw-throw),
per-tenant isolation, OnOpen seed, slug traversal-containment (real SQLite).
- clients/captable: the REAL embedded bundle vs REAL SQLite through the whole
lifecycle (create stakeholder → issue share class → issue shares → priced round
+ investment dilution → transfer → cap table), and an HTTP wire test via the
zip/Fiber test client (trusted headers) proving create→read-back, issuance→cap
table, the bundle's OWN 404 (not a proxy 502), the 403 principal gate, and
cross-tenant isolation.
- Live binary boots under CLOUD_ENABLE=captable and serves /v1/captable/health
200 in-process (a proxy would 502).
go build ./... + go build -tags "libsqlite3 sqlite_fts5" ./cmd/cloud green.
go.mod pins github.com/hanzoai/captable at its merged main commit; go.sum authentic.
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The server side of the ex-/v1/arcd surface: one native build API on the
runner fabric that hanzo build, git-push-to-deploy, and cloud's own
self-release all call — no GitHub builders, no Actions.
- runner.go: POST /v1/runner. Privileged (caller supplies the output
image), so gated by a constant-time build-callback token AND an
image-ref allowlist (ghcr.io/{hanzoai,luxfi,zooai}/*). Fails closed
when no token is configured.
- k8s.go: launchDirectBuild — validated at the same choke point as the
tenant build; plus buildFrontendCmd + buildJobSpec extracted so tenant
and direct builds share ONE Job spec and ONE frontend selector.
- hanzoai/pack is the default BuildKit frontend (zero-config, gateway.v0);
a Dockerfile is the explicit escape hatch (dockerfile.v0).
Tests: token 503/403, missing-field 400, image-allowlist 403, happy-path
202. Full platform suite green; whole cloud module builds.
Before: subsystems opened SQLite inconsistently. clients/code used the good
per-ORG-file pattern ({DataDir}/orgs/{slug}/code.db, resolved per-request), while
git/functions/tracker each opened ONE shared DB at Mount ({DataDir}/git.db,
functions.db, tracker.db) scoped only by an org column per-row — a decomplected
tenancy gap where the physical boundary was a single file for every tenant.
After: ONE resolver — cloud.TenantDB(dataDir, org, project, subsystem) — is the
single way any subsystem opens a tenant SQLite DB. Path convention:
project-scoped: {DataDir}/orgs/{orgSlug}/projects/{projectSlug}/{subsystem}.db
org-scoped: {DataDir}/orgs/{orgSlug}/{subsystem}.db
It MkdirAll 0700s, opens via the sole "sqlite" driver (github.com/hanzoai/sqlite),
applies the shared single-writer + WAL pragmas, and folds org/project through the
injective SanitizeOrg slugger so distinct tenants can never share a file and no
segment can traverse. A generic cloud.TenantStore[T] caches per-tenant stores
(opened once each) so the hand-rolled per-subsystem map is DRY'd into one value.
SanitizeOrg (the one injective org-slug normalizer) moves to the root cloud
package beside OrgHasUnsafeRune and TenantDB; provisioning.SanitizeOrg delegates
to it, byte-identical, so S3/KMS/knowledge slugs are unchanged.
Subsystems migrated onto the resolver:
- code: adopts the helper; stays ORG-scoped (no project axis) — same path.
- git: single-shared git.db -> per-ORG file. Kept org-scoped (NOT project)
because /v1/git/usage is a deliberate org-wide rollup across every
project; the project stays a row column.
- functions: single-shared functions.db -> per-ORG file (no project axis).
- tracker: single-shared tracker.db -> per-(org, IAM-project) file — tracker is
project-scoped (principal.Project); its KEY-based projects are rows
WITHIN each per-project file.
- tasks: left as-is — it opens NO SQLite (delegates to the shared durable
engine owned by durable.go), so there is nothing to migrate.
Data safety: the single-shared -> per-tenant switch is fail-closed (an invalid
org/project errors rather than falling through to another tenant's file). These
are new subsystems with little/no production data; existing rows in a prior
shared *.db would live under {DataDir}/{subsystem}.db and are NOT auto-migrated —
a deployment carrying such data must relocate rows into the per-tenant files
before cutover. No silent data drop.
Tests prove isolation: two orgs -> two files, no cross-read; two projects under
one org -> two nested files; project-scoped path nests; the cache opens each
tenant once. Root tenantdb_test.go plus per-subsystem end-to-end file-isolation
tests (git/functions org, tracker project).
Co-authored-by: zeekay <ai@hanzo.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
CONSOLE_CACHEBUST was the cloud sha, so a console-only push could not trigger a
fresh embed without a cloud commit — the whole point (freshness) leaked between
cloud pushes. Resolve hanzoai/console main HEAD (git ls-remote, extraheader
cleared since actions/checkout's GITHUB_TOKEN 404s cross-repo; gh absent on the
runner) and use it as the cachebust; fall back to the cloud sha if empty. A
console change now moves the cache key on its own.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* controlplane inc-2 seam (a): per-pod ML-DSA-65 identity keys
Replace the symmetric-HMAC proof-of-possession with per-pod asymmetric ML-DSA-65 (github.com/luxfi/crypto/mldsa, MLDSA65, FIPS 204 Level 3) identity keys — the same key material that becomes the cert-signing key in seam (c). idKey is now a fresh random keypair (crypto/rand), NEVER seed-derived; the registry stores the public key; signPoP uses the FIPS 204 5.2 deterministic variant with a domain-separation context; VerifyPoP is asymmetric VerifySignatureCtx.
Acceptance: the two byzantine-safety tests TestRed_B_DerivablePoPForgesHonestLeg and TestRed_B_LoneNodeForgesFullQuorum are un-skipped and now GREEN (an attacker rebuilding a pod's custody from public inputs gets a different key whose PoP fails); TestSafety_RogueAndForgedLegs_Rejected is re-armed to forge a CORRECTLY-DERIVED leg under a real attacker key (not a bit-flip) and still rejects it.
Scope: seam (a) only. ProductionBCCSigningReady() stays false and all other stubs remain — the flag flips later in the same change that lands the real cert (seam c) and deletes the stub types. Full suite green (zero skips), -race clean, gofmt clean, containment intact (no-tag build still matches no packages).
* controlplane seam (a): R3 — doc.go PoP narrative now reflects discharged crypto
RED review R3: refresh the stale CLASS-B caveat + ShareCustody stub-catalog entry so prose matches the code. Per-pod identity keys are real random ML-DSA-65 (not seed-derived); the two TestRed_B_* are un-skipped + green; only the z-share / cert crypto remains stub (ProductionBCCSigningReady stays false until seam c). Doc-only; no code change.
Adds the write path for LLM-observability events (traces/observations/scores)
that the retired console-worker (Node BullMQ->Valkey->Datastore) used to
provide, folded into the cloud binary as a normal o11y subsystem.
- POST /v1/o11y/ingestion (validated tenant via principal.Tenant) -> parse batch
-> route by type -> per-table batch insert into the Hanzo Datastore via the
branded github.com/hanzoai/datastore-go/v2 client (promoted to a direct dep;
first direct user in cloud). ZERO clickhouse imports/identifiers -- datastore-go
brings the same upstream ch-go line (MVS-unified v0.71.0) the SigNoz o11y query
runtime uses, so it coexists in one binary (verified).
- Grounding: the embedded o11y runtime (github.com/hanzoai/o11y) is a SigNoz fork
serving INFRA o11y and mounts app.All("/v1/o11y/*") at order 70; it has no
LLM-obs ingestion. This is a cloud-native SPECIFIC route registered at order 68
so Fiber's in-order match binds it AHEAD of the wildcard (same rule scope.go
uses for /v1/o11y/{logs,metrics,status} at 69). A route at the OTLP-ingest
order (72) would be swallowed by the proxy.
- Oversized event bodies overflow to object storage (blob ref stored inline),
threshold via CLOUD_O11Y_INGEST_BLOB_BYTES (default 1 MiB).
- Always mounted (no feature flag); the Datastore (O11Y_DATASTORE_DSN) is a
required dependency. Fail-soft: no DSN / unreachable datastore -> unmounted,
never blocks boot. Inert in prod until the console producer repoints here.
Tests: go test -tags 'cloud cloud_mount' ./clients/o11y/... green (8 cases:
routing, batching, blob overflow/disabled, sink+blob error propagation, threshold).
Full build green: go build -tags 'cloud cloud_mount' ./...
FLAGGED for cutover review:
- ASSUMED table/column schema (worker source ships dist-only; reconcile
traces/observations/scores columns with 002_llm_observability.sql before the
console producer is repointed).
- Durable EmbeddedTasks hand-off: flush runs INLINE today (removes BullMQ+Valkey);
the durable enqueue->activity path is the next reviewed step (a Datastore insert
must be a durable Activity, not run in a workflow fn).
Pre-existing stray direct modernc.org/sqlite import in a test file (from #201).
Swap to the canonical _ "github.com/hanzoai/sqlite" (its !cgo backend IS modernc,
identical behavior) so NO file anywhere imports modernc directly. Test-only —
not in the shipped binary — but keeps 'hanzoai/sqlite only' truly airtight.
ONE data model, MANY adapters: handlers keep returning structured JSON; this
middleware re-serializes successful application/json responses through
zap-proto/md when the caller asks (Accept: text/markdown or ?format=md), so
token-efficient markdown is a request-time choice, not a second code path.
/v1/code/ + /v1/agents/ may default to markdown; caller override always wins.
Fail-safe: md render error leaves JSON untouched (never a 500); streams/HTML/
bytes pass through.
Native, per-org code-intelligence subsystem for AI coding agents and hanzo.app.
Retrieval is HYBRID — three orthogonal tiers fused with reciprocal-rank fusion
(the SOTA lesson that embeddings alone under-serve code search):
- lexical — FTS5 trigram over code-tokenized text (camelCase/snake_case split,
operators kept); substring + regex via trigram pre-filter + regexp verify (Zoekt).
- symbolic — go/parser for Go (real def→ref call edges) + compact lexical
extractors for TS/JS/Python/Rust/Solidity; go-to-symbol + edge table.
- semantic — AST-boundary chunks embedded via the SAME gateway /embeddings
clients/knowledge uses; cosine KNN over a float32 vector table.
Storage is ONE SQLite file per org at {DataDir}/orgs/{slug}/code.db (HIP-0302):
the tenant boundary is PHYSICAL. Every request is principal-gated (principal.Tenant)
— no validated principal ⇒ 403, a client X-Org-Id is never trusted.
Routes (order 134, before the AI /v1/* catch-all):
GET /v1/code/search ?q=&type=text|regex|symbol|semantic|hybrid&repo=&limit=
POST /v1/code/context {query,budgetTokens,repo} → budget-packed context bundle
GET /v1/code/ask ?q=&repo= (or POST) → cited RAG answer (deps.AI)
POST /v1/code/index {repo,files,prune} → (re)index, incremental by hash
Parsing is pure-Go and vectors are brute-force cosine because the repo's canonical
build is CGO_ENABLED=0 (Makefile); CGO tree-sitter and the sqlite-vec loadable
extension would break `go build ./...`. The vectors table is the schema-compatible
sqlite-vec `vec0` drop-in seam. Builds + tests green under both CGO=0 (modernc) and
CGO=1 (sqlite_purego).
listMachines/listGpus already log+fall-through to a BYO-only list when Visor is
down; listClusters alone returned the error, which surfaced as a 502 + a console
error on the Clusters/GPUs page for every org where Visor isn't deployed
(visor.hanzo.svc unresolvable). Mirror the graceful fold: log, drop managed
pools, still return the org's BYO clusters. 200, honest empty, no page error.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Unblocks the embedded IAM subsystem. v1.31.18 getPermissionEnforcer called
authz.NewEnforcer(&DefaultLogger{}, false) — the (logger,bool) form Casbin
type-switches params[1]->persist.Adapter, panicking "bool is not persist.Adapter:
missing method AddPolicy" the moment InitEmbed runs against a FRESH store (exactly
the unified cloud iam subsystem). v1.31.19 (fe50caf7) uses NewEnforcer()+SetLogger.
Proven on a fresh store: CLOUD_ENABLE=iam,ai boots green ("iam embedded in-process")
and serves /v1/iam/.well-known/openid-configuration 200 (was fail-closed 503/404);
iam + ai co-reside, process listens :8080/:9653/:9090.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Finishes the native web-search half of /v1/websearch (HIP-0106: no external
search SaaS, one fewer non-Go dependency). The route no longer reverse-proxies
to the retired SearXNG pod; it runs metaSearch in-process:
- search.go: keyless meta-search over public engines (Bing default, DDG opt-in),
parses HTML with x/net/html, merges+dedupes by normalized URL, returns the
exact SearXNG {query,number_of_results,results[]} envelope the LibreChat
searxng client decodes verbatim. A failing/bot-challenged engine contributes
zero and never fails the request (degrades to fewer results, never a 5xx).
- websearch.go: /v1/websearch/search now serves searchNative; removed
newSearchProxy + searchUpstream + WEBSEARCH_UPSTREAM. Auth unchanged (F2):
validated principal OR shared X-API-Key; neither ⇒ 401/503, never open.
- tests: converted the proxy route/guard tests to native (mocked engine via
WEBSEARCH_BING_URL fixture); added metaSearch parse + graceful-degrade tests.
go test ./clients/websearch/... green; full binary builds.
Wire the metrics board handler and route that the completed data layer was
missing. metrics.go already had the ClickHouse ledger + GenAI-span aggregation
(assembleTotals/Series/ByModel, usageWhere, latency percentiles) and the
in-memory honest-empty path; this adds:
- metricsBoard handler: principal gate (403 without a validated tenant),
SuperAdmin all-orgs via c.IsAdmin(), range preset -> window/bucket
(24h|7d|30d, ?interval override), ?project threaded. nil telemetry or a
non-default project -> honest-empty board (never 503, never fabricated).
- Telemetry.Metrics added to the interface (dsTelemetry + memTelemetry already
implement it).
- Route registered in Mount() and the test mountApp().
All clients/eval tests pass, including the three handler tests that previously
404d (TestMetricsHandlerHonestEmpty / RequiresPrincipal / NonDefaultProjectEmpty).
Co-authored-by: hanzo-dev <dev@hanzo.ai>
* feat(eval): drop cloud_usage-as-observations read; collapse to o11y span plane
Step 1 of the unified AI-observability plan: the observation of record is the
o11y gen_ai span plane (/v1/o11y/observations), not a second projection of the
metering warehouse. evalsvc no longer reads hanzo.cloud_usage as 'observations'.
- Remove Telemetry.ListObservations + Observation/ObservationFilter types + the
dsTelemetry (cloud_usage) and memTelemetry impls + the now-dead asInt64 coercer.
- Remove GET /v1/evals/observations route, its handler, and observationView/
toObservationView. The console Observe > Observations view now reads o11y.
- cloud_usage stays the metering warehouse (read by /v1/usage + billing) — only
the duplicate obs projection is gone. eval keeps its unique datasets/evaluators/
runs and its own eval_traces/eval_scores tables.
- Bump ai dep to the enriched-gen_ai-span build (forward-only from v1.804.0).
Build+vet+test ./clients/eval green.
* cloud(cli): compute ladder — hanzo run/agent/bot verbs (thin /v1 clients)
Preserve in-flight CLI work: hanzo run (artifact/function), hanzo agent
(headless managed agent -> /v1/agents/:ref/run), hanzo bot (computer-using
agent -> operative/visor). Thin clients over IAM token + cloud /v1.
Claude-Session: https://claude.ai/code/session_01EjRSpFBvbjxTqaYVbds9bA
world.hanzo.ai is the OIDC client `hanzo-world`; IAM stamps its access
tokens with aud=hanzo-world (each app's aud is its client_id). That
audience was missing from cloud's baked identity-sanitizer allowlist, so
signed-in world tokens resolved anonymous and api.hanzo.ai returned 401.
Append the client_id (forwards-only) and add a membership + resolved-env
acceptance test.
Claude-Session: https://claude.ai/code/session_013jh8aka8q8RvhhVQ1psMeW
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Cloud now serves /v1/commerce/* + /_/commerce/* ITSELF via a new
clients/commercesvc leaf that wraps commerce.Embed's gin handler — the same
wrap-don't-rewrite fold clients/iam and clients/kms use — instead of proxying to
a remote commerce pod. deps.Commerce resolves to the in-process CommerceClient
(CommerceInProcess) when commerce is co-resident.
Why a leaf (not the upstream commerce.Mount): commerce's own Mount/init sit
behind //go:build cloud, and cloud builds without that tag, so cloud's plain
build never compiled the registration — commerce was blank-imported yet absent
from cloud.Registry (proxied at runtime). The leaf lives in-repo and imports only
commerce's cloud-free surface (Embed, api.Route, Version), so the upstream
commerce.Mount that imports cloud stays tagged out — no cloud<->commerce import
cycle. Zero go.mod/go.sum churn (commerce v1.46.40 already required).
STAGED (prod-safe): commerce joins iam/ingress in stagedSubsystems, so the
mount-all default is unchanged — the in-process cutover happens only on explicit
CLOUD_ENABLE=...,commerce. Phase 2 flips the default once validated in prod. The
remote proxy seam (CLOUD_COMMERCE_HTTP_URL / CLOUD_COMMERCE_ZAP_ADDR) is
untouched; the disabled/RPC fallbacks still compile.
GetTenantConfig is answered in-process (org + brand); CheckEntitlement fails
closed until commerce exports its subscription->plan->features resolver (Phase 2)
— the specified "cannot verify => never open" default clients/entitlements relies
on.
Proven: CLOUD_ENABLE=commerce -> GET /v1/commerce/tenant, /v1/commerce/catalog,
/_/commerce/healthz all 200 from the embedded gin engine; an unknown
/v1/commerce path returns gin's own 404 (a proxy would 502).
Claude-Session: https://claude.ai/code/session_016yg7GPhYdWCh9vpp4HEwLZ
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Served sites typed .wasm/.data/.mem/.unityweb/.pck via mime.TypeByExtension only,
which returns "" for the engine payloads (empty Content-Type) and can mis-type
.wasm — breaking WebAssembly.instantiateStreaming (requires application/wasm) and
the loader's streaming fetch of Unity/Emscripten .data/.mem and Godot .pck. Pin
those in one gameAssetType map; everything else defers to the stdlib table.
Unblocks hosting WebGL game builds. Tested (TestGameAssetContentType).
'arcd' is the client-side GitHub-Actions BYO product (github.com/arc-runner);
the platform's own native CI/compute pool is 'runner'. Rename the enqueue path
and de-brand the build command help/comments accordingly. Pairs with the
platform route move pages/api/v1/arcd/enqueue.ts -> pages/api/v1/runner.ts.
The 4 go invocations (mod download, sqlite double-register gate, encryption-proof
test, final build) had no cache mount, so every release recompiled the full CGO
graph (k8s + otel-collector + sqlcipher, CGO=1) from scratch on the ephemeral ARC
runner — the Build step was ~1316s/22min, dwarfing every other step. Mount
/go/pkg/mod + /root/.cache/go-build (sharing=locked) so the persistent ARC dind
BuildKit cache keeps the Go build+module cache warm. First build cold; subsequent
builds reuse compiled artifacts — target single-digit-minute rebuilds.
CreateTraces opens the ClickHouse conn + spawns the writer's ticker goroutine, so
a failed Start must Shutdown to release them rather than leak on the fail-soft
mount path.
The validated-org-principal check asserted ==200, coupling this cross-package
gate test to the orgs/users/me handler's downstream success (IAM/datastore),
which flaked under full-suite parallel load. Assert the gate/shadow decision
only — admitted (not 403) and mounted (not 404) — since tenant-scoped data
correctness is proven in clients/admin/scope_test.go. Anonymous->403 and the
SuperAdmin-only platform routes are unchanged.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The prior two attempts (c97af12 sha-pin via git ls-remote, 4a7e533 via gh api)
both FAILED at version-compute: the ARC runner has no gh CLI, and git ls-remote
404s because actions/checkout installs a global http.extraheader carrying THIS
repo's GITHUB_TOKEN (scoped to hanzoai/cloud), overriding URL creds on the
cross-repo hanzoai/console lookup.
Bulletproof instead: no console-HEAD resolution at all. release.yml passes the
cloud commit sha as --build-arg CONSOLE_CACHEBUST (unique per push); the
Dockerfile references it in the proven `git clone --depth 1 --branch main`
RUN, so the layer cache key changes every build and re-clones console main HEAD
fresh. No gh, no ls-remote, no extraheader. Correctness over cache reuse — the
console stage rebuilds each release, but the embed is never the frozen snapshot.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- clients/base: register /v1/base/health in Mount BEFORE the CLOUD_BASE_EMBED
gate and mark cloud.HealthOwner — same always-on liveness pattern as
clients/plan + clients/pricing. Fixes cmd/cloud TestMountAllAndServeHealth
(base was the only listed subsystem not self-serving health).
- clients/kmssvc red_dualmount_test: #192 made the admin cockpit two-scope —
orgs/users/me are org-scoped (guardScoped: a validated org principal is
admitted + hard-scoped to its own org; anonymous still 403), while
audit/roles/finance/flags/revenue stay SuperAdmin-only (guard: 403 for a
non-admin principal). Assert both, plus kms's public /v1/kms/config never
shadows either. No production code semantics changed — the stale test tracked
the pre-two-scope admin-only contract.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
git ls-remote failed 'Repository not found' on the cross-repo hanzoai/console
lookup: actions/checkout installs a global git http.extraheader carrying THIS
repo's GITHUB_TOKEN (scoped to hanzoai/cloud only), which overrides the URL
creds and 404s. gh api honors GH_PAT (org read) and is unaffected. Unblocks the
console-embed-freshness fix (c97af12) — the release aborted at version-compute
before building.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cloud already embeds the o11y trace write side (chtraces), so shipping its OWN
spans over the ZAP wire to a collector that then writes the same store is pure
waste. Route them through the ZAP locality-adaptive Router (luxfi/zap v1.2.1):
when sender and sink share this binary, the Cost-0 InProcessInterface wins and
the LIVE proto batch is handed to the sink by value — zero ZAP-wire serialize,
zero socket, no second collector hop.
- clients/o11y/tracesink.go: Router + Cost-0 InProcessInterface on Destination
"hanzo.o11y.traces"; a chtraces exporter (the REAL o11y_index_v3 writer, reused
as a consumer.Traces — its pdata->SpanV3 conversion is unexported) fed in
process. Handler bridges SDK-exporter proto spans -> pdata via one in-memory
OTLP round-trip. OPT-IN (O11Y_TRACES_ZAP_INPROCESS) + fail-soft: any error
leaves cloud's spans on the wire; can never take cloud down. NewTraceExporter +
routerTraceClient own the transport; cmd/cloud stays the composition root.
- cmd/cloud/telemetry.go: install the ONE tracer provider over the Router
(in-process primary, ZAP wire fallback when the sink isn't registered). Enable
when the in-process sink is on OR a wire endpoint is set. Composition-root
single-provider invariant (ai's GenAI tracer inherits it) preserved.
- go.mod: github.com/luxfi/zap v1.2.0 -> v1.2.1 (adds Router/InProcessInterface).
TDD: span from cloud's provider reaches the in-process handler with no wire
client and no socket; proto->pdata round-trip preserves the span; router prefers
in-process, falls back to wire on ErrNoRoute, surfaces ErrNoRoute when neither.
The kmssvc dir was an artificial split: clients/kms is the KMS library
(embeds luxfi/kms + SecretStore + in-process client), clients/kmssvc was
the Fiber subsystem mounting /v1/kms/* — and it was ALSO package kms, dir-
named kmssvc only to dodge a dir-name collision. That svc suffix is a
workaround, not a concept.
Move every kmssvc file into clients/kms (kmssvc.go → mount.go; login.go,
env_required/kms/login_ratelimit/paas_sync/red_*/v6 tests). subsystems.go
imports clients/kms (order 10, /v1/kms/*); exactly one cloud.Register("kms").
Zero kmssvc refs remain. Full cloud build + kms package tests (incl red_*
adversarial) green. (Pre-existing clients/kms/replication test build failure
is unrelated — broken on main before this change.)
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Composes the ha lease-round (v0.1.1) and vfs/replica.FencedStore (v0.6.3) into
the cloud per-org substrate, and adds the request-layer exactly-once dedup, so
per-org SQLite is safe under a multi-replica Deployment (not just replicas:1).
Four orthogonal concerns, one home each:
internal/org/fence.go CASFencer: the INTERIM monotone round source. A per-org
writer lease {round,owner} over the object store's CAS
(a single linearizable register); takeover strictly
bumps the round, renewal keeps it. Implements ha.Fencer,
so the Lux BFT round drops in behind the same seam later.
HRW is only an optimization (cuts contention); safety
does not depend on a fresh/agreed membership view — a
split view costs liveness, never safety, because the
fence backstops it.
internal/org/condstore.go MinioConditionalStore: the concrete atomic-CAS store
(minio If-Match against the SeaweedFS gateway), promoted
from the orphaned internal/writefence. satisfies
replica.ConditionalStore.
internal/idem/ exactly-once request execution: request-id PK written in
the SAME txn as the effect (atomic dedup+effect), shipped
in the per-org snapshot so a retry re-routed after a
rolling upgrade is deduped on the successor. 'fail if
already done' via ErrAlreadyApplied.
internal/org/shared.go re-export FencedStore/ConditionalStore/Lease/Fencer/
Round/ErrStaleRound so the org API stays one surface.
Deletes internal/writefence (was orphaned, zero importers): its fence primitive
is promoted to vfs/replica.FencedStore (the storage substrate's rightful home),
its minio store to condstore.go — one and one way, forward-only.
Safety (no data loss + no double-exec) rests on the composition, proven by
handoff_test.go against the four hazards: (a) partition minority cannot advance
the round -> cannot write; (b) rolling-upgrade handoff -> successor CarryForwards
the predecessor's last landed write + dedups; (c) duplicate request -> idem runs
once; (d) deposed writer -> refused by election, and if it still ships, fenced by
FencedStore. Ship-before-ack: a request is 'done' only once its fenced ship lands.
Interim round source = single linearizable register (object CAS) = crash-fault
tolerant. Roadmap: replace readLease/claim with the quasar PQ-BFT agreed round
(Byzantine-tolerant, deterministic finality, 3/5 quorum) — same ha.Fencer seam,
same FencedStore admission.
Cross-repo: pins ha@fence-lease + vfs@fenced-store (pseudo-versions); retag to
ha v0.1.1 + vfs v0.6.3 once those merge. go.sum touches only ha+vfs; the
pre-existing luxfi/keys@v1.2.0 re-tag mismatch (blocks `go mod tidy` on main
today) is unrelated and untouched.
The console-clone+build layer was keyed only on static text ('git clone
--branch main'), so on the persistent ARC dind BuildKit cache EVERY cloud
build re-embedded the SAME stale console snapshot. New console work — the
native Tracker module, and everything since the cache was first warmed —
silently never shipped: console.hanzo.ai/tracker rendered an old surface
with zero /v1/tracker calls even on a freshly-deployed image.
Fix (values, not places): release.yml resolves hanzoai/console main HEAD
(git ls-remote) at build time and threads it through --build-arg CONSOLE_REF;
the Dockerfile fetches that exact ref (init+fetch+checkout, sha- or branch-
capable). A changed sha moves the layer cache key, so each build embeds the
live console commit — deterministically pinned, never frozen.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
admin.hanzo.ai as ONE cockpit for BOTH tiers off ONE identity predicate. scope.go decomplects the rule into a single place (resolveScope/scopedOrgs/descendants): owner==admin (c.IsAdmin, SuperAdmin) => cross-tenant, all orgs; any other validated admin => their OWN org, hard-scoped server-side. guardScoped admits a SuperAdmin OR a validated org-pinned caller and the handler scopes the data; the platform control plane (roles/audit/finance/revenue + launch/release/flags/access) stays s.guard (SuperAdmin only). me/overview/orgs/users/usage/analytics/bases are org-scoped; a non-super caller can never read another tenant for any input (their org is the sanitized, un-forgeable c.Org()).
Feature flags / launch / access via Hanzo Insights (one engine, not two): clients/featureflags is a hot-apply evaluation seam over Insights /flags (env = fallback default, 15s TTL, fail-safe degrade); /v1/admin/flags surfaces the launch switches (public_signup, waitlist_open, waitlist_access_capacity, ...) with deep-links to the Insights flag manager + activity log. /v1/admin/waitlist + /boost proxy the Base waitlist engine (server-authed, KMS secret, audited grant). /v1/admin/bases is the scoped tenant-Base panel seam (honest-empty until the Base engine is embedded).
Fix: waitlist.go shadowed the ok() envelope writer with a local bool (compile error) — renamed to configured. Tests: scope_test.go proves the two-scope invariant (super sees all; org-admin hard-pinned to own org; platform routes 403 an org-admin; users read pinned to own org); featureflags_test.go proves hot-apply + env fallback. go build ./... green; go test ./clients/admin/ + ./clients/featureflags/ green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
* feat(edge): embed the gateway CORS + per-IP rate-limit role in cloud
Fold the hanzoai/gateway edge role into cloud so it can serve api.hanzo.ai
directly, dropping the redundant KrakenD hop. cloud already validates the IAM
JWT + strips/re-mints identity headers (SanitizeIdentity), runs the per-tenant
ScopeRateLimit, and owns balance/spend-cap quota (BillingGate) — the gateway
duplicated the JWT/identity role and added only CORS + a per-IP flood cap.
Adds middleware_edge.go (package cloud), two orthogonal middlewares wired into
the serve.go chain AFTER Logger/sites and BEFORE identity:
- EdgeCORS: credentialed reflect-Origin CORS matching the gateway's policy
(methods/headers/max-age). DEFAULT OFF (empty CLOUD_CORS_ORIGINS) so the
shared Traefik ingress `cors-allow-all` stays the sole CORS authority on the
recommended rollout — enabling both would double the ACAO header. Set
CLOUD_CORS_ORIGINS only on a direct DO-LB->cloud edge. Handles + short-circuits
the OPTIONS preflight (204) before any auth work.
- EdgeRateLimit: per-client-IP fixed-window flood cap (default 100/1s, gateway
service-tier parity, strategy:ip) that runs BEFORE identity — the one gap
ScopeRateLimit (keyed on the validated tenant) structurally can't see: an
anonymous flood with no valid JWT. Keyed on the leftmost X-Forwarded-For;
in-cluster direct callers (no XFF) are exempt, matching the standalone
gateway's public-only scope. Opportunistic eviction keeps the bucket map
bounded at edge IP cardinality (default ON, CLOUD_EDGE_RATELIMIT=false to
disable). This preserves the gateway's protection rather than dropping it.
Config: CORSOrigins, EdgeRateEnabled/PerIP/WindowSec (config.go).
Tests: TestOriginMatcher, TestEdgeCORS_*, TestEdgeRateLimit_* (all green).
CGO_ENABLED=0 go build ./... + go test . green.
* feat(gateway): /v1/gateway runtime-mutable edge-policy plane
Make the embedded gateway role RUNTIME-CONFIGURABLE instead of baked into static
config: an operator/SuperAdmin retunes CORS, the per-IP flood cap, or a tenant's
rate ceiling via PUT /v1/gateway/config with NO redeploy — replacing the gateway's
image-baked KrakenD config.
clients/gatewaypolicy (leaf pkg, stdlib + hanzoai/sqlite only, no cloud import so
both the middleware and the HTTP subsystem share it cycle-free):
- Policy{CORSOrigins, PerIPRPM, WindowSec (platform), OrgRPM (per-org)} — every
field is enforced by a consumer; no stored-but-ignored knob.
- Store: one encrypted per-tenant SQLite (gateway.db), org-keyed rows. The admin
org row is the PLATFORM policy, layered over the static env/flag boot defaults.
Cached resolvers Platform()/OrgRPM()/Effective() (5s TTL, fail-open to static),
merge(base,over) makes a partial PUT additive. Fail-soft: a store-open error
degrades to static-only (reads work, writes error) — the edge never goes down.
clients/gatewaysvc: the /v1/gateway subsystem (order 139) — GET/PUT config over
the SAME store, IAM-gated like clients/pricing/enablement.go:
- platform fields (CORS/per-IP) writable ONLY by SuperAdmin (c.IsAdmin()); routed
to the platform row explicitly (PutPlatform) so an org-switched SuperAdmin still
lands on it.
- per-org OrgRPM writable by the org admin (own org via principal.Tenant, never a
raw header) or a SuperAdmin targeting ?org=<slug>.
Wiring: deps.GatewayPolicy (BuildDeps constructs it, layered over staticEdgePolicy;
serve.go closes it at shutdown). EdgeCORS/EdgeRateLimit now read the PLATFORM policy
LIVE (recompiling the CORS matcher only when the allowlist changes; the per-IP
limit/window per request). ScopeRateLimit gains the runtime per-org OrgRPM
override (most-restrictive-wins with the commerce-configured ceiling).
Tests: gatewaypolicy (static-only, platform layering, additive merge, per-org +
platform-default OrgRPM, persist-across-reopen); gatewaysvc (principal required,
org-self OrgRPM, org-admin platform 403, SuperAdmin platform, org-switched-still-
platform, empty-body 400). CGO_ENABLED=0 go build ./... + go test . green.
Mounts /v1/waitlist/* served in-process off the embedded hanzoai/base app over
the durable cloud PVC — the in-binary replacement for the standalone superbase
pod. STAGED + fail-closed: no-op unless CLOUD_BASE_EMBED=1. Registered as the
"base" subsystem (order 60) in clients/base.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The datastore connects ASYNCHRONOUSLY: ai/object.InitDatastore flips
DatastoreEnabled true only AFTER Mount returns. Mount ran the CREATE TABLE
DDL only when DatastoreEnabled() was already true, so in prod it was skipped
and never retried -> GET /v1/sbom/{ref} 502 'Unknown table expression
identifier hanzo.sbom_component' while /v1/sbom/health reported datastore:true.
Add a lazy, idempotent ensureTable(ctx) guarded by a mutex+bool that latches
ONLY success (a transient failure retries; sync.Once would cache the failure).
It CREATE DATABASE IF NOT EXISTS hanzo then CREATE TABLE IF NOT EXISTS, and is
called from ingest and resolve right after requireDatastore() passes; on error
they return a retryable 503. Mount now routes its best-effort boot DDL through
the same ensureTable and is non-fatal (a Mount-time miss no longer aborts the
subsystem).
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The live cloud console runs on <brand>.cloud hosts that route straight to the
cloud Service (console.lux.cloud, console.zoo.cloud, …). BrandForHostOK only
matched a brand's primary marketing Domain (lux.network, zoo.ngo), so a request
Host on lux.cloud/zoo.cloud fell through to the deployment brand — emitting Hanzo
branding on a Lux/Zoo surface (agent-skills catalogue + any Host-branded reply).
Add AltDomains per brand (lux→lux.cloud; zoo→zoo.network,zoo.cloud;
hanzo→hanzo.cloud,hanzo.app; pars→pars.ai) and match them in BrandForHostOK.
Base-URL/issuer scoping still uses the primary Domain.
Claude-Session: https://claude.ai/code/session_01CDooqWJiB7yNNaSjGQjdL7
Co-authored-by: hanzo-dev <dev@hanzo.ai>
New subsystem clients/agentskills serves the Agent Skills Discovery surface from
a catalog GENERATED by hanzoai/openapi's skills.py and embedded via go:embed:
GET /.well-known/agent-skills/index.json the brand's MASTER catalogue
GET /.well-known/agent-skills/<skill>/SKILL.md one skill document
WHITE-LABEL: the brand is decided per request from the Host (new BrandForHostOK,
mirroring platform.ts getWhiteLabelBrand) — api.hanzo.ai serves Hanzo,
api.lux.network serves Lux (lux.id), api.zoo.ngo serves Zoo — never Hanzo
branding on a Lux/Zoo surface. An unmatched Host degrades to the deployment brand
(CLOUD_BRAND), not blindly Hanzo. Order 8 registers these exact routes BEFORE
IAM's /.well-known/* wildcard (50) and the console catch-all, so they win Fiber's
first-match. Public, GET-only, no secrets.
The binary does not re-derive skills — it serves the embedded bytes, so the
sha256 digests in index.json match the served SKILL.md exactly. Only a tiny,
self-consistent `ai` fallback is committed (catalog/.gitignore); `make
agentskills` / the Dockerfile `skills` stage regenerate the FULL catalog (all 68
services × hanzo/lux/zoo) from the openapi SOT before `go build`, mirroring
webui/dist. End-to-end serve test drives the real router (index + SKILL.md +
white-label + digest + 404); cmd/cloud links clean.
Claude-Session: https://claude.ai/code/session_01CDooqWJiB7yNNaSjGQjdL7
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Add clients/sbom: a self-contained subsystem riding the ONE shared
ClickHouse client (ai/object.Datastore*) — no second connection — that
ingests CycloneDX SBOMs from CI and serves them by image digest/ref.
The store (hanzo.sbom_component, ReplacingMergeTree) is GLOBAL/cross-tenant
by design: an SBOM belongs to a content-addressed image digest, not a
tenant, so any tenant deploying that image resolves the same component set.
Ingest is gated to the canonical cloud super-admin check (c.IsAdmin(),
owner==AdminOrg) which the build fleet carries; resolve exposes only an
image's immutable bill-of-materials (no tenant data).
POST /v1/sbom ingest (super-admin/CI): flatten document.components[]
GET /v1/sbom/{ref} resolve by digest OR ref (FINAL dedupe, type,name order)
GET /v1/sbom/health liveness + datastore bool (not JWT-gated)
Registered id "sbom" order 137 with cloud.HealthOwner (binds before the ai
/v1/* catch-all at 150). Mirrors clients/analytics for structure, coercers,
and the honest-503 datastore gate.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Clean-semver pin bringing three ai releases into the cloud binary:
- v1.803.1 fix(account): cookie-session self-heal (#71) — a signed-in admin
whose beego session already holds a guest u-<hash> is rebound to the
canonical identity from hanzo_iam_token, so /v1/admin/* stops 403ing.
- v1.804.0 refactor(authz): ONE super-admin rule — membership in the `admin`
org (owner == AdminOrg); drops the configurable globalAdminOrgs + built-in.
Matches cloud's clients/admin (isSuperAdmin canonical, isGlobalAdmin alias)
and the console isSuperAdminAccount gate.
Supersedes the pseudo-version pin (#197) and the intermediate v1.803.1 pin
(#198, closed). Verified: go build -tags "libsqlite3 sqlite_fts5" ./cmd/cloud.
Claude-Session: https://claude.ai/code/session_01VZbTTNtf4y8y3XUr9wMqTX
Co-authored-by: hanzo-dev <dev@hanzo.ai>
* feat(platform,sites): pure-Go zip/tar.gz static-site upload + custom-domain serving
Adds a self-service static-site deploy to the unified cloud binary's PaaS
surface and lets the site edge serve a customer's own domain from S3.
- projects: walkArtifact accepts a ZIP (archive/zip) as well as tar(.gz),
sniffed by magic bytes; one deploy contract (index.html at root, same size
and traversal guards), three container formats. A single wrapping top-level
directory (a zip made from a project folder) is stripped so index.html lands
at the root.
- projects: the deploy handler reads the artifact from a multipart file upload
(a browser <input type=file>) OR the raw request body (a curl one-liner).
- sites: the host edge now serves a bound CUSTOM domain (a customer apex/host
pointed at this edge) from that project's S3 prefix, resolved by the full
host. Only external hosts (never one of our self domains) with a LIVE binding
are served; every other host — our api/console hosts, or an unbound host
routed here — Continues to the normal pipeline, so the API path pays no
per-request lookup and a customer binding can never shadow a real Hanzo host.
- projects: POST/GET .../domains binds and lists a site's custom domains
(admin-gated until DNS-ownership verification is wired here).
- surface: the static engine is exposed under /v1/platform/sites/* (the PaaS
namespace) in addition to /v1/projects/*, so the one user flow is create a
site -> upload a zip -> bind a domain -> live.
Pure Go, CGO-off. New unit tests cover the zip walker, format dispatch,
single-root strip, custom-domain routing (served/passthrough/self-host/not
-live), hostname validation, and host binding.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(projects): authorize custom-domain binding by the platform-operator org
A custom-domain bind is authorized for a global admin OR the platform-operator
org (the deployment's brand org, env CLOUD_PLATFORM_OPERATOR_ORGS, default the
brand). The operator manages customer DNS until per-tenant DNS-ownership
verification is wired here. Safe because a bound domain is inert until its owner
points DNS at this edge — the real gate is DNS control.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* debrand: signoz -> o11y (branding) + repoint collector module
Drop "SigNoz"/"signoz" where it is BRANDING (comments, docs, prose) to o11y,
and repoint cloud's direct collector import to the renamed module.
- go.mod/go.sum: github.com/hanzoai/signoz-otel-collector v0.144.6 (direct)
-> github.com/hanzoai/otel-collector v0.144.7 (direct). The old module stays
as an // indirect dep because hanzoai/o11y v1.5.2 (separate repo, out of
scope) still imports it; the fork also pulls upstream
github.com/SigNoz/signoz-otel-collector v0.144.5 // indirect.
- clients/o11y/ingest.go: chlogs/chtraces imports -> otel-collector.
- Branding prose swapped to o11y in telemetry.go, subsystems.go, embed.go,
logs.go, metricsread.go, scope.go, ingest_test.go, agents.go, LLM.md,
docs/consolidation.md.
KEPT (not branding):
- ClickHouse schema read by cloud (written by the deployed collector):
signoz_traces / signoz_logs / distributed_signoz_index_v3, severity_text
columns. Renaming reads without migrating the live schema breaks them; the
v0.144.6->.7 patch bump does not migrate table names.
- Upstream API in hanzoai/o11y/pkg/signoz: import path, alias, type SigNoz,
and signoz.New / SigNoz.Start references (that repo is debranded separately).
- Upstream package names: signozclickhousemetrics.
- Honest attribution: "SigNoz's dd-sketch fork of ch-go".
Build: go build ./... = 0, go vet = 0, go test ./clients/o11y ./clients/admin
= ok. go mod tidy is blocked by a pre-existing luxfi/keys@v1.2.0 go.sum
checksum mismatch (identical on origin/main) -> CI-authoritative.
* o11y: read o11y_* ClickHouse tables + bump collector to v0.144.8
Direct ClickHouse table reads renamed signoz_* -> o11y_* to match the
o11y read plane and the data-preserving RENAME migration (hanzoai/o11y#28):
o11y_traces.distributed_o11y_index_v3 (was signoz_traces.distributed_signoz_index_v3)
o11y_logs.distributed_logs_v2 (was signoz_logs.distributed_logs_v2)
Files: clients/o11y/logs.go, clients/o11y/metricsread.go,
clients/o11y/ingest.go, clients/admin/o11y.go (+ o11y_test.go).
Bump github.com/hanzoai/otel-collector v0.144.7 -> v0.144.8 (writer side
now CREATEs/WRITEs the same o11y_* physical schema). Collector go.mod is
unchanged between the two tags (identical go.mod hash) — pure source
rename, so the module graph is unchanged; go mod tidy left to CI
(pre-existing luxfi/keys tidy block is CI-authoritative).
go build ./... = 0. clients/admin + clients/o11y tests green (SQL
assertions now match o11y_* target names). Lockstep deploy: collector
v0.144.8 -> o11y#28 RENAME migration -> o11y+cloud readers.
* cloud: embed o11y v1.5.4 — version-less /v1/o11y + o11y_ schema reads + debrand
Bumps hanzoai/o11y v1.5.2→v1.5.4 (version-less surface + o11y_ ClickHouse table
reads + the signoz→o11y debrand) and repoints embed.go to the renamed runtime
package pkg/signoz→pkg/o11y (type SigNoz→O11y). Pairs with otel-collector v0.144.8
(writes o11y_) + the lockstep cutover migration. go build ./... = 0.
---------
Co-authored-by: hanzo <z@hanzo.ai>
Embeds hanzoai/o11y#26: the mount normalizes the /v1/o11y/<resource> public
contract onto the internal SigNoz /api/vN routes (kills the /api/ leak, fixes
the llmobs /v1/o11y/* 404). Pairs with the cloud CR O11Y_GLOBAL_EXTERNAL__URL=""
change (universe#461) — deploy together.
Co-authored-by: hanzo <z@hanzo.ai>
The ai (casibase) layer SETS the httpOnly hanzo_iam_token cookie (the IAM JWT) after
login (ai/controllers/account.go iamTokenCookieName), but cloud's own identity
middleware only read [iam_access_token, access_token, hanzo_token] — NOT
hanzo_iam_token. So the embedded console (browser holds ONLY that cookie, no
Authorization header) resolved to no validated principal → every org-scoped /v1
endpoint (agents, gpus, machines, platform, orgs, entitlements, …) 403'd
'X-Org-Id required', and modules rendered empty. Add hanzo_iam_token (first) to
cookieTokenNames so cloud reads the SAME cookie the ai layer sets → validates the
JWT → X-Org-Id from owner → org-scoped surfaces authorize. Verified: the JWT is
present in the browser (1533-char httpOnly hanzo_iam_token); only the name was wrong.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pulls the get-account cookie-path fix (hanzoai/ai#81): a signed-in admin
whose beego session already holds an anonymous guest u-<hash> is now
self-healed from the hanzo_iam_token credential to its canonical identity,
so /v1/admin/* stops 403ing under the console cookie session. Verified:
cloud binary builds with -tags "libsqlite3 sqlite_fts5".
Claude-Session: https://claude.ai/code/session_01VZbTTNtf4y8y3XUr9wMqTX
Co-authored-by: hanzo-dev <dev@hanzo.ai>
internal/org/shared.go split its aliases along the real seam: election
(Member/Owner/IsOwner/Replicas) now re-exports github.com/hanzoai/ha; the
Replicator/Store/DB/DBPath stay github.com/hanzoai/vfs/replica. membership.go
and cipher.go are unchanged (the alias types line up: org.Member = ha.Member).
writefence doc updated to name ha as the election primitive.
One concern, one home: who-writes (ha) vs how-state-ships (vfs). No behavior
change; internal/org + writefence pass with -race.
NOTE: `go mod tidy` is blocked in this repo by a PRE-EXISTING, unrelated
luxfi/keys@v1.2.0 go.sum checksum mismatch; ha was added via `go get` +
marked direct by hand. Re-run tidy once that pin is fixed.
Co-authored-by: hanzo <z@hanzo.ai>
Per RED's conditional GO on the encryption image:
- GOFLAGS -mod=mod -> -mod=readonly: the committed go.sum is the SOLE source
of truth; any needed-hash drift FAILS the build instead of silently
re-recording an unverified hash. Verified go.sum is complete + sumdb-
consistent (go build/download clean with GOSUMDB ON).
- Drop GOSUMDB=off: a money image must not blanket-disable the checksum
database. GONOSUMDB scoped to zap-proto/* only (first-party-direct).
- Digest-pin the three base images (node:24-alpine, golang:1.26-alpine3.22,
alpine:3.22) @sha256 for a reproducible money image.
The ldd/readelf link proof stays belt-and-suspenders behind the ciphertext
proof (verified at build time on the musl image).
The limiter guards the OFF-GATEWAY path where nothing trusted stamps
X-Forwarded-For, so keying on the client-settable XFF let an attacker send a fresh
value per request and reset the 30/min bucket at will. These are all post-auth
money-write routes, so key on the un-spoofable VALIDATED principal (X-Org-Id/
X-User-Id, minted by SanitizeIdentity from a verified JWT); fall back to the socket
peer (L4 RemoteAddr) for an unauthenticated request (which the handler 403s anyway).
Test now rotates XFF during the flood (must NOT reset) and asserts a second principal
keeps its own bucket.
Cloud-direct/off-gateway money path loses the edge WAF/limiter and the embed
session-bridge's Sec-Fetch-Site gate passes VACUOUSLY when Origin/Referer/SFS are
all absent (RED). Adds two positive controls to the console write surface:
CSRF (csrf.go) — GET /v1/console/csrf issues a token bound to the validated
principal (X-User-Id+X-Org-Id), MAC'd with keyed BLAKE3 (luxfi/crypto,
blake3.KeyedHash) under a server-only KMS key (CONSOLE_CSRF_KEY; ephemeral
per-process fallback). requireCSRF enforces X-CSRF-Token on the AMBIENT-cookie path
ONLY (no Authorization + a Cookie present) — Bearer/Basic/gateway/API callers are
immune to CSRF and skip it, so nothing non-browser breaks. A cross-site page cannot
read the same-origin token (SOP) nor set the custom header (no CORS preflight
granted), and the token is identity-bound so it can't be replayed as another user.
Rate limit (ratelimit.go) — per-IP token bucket (30/min) on mint/rotate/revoke key
+ wallet top-up; distinct from commerce's spend-cap, restores frequency protection
lost off-gateway. Keyed on XFF first-hop.
Wraps POST/DELETE keys, POST onboard, POST topup, POST billing, POST/PUT/PATCH/DELETE
commerce. Reads stay open. Tests: ambient-no-token 403, valid-token 200,
cross-identity replay 403, Bearer-skips-CSRF 200, rate-limit 429; existing suites
unchanged (no cookie ⇒ CSRF skipped). luxfi/crypto for the MAC (NOT stdlib/JWT).
Needs arcd build + RED review; CONSOLE_CSRF_KEY to be provisioned via KMS for
restart/multi-replica-stable tokens (coordinate ac742480).
The in-binary direct-Bearer path (console SPA -> cloud, gateway bypassed) stamps
X-User-Id = the JWT subject (a UUID) via idClaims.userID(). The console key ops
built the IAM id as <owner>/<X-User-Id> = hanzo/<uuid>, but IAM's mint-user-keys /
get-user resolve only <owner>/<name> (hanzo/z) -> 'password or code is incorrect'
-> hk- mint 502 on the cloud-direct path. (The gateway path worked because the
gateway minted X-User-Id == username.)
Fix (narrow blast radius, per RED-preferred approach — does NOT reorder userID()):
- idClaims.username(): the IAM username (name claim, then preferred_username),
NEVER the subject.
- SanitizeIdentity stamps X-User-Name from the validated username, DISTINCT from
X-User-Id. X-User-Name is already an authorityHeader (stripped on ingress,
re-injected only from validated claims -> forgery-proof).
- resolveCaller carries a distinct caller.username (X-User-Name, fallback to
X-User-Id for the gateway path); new caller.keyID() = <owner>/<username> is used
ONLY by the user-key ops. caller.id / caller.name are UNCHANGED, so the
billing/topup/commerce subjects are byte-identical -> zero money-path impact.
Tests: TestKeys_DirectBearerPath_MintsByUsernameNotUUID (mint targets hanzo/z, not
the UUID), TestSanitizeIdentity_StampsUserName, TestSanitizeIdentity_UserNameForgeryStripped;
existing key + identity suites unchanged (gateway path falls back to owner/name).
Needs arcd/CI image build (no local docker) + RED review before deploy.
The unified binary embeds IAM's per-org SQLCipher store and commerce's
per-tenant money DBs; the prior CGO_ENABLED=0 build shipped pure-Go
modernc — PLAINTEXT at rest. Rebuild CGO=1 against system libsqlcipher
(hanzoai/iam's proven recipe: libsqlite3 tag + libsqlcipher symlink +
-DSQLITE_HAS_CODEC), runtime base scratch -> alpine:3.22 + sqlcipher-libs
(CGO needs libc + the codec .so).
Baked-in RED gates (a failing gate = NO image):
- modernc double-registration guard: 0 modernc in the CGO=1 ./cmd/cloud
graph (the one 'sqlite' driver is mattn/SQLCipher).
- TestEncryptionProof: real ciphertext-at-rest under SQLITE_REQUIRE_CODEC=1.
- cek.go golden-vector KAT (TestUnwrapGoldenFixture + round-trip): a frozen
pre-luxfi-swap 61-byte DEK sidecar still decrypts under the shipped
luxfi/crypto-AEAD code — existing encrypted stores stay readable.
- readelf/ldd link proof: the binary binds sqlite3_* to libsqlcipher, never
a plaintext libsqlite3.
Console embed stage unchanged (same-origin console). RED must review before
the image ships.
Bump the six hanzo modules to their driver-converged releases so the CGO=1
unified binary has EXACTLY ONE database/sql 'sqlite' registration
(mattn/SQLCipher), ending the 'sql: Register called twice for driver
sqlite' panic:
sqlite v0.1.5 -> v0.2.3 (SetPersistWAL + OpenPragma primitives)
orm v0.5.2 -> v0.6.1
base v1.4.6 -> v1.5.7 (+ replicate v0.9.5, the last modernc leak)
commerce v1.42.29 -> v1.46.40 (+ go:embed plans fix)
o11y (pseudo) -> v1.5.1
replicate v0.8.0 -> v0.9.5
Retarget the mattn v2.0.3+incompatible replace v1.14.16 -> v1.14.47 (the
SetFileControlInt/SQLITE_FCNTL_PERSIST_WAL-capable version hanzoai/sqlite
v0.2.3 needs for SetPersistWAL).
Verified CGO=1: 0 modernc in the ./cmd/cloud dep graph; the 517MB binary
builds and boots (--help) with NO double-register panic.
studio.render now (1) writes any inputs shipped with the job into the
local studio input dir via its own /upload/image, so an uploaded photo
(which lives in orgs/{org}/input on the dispatching pod, unreadable here)
resolves for LoadImage before the render; and (2) forwards the job's
active org as the studio_active_org cookie on /upload/output, so the
finished render lands in that org's gallery even when the worker token's
home org differs (a@hanzo.ai home=hanzo, rendering for karma).
SuperAdmin: /v1/admin/me and /v1/admin/users now emit the canonical
`isSuperAdmin` key alongside the deprecated back-compat alias `isGlobalAdmin`
(both populated with the SAME fact — owner == AdminOrg). The console may read
either during the rename migration and sees the same truth. No DB change: the
signal was always a derived boolean, never a stored column.
Entitlements: new clients/entitlements subsystem (order 139) — the per-org
product-enablement plane the console's paid-product sidebar reads.
GET /v1/orgs/:org/entitlements -> { "enabled": [...] }
POST /v1/orgs/:org/entitlements { add?, remove? } -> { "enabled": [...] }
Two authorities, never braided: ENABLEMENT (this store: durable per-tenant
SQLite, (org,product) key, settings-store discipline) vs ENTITLEMENT (commerce:
deps.Commerce.CheckEntitlement at write time). A non-super-admin may only enable
a product the org's plan already grants (402 otherwise); disabling is never
gated; a super admin bypasses the commerce gate and may target any :org. Org
scoping mirrors clients/kms: :org must equal the validated owner claim unless the
caller is a super admin; a bearer-less forge fails the principal gate (403).
Tests (TDD, all green): store tenant-isolation + all-or-nothing Apply;
forged-request 403; cross-org 403; malformed org/product 400 (commerce not
consulted); entitled enable 200; unentitled enable 402 (nothing persisted);
super-admin bypass 200 (commerce not consulted); nil-commerce member-add 503
(fail-closed); remove never gated; empty mutation 400. Plus admin_test asserts
isSuperAdmin present and equal to isGlobalAdmin on both /me and /users.
Bumps @hanzo/plans to the World-pricing catalog (world-enterprise tier +
world.model_api gate) and adds the single-sourced enforcement contract for
the /v1/world data plane.
- clients/plan: export Entitlements(ctx, id) — the one Go seam to read a
plan's canonical entitlement block from the @hanzo/plans catalog (runs the
bundle 'entitlements' route; no data duplication, no fromLegacy re-impl).
- clients/world/entitlement.go: WorldLimits + WorldLimitsFromEntitlements
(pure) + ResolveWorldLimits(ctx, planID) — values sourced from world.*
entitlements, never hardcoded. FreeWorldLimits is the fail-closed floor
(catalog outage degrades to Free, never grants model/stream).
- GET /v1/world/limits?plan=<id>: machine-readable contract echo so agents/
dashboard self-config against the live catalog instead of hardcoding tiers.
- Tests: contract mapping (all tiers), fail-closed on unmounted catalog, and
end-to-end Entitlements against the real embedded bundle (world.model_api
present on pro/enterprise, absent on free).
Per-request enforcement (org->plan resolution + rate limiter wiring) is the
documented follow-up owned with feat/world-model-engine; both gates resolve
through ResolveWorldLimits so policy stays single-sourced.
Proxy capture of cloud->ring proved the Safe flow hits the ring's commit-after-
response read-after-write race TWICE, not once: createVault->createWallet ('vault
not found', already retried) AND createWallet->deploy ('wallet not found', which
502'd custody=safe). With ALL requests pinned to one node (via a debug proxy) the
deploy STILL 404'd, so it is a Postgres commit-visibility lag, not node affinity.
Extract doRetryNotFound(...notFound) (bounded 6x/250ms linear, ctx-aware, fail-fast
on any other error; do() only unmarshals on 2xx so out is safe across retries) and
use it for BOTH createWallet ('vault not found') and deploySafe ('wallet not
found'). go test ./clients/wallets/... green; cmd/cloud builds.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The embedded KMS write path POST /v1/kms/orgs/{org}/secrets defaulted a
missing env to "default" (envOr), committing the write to a bucket that
project/env/path readers (the kms-operator, cluster syncs) never resolve.
That split is what let an IAM z-password land in env=default while prod kept
serving the stale value. env is a first-class component of the storage key
(kms/secrets/{path}/{env}/{name}) and cannot be aliased, so a write with no
env now fails loud (400). GET/DELETE/LIST keep the envOr compat default (a
read/delete can't plant a value another reader trusts; legacy readers that
omit env must keep working). No PATCH route exists on this surface.
Regression tests: write without env -> 400 (and lands nowhere); write
env=prod is readable via the operator's project/env/path resolution (sha256
round-trip, values never printed) and is not visible in env=default. The
fail-closed-without-master-key test now sends a valid env so it still
exercises the 503 master-key gate rather than 400-ing on input.
Claude-Session: https://claude.ai/code/session_01D4FSvT3UfhrFJNQjrctjEj
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Completes the #71 auth repair as a clean dep bump (the fix lives in ai + iam, not
the cloud tree):
- github.com/hanzoai/ai v1.802.0 -> v1.802.1-0.20260708185316-0321c35877f0
(ai#79 0321c358: self-heal get-account identity — stop degrading real logins to
u-<hash> guests; fail-closed 401). Pseudo-version pins the commit while the
semantic-release patch tag mints (1 commit ahead of v1.802.0).
- github.com/hanzoai/iam v1.31.18 already pinned in main (iam#109 fail-closed
guest-mint gate) — MVS keeps it over ai's older iam pin.
go mod tidy added the authentic gopsutil/v4 transitive hashes (iam util); go mod
verify OK; -mod=readonly CGO_ENABLED=0 go build ./cmd/cloud green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Makes console.hanzo.ai (go:embed console) authenticate its money surfaces: the
first-party IAM session cookie → validated principal (sessionAccessToken → v.validate),
RED-hardened (H2 Secure cookie, H3 same-origin bridge gate), on iam v1.31.18 (H1
session-regeneration + iam-main security fixes). Pairs with console v8.4.122 which
addresses billing/commerce/keys at the canonical bare /v1 in embed mode.
v1.31.18 is iam main (guard-leak stamp #108, guest-signin fail-closed #109, capauth
PermAttenuate #106) UNION the InitEmbed line UNION RED H1 (SessionRegenerateID on the
sign-in transition). The prior embed pin v1.31.17 had diverged off an old base and was
MISSING those iam-main security fixes, so pinning v1.31.17+H1 would have shipped the
money embed without them. v1.31.18 ships H1 + guard-leak/guest/capauth + InitEmbed
atomically into this binary's in-process IAM. Transitive indirect bumps (purego/
plan9stats/locafero/gopsutil-v4/viper/tidwall-match) are MVS-driven by iam v1.31.18.
H2 (HIGH) — pin the IAM session cookie Secure. clients/iamsvc/iamsvc.go derived
Secure from web.BConfig.Listen.EnableHTTPS, which is FALSE (the binary listens plain
:8000 behind the TLS-terminating ingress) → the session cookie shipped non-Secure.
The embed bridge turns that opaque sid into a money bearer (hk- mint, balance/top-up),
so a non-Secure cookie is capturable off any plaintext leg and replayable. Pinned
Secure: true (the deployed edge is always HTTPS).
H3 (MED-HIGH) — gate the ambient-cookie bridge to same-origin. billing.go/commerce.go
forward GET verbatim to commerce; a SameSite=Lax cookie still rides a top-level GET, so
a cross-site link could drive the victim's own money action if any commerce GET mutates.
validatedPrincipal now fires the session bridge ONLY for a same-origin request
(sessionBridgeSameOrigin: Sec-Fetch-Site same-origin|none, else Origin/Referer
host==Host) — refusing cross-site AND sibling-subdomain (same-site). Bearer/JWT-cookie
paths (non-ambient) are unaffected. +TestSessionBridgeSameOrigin (7 cases) green.
REMAINING for money: H1 (session-fixation — SessionRegenerateID on the IAM sign-in
transition) lands in hanzoai/iam (compiled into this binary); coordinating.
The go:embed console (console.hanzo.ai → cloud:8000) authenticates against the
in-process IAM, which sets an OPAQUE, httpOnly session cookie (cloud_session_id)
and stores the user's IAM-minted access-token JWT SERVER-SIDE against that session.
The console's Next BFF token-minting routes are stripped by the static export, so a
browser request to a cloud-native route (/v1/console/keys, /v1/billing/*) carries
only the session cookie — no bearer — and validatedPrincipal refused it, 401ing
every authenticated surface (API keys, billing, every product page = shell).
validatedPrincipal now resolves that session cookie to the server-stored access
token (sessionAccessToken via web.GlobalSessions) as a LAST RESORT (after Bearer/
Basic/JWT-cookie), then validates it through the SAME v.validate (sig/iss/aud/exp).
Identity is bound to the VALIDATED session: the client holds only an unguessable,
httpOnly sid; the session never asserts identity itself. No-op on gateway-fronted
binaries (a bearer is present) and on binaries with no IAM session manager
(web.GlobalSessions == nil) — tested. CSRF: cloud_session_id is SameSite=Lax, so a
cross-site request never carries it; and cookieTokenNames already establishes cloud's
JWT-cookie auth posture. This is the v8.4.5-flagged 'set the cookie the sanitizer
looks for' path, done cloud-side from the session store (no cross-repo IAM release).
RED review requested before it fronts money (session-fixation / CSRF surface).
topupConfig defaulted to the placeholder 36900; align to the
genesis-canonical Hanzo mainnet chain id 36963 (lux/genesis, and the
rest of cloud clients/treasury+wallets already use 36963). Still
env-overridable via HANZO_CHAIN_ID.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The observability surface was mounted THREE ways over the same /v1/o11y/* paths:
clients/observe (order 44), clients/o11y's o11yscope (order 69), and the
hanzoai/o11y wildcard runtime (70/71). observe also served /v1/settings/:product,
which is console product config, not observability. Collapse to one and one way.
- ONE owner of /v1/o11y/{logs,metrics,status}: clients/o11y's o11yscope (order 69).
observe's richer logic is folded IN so nothing is lost — the REAL per-org RED
metrics + LLM usage (metricsread.go, was a stub in o11y) and the two-view logs
(admin infra stdout / per-org request-from-traces). Tenant isolation preserved:
org is principal.Tenant bound as a positional ClickHouse param, the product is
shape-validated → alias-mapped (console slug → workload) → allowlisted
(knownServices, SSRF/injection boundary). observe's productAlias merged into
resolveService so no product loses data. Admin god-view gates on c.IsAdmin()
(== owner=="admin" SuperAdmin after SanitizeIdentity), never a per-org isAdmin.
- /v1/settings/:product moved OUT of observe into clients/settings (it is NOT
observability). Behavior/contract preserved verbatim from observe: {config,
secretKeys} shape, KMS ref orgs/{org}/settings/{product}/{key}, (org,product)
store isolation, secrets-to-KMS-or-fail-closed. Replaces the orphaned, divergent
clients/settings stub with the live behavior and wires it in (order 138).
- /v1/query does not exist (no registrant, no consumer) — nothing to fold.
/v1/observe/health was the auto-derived GET /v1/<id>/health for id "observe";
it vanishes with the subsystem (o11yscope gets /v1/o11yscope/health; the runtime
serves its own gate-exempt /v1/o11y/api/v*/health*). Both documented.
- DELETE clients/observe; drop its import; add clients/settings; fix the stale
subsystems.go o11y comment ("reverse proxy to the dedicated o11y Deployment" →
the embedded reality: scoped reads 69 + in-process runtime 71 + OTLP ingest 72).
Net -1037 LoC. cmd/cloud + cmd/hanzo build; clients/o11y + clients/settings tests
pass (20/20), covering tenant isolation, secrets-never-plaintext, product
validation, alias resolution, and route precedence over the wildcard proxy.
One coherent change to the subsystem-registration layer. Concrete types over
`any` at the call sites, indirection deleted, generics only where they remove
real duplication.
CHANGE 1 — kill the per-subsystem `any`-unwrap boilerplate
Every in-repo subsystem's init() hand-wrote the identical
func(app any, deps cloud.Deps) error { a, ok := app.(*zip.App); if !ok {…}; return Mount(a, deps) }
Add ONE adapter, cloud.Typed(func(*zip.App, Deps) error) MountFunc, that does
the *zip.App recovery in a single place (fail-closed, never panics). All ~50
subsystems collapse to `cloud.Register("x", n, cloud.Typed(Mount))` /
`cloud.RegisterWithShutdown(..., cloud.Typed(Mount), Shutdown)`. Redundant
shutdown wrappers dropped where Shutdown already matches ShutdownFunc; kept
only where a no-arg Shutdown() needs signature adaptation.
MountFunc's param STAYS `any` on purpose: the pinned external subsystem modules
(hanzoai/ai, authz, base, commerce, metrics, o11y, licensing) register with
`func(any,…)`, and a `func(any,…)` literal is not assignable to a
`func(*zip.App,…)` parameter — retyping MountFunc would break those modules at
compile time. The assertion is now central, not per-subsystem. MountAll takes
the concrete *zip.App (threaded from Serve).
CHANGE 2 — OwnsHealth flag replaces the "<name>svc" health kludge
Some subsystems serve their OWN fail-closed /v1/<name>/health; the generic
always-ok liveness route in Serve would shadow it. The old fix encoded routing
policy in the id ("kmssvc" parked the generic route at an unrouted path). Now
Register/RegisterWithShutdown take `opts ...Option`; cloud.HealthOwner sets
MountSpec.OwnsHealth, and Serve's generic-health loop skips a HealthOwner. The
id is once again the clean route name. Invariant now uniform and checkable:
a subsystem serves /v1/<name>/health IFF it registers cloud.HealthOwner.
Migrated every health-owner to it: kms, paas, s3 (named in scope) plus
analytics, console, platform, ml (same kludge) and notify, plans, pricing,
security (had coincidental id==route; security's real probe reports a rule
count the generic route was silently dropping). pickKMSClient gate + all
tests + stale comments updated from the "kmssvc"/"s3svc"/… ids to kms/s3/….
CHANGE 3 — clean package renames (no collision)
clients/paassvc → clients/paas, clients/projectsvc → clients/projects
(package decls, filenames, the sole importer, error strings, userAgent, and
doc refs repo-wide). clients/kmssvc + clients/tasksvc KEEP their package names
— the `svc` disambiguates the subsystem from the same-named library it imports
(clients/kms, hanzoai/tasks); their ids are already clean (kms via CHANGE 2,
tasks).
CHANGE 4 — generic pick[T]
The five identical co-resident-or-RPC-or-disabled resolvers (IAM, Base,
Commerce, O11y, MQ) collapse into one
pick[T](cfg, log, name, label, zapAddr, rpc func(string)T, disabled func()T) T.
KMS/AI/VFS/Payments/Vault keep bespoke pickers — their construction genuinely
differs (embedded store / gateway preference / S3-admin backend / never
co-resident), so they are left alone.
Verified: CGO_ENABLED=0 go build ./cmd/hanzo/ and ./cmd/cloud/ both exit 0;
go vet clean on every changed package; `hanzo --help` lists kms/paas/projects/
s3/tasks svc-free; cloud root + renamed + health-owner package tests pass; new
build_registration_test.go covers Typed + HealthOwner. Net −199 lines.
Picks up the increment-2 crypto hygiene: the PartyID<=ValidatorSetSize DoS
bound on the quasar/pulsar Finalize path (Item7a) + the structural-Verify lock
(Item7b). v1.35.32 corrects a 1-based off-by-one in v1.35.31 that rejected the
Nth validator; verified the controlplane N=7 ceremony finalizes under -race.
LOW severity (ingestLeg bounds PartyID upstream) but the fix is now live-pinned.
Add clients/ingress — an embedded edge plane in the cloud binary so the ONE
binary can BE the fleet edge: terminate TLS, run ACME, and reverse-proxy by Host
to upstreams, configured LIVE over /v1/ingress with no static routes.yaml and no
restart to change a route (hot-apply via an atomic engine snapshot swap).
Control plane (zip): /v1/ingress/{routes,services,middlewares,tls,status},
SuperAdmin-gated, per-tenant SQLite persistence, route Host globally unique;
every mutation reloads the engine.
Data plane (net/http): :80 (ACME HTTP-01 + router) and :443 (SNI TLS termination
via x/crypto/acme/autocert + router). Started only in edge role
(CLOUD_INGRESS_EDGE_ENABLED); app role keeps the listeners off — role = runtime
config, one binary.
Proxy: github.com/vulcand/oxy/v2 (Traefik lineage) weighted round-robin, the
Traefik router->service->middleware model. Middlewares: redirectScheme,
stripPrefix, addPrefix, headers.
STAGED subsystem (config.stagedSubsystems): linked but mounts ONLY when named in
CLOUD_ENABLE, so prod is untouched. Orthogonal to /v1/gateway (auth/rate-limit).
Build: CGO_ENABLED=0 go build ./... green; go test ./clients/ingress green (11 tests).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Red-team findings on the write-fence primitive:
- HIGH: mirrorKey(plugin,shard) = "writefence/"+plugin+"/"+shard was not
injective — mirrorKey("kms/tenant-a","secrets") collided with
mirrorKey("kms","tenant-a/secrets"), so a push framed as one shard could
overwrite another's epoch/writer/payload. Real shard scopes carry '/'
(vfs replica.DBPath yields "projects/site"), so it is reachable. Fixed with
a %d:-length-prefixed key; TestMirrorKeyInjective_NoCrossShardAliasing locks
it (was red's failing PoC, now green).
- LOW: MinioConditionalStore.Get did StatObject+GetObject (two round trips);
tightened to one GET whose obj.Stat() ETag is consistent with the read bytes
— closes the window rather than relying on the CAS to absorb a stale version.
Core CAS/epoch soundness unchanged (red GO: N=64 same-epoch race → one winner,
retry-bounded, strict-> rejects epoch==recorded). Still shadow-only, unwired.
go test -race -count=200 green.
Red-pass finding: the check-#1 grep char class [A-Za-z0-9_, ] missed the
build-constraint negation form `-tags '!x,controlplane'` (the `!` broke the
match before reaching controlplane). Add `!` to the class so it is caught.
Verified the deeper guarantees hold (evasion-agnostic), so this is belt-only:
- ZERO non-test importers of clients/controlplane (grep-confirmed).
- The package has ZERO untagged files, so importing it into serve code fails
the untagged `go build ./...` — check #3 catches ANY -tags syntax, incl.
GOFLAGS=-tags=controlplane (verified: the pkg becomes buildable => check #3's
'matched no packages' assertion fails => CI red).
Runtime asserts + external-cert selfComposedCert seam confirmed wired through
guarded constructors. No stub-crypto path reaches a serve binary.
Self-review finding: containment.go's runtime guard trusts testing.Testing(),
which is backed by a linker-set string var (testing.testBinary, set by `go
test` itself per cmd/go/internal/load/test.go). Confirmed locally that
`go build/run -ldflags="-X testing.testBinary=1"` spoofs it to true in a REAL
(non go-test) binary — verified with a throwaway program before writing this.
containment.yml's grep step now also fails the build on any reference to
`testing.testBinary` outside the Go toolchain itself, so a build path that
tried to ship that spoof gets caught the same way a `-tags controlplane`
build path does. Documented as a known residual in the workflow's header:
this is a mitigation (CI catches it), not a cryptographic close — that needs
increment-2's real signing, tracked in doc.go.
Also fixed the exclusion patterns to be grep-implementation-agnostic (some
recursive greps don't prefix paths with "./"), verified against a planted
violation for both checks.
Closes the same-epoch double-write on the HIP-0107 data-plane push path
(github.com/hanzoai/vfs/replica, wired in internal/org): today the only
admission checks are replica.IsOwner (a pure local computation over a
possibly-stale membership view) and the StatefulSet Recreate deployment
shape (role.Role) — both comment-only, non-atomic, and the underlying
Store/Backend.Put is an unconditional overwrite ("Overwriting is allowed").
A deposed/partitioned writer and a freshly-elected one can both push.
internal/writefence/fence.go adds Fence.Push: a single atomic
read-check-CAS that (1) rejects any candidateEpoch <= the epoch currently
recorded for the shard (strict >, closing the same-epoch case) and (2)
performs the epoch-advance and payload append in ONE conditional write
against the store's live version token, so two racing writers cannot both
land — the store is the sole arbiter, never an in-memory cache. Retries
once on a lost CAS race, re-checking strict monotonicity against the new
state, so a same-epoch racer's retry fails ErrStaleEpoch rather than
silently duplicating the admit.
EpochSource is the pluggable seam clients/controlplane's lease epoch drops
into once it graduates from shadow (Stage 1 today) — this package imports
nothing from controlplane. ConditionalStore models the S3 If-Match / GCS
generation-match primitive; store.go backs it for real with minio-go's
native SetMatchETag/SetMatchETagExcept (already vendored at v7.0.100, no
go.mod bump). fake_test.go models the same semantics in-process with a
barrier hook that deterministically reproduces the concurrent-CAS race.
Tests prove: strict-epoch rejection of a same-epoch retry (same and
different writer), the raw CAS rejecting a race loser, the full
concurrent-Push race resolving to exactly one winner, a legitimately
higher epoch being admitted, a stale lower epoch being rejected, and
per-shard scoping. Not yet wired into the live push path (that remains
gated by controlplane's shadow flag per HIP-0116); this is the fence
primitive plus a precise wiring recommendation for hanzoai/vfs's block
layer, which currently exposes no conditional-write capability to adopt.
Stage-1 ceremony's crypto is stub/forgeable by design (doc.go); this closes
the drift risks doc.go's increment-2 worklist flagged:
- .github/workflows/containment.yml (PR-gated): greps every build/release
surface in the repo for `-tags controlplane` and fails the build if found,
plus a positive proof that `go build ./...` links clients/controlplane into
no cmd/ main and that the package still matches zero packages with no tag.
- containment.go: mustHarnessOnly fail-closed panics the moment this
package's stub crypto is touched (package-import-time for the
PartialZVerifier registration, construction-time for NewSigner/
NewStubComposer) unless ProductionBCCSigningReady() (hardcoded false) or
testing.Testing() (the Go toolchain's own go-test signal, unspoofable by a
real build) holds. Proven end-to-end via a real subprocess
(TestContainment_NonHarnessProcessRefuses), not just in-process logic.
- selfComposedCert typed seam (driver.go/signer.go): CertComposer.Compose now
returns an unexported wrapper only it can produce; verifyOwnCertStructure
accepts only that type, never a bare *quasar.QuasarCert. An externally-
received cert has no way to become one, so it cannot reach the structural
check even by mistake. VerifyExternalCert is the sole seam for such a cert
and fails closed (increment-2 crypto not implemented). Locked from a
black-box vantage in external_cert_test.go.
Containment verified unchanged: `go build ./clients/controlplane/...` (no
tag) still matches zero packages; `go build ./...` still links no cmd/ main
to the package; full `-tags controlplane -race` suite green, no test weakened.
The cloud-embedded console (console.hanzo.ai + team) is built from hanzoai/console
build:embed. hanzoai/console now ships <HanzoAnalytics/> (env-gated on
NEXT_PUBLIC_ANALYTICS_WEBSITE_ID). Default it to the console.hanzo.ai property
(7dce54ee-41f6-4751-96bf-fe005067c7c7, public per-site) in the console build stage
so the one native analytics tag renders on the next cloud build. GA4/Pixel off.
The luxfi/mpc threshold signer returns a NON-canonical r|s: s is frequently in the
upper half (s > N/2). luxfi/geth's tx validation (ValidateSignatureValues,
homestead=true) REJECTS high-S signatures, so the anchor's MPC-signed self-tx
failed on submit with 'invalid sender' (live: POST /v1/admin/treasury/anchor ->
status error, note 'submit: send tx: invalid sender'). recoverableSig now
canonicalizes s to N-s when it exceeds N/2 before searching the recovery id, so
the 65-byte r|s|v it hands tx.WithSignature is EIP-2-valid and recovers to the
treasury MPC wallet. Tests: TestRecoverableSig_LowSNormalization (forced high-S ->
low-S, still recovers). go test ./clients/wallets/... green; cmd/cloud builds.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Adversarially verify blue's class-A fixes hold under op COMPOSITION inside a
single block (which the original red suite exercised only as separate blocks or
single ops). Six hostile compositions — bare reassign, release+reassign,
release+assign, membership-remove+reassign, remove+release+assign, assign-steal
— are each refused end-to-end through the N=7 ceremony, and the live writer is
unchanged across every voter with its lease mirror consistent. Plus: the
authorized proven-dead handoff stays single-valued under redundant reassigns,
and assign+release of a fresh resource leaves no orphan writer (mirror desync
would be a second authority). GO: the double-write class is fully closed.
The ring's :8081 commits a newly-created vault to its DB AFTER writing the
createVault 201 response, so cloud's back-to-back createVault->createWallet (fired
microseconds apart on one keep-alive connection) races the commit and read-misses
the just-created vault -> 404 'vault not found' -> custody=safe 502. A slower
client (curl, separate processes) never observes the gap, which is why manual
repro succeeded. Bounded retry (6x, linear 250ms backoff, ctx-aware) on exactly
that 404; every other error still fails fast. Idempotent per attempt (fresh body).
go build ./cmd/cloud green; go test ./clients/wallets/... green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Wires the #162 BindAnchorSigner seam to a real quorum signer. New:
- wallets.TreasuryAnchorSigner(org,chain): resolves-or-provisions the org's stable
KindTreasury wallet on the ring (reserved account 'treasury' / wallet
'reserve-anchor', idempotent) and returns its EVM address + a sign closure.
- The closure produces an EVM-recoverable r‖s‖v signature: the ring returns a bare
r‖s (64B, no recovery id) but tx.WithSignature needs 65B, so recoverableSig finds
the v whose recovery yields the wallet address (fails closed otherwise).
- POST /v1/admin/treasury/bind-anchor (global-admin): calls TreasuryAnchorSigner +
BindAnchorSigner, so subsequent /v1/admin/treasury/anchor commits the ledger root
signed by the treasury MPC wallet, not the lone KMS key. Returns the bound address
(fund it for gas on the Hanzo L1).
Tests: TestRecoverableSig (both parities recover to the signer) + _NoMatch (fail
closed). go build ./cmd/cloud green; go test ./clients/wallets/... green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The o11y-scope landing added clients/o11y/{scope,status}.go referencing a metrics-
query layer (vmClient, newVMClient, promLabel, metricsQuery, queryMetrics,
metricsResult, metricPoint, usageRollup, boundRangeMinutes) whose source file was
never committed → `go build ./cmd/cloud` failed (undefined symbols), taking the
whole deploy plane down (no new cloud image buildable from main). Restore the file
to the surface's own honest-empty contract: newVMClient reads O11Y_VM_URL and an
unset/unreachable VM degrades every query to an honest-empty series (never a
fabricated point); status.go's VM up-inventory works when VM is wired. queryMetrics
returns the honest-empty RED series until the VM query_range wiring lands. Full
`go build ./cmd/cloud` now links; go test ./clients/o11y passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Brings the embedded ai subsystem up to v1.802.0:
- #76 opt-in auto-routing (virtual auto/zen-router model, X-Routed-Model)
- #77 per-org enable/disable (OrgSettings precedence)
- #78 admin-settable defaults (reserved "*" row, /v1/get-routing-defaults)
+ RoutingEvent collection (no prompt text) + /v1/export-routing-ledger
Edge contract unchanged; auto_routing_billing_test green against the new
module (ok github.com/hanzoai/cloud). Pre-existing clients/o11y compile
break on main is untouched (fix lands separately).
Kill the second automation surface. /v1/auto was a per-org reverse proxy
(clients/auto + clients/auto/proxy) to the standalone hanzoai/auto engine
(auto.hanzo.svc) — a duplicate of the native, in-process /v1/automations
Connectors+Automations engine (clients/automations, cloud.EmbeddedTasks,
706-piece catalogue). One engine, one surface: /v1/automations is the ONE
native automation engine. The external engine + its console link-out are
retired (console + universe in paired PRs).
- Remove the order-140 blank import of clients/auto from subsystems.
- Delete clients/auto/ (auto.go + proxy/).
No functional loss: /v1/automations already serves flows/versions/runs/
pieces/MCP natively. clients/kb keeps its own AUTO_UPSTREAM piece-runner
coupling (a separate, pre-existing bridge to a never-implemented engine
endpoint) — reported for a follow-up, not touched here.
go build ./... green, go vet green, go test ./clients/automations + root ok.
* refactor(automations): rename connector catalogue pieces -> connectors (HIP-0125)
The automations connector CATALOG surface drops the ActivePieces term "pieces" for the ONE Hanzo term "connectors":
- GET /v1/automations/pieces -> /v1/automations/connectors; /pieces kept as a
byte-identical back-compat alias (same handler) so live clients never break.
- Catalog{PieceCount,Pieces} -> {ConnectorCount,Connectors}; PieceMetadata/
PieceAuth/PieceAction/PieceTrigger -> Connector*; JSON tags pieceCount/pieces
-> connectorCount/connectors; embedded catalog.json + OpenAPI updated to match.
- Test proves the /pieces alias mirrors /connectors byte-for-byte.
Deliberately UNCHANGED (persisted @xyflow builder wire contract; renaming would
break live clients + stored flows): the flow-step protocol PieceName/pieceName,
PIECE/PIECE_TRIGGER, corePiece. Aligning those is a staged migration (HIP-0125).
* chore(automations,git,framework): scrub AI-slop placeholder comments (Rob Pike pass)
Comment-only, zero behavior change. Removes agent-note narration and future-work hedges, keeps the real WHY:
- automations.go: drop "a separate agent later OVERWRITES this file" narration; keep the Catalog-is-the-wire-contract invariant.
- framework/naming.go: "value for now" -> "value derived from now" (it reads the now arg, not a hedge).
- git/git.go: drop TODO(billing) + "in the MVP" hedge; state the git.usage meter fact.
- git/storage.go: drop TODO(vfs)/MVP/follow-up narration; keep the WHY osfs (not vfs) is used (vfs.FS does not implement go-billy).
Kept as real WHY/invariants (not slop): connector_core.go loopback-test SSRF guard, connector_slack.go httptest override, affiliates/store.go sentinel + PendingCents; types.go was already cleaned in the rename commit.
* docs(automations): point connector-rename references at HIP-0126 (0125 was taken)
---------
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
CTO decision: ONE automation engine = the native Go /v1/automations
(clients/automations on cloud.EmbeddedTasks). This removes the redundant
/v1/auto reverse-proxy subsystem (clients/auto), a per-org proxy to the
standalone ActivePieces Deployment (auto.hanzo.svc).
- delete clients/auto/ (auto.go + proxy/)
- drop the order-140 blank import from subsystems.go
Safe: no live caller of cloud/v1/auto — console link-outs to auto.hanzo.ai,
and clients/kb calls the engine directly via its own AUTO_UPSTREAM client
(untouched here). The native /v1/automations surface is unaffected.
NOTE (does NOT retire the ActivePieces Deployment): clients/kb/sync_piece.go
still executes connector pieces via the engine at /v1/auto/pieces/{piece}/run;
the native engine exposes the piece CATALOGUE but not piece EXECUTION yet, so
auto.hanzo.svc must stay until native reaches piece-run parity.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Comment-only tightenings, zero behavior change:
- pubsub/o11y: drop the misleading "GC won't collect" narration on the
package-level server/collector refs; state the real reason (shutdown
reachability) or the actual invariant (metrics ref is a write-only keepalive).
- iamsvc: condense the 11-line InitEmbed block that verbatim-restated the
package doc down to the fail-closed WHY that matters at the call site.
No code changed (git diff: comments only).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Safe deploy (POST /v1/wallets/{id}/smart-wallet) resolves the owner wallet by its
db.Wallet PRIMARY KEY (orm.Get). The :9800 internal /keygen mints a threshold key
but persists NO db.Wallet row, so deploy 404'd 'wallet not found' (live: every
custody=safe create -> 502). The ring's Safe surface is VAULT-scoped: the only
create path that persists a db.Wallet AND returns its id is
POST /v1/vaults/{id}/wallets.
safeCustody.Provision now: createVault -> createWallet (vault-scoped, returns db
id + internal WalletID + EOA) -> deploySafe(dbId). KeyRef stays
<internalWalletId>|<smartWalletId> (owner-sign via :9800 uses the internal id;
propose via :8081 uses the smart-wallet id); the db id is only needed for the
one-time deploy. safeclient gains createVault + createWallet; the stub test now
emulates the vault/wallet-create routes. go test ./clients/wallets/... green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Add two org-scoped cloud-api surfaces for the enterprise console:
- /v1/usage/summary (clients/usage): the org's unified footprint roll-up —
spend by category over time + wallet (from the commerce ledger) plus LLM
usage totals (from the warehouse). Composes existing sources server-side;
each degrades independently to honest zeros with a source marker. Org from
the validated bearer only (principal.Tenant); a forged X-Org-Id with no
principal 401s and never reaches commerce.
- /v1/audit (clients/auditlog): the per-org twin of the admin god-view — an
org admin reads ONLY their own org's events off the SAME tamper-evident,
hash-chained store. Org PINNED server-side (a client ?org is ignored);
filters time/actor/action/resource/resourceId/result + pagination.
- audit: extract the shared audit.Wire projection (used by both the admin and
org routes, one JSON contract) and add a ResourceID filter to audit.Query.
Tests: usage (pure roll-up/categorization + HTTP scoping/honest-zeros),
auditlog (real in-memory recorder: scope isolation, filters, pagination,
401/501), audit (ToWire + ResourceID). CGO_ENABLED=0 go build + go test green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Reviewing my own displacement fix adversarially: red tested release+reassign,
but a standalone release of a LIVE holder was still admitted, and after it the
shard is unowned — so release(victim) at H then assign(attacker) at H+1 puts a
second live writer on the shard (same class, pure policy, survives real crypto).
Fix: a LIVE holder's lease is immutable — releasable only when the holder is
proven dead (ErrUnauthorizedRelease), symmetric with reassign. Closes the whole
double-write class, not just red's two tested paths. + TestPolicy_ReleaseLiveHolderRefused;
TestRSM_DeterministicConvergence now marks the holder proven-dead (out-of-band)
before releasing. Suite green.
* treasury: anchor signs through a quorum-gateable seam, not a lone key
anchor_evm.go held the signer's private key in-process and did types.SignTx.
Decouple WHERE the key lives from the tx builder via a txSigner seam:
- keySigner — the existing local KMS-provisioned key (default; unchanged result,
proven byte-identical to types.SignTx).
- mpcSigner — delegates the 32-byte EVM signing hash to a quorum-gated custody
backend (the reserve's 3-of-5 treasury MPC wallet), bound via BindAnchorSigner
(the finance seam). The bound signer wins over any local key.
submit() now hashes the tx, delegates the hash to the resolved signer, and
applies the recoverable signature — agnostic to single-sig vs threshold. Fails
closed when neither signer is available (never fabricates a signature).
Test proves both paths recover to the correct sender and the quorum signer is
invoked exactly once; the live ring is a config swap.
* feat(gpu): BYO-GPU worker uploads render outputs to the org gallery
After studio.render completes on the local GPU, the worker fetches each finished
output from the local studio (/view) and POSTs it to the org studio's /upload/output
with the user's IAM bearer — landing it in orgs/{org}/output (S3-mirrored to the
gallery). No S3/rclone credentials ever touch the box; the session token is the only
credential. Upload target resolves from input.uploadUrl, then HANZO_STUDIO_UPLOAD_URL,
then studio.hanzo.ai. Proven end-to-end against studio 0.14.9 (aud hanzo-console).
* feat(gpu): per-machine share policy — advertised on the fleet record, enforced at claim
A linked GPU can be shared to specific orgs/projects/job-types/models with limits via
ONE policy object on the machine record (SharePolicy). It rides in the fleet
registration (input.policy) and is enforced ONCE, at claim: a job outside the policy
is failed back so an eligible worker takes it. nil/zero policy = fully permissive
(unchanged behaviour). Loaded from HANZO_GPU_POLICY (inline JSON) or
HANZO_GPU_POLICY_FILE. Unit-tested (reject matrix + loader).
Server-side multi-org queue fanout + metering-to-org+project remain follow-ups; the
worker enforces its own policy today (workers still claim their own org's queue).
* feat(world): GDELT + allowlisted-RSS news data plane (clients/world)
First vertical slice of the World news backend in the unified cloud binary:
GET /v1/world/news merged, filtered, freshest-first feed -> {items:[…]}
GET /v1/world/pipeline per-(org,project) pipeline config
PUT /v1/world/pipeline upsert feeds + keyword/region/source filters
GET /v1/world/stream SSE live refresh (ZAP-native, org+project scoped)
- Ports world/api/{gdelt-doc,rss-proxy}.js: GDELT 2.0 Doc artlist + host-
allowlisted RSS/Atom (~180-domain SSRF allowlist, enforced at PUT boundary,
at fetch time, and on redirect targets).
- Org/project isolation on every path (principal.Tenant/Project); SQLite
pipelines table PK(org,project); in-memory TTL feed cache (10m).
- RegisterWithShutdown order 142; one blank-import line in subsystems.go.
- Tests: httptest-stubbed upstreams (deterministic/offline) + live-verified.
---------
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Follow-up rails from the red pass + the cryptographer audit (all on top of the
class-A fixes):
- checkQuorumSafety(N,quorum,f) asserts N>=3f+1, quorum>=2f+1, 2q>N+f at cluster
construction (fail closed). The 2q>N+f margin at N=7 is exactly 1 and is the
whole basis of the no-fork property, so a future sizing change can never
silently break safety. + TestSafety_QuorumParametersAreByzantineSafe.
- verifyOwnCertStructure: renamed the driver's structural self-composed cert
check away from 'independent triple-gate verification' and documented that an
external cert must go through the cryptographic VerifyUnderPolicy (increment-2),
never this structural path (red #8).
- commitZ: domain-separate the z-share commitment by session + party
(H(cp-commit||sid||party||z)) so a commitment cannot be replayed across
sessions/parties (red #4 hardening).
- doc.go: record the red->blue outcome (no-fork core held; 4 class-A closed), the
CLASS-B caveat (stub secrets are public-seed-derivable -> safety suite meaningful
only under real crypto), and the increment-2 security worklist (distributed DKG,
authenticated handoff + KMS fence, RSM-level authz re-verify, external-cert
crypto verify, CI guard against -tags controlplane releases).
Suite green under -tags controlplane -race; default build unaffected.
Red found a CRITICAL double-write + 3 more class-A breaks (pure orchestration/
policy, survive real crypto) with failing exploit tests. All closed; red's 4
class-A tests now pass without weakening them; class-B (stub-crypto-forgeable)
deferred to the real-crypto increment with explicit t.Skip TODOs.
#1/#2 CRITICAL double-write (policy.go, placement.go): displacement of a LIVE
shard writer now requires proven-death by out-of-band evidence. A proposer-
written same-block release authorizes nothing (it is not holder consent), and
membership removal no longer manufactures proven-dead. Fail-closed increment-1
posture; authenticated graceful handoff + KMS fence are increment-2.
#3 HIGH barrier forgeable (driver.go, custody.go, signer.go, transport.go):
Round1 commitments are now proof-of-possession authenticated exactly as Round2
legs, so one node cannot forge a quorum of spoofed commitments to defeat
commit-before-reveal.
#4 MEDIUM apply fork gate (rsm.go): RSM.Apply re-checks ParentRoot == the
applied-state commitment, so a block that does not extend local state can never
mutate it (defense-in-depth for a future recovery/gossip path).
Corrected TestPolicy_ShardReassign_WithRelease (it asserted the vulnerable
same-block-release-authorizes-displacement behavior) to assert the fix. Updated
rsm_test blocks to extend state properly (the new parent-root gate). Suite green
under -tags controlplane; default build unaffected (package is tag-gated).
New KindSafe custody composes the ring's TWO planes without importing luxfi/mpc:
- :9800 internal threshold API (mpcclient) — keygen the owner MPC EOA + owner-sign
- :8081 product API (new safeclient) — CREATE2 Safe deploy + EIP-712 Safe-tx propose
safeclient.go mints a SHORT-LIVED HS256 ring JWT (iss=mpc.lux.network, aud=mpc-api,
role=admin, org-scoped) hand-rolled (crypto/hmac, no jwt dep) from the ring's
MPC_JWT_SECRET — resolved from cloud's in-process KMS via
CLOUD_WALLETS_MPC_JWT_SECRET_REF, NEVER a plaintext env value. The deploy route is
role-gated (owner|admin), so role=admin clears it.
safeCustody.Provision: keygen (owner EOA) -> deploy Safe(owners=[EOA], threshold=1)
on the wallet's EVM chain (per-wallet, default Hanzo L1 36963); KeyRef encodes both
ring handles (<mpcWalletId>|<smartWalletId>); Address = predicted Safe contract.
Sign: owner-approval signature via :9800 (uniform /v1/wallets/:id/sign). New route
POST /v1/wallets/:id/safe-tx composes the ring propose (EIP-712 MPC-sign) via a
safeProposer capability type-assert (no Kind switch). Fails closed
(ErrMPCNotConfigured) until CLOUD_WALLETS_MPC_API_ADDR + the JWT secret are wired.
Tests: TestSafeCustody drives a stub emulating both ring planes (asserts the minted
JWT is HS256-valid with correct iss/aud/role/org) + TestSafeCustody_FailClosed.
go test ./clients/wallets/... green; CGO_ENABLED=0 go build ./cmd/cloud green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two coupled changes so main builds green AND the MPC custody surface is live:
1. Fix the broken build on main. #160 (CLOUD_ROLE writer/reader HA split) and
#163 (Stage-0 control-plane inert config) each added a `Role` field to the
SAME Config struct on separate branches; the merge left Config.Role
redeclared (role.Role vs string) + a duplicate struct-literal key, so
`go build ./cmd/cloud` failed (release lane stuck at v1.786.124). Rename the
inert #163 field to ControlPlaneRole (env ROLE, consumed by nothing yet). The
HA Role (role.Role, CLOUD_ROLE, used by serve.go/build.go) is unchanged.
2. Register the wallets subsystem. clients/wallets (#151/#161) was never blank-
imported into subsystems.go, so its init() never ran and /v1/wallets was
unrouted (404) despite the code shipping. Add the order-127 blank import so
the accounts/wallets/custody/keys/sign surface mounts — KMS custody always
on; mpc/treasury fail closed until CLOUD_WALLETS_MPC_ADDR +
CLOUD_WALLETS_MPC_API_KEY_REF are wired. This is the seam the treasury anchor
(#162 BindAnchorSigner) binds through.
Verified: CGO_ENABLED=0 go build ./cmd/cloud green; wallets + config tests ok;
local boot logs 'wallets mounted' (defaultCustody=kms) then 'listening', no panic.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Two PRs landed on main that both added a Config.Role field — the #160
HA writer/reader role (role.Role, load-bearing in Serve) and the Stage-0
control-plane role (string). The text-merge compiled to a duplicate field
and broke the default build. Rename the inert control-plane field to
ControlPlaneRole (env ROLE unchanged); the HA Role keeps its name and all
cfg.Role.IsReader()/String() consumers are untouched.
* refactor(kms): de-alias badger→zapdb (the embedded store IS ZapDB)
clients/kms/kms.go imported the store as `badger "github.com/luxfi/zapdb"`.
The store is luxfi/zapdb — the canonical Lux embedded KV, a hardened Badger
fork whose Go package is still literally `package badger`. The alias made call
sites read like raw dgraph-io/badger. Rename the alias to `zapdb` so every call
site is self-documenting; behaviour is byte-identical (same package, same API).
Confirms the invariant: `grep -rn dgraph-io/badger` across cloud = 0. There is
no raw Badger anywhere; the one embedded store is ZapDB.
* feat(cloud): CLOUD_ROLE writer/reader split + read-only KMS reader + writer-pin
Introduces an explicit HA role so read replicas can be added WITHOUT ever
risking a second writer opening the RWO stores. Default is byte-identical to
today: unset CLOUD_ROLE ⇒ Writer ⇒ the single pod that owns the RWO PVC.
- role: CLOUD_ROLE ∈ {writer(default), reader}. Serve fails CLOSED on an
explicitly-invalid value (a wrong guess demotes the real writer or risks a
second one). Pure, tested, imports nothing from cloud.
- kms: Config.ReadOnly opens the ZapDB store READ-ONLY with the lock guard
BYPASSED — a reader serves secrets off a restored replica and NEVER takes the
exclusive write lock (the mechanism proven safe by luxfi/zapdb's
WithReadOnly + BypassLockGuard; zapdb-replicate uses the same to coexist with
a live writer). Reader with no restored store / no key fails closed. Tested
round-trip: writer writes → reader reopens read-only → reads back; reader
writes rejected.
- writerpin: the single-writer election seam. SingleWriter (production-correct
for StatefulSet replicas:1) is the default; ConsensusPin (Quasar leaderless
election) is an HONEST stub that fails closed with ErrNotImplemented rather
than fabricating a pin. Tested.
- wiring: Serve resolves+logs the role and the backing pin; pickKMSClient opens
KMS read-only for readers. Writer path unchanged.
NOT YET wired (reported for Red/CTO): consensus election (writerpin gates no
store-open yet — k8s guarantees the single writer); reader gating of the
audit chain / durable tasks / per-tenant SQLite (still open writable) — the KMS
reader path is the completed slice. Data replication runs as sidecars at the
manifest layer (hanzoai/replicate for SQLite, luxfi/zapdb-replicate for ZapDB),
not via in-process import.
* feat(ha): fail-closed reader write-guard + prove in-process KMS backup
ReaderGuard: one boundary middleware rejects mutating verbs on a Reader
(405), gating EVERY store (KMS+audit+tasks+SQLite), not just KMS's
read-only open — a mis-routed write can no longer silently persist to a
reader's ephemeral dir and vanish on restart (H4). No-op on a Writer.
replication_test: real *zapdb.DB writer streams incremental age-encrypted
db.Backup blocks WHILE live; reader Restores into its OWN separate dir —
refutes the C1 'second-process open fails' path and proves the producer.
Fail-closed test: no recipient => no block (never plaintext to S3).
* test(ha): reader-guard verb matrix + replication edge cases
ReaderGuard: GET/HEAD/OPTIONS reach the store, POST/PUT/PATCH/DELETE all
405 without reaching it; Writer path (guard unmounted) serves every verb.
replication: wrong-identity restore fails closed; restore requires manifest
+ identity (unhydrated store never serves empty); repeated/no-op/overwrite
backups restore to the exact latest value (chain-correctness invariant).
* test(config): align IAM single-replica test with staged-subsystem contract
The 'empty list -> iam-enabled' subtest predates IAM becoming a STAGED
subsystem (stagedSubsystems["iam"]=true): the empty-Enable mount-all
default deliberately does NOT mount IAM (it corrupts the shared Beego
global and crashes `ai` with SQLITE_CANTOPEN). So empty list is
iam-DISABLED and >1 replica is allowed; the guard fires only when iam is
EXPLICITLY enabled. Code was correct; the test asserted the pre-staging
behavior. Pre-existing red on main, unrelated to the HA change.
Promote the luxfi consensus stack (consensus v1.25.15, bft v0.1.5,
p2p v1.21.1, validators v1.2.0) from indirect to direct requires, and add
four INERT control-plane config fields. Zero behavior change, reversible.
Deps + inert config only — no engine imported/started, no routes, no
serve.go/build.go behavior change.
v1.25.15 is the minimal clean tag: it already carries NewBFT (consensus.go:168)
+ engine/bft, its graph pulls validators v1.2.0 (Manager), it requires exactly
pulsar v1.1.1 (which stays v1.1.1 — zero drift), and it is the MVS-selected
version, so promotion is a no-op to the compiled graph. A lower tag would
downgrade the whole build's consensus (behavior change); a higher tag drifts
pulsar + consensus code.
The four are held direct by controlplane_deps.go: blank imports behind the
never-set //go:build controlplane_deps tag, so nothing links into the binary.
go mod tidy keeps them direct (it reads all build tags); deleting the file
reverts them to indirect. NodeID/Peers/Role/ControlPlaneQuorum parse in
LoadConfig (NODE_ID/PEERS/ROLE/CONTROL_PLANE_QUORUM) but no subsystem reads them.
Architecture direction (proposed, not shipped): the control plane is designed
to run Quasar (post-quantum BFT, protocol/quasar Submit->Finalized) under a
strict-PQ cert profile with a Pulsar RoundSigner threshold signer.
tidy also corrected pre-existing drift on main (nats-io/nats.go indirect->direct
via clients/kafka/interop_test.go; pruned 7 superseded go.sum lines) — verified
identical on pristine origin/main.
Red review findings:
- go.sum: luxfi/precompile v0.5.37 zip hash disagreed with sum.golang.org
(h1:Yh3dJ+... vs authoritative h1:2v0z...) → cold-cache CI SECURITY ERROR.
Corrected to the sumdb-vouched hash.
- telemetry.go: remove the plaintext OTLP-HTTP fallback (newTraceExporter) that a
stray/standard OTEL_EXPORTER_OTLP_ENDPOINT could use to silently downgrade
tenant-carrying trace spans to cleartext. ONE wire now: ZAP. Dropped the
otlptracehttp import (also severs its transitive grpc pull) and the dead
otlpEndpoint parameter. OTLP stays only the collector's interop receiver.
The ai subsystem serves a virtual `auto`/`zen-router` model that resolves to a
concrete model id before pricing/billing, meters its own token cost keyed on the
SERVED model, and reports it via the X-Routed-Model header. The cloud edge prices
/v1/ai/* by PATH (0, self-metered), never by the request model, so `auto` bills
as whatever it resolved to — and the edge passes X-Routed-Model through untouched.
- auto_routing_billing_test.go: TestAutoRoutingBillsAsResolvedModel (edge does not
double-bill /v1/ai/* + header pass-through) and TestDefaultPriceAiPathModelAgnostic.
- AUTH_BILLING_CONTRACT.md §4a: document the binding.
No code change needed — cloud already meters ai from the subsystem's own usage
record (which keys off the resolved request.Model), so the edge binds correctly.
Route ExportTraceServiceRequest wire encoding through github.com/zap-proto/zap2pb
(the sanctioned ZAP<->protobuf boundary) instead of importing
google.golang.org/protobuf {proto,encoding/protowire} directly. Wire bytes are
byte-identical (repeated ResourceSpans under field 1); TestUploadTracesOverZAP
still decodes the spans over the real ZAP transport.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
anchor_evm.go held the signer's private key in-process and did types.SignTx.
Decouple WHERE the key lives from the tx builder via a txSigner seam:
- keySigner — the existing local KMS-provisioned key (default; unchanged result,
proven byte-identical to types.SignTx).
- mpcSigner — delegates the 32-byte EVM signing hash to a quorum-gated custody
backend (the reserve's 3-of-5 treasury MPC wallet), bound via BindAnchorSigner
(the finance seam). The bound signer wins over any local key.
submit() now hashes the tx, delegates the hash to the resolved signer, and
applies the recoverable signature — agnostic to single-sig vs threshold. Fails
closed when neither signer is available (never fabricates a signature).
Test proves both paths recover to the correct sender and the quorum signer is
invoked exactly once; the live ring is a config swap.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The prior mpcclient targeted a DECIDED-but-nonexistent dashboard route tree
(/v1/wallets/{id}/sign, /v1/treasury/*) authed with a hand-minted HS256 JWT.
The deployed luxfi/mpc ring's real, working server-to-server custody surface is
the internal threshold API (cmd/mpcd/main.go, :9800): POST /keygen + POST /sign,
gated on the static MPC_INTERNAL_API_KEY bearer token — the exact contract the
ring's own /sign handler documents for a custody adapter.
Reconcile cloud to that contract:
- mpcclient.go: keygen + sign over the internal API; static bearer key (KMS),
no JWT/dependency; deterministic idempotency key per (org,wallet,digest).
- custody.go: mpc + treasury provision via keygen, sign via /sign with the
wallet's EVM chain id; Rotate preserves the address (ring-managed shares).
Treasury quorum governance moves to the finance policy layer over this same
primitive (no separate ring route).
- wallets.go: CLOUD_WALLETS_MPC_API_KEY_REF (KMS ref of the bearer key).
- test: stub emulates the internal /keygen+/sign contract.
Feature-flagged: unset CLOUD_WALLETS_MPC_ADDR ⇒ mpc/treasury fail closed.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The console per-product Metrics dashboard groups usage on metadata.product /
metadata.agent, but commerce RecordUsage persists only provider/model (no product
field), so the breakdowns rendered honest-empty even though every non-LLM product
already meters+gates per-org via ResourceMeter (provider=<product>, default fee
$1.00, fail-closed 402 on zero balance).
clients/billing/usage.go is the ONE read-side adapter: usage() injects a canonical
metadata.product onto each ledger row (agent->agents, provisioning->kind,
token-metered->inference, else provider) from the SAME charged ledger, and honors
the previously-ignored ?product=<id> (server-side filter) and ?groupBy=product
(per-product spend rollup {product,requests,amountCents}). A row already carrying
metadata.product/agent wins, so it degrades to a no-op once the meter/commerce
persist them natively (forward-compatible).
No change to what is charged or gated; the balance floor stays enforced by default.
scopedBillingQuery is extracted so proxy() and usage() build the subject boundary
one way. AUTH_BILLING_CONTRACT.md documents coverage + the native-field checklist.
Tests: productOf table + enrich/filter/group units + handler-level ?product= /
?groupBy=product through the real route (33 billing tests green).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The console image stage now FAILS the build when build:embed does not emit a
real static bundle (non-empty out/index.html + out/_next), instead of silently
degrading to the committed fallback shell. A broken console export can no longer
ship the placeholder to prod. Escape hatch: --build-arg ALLOW_PLACEHOLDER=1 for
a pure-Go dev image with no Node console.
hanzoai/console build:embed produces a real 7.7M static export (361KB index.html
+ 4.3M _next chunks); //go:embed bakes it into the ONE cloud binary. Also drops
the last console2 references (repo is hanzoai/console).
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The frontend repo is hanzoai/console (console2 was renamed away). Kill the
dead name across the build path + source so there is one name, one way:
- Dockerfile: clone hanzoai/console.git; ARG CONSOLE_REPO / CONSOLE_REF
- Makefile: CONSOLE_DIR; webui + build-standalone targets
- config.go: drop dead console2.hanzo.ai from the ZAP-WS origin allowlist
- comments across clients/* reference the console repo + its TS modules by
their real name
No behavior change beyond dropping one unused CORS origin. Root pkg builds.
Red review = SHIP; these close the 4 cloud-side low findings so the PR lands
with no known edges.
low-1 (rollback atomicity): createDedicated's inject-failure branch now calls
removeAddonURL BEFORE tearing the backend down. injectAddonURL is not atomic —
a strategic-merge PATCH can LAND server-side yet still return err (dropped
response / post-commit timeout); scrubbing the maybe-written <KIND>_URL first
means a committed-but-errored inject can't leave the instance pointing at a
deleted backend (a dangling DSN is worse than Base). Proven by
TestDedicated_InjectPartialWriteRollsBackOrphanKey (fake now models the
write-then-error partial failure; asserts inject THEN remove ran, key gone).
low-2 (kv fail-open corner): TestDedicatedKV_RequirepassEnforced boots the REAL
ghcr.io/hanzoai/kv image with the exact engine.args + mounted requirepass
config and asserts an UNAUTHENTICATED PING is REJECTED, then that default:<pw>
authenticates — locking down the one corner where, if the image ignored the
positional config, the instance would boot unauthenticated. Raw RESP over TCP
(zero new client deps); gated on CLOUD_KV_SMOKE_IMAGE + docker so the default
suite stays green, real in CI.
low-3 (strategic-merge sibling preservation): TestPatchAddonSecret_RealAPIServer
runs the ACTUAL k8sOrchestrator addon methods against a REAL kube-apiserver
(controller-runtime envtest) — inject KV_URL then SQL_URL => BOTH survive in
.data; RemoveAddonSecretKey drops one, keeps the other; idempotent on absent
key/Secret. Replaces the fake orchestrator's assumption with a server-proven
fact. Gated on KUBEBUILDER_ASSETS (skip without envtest binaries). Adds
controller-runtime v0.23.3 as a TEST-ONLY dep — pinned to the release that
keeps k8s.io at v0.35.3 (NO production client-go bump).
low-4 (datastore tag symmetry): dedicated datastore image tag floating ':26' ->
env("CLOUD_DEDICATED_DATASTORE_TAG", "26.2.3.2"), symmetric with sql/kv/docdb.
A floating ':26' resolves to whichever datastore lineage (bridge vs fork, distinct
data dirs) last pushed under it — a per-org instance must boot a deterministic
image.
go build ./... green; go test ./clients/provisioning/... green (envtest PASS
against a live apiserver, kv-smoke skips without docker).
(cherry picked from commit 04d841c4906b58fa06b1bc407b55c97e6661f169)
* feat(o11y): wire native datastore metrics ingest into the embedded runtime
Bumps hanzoai/o11y to the native datastore metrics driver and starts an
in-process ZAP metric receiver (clients/o11y/metrics.go) that writes metrics to
the datastore over upstream ch-go via o11y/pkg/datastoremetrics — no histogram
fork. Reuses the embedded runtime.TelemetryStore.ClickhouseDB() connection, so
the query plane (read) and metrics (write) share one datastore conn.
Opt-in + fail-soft: gated on O11Y_METRICS_ZAP_LISTEN, a no-op until set, errors
logged and swallowed so metrics ingest can never take the query plane down. This
unblocks retiring the standalone signoz-otel-collector metrics path once verified
(verify-then-cutover). CGO_ENABLED=0 build + vet + existing o11y/observe tests green.
* chore: re-pin o11y@main (native datastore metrics driver merged)
---------
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Red review = SHIP; these close the 4 cloud-side low findings so the PR lands
with no known edges.
low-1 (rollback atomicity): createDedicated's inject-failure branch now calls
removeAddonURL BEFORE tearing the backend down. injectAddonURL is not atomic —
a strategic-merge PATCH can LAND server-side yet still return err (dropped
response / post-commit timeout); scrubbing the maybe-written <KIND>_URL first
means a committed-but-errored inject can't leave the instance pointing at a
deleted backend (a dangling DSN is worse than Base). Proven by
TestDedicated_InjectPartialWriteRollsBackOrphanKey (fake now models the
write-then-error partial failure; asserts inject THEN remove ran, key gone).
low-2 (kv fail-open corner): TestDedicatedKV_RequirepassEnforced boots the REAL
ghcr.io/hanzoai/kv image with the exact engine.args + mounted requirepass
config and asserts an UNAUTHENTICATED PING is REJECTED, then that default:<pw>
authenticates — locking down the one corner where, if the image ignored the
positional config, the instance would boot unauthenticated. Raw RESP over TCP
(zero new client deps); gated on CLOUD_KV_SMOKE_IMAGE + docker so the default
suite stays green, real in CI.
low-3 (strategic-merge sibling preservation): TestPatchAddonSecret_RealAPIServer
runs the ACTUAL k8sOrchestrator addon methods against a REAL kube-apiserver
(controller-runtime envtest) — inject KV_URL then SQL_URL => BOTH survive in
.data; RemoveAddonSecretKey drops one, keeps the other; idempotent on absent
key/Secret. Replaces the fake orchestrator's assumption with a server-proven
fact. Gated on KUBEBUILDER_ASSETS (skip without envtest binaries). Adds
controller-runtime v0.23.3 as a TEST-ONLY dep — pinned to the release that
keeps k8s.io at v0.35.3 (NO production client-go bump).
low-4 (datastore tag symmetry): dedicated datastore image tag floating ':26' ->
env("CLOUD_DEDICATED_DATASTORE_TAG", "26.2.3.2"), symmetric with sql/kv/docdb.
A floating ':26' resolves to whichever datastore lineage (bridge vs fork, distinct
data dirs) last pushed under it — a per-org instance must boot a deterministic
image.
go build ./... green; go test ./clients/provisioning/... green (envtest PASS
against a live apiserver, kv-smoke skips without docker).
(cherry picked from commit 04d841c4906b58fa06b1bc407b55c97e6661f169)
Extend the dedicated-instance strategy so all four on-demand data add-ons —
Hanzo KV / SQL / DocDB / Datastore — route through ONE mechanism, and bind an
enabled add-on to an app instance by injecting its DSN as <KIND>_URL into the
instance's addons Secret (disabling reverts to Base).
- store: additive instance column (idempotent ALTER, threaded through Resource/
cols/scan/Insert) + ListByInstance(org,instance).
- dedicated: add sql (Datastore type=postgresql, POSTGRES_* env, PGDATA subdir)
and kv (type=valkey, per-instance requirepass via a MOUNTED config Secret since
the kv-server binary reads no password from env; DSN user=default). Engine
gains adminUser/env/args/secretMount so the CR builder stays one code path.
- addon_inject: injectAddonURL/removeAddonURL + orchestrator PatchAddonSecret
(strategic-merge, create-if-absent, key-preserving) / RemoveAddonSecretKey
(JSON-merge delete, idempotent). Reloader annotation + rev bump on the Secret.
- create: instance bind field (validated); inject AFTER the row Insert as part of
the atomic provision (rollback on failure). drop: revert to Base BEFORE tearing
the backend down.
- sql/kv move off the shared-logical registry (each org OWNS its instance); the
orphaned shared postgres/redis provisioners + pgx/go-redis direct deps removed.
Tests: instance column round-trip + ListByInstance isolation; sql/kv DSN + CR
shape; inject merges (second add-on never clobbers the first); un-bound create
skips injection; drop removes URL before teardown; inject failure rolls back the
whole provision. go build/vet/test green.
Pulls ai's additive isGlobalAdmin field on /get-account so console
recognizes global admins. Pure dependency bump: re-pins re-tagged
luxfi/* modules from source (GOPRIVATE, sumdb-bypassed) after the
documented content-hash drift, prunes cloud.google.com/go/compute and
stale hanzoai/iam v1.31.16 (ai dropped the GCP SDK and requires iam
v1.31.17). go build ./... green (CGO_ENABLED=0).
Fold the standalone otel-collector Deployment into the unified cloud binary:
an in-process OpenTelemetry Collector accepts OTLP (grpc :4317, http :4318) and
writes spans+logs into the same ClickHouse datastore cloud already reads for the
o11y query plane (signoz_traces / signoz_logs, cluster insights). Consumers point
at cloud.hanzo.svc instead of otel-collector.hanzo.svc.
Trimmed, driver-compatible pipeline (reuses the signoz clickhouse exporters that
compile against cloud upstream clickhouse-go v2.44.0):
otlp -> memory_limiter, resource(namespace=hanzo, env), batch
-> clickhousetraces (traces), clickhouselogsexporter (logs)
- OFF by default (CLOUD_OTLP_INGEST_ENABLED); fail-soft; ShutdownFunc flushes.
- DSN via env (envprovider), never on disk; metrics self-telemetry off so only
:4317/:4318 bind (no :9090 class clash).
- telemetry.go: add OTLP-HTTP exporter path so cloud can loop back to the
in-process ingest at localhost:4318 (ZAP stays default/canonical).
DEFERRED: metrics pipeline (signozclickhousemetrics) needs SigNoz dd-sketch
ch-go fork (chproto.DD/Store/IndexMapping) that will not compile against cloud
upstream ch-go; metrics ingest stays on the standalone collector. See
clients/o11y/LLM.md.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
One custody seam over three orthogonal signing backends selected per-wallet by
Kind:
- KindKMS single-sig custody IN-PROCESS via the embedded luxfi/kms client
(deps.KMS). The fully-exercised spine: a real secp256k1 key is generated, its
private bytes sealed under the KMS envelope, and every Sign recovers to the
wallet address. No network hop.
- KindMPC / KindTreasury custody DELEGATE over HTTP to the deployed luxfi/mpc
cluster via a thin typed REST client (the clients/mpcseal precedent). cloud
never imports github.com/luxfi/mpc. Unconfigured -> fail closed
(ErrMPCNotConfigured); a signature is never fabricated.
Config seam: KMS always available; mpc/treasury built only when
CLOUD_WALLETS_MPC_ADDR is set and the HS256 JWT secret resolves from a KMS ref
(never a plaintext env). Per-tenant SQLite (org column on every row, every query
filtered by org). Finance seam (WalletForLedgerAccount) is a pure lookup only.
Tests (incl -race): KMS single-sig end-to-end (sig recovers to address, sealed
at rest, rotate changes address), per-tenant isolation, custody seam selects
backend (fail-closed mpc/400 unknown), mpc path wired against a faithful stub.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Reconciles TestGlobalAdminGate_RequiresAdminOrgAndIsAdmin with the task #51 decision
to pin IAM_ADMIN_ORG to the operator org (hanzo). The assertions are unchanged — the
gate is owner==adminOrg AND isAdmin — but the comment/labels no longer editorialize
that owner==admin is the only valid adminOrg. adminOrg is deployment config; the test
pins it to "admin" hermetically and proves the two invariants that hold for ANY
adminOrg: isAdmin is required (a non-admin in the admin org gets nothing) and owner is
required (an admin of any OTHER org gets nothing). Renamed the different-org case off
"hanzo" (which prod now pins AS the admin org) to a neutral "globex" so it reads
unambiguously.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The finance ledger of record ran on a single process-wide {DataDir}/treasury.db.
Select the store per request from the validated IAM owner instead, so every
tenant's books live on their OWN Hanzo Base file and one tenant's writes can
never appear in another's read.
- sqlstore.Manager: opens+caches one *Store per tenant (mutex-guarded map). The
house/reserve ledger is one fixed file ({DataDir}/treasury.db, preserved — no
migration of live reserve capital); customer ledgers are {DataDir}/finance/{slug}.db.
- tenantSlug: injective (never folds acme/ACME), path-traversal-guarded, reserves
the house slug. Verbatim stem for a DNS-ish org, else a sha256 slug. Consumes the
treasury's canonical hanzoai/sqlite opener (Open) — ledgercore's per-tenant opener
is a test-only helper that would double-register the sqlite driver.
- treasury.Mount binds the ledger of record to the HOUSE store; myAccounts reads the
caller's OWN per-tenant file (house scope still honours the Formance/Postgres opt-in).
- StorageDriver(): one place decides the driver — sqlite (default, what prod runs) or
postgres (opt-in via FORMANCE_LEDGER_URL). Postgres option preserved, never the default.
Tests: per-tenant isolation (A's write never in B's read; distinct files; cache
identity; traversal stays in-dir; no case-fold) + default-driver=sqlite + opt-in
preserved. go build ./cmd/cloud green; ./clients/treasury/... green (incl -race).
Co-authored-by: hanzo-dev <dev@hanzo.ai>
* fix(config): stage IAM off the mount-all default — unblock the release boot smoke
Every cloud release since the IAM embed (#142) has failed its boot smoke and
the fleet stayed pinned to a pre-embed image (v1.786.110), so the treasury +
finance merges (#143/#144/#145/#147) never shipped.
Root cause (from the failed release smoke logs): with CLOUD_ENABLE unset the
binary mounts every registered subsystem, so iamsvc.Mount now runs
iamserver.InitEmbed(). In the smoke/Docker env InitEmbed panics opening its own
SQLite (IAM_DATA_DIR=/data/iam absent on the tmpfs) and is recovered to a
fail-closed 503 — but IAM and the ai subsystem are sibling casibase/casdoor
forks linked against the SAME beego module, so InitEmbed's half-initialised
shared process-global (web.BConfig / xorm adapter) then makes ai's own
bootstrap fail identically:
iam ERROR iamserver.InitEmbed: bootstrap panicked: unable to open database file (14)
ai INFO ai: initializing runtime
cloud: mount: mount ai: ai: bootstrap: unable to open database file (14) -> SMOKE FAIL
This would crash api.hanzo.ai in prod too (CLOUD_ENABLE is unset there), not
just the smoke.
Fix: make IAM a STAGED subsystem — excluded from the empty-Enable mount-all
default, mounted ONLY when named in CLOUD_ENABLE. This is exactly the HIP-0106
staged-rollout contract iamsvc already documents ('operator adds iam to
--enable only after the fold is verified'), now enforced in code. It restores
the pre-#142 mount-all set (iamsvc is the only subsystem #142 added to it), so
the boot smoke goes green again; hanzo.id keeps being served by the standalone
iam pod until an explicit, verified cutover. Local mount-all boot now reaches
'listening' with iam 'subsystem disabled' and ai mounted clean.
pickIAMClient already falls back to the remote/disabled IAM client when
Enabled("iam") is false (build.go), which is current prod behaviour, so no
deps.IAM regression. One activation mechanism (the enable-list), one place.
* test(identity): lock global-admin = owner==adminOrg AND isAdmin
The cloud admin surfaces (incl. /v1/admin/treasury/*) grant global admin only
to a validated principal whose org IS the admin org AND whose token carries
isAdmin. This locks that invariant end-to-end through the real JWKS-validated
SanitizeIdentity boundary, with the two cases that matter for the treasury flip:
- a hanzo-org ADMIN (owner=hanzo, isAdmin=true) -> NOT global admin
- a NON-admin in the admin org (owner=admin, isAdmin=false) -> NOT global admin
The sole global admin z@hanzo.ai is global admin because IAM promotes @hanzo.ai
into the admin org (owner==adminOrg), NOT because it lives in 'hanzo'. This test
is the guard proving the boundary must NOT be widened to owner==hanzo (e.g.
IAM_ADMIN_ORG=hanzo), which would elevate every hanzo-org admin to see all
tenants' finances. The gate stays owner==adminOrg AND isAdmin; the fix for z is
that its token carries owner=admin, never a wider gate.
---------
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The hanzo CLI (cmd/hanzo) had no build target, so a naive
`go build ./cmd/hanzo` used the machine default CGO_ENABLED=1 and panicked
at init: "sql: Register called twice for driver sqlite".
Root cause: with CGO on, github.com/hanzoai/sqlite (the canonical Hanzo
driver, imported by ~15 clients/*/store.go) compiles its mattn/SQLCipher
backend and registers "sqlite"; the embedded upstream deps that import
modernc.org/sqlite directly (base/core, o11y, commerce/db, orm/db) register
"sqlite" a second time -> panic.
Fix: build cmd/hanzo the same pure-Go way cmd/cloud and the Dockerfile
already ship (CGO_ENABLED=0). hanzoai/sqlite's !cgo backend IS modernc, so
the fork and every modernc importer resolve to a single registration. This
extends the existing CGO_ENABLED?=0 policy (see Makefile header) to the new
binary instead of adding a second way to build; no dependency is dropped and
hanzoai/sqlite stays canonical.
make hanzo # -> ./bin/hanzo, pure Go, one 'sqlite' registration
clients/treasury/anchor{,_evm}.go (Phase 2 ledger-root anchor) import
luxfi/geth and luxfi/crypto directly, so they are no longer indirect.
go mod tidy result; no version change.
Collapse the treasury's separately-written double-entry SQL onto ledgercore
(github.com/hanzo-fi/ledger) — the SAME engine the ledger's own store uses — so
there is exactly one double-entry implementation across the stack (church of
Rich Hickey: one double-entry value, not three places).
- Reimplement clients/treasury/ledger/sqlstore to back the ledger.Store/ledger.Tx
port with ledgercore instead of hand-rolled treasury_postings SQL. The
accounting truth — every balance and the reserve overdraw guard — is now
ledgercore's (postings -> moves -> balances + hash-chained log, idempotency-key
dedup, WithTx atomic read-then-write). The adapter only maps the treasury's
vocabulary (int64 cents, Kind/Program/Ref key, signed-Posting Entry) onto it.
- KEEP the port/adapter seam: Open()'s signature is unchanged, so treasury.go and
the Formance-HTTP opt-in are untouched — native (ledgercore) stays the default
backend. The engine (ledger.go) and the on-chain Root are UNCHANGED: each Entry
is round-tripped verbatim (as ledgercore transaction metadata), so the Root is
byte-identical to the previous store's, independent of ledgercore's own postings.
- Policy (revenue-share bps) stays in a small side table — it is Hanzo config, not
double-entry accounting, so it does not belong in the shared engine.
- Pin bun to v1.2.9 (replace): ledger-fi floors v1.2.18, which removed
schema.Formatter/NewFormatter/Append that hanzoai/o11y still uses; ledgercore's
compiled closure uses no v1.2.18-only API, so v1.2.9 satisfies both. hanzo-fi/ledger
is pinned to the PR-3 branch commit until it merges.
Tests (all green, incl. -race): overdraw guard, at-most-once payout, snapshot
reconcile, scope isolation, and Tx rollback all pass unchanged against the
ledgercore-backed store. The whole cloud module builds under -mod=readonly, and
the treasury test binary links NO modernc driver (so it does not reintroduce the
"sqlite registered twice" panic).
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The 36963 coreth fee market pins a 25 gwei min base fee, so a legacy tx priced
at base+1 strands the moment the base fee ticks up. anchor_evm.go now submits a
DynamicFeeTx (1 gwei tip floor, 2x-base-fee cap) — proven accepted on-chain as a
type-2 tx.
Adds clients/treasury/cmd/anchorctl: a one-shot in-cluster tool that provisions
the KMS-held signer (key -> KMS, only the address printed), funds it from a
genesis account, deploys contracts/TreasuryAnchor.sol, and can send anchor(bytes32).
Includes the compiled TreasuryAnchor.bin (solc 0.8.26, optimizer 200, cancun).
Deployed live: contract 0x53141dF42DF13Aad0512f2F08c3E3216EEFac5F2, owner = signer
0x703D4227d58d0b6A20BD721c940CED170470f634 (KMS ref hanzo/treasury-anchor/TREASURY_ANCHOR_SIGNER_KEY).
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The cloud registers the "sqlite" driver exactly once in every build mode EXCEPT
a naive `go test -race`: -race forces CGO=1, which links the fork's mattn
"sqlite" (github.com/hanzoai/sqlite) ALONGSIDE the embedded deps that import
modernc directly (ai/base/commerce/o11y/orm), so both register "sqlite" and the
binary panics at init ("sql: Register called twice for driver sqlite") — the
pre-existing failure in clients/{graph,kmssvc,o11y}.
`make test` (CGO=0) and `make test-cgo` (-tags sqlite_purego) already avoid this
by resolving the whole binary to modernc's single registration. This adds the
missing peer for the race detector: `make test-race` runs
`CGO_ENABLED=1 go test -race -tags sqlite_purego ./...` — CGO on for the race
instrumentation, but the fork forced to its pure-Go backend so mattn never
registers and "sqlite" is registered exactly once. The ONE way to race-test the
cloud.
Proof: `go test -race ./clients/o11y/` panics; `go test -race -tags sqlite_purego
./clients/{graph,kmssvc,o11y}/` all pass.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The finance.hanzo.ai + console Finance surfaces render real per-org data
instead of preview stubs. This adds no billing system — it PROJECTS the two
that already exist (the commerce customer wallet + the treasury reserve fund)
into the @hanzo/finance-ui contract (USD cents, optional-safe), scoped to the
validated IAM owner.
clients/billing/finance.go — six commerce-projected reads, reusing this
package's commerceProxy + per-org subject-pinning (one commerce read path):
GET /v1/finance/balance commerce balance (holds -> pendingCents)
GET /v1/finance/credits commerce deposit rows (grants, positive)
GET /v1/finance/usage?range= commerce withdraw rows -> series+lines+total
GET /v1/finance/invoices honest empty (no invoice ledger exists yet)
GET /v1/finance/payment-methods commerce portal, masked to brand+last4
GET /v1/finance/ledger?range= commerce ledger -> signed per-org postings
clients/treasury/treasury.go — GET /v1/finance/treasury reshaped from the
reserve Report into the TreasurySummary shape (reserve/committed/available +
honest Hanzo L1 anchor); the transparency policy rides along additively.
Tenant isolation: org A never sees org B (per-org subject pinned server-side,
client cannot widen scope); payment methods re-masked defensively so a PAN can
never leak. Honest empty/typed shapes where a data source does not exist yet.
Tests: go test -race ./clients/billing/... ./clients/treasury/... green;
go build ./cmd/cloud green.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
* feat(treasury): native double-entry reserve fund + backed-payout seam (#treasury)
The platform's OWN fund/reserve accounting, one layer ABOVE the per-org commerce
credit ledger. A store-agnostic, cloud-decoupled double-entry engine
(clients/treasury/ledger) — the SEED of the native hanzoai/finance central ledger
(the Go replacement for the Formance stack) — plus a Base/SQLite adapter
(ledger/sqlstore) and the cloud client (clients/treasury).
Core (clients/treasury/ledger): accounts + balanced journal entries (Σ postings==0,
refused otherwise), ONE shared fund:reserve pool with per-program payout sinks,
revenue-share policy (bps, one place), and the reserve GUARD — a fund debit that
would overdraw is refused, atomically, so growth-loop payouts are backed capital not
unbounded minting. Zero cloud/zip/SQLite imports; persistence is the Store/Tx port,
so it lifts to hanzoai/finance as a directory move.
Surface: GET /v1/treasury (org transparency), GET /v1/admin/treasury (report +
journal + anchor), POST /v1/admin/treasury/{policy,sweep,seed,anchor} (global-admin).
treasury.Reserve(program,ref,memo,cents) is the ONE seam the 3 loops call: backed →
proceed to credit; not backed → honestly pending; unmounted → passthrough
(backward-safe). Idempotent by ref (at-most-once fund debit). ledger.Root commits the
whole journal for the Hanzo L1 anchor (Phase 2 wires the KMS-signed submit).
Tests (-race, green): double-entry balances, revenue-share accrual + per-period
idempotency, reserve guard (backed→blocked), at-most-once, concurrent no-overdraw,
admin gate, Reserve passthrough+enforced, sqlstore round-trip + tx rollback.
* feat(finance): Formance ledger-of-record backend + backed payouts + scope-aware /v1/finance/*
Adopt Formance as the ledger of record behind a ledger.Backend PORT, without
reimplementing double-entry: two adapters satisfy the port — the native Base/SQLite
engine (offline/default, ships the reserve fund today) and clients/treasury/formance
(a real HTTP client to the Postgres-backed Formance Ledger v2 API: world→fund accrual,
fund→payout debit, 400 INSUFFICIENT_FUND→not-backed=the overdraw guard Formance
enforces, reference→idempotency). Select by FORMANCE_LEDGER_URL — a config flip. Root
computed via a SHARED hash so the L1 anchor is backend-agnostic.
Back the growth-loop payouts: referrals/affiliates/authors now DEBIT the reserve fund
via the ONE treasury.Reserve seam before crediting the recipient wallet — fund down,
wallet up, reconciled. Not backed → honestly pending (referrals) or 402 + VoidPayout
restores pending (affiliates/authors). Idempotent by ref (at-most-once). Unmounted →
passthrough (backward-safe; existing loop suites stay green).
Scope-aware /v1/finance/* — ONE engine, three tenancy surfaces (admin/console/finance
product): tenant derived from IAM, house/reserve locked to global-admin under
/v1/admin/finance/*, per-org callers see ONLY their own org:<tenant>:* accounts.
GET /v1/finance/accounts (per-org; admin ?scope=house|?org=<t>). Storage tiers doc'd:
authoritative OLTP ledger (native/Formance) + ClickHouse OLAP projection over the same
o11y event stream (audit mirror — no second metering pipeline).
Tests (-race, green): Formance adapter (accrual+idempotency, debit guard+replay,
snapshot) via a fake Formance server; scope isolation (per-org never sees house);
backed-payout enforced+blocked+at-most-once; VoidPayout restores pending.
* feat(treasury): Phase 2 — Hanzo L1 (36963) ledger-root anchor (contract + luxfi/geth submit + KMS signer)
Make the off-chain books tamper-evident on the LIVE Hanzo L1 (verified running:
network/chainId 36963, hanzod-0 producing blocks, EVM at network-36963).
- contracts/TreasuryAnchor.sol: minimal immutable witness — owner-gated anchor(bytes32)
appends a timestamped root + emits Anchored; latest()/count for cheap verification.
No upgradeability, no token — one job.
- anchor_evm.go: real luxfi/geth submitter — dial → chainID/nonce/gasPrice → sign a
LegacyTx (anchor(bytes32) call when TREASURY_ANCHOR_CONTRACT set, else a 0-value
self-tx carrying the root) with types.SignTx → send → await receipt → persist. The
signer key is provisioned from KMS (KMSSecret → env TREASURY_ANCHOR_SIGNER_KEY,
ref TREASURY_ANCHOR_SIGNER_KMS_REF) — NEVER plaintext in code/manifest.
- ledger.Root/ComputeRoot: deterministic SHA-256 hash-chain over the whole journal +
reserve, shared by both backends so the anchor is backend-agnostic. A change to any
historical posting changes the root.
- POST /v1/admin/treasury/anchor submits when wired; else returns the root that WOULD
be committed + the EXACT remaining step. GET /v1/admin/treasury shows last anchored
root/tx/block + synced flag. Persisted across restart (treasury_anchor.json).
Honest status: the on-chain submit is COMPLETE + compiling + config-gated but NOT
driven live this pass — the node's external JSON-RPC is unreachable from the build
env and needs an operator to: deploy TreasuryAnchor on 36963, provision the KMS
signer (fund it), set TREASURY_ANCHOR_{RPC_URL,CONTRACT,SIGNER_KEY}. In-cluster the
cloud binary reaches hanzod-rpc-internal:9630, so it's one deploy-config away.
Builds green (cmd/cloud links luxfi/geth); tests -race green; gofmt clean.
---------
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Manage a local hanzo-engine (the `hanzoai` OpenAI + Anthropic model server)
from the canonical `hanzo` CLI:
- install: runs the canonical install.sh / install.ps1 (single source of truth
for platform detection + signature verification — no re-implementation).
- serve: launches the installed binary (`hanzoai --port P run -m MODEL`);
syscall.Exec on Unix so signals + exit code flow through.
- status: probes the local engine, reusing the /v1/models probe that
`hanzo gpu connect --serve-engine` advertises with.
Tests cover wiring, ready/unreachable status, and binary discovery.
Make the off-chain books tamper-evident on the LIVE Hanzo L1 (verified running:
network/chainId 36963, hanzod-0 producing blocks, EVM at network-36963).
- contracts/TreasuryAnchor.sol: minimal immutable witness — owner-gated anchor(bytes32)
appends a timestamped root + emits Anchored; latest()/count for cheap verification.
No upgradeability, no token — one job.
- anchor_evm.go: real luxfi/geth submitter — dial → chainID/nonce/gasPrice → sign a
LegacyTx (anchor(bytes32) call when TREASURY_ANCHOR_CONTRACT set, else a 0-value
self-tx carrying the root) with types.SignTx → send → await receipt → persist. The
signer key is provisioned from KMS (KMSSecret → env TREASURY_ANCHOR_SIGNER_KEY,
ref TREASURY_ANCHOR_SIGNER_KMS_REF) — NEVER plaintext in code/manifest.
- ledger.Root/ComputeRoot: deterministic SHA-256 hash-chain over the whole journal +
reserve, shared by both backends so the anchor is backend-agnostic. A change to any
historical posting changes the root.
- POST /v1/admin/treasury/anchor submits when wired; else returns the root that WOULD
be committed + the EXACT remaining step. GET /v1/admin/treasury shows last anchored
root/tx/block + synced flag. Persisted across restart (treasury_anchor.json).
Honest status: the on-chain submit is COMPLETE + compiling + config-gated but NOT
driven live this pass — the node's external JSON-RPC is unreachable from the build
env and needs an operator to: deploy TreasuryAnchor on 36963, provision the KMS
signer (fund it), set TREASURY_ANCHOR_{RPC_URL,CONTRACT,SIGNER_KEY}. In-cluster the
cloud binary reaches hanzod-rpc-internal:9630, so it's one deploy-config away.
Builds green (cmd/cloud links luxfi/geth); tests -race green; gofmt clean.
`hanzo gpu connect --serve-engine` advertises a local hanzo-engine (the OpenAI +
Anthropic model server on :1234) on the org fleet, alongside the existing
studio.render worker. The worker probes GET {engine-url}/v1/models, publishes the
endpoint + model list in its presence record, and prints (or with --register-provider
POSTs) the /v1/add-provider call that routes api.hanzo.ai model traffic to this GPU as
an OpenAI-compatible (Type=Local) provider.
- cli/gpu.go: --serve-engine/--engine-url/--engine-endpoint/--register-provider;
probeEngine, refreshEngine, engineAdvertisement, capabilities, provider hint;
`hanzo gpu status` shows the engine endpoint.
- clients/visor/fleet.go: byoWorker + fleetRegistration carry capabilities + engine;
GET /v1/fleet/workers advertises the endpoint (additive, omitempty).
- docs/bring-your-gpu.md: Connect (BYO) vs Deploy (cloud) -> engine.serve + studio.render.
- tests: probe/advertise/registration + full stub-cloud round-trip (no model needed).
One fleet, two job types: engine.serve (model serving) + studio.render (diffusion).
Adopt Formance as the ledger of record behind a ledger.Backend PORT, without
reimplementing double-entry: two adapters satisfy the port — the native Base/SQLite
engine (offline/default, ships the reserve fund today) and clients/treasury/formance
(a real HTTP client to the Postgres-backed Formance Ledger v2 API: world→fund accrual,
fund→payout debit, 400 INSUFFICIENT_FUND→not-backed=the overdraw guard Formance
enforces, reference→idempotency). Select by FORMANCE_LEDGER_URL — a config flip. Root
computed via a SHARED hash so the L1 anchor is backend-agnostic.
Back the growth-loop payouts: referrals/affiliates/authors now DEBIT the reserve fund
via the ONE treasury.Reserve seam before crediting the recipient wallet — fund down,
wallet up, reconciled. Not backed → honestly pending (referrals) or 402 + VoidPayout
restores pending (affiliates/authors). Idempotent by ref (at-most-once). Unmounted →
passthrough (backward-safe; existing loop suites stay green).
Scope-aware /v1/finance/* — ONE engine, three tenancy surfaces (admin/console/finance
product): tenant derived from IAM, house/reserve locked to global-admin under
/v1/admin/finance/*, per-org callers see ONLY their own org:<tenant>:* accounts.
GET /v1/finance/accounts (per-org; admin ?scope=house|?org=<t>). Storage tiers doc'd:
authoritative OLTP ledger (native/Formance) + ClickHouse OLAP projection over the same
o11y event stream (audit mirror — no second metering pipeline).
Tests (-race, green): Formance adapter (accrual+idempotency, debit guard+replay,
snapshot) via a fake Formance server; scope isolation (per-org never sees house);
backed-payout enforced+blocked+at-most-once; VoidPayout restores pending.
The platform's OWN fund/reserve accounting, one layer ABOVE the per-org commerce
credit ledger. A store-agnostic, cloud-decoupled double-entry engine
(clients/treasury/ledger) — the SEED of the native hanzoai/finance central ledger
(the Go replacement for the Formance stack) — plus a Base/SQLite adapter
(ledger/sqlstore) and the cloud client (clients/treasury).
Core (clients/treasury/ledger): accounts + balanced journal entries (Σ postings==0,
refused otherwise), ONE shared fund:reserve pool with per-program payout sinks,
revenue-share policy (bps, one place), and the reserve GUARD — a fund debit that
would overdraw is refused, atomically, so growth-loop payouts are backed capital not
unbounded minting. Zero cloud/zip/SQLite imports; persistence is the Store/Tx port,
so it lifts to hanzoai/finance as a directory move.
Surface: GET /v1/treasury (org transparency), GET /v1/admin/treasury (report +
journal + anchor), POST /v1/admin/treasury/{policy,sweep,seed,anchor} (global-admin).
treasury.Reserve(program,ref,memo,cents) is the ONE seam the 3 loops call: backed →
proceed to credit; not backed → honestly pending; unmounted → passthrough
(backward-safe). Idempotent by ref (at-most-once fund debit). ledger.Root commits the
whole journal for the Hanzo L1 anchor (Phase 2 wires the KMS-signed submit).
Tests (-race, green): double-entry balances, revenue-share accrual + per-period
idempotency, reserve guard (backed→blocked), at-most-once, concurrent no-overdraw,
admin gate, Reserve passthrough+enforced, sqlstore round-trip + tx rollback.
* feat(iam): embed IAM in the unified cloud binary as an in-process subsystem
Folds Hanzo IAM -- the identity provider serving hanzo.id (login/authorize/
token/jwks/userinfo, /v1/iam/* admin, OAuth2/OIDC, LDAP/RADIUS) -- into the
unified hanzoai/cloud binary as the LAST binary-consolidation piece
(HIP-0106: "one Go binary embeds IAM + KMS + o11y").
clients/iamsvc wraps IAM's own Beego runtime: iamserver.Init() runs the full
bootstrap without binding a listener, and web.BeeApp.Handlers is mounted
verbatim on cloud's zip.App at every prefix IAM owns (/v1/iam/*,
/.well-known/*, /login/oauth/*, /_/iam/*, /cas/*, /scim/*). No auth logic is
reimplemented -- the same controllers answer, so OAuth/OIDC semantics
(authorize clientId org-resolution, JWT audiences, SuperAdmin owner=="admin",
argon2id password hashing) are preserved byte-for-byte. Registered at order 50
(identity authority, mounts before dependents).
- go.mod: pin hanzoai/iam v1.28.12 -> v1.31.16 (latest; carries the
authorize-login org-resolution fixes #95/#96 the operator SSO chain needs).
- subsystems.go: blank-import clients/iamsvc; IAM no longer "NOT fused in".
Auth-critical middleware interactions verified: /v1/iam/* prices to 0 in
DefaultPrice (ungated -- the M2M /v1/iam/oauth/token mint is never charged);
SanitizeIdentity strips only forgeable X-User-*/X-Org-* headers, never the
Authorization bearer or iam_session_id cookie IAM's session/oauth logic reads.
Activation is STAGED via the enable-list gate: "iam" is NOT added to the live
--enable until IAM config is present in the cloud runtime and the fold is
verified (login/authorize/token/jwks + operator SSO chain). The standalone iam
pod keeps serving hanzo.id via ingress until then.
Build gate: CGO_ENABLED=0 go build ./... && go test . green. clients/iamsvc
tests prove registration (order 50) + full-path preservation through the mount.
* fix(iam): red-review — embed-mode bootstrap, fail-closed, single-replica guard
Addresses the red review of cloud#142 (mount mechanism approved; activation
blocked on standalone-only side effects in the wrapped entrypoint).
1. [HIGH] Embed-mode bootstrap. iamsvc now calls iamserver.InitEmbed (new in
iam v1.31.17) instead of the standalone Init: skips StopOldInstance
(lsof/SIGKILL — panics on distroless, kills a co-resident on shared netns),
skips LDAP/RADIUS listeners (RADIUS binds unmanaged UDP with an empty shared
secret), skips export/os.Exit, binds no listener. Standalone hanzo iam / iamd
is byte-for-byte unchanged (Init delegates to the same shared bootstrap with
every flag on). Also covers [MED] #3 — directory listeners never start
in-process.
4. [MED] Fail-closed, not fail-loud. InitEmbed returns an error (recovers
bootstrap panics); a broken/misconfigured IAM degrades THIS subsystem to a
503 fail-closed on every IAM prefix (mountFailClosed) — every co-resident
subsystem (KMS, o11y) stays up. Mirrors the KMS "no master key -> health-only"
blast-radius isolation.
2. [HIGH] Single-replica enforcement. Embedded IAM uses Beego's process-local
"memory" session store. Config.Validate now REFUSES to boot iam-enabled above
CLOUD_REPLICAS=1 (a real runtime guard, not convention); the helm chart pins
replicas=1 + injects CLOUD_REPLICAS whenever "iam" is in --enable.
5. [MED] Bump verified iam v1.31.16 -> v1.31.17. The slim-JWT change keeps every
claim cloud reads (owner, isAdmin, email, name kept; aud is a registered
claim, untouched) — IdentityMiddleware unaffected. authz v1.10.4 policy-API
swap is IAM-internal (cloud builds green, no direct use). redirect_uri
exact-match + AutoSignin CC-JWT normalization are version-skew CUTOVER gates:
version-match the standalone pod + verify registered redirect_uris are exact
before adding "iam" to the live --enable (runtime-data checklist, not code).
Tests (CGO_ENABLED=0):
- TestIAMEmbedBehindMiddlewareChain — unauth POST /v1/iam/oauth/token, /login,
jwks + /login/oauth/authorize return 2xx through the REAL SanitizeIdentity +
BillingGate chain (never 402/503); forged X-User-IsAdmin is stripped; a priced
control path is denied at zero balance (proves the gate is engaged).
- TestDefaultPriceExemptsIAM — every IAM prefix prices to 0.
- TestValidateIAMSingleReplica — iam + replicas>1 refused; 1/unset/off ok.
- TestMountFailClosed503 — the fail-soft path serves 503 on every IAM prefix.
Build+test green; standalone hanzo iam still links; helm renders replicas=1 for
iam-enabled, replicaCount otherwise. Depends on iam v1.31.17
(hanzoai/iam#feat/iam-embed-entrypoint). STILL STAGED — the standalone iam pod
serves hanzo.id until red GREEN + runtime e2e.
---------
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Addresses the red review of cloud#142 (mount mechanism approved; activation
blocked on standalone-only side effects in the wrapped entrypoint).
1. [HIGH] Embed-mode bootstrap. iamsvc now calls iamserver.InitEmbed (new in
iam v1.31.17) instead of the standalone Init: skips StopOldInstance
(lsof/SIGKILL — panics on distroless, kills a co-resident on shared netns),
skips LDAP/RADIUS listeners (RADIUS binds unmanaged UDP with an empty shared
secret), skips export/os.Exit, binds no listener. Standalone hanzo iam / iamd
is byte-for-byte unchanged (Init delegates to the same shared bootstrap with
every flag on). Also covers [MED] #3 — directory listeners never start
in-process.
4. [MED] Fail-closed, not fail-loud. InitEmbed returns an error (recovers
bootstrap panics); a broken/misconfigured IAM degrades THIS subsystem to a
503 fail-closed on every IAM prefix (mountFailClosed) — every co-resident
subsystem (KMS, o11y) stays up. Mirrors the KMS "no master key -> health-only"
blast-radius isolation.
2. [HIGH] Single-replica enforcement. Embedded IAM uses Beego's process-local
"memory" session store. Config.Validate now REFUSES to boot iam-enabled above
CLOUD_REPLICAS=1 (a real runtime guard, not convention); the helm chart pins
replicas=1 + injects CLOUD_REPLICAS whenever "iam" is in --enable.
5. [MED] Bump verified iam v1.31.16 -> v1.31.17. The slim-JWT change keeps every
claim cloud reads (owner, isAdmin, email, name kept; aud is a registered
claim, untouched) — IdentityMiddleware unaffected. authz v1.10.4 policy-API
swap is IAM-internal (cloud builds green, no direct use). redirect_uri
exact-match + AutoSignin CC-JWT normalization are version-skew CUTOVER gates:
version-match the standalone pod + verify registered redirect_uris are exact
before adding "iam" to the live --enable (runtime-data checklist, not code).
Tests (CGO_ENABLED=0):
- TestIAMEmbedBehindMiddlewareChain — unauth POST /v1/iam/oauth/token, /login,
jwks + /login/oauth/authorize return 2xx through the REAL SanitizeIdentity +
BillingGate chain (never 402/503); forged X-User-IsAdmin is stripped; a priced
control path is denied at zero balance (proves the gate is engaged).
- TestDefaultPriceExemptsIAM — every IAM prefix prices to 0.
- TestValidateIAMSingleReplica — iam + replicas>1 refused; 1/unset/off ok.
- TestMountFailClosed503 — the fail-soft path serves 503 on every IAM prefix.
Build+test green; standalone hanzo iam still links; helm renders replicas=1 for
iam-enabled, replicaCount otherwise. Depends on iam v1.31.17
(hanzoai/iam#feat/iam-embed-entrypoint). STILL STAGED — the standalone iam pod
serves hanzo.id until red GREEN + runtime e2e.
The ZAP-native trace exporter marshaled its payload via the generated
otlp/collector/trace/v1.ExportTraceServiceRequest, whose sibling
trace_service_grpc.pb.go (no build tag) drags google.golang.org/grpc into the
graph — contradicting the exporter's own contract (ZAP wire, never gRPC).
Encode the ExportTraceServiceRequest envelope directly from the grpc-free trace
messages with protowire: it is a single 'repeated ResourceSpans resource_spans
= 1', so appending each ResourceSpans under field 1 is byte-identical to the
generated marshaler (proven by the existing round-trip test, which still decodes
with the canonical collector type). go list -deps ./zaptrace now shows no grpc.
Hanzo services speak ZAP/HTTP/WS, never gRPC.
Caveat: the cloud module still pulls google.golang.org/grpc transitively via
hanzoai/ai (sibling-owned), hanzoai/o11y (embedded SigNoz — intrinsically an
OTLP/gRPC collector) and hanzoai/base (GCS gRPC transport). Not removable by a
cloud-local change; tracked separately. go.sum: incidental tidy prune of stale
vfs/age checksums.
Folds Hanzo IAM -- the identity provider serving hanzo.id (login/authorize/
token/jwks/userinfo, /v1/iam/* admin, OAuth2/OIDC, LDAP/RADIUS) -- into the
unified hanzoai/cloud binary as the LAST binary-consolidation piece
(HIP-0106: "one Go binary embeds IAM + KMS + o11y").
clients/iamsvc wraps IAM's own Beego runtime: iamserver.Init() runs the full
bootstrap without binding a listener, and web.BeeApp.Handlers is mounted
verbatim on cloud's zip.App at every prefix IAM owns (/v1/iam/*,
/.well-known/*, /login/oauth/*, /_/iam/*, /cas/*, /scim/*). No auth logic is
reimplemented -- the same controllers answer, so OAuth/OIDC semantics
(authorize clientId org-resolution, JWT audiences, SuperAdmin owner=="admin",
argon2id password hashing) are preserved byte-for-byte. Registered at order 50
(identity authority, mounts before dependents).
- go.mod: pin hanzoai/iam v1.28.12 -> v1.31.16 (latest; carries the
authorize-login org-resolution fixes #95/#96 the operator SSO chain needs).
- subsystems.go: blank-import clients/iamsvc; IAM no longer "NOT fused in".
Auth-critical middleware interactions verified: /v1/iam/* prices to 0 in
DefaultPrice (ungated -- the M2M /v1/iam/oauth/token mint is never charged);
SanitizeIdentity strips only forgeable X-User-*/X-Org-* headers, never the
Authorization bearer or iam_session_id cookie IAM's session/oauth logic reads.
Activation is STAGED via the enable-list gate: "iam" is NOT added to the live
--enable until IAM config is present in the cloud runtime and the fold is
verified (login/authorize/token/jwks + operator SSO chain). The standalone iam
pod keeps serving hanzo.id via ingress until then.
Build gate: CGO_ENABLED=0 go build ./... && go test . green. clients/iamsvc
tests prove registration (order 50) + full-path preservation through the mount.
The THIRD growth loop next to referrals (one-time credit) and affiliates
(partner commission): pays open-source AUTHORS a royalty on the metered platform
spend of orgs who DEPLOY their projects on Hanzo. Mirrors clients/affiliates
exactly — one SQLite store, server-side tenant isolation, one Mount (HIP-0106),
the SAME commerce ledger path (a credits payout is a grant, tag grant:author),
and an at-most-once accrual latch.
Flow: connect GitHub (IAM-linked account or supplied login) → verify repo
ownership (OAuth admin-check OR a hanzo.json verify-code file) → a deploy of a
verified author repo by ANY org is recorded (provenance) → sweep accrues 5% of
that org's month-to-date spend, at-most-once per (author, deploying-org, period),
self-deploys excluded → staff pay out as credits (real grant) or cash (record-only),
never exceeding pending.
Surface: GET /v1/authors, POST /v1/authors/{connect,repos/verify,deploys/record};
GET /v1/admin/authors, POST /v1/admin/authors/{sweep,:id/approve,:id/suspend,:id/payout}.
10 tests, all -race green: repo canonicalization, both verify methods, deploy
attribution + idempotency, spend×share accrual + at-most-once, lazy dashboard
sweep, credits-one-grant/cash-record-only/pending-guard payout, admin gate.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Mirrors clients/referrals: one SQLite store, server-side tenant isolation, one
HIP-0106 Mount, admin surface global-admin-gated + enveloped for the console
proxy. Affiliates earn an ONGOING commission (default 20%) on the metered spend
of the customers they refer — the recurring, partner-revenue growth loop beside
referrals' one-time both-sides credit.
- apply (org) -> status applied; staff approve mints the code (vanity opt-in,
uniqueness-enforced, else a derived slug) + sets the rate.
- attribute (?aff capture) records referred_org->affiliate (first-touch, one per
referred org, self blocked; approved affiliates only).
- accrual sweep: commission = referred org spend this period x rate, latched
at-most-once per (affiliate, referred_org, period) in one txn; also lazy on the
affiliate's own dashboard read.
- payout: a credits method issues a commerce grant (tag grant:affiliate); cash
methods are record-only; can never exceed pending (accrued - paid), reserved
atomically before any grant.
Tests (go test -race, 9 green): apply->approve, vanity uniqueness (409),
accrual = spend x rate, idempotent-per-period sweep, payout-as-credits issues one
grant + cash record-only + pending guard, admin gate 403, attribution
self/unknown/first-touch, Mount.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
* feat(referrals): native /v1/referrals viral loop over the commerce ledger
Per-org referral program mirroring clients/crm's structure (one SQLite store,
server-side tenant isolation, HIP-0106 Mount). Grants promo credit through the
SAME commerce deposit path as clients/admin.grantCredit (trial/Credit bucket,
tag grant:referral).
- Stable deterministic referral code per org (base32 of a hash of the org id) +
a white-labeled ?ref link; persisted directory for O(1) reverse lookup.
- POST /v1/referrals/claim: record referrer<->referee (referee = validated
caller), status signed_up. Self-referral blocked, one-per-referee idempotent
(first-touch wins).
- Qualify signal = referee metered spend (honest 'actually used the product').
On qualify, grant BOTH sides: referrer +$10, referee +$5. At-most-once via a
credited_at latch — no sweep and no concurrent read can double-pay.
- Trigger: lazy on the referrer's GET /v1/referrals + POST /v1/admin/referrals/
sweep (cron path). GET /v1/admin/referrals directory, both global-admin gated.
- Constants (bonus amounts + ledger tag) in one place. Commerce behind an
interface for testable double-grant/idempotency proofs.
Tests: code derivation, self-ref block, idempotent claim, qualify->double-grant
with balances moving through the (fake) ledger, at-most-once idempotency, lazy
qualify on read, admin gate + directory, real Mount. All green (go test -race).
* feat(referrals): envelope the /v1/admin/referrals surface for the console admin proxy
---------
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud's internal/org was the ORIGIN of the HA-SQLite machinery; it's now promoted to the
shared hanzoai/vfs/replica lib that every service adopts. Delete the duplicated impls
(replica.go Replicator+Store+DB+DBPath, owner.go Member+Owner+IsOwner+Replicas+HRW) and
re-export them as aliases (shared.go). cloud-specific pieces stay: membership.go (live IAM
source), cipher.go (KMS envelope — already satisfies replica.Cipher), vfsstore.go (Store over
deps.VFS, now using the exported replica.Version). One and one way: ONE Replicator + election,
in vfs/replica, used by cloud AND visor. Builds + org tests green (vfs v0.6.2).
Zen models were capped at the 4096 fallback in getContextLength, so every
console chat (grounded assistant ~4190-token system prompt) 402'd
'exceeds maximum token count: 4096'. v1.800.9 special-cases the zen* prefix
to 131072. Fixes the P0 console-chat gate.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The embedded o11y runtime's OTel instrumentation defaults to a Prometheus pull
reader bound to 0.0.0.0:9090 (pkg/instrumentation) — the SAME port as cloud's
health listener (CLOUD_HEALTH_LISTEN=:9090). Activating the embed therefore made
the whole cloud process crash-loop with 'listen tcp :9090: bind: address already
in use' (verified on the canary), taking down all of api.hanzo.ai — a listener the
standalone o11y pod never contended for.
buildEmbeddedHandler now defaults O11Y_INSTRUMENTATION_METRICS_ENABLED=false (via
setenvDefault, operator-overridable to a free port) before construction. Cloud owns
process-level observability (exports its own OTel telemetry), so the embed serves
/v1/o11y in-process without a second metrics listener. Extracted the env defaults
into applyEmbedEnvDefaults + TDD (guard + operator-override).
Verified on the cloud-unified-canary: with this default the .104 embed goes Ready
and serves /v1/o11y in-process (health 200); without it the pod crash-loops on :9090.
CGO=0 go build/test green.
Refactor clients/o11y/embed.go onto o11y v1.5.0's shared builder community.NewServer
+ community.NewConfig — the EXACT construction the standalone o11y pod runs — so
the in-process runtime cannot drift from the pod's auth (pkg/identn/iamidentn,
Hanzo IAM gateway-header identity). Collapses the duplicated ~70-line signoz.New
factory list (drift risk) to one call. Enable signal now reads the flat operator
knob O11Y_DATASTORE_DSN (what the pod sets), falling back to the structured
O11Y_TELEMETRYSTORE_DATASTORE_DSN.
Exempt liveness/readiness paths from gate(): the runtime serves them without
identity (k8s probes pass that way), so gating them only breaks unauthenticated
health probes (admin System Health CLOUD_O11Y_HEALTH_URL, the external o11y.*
hosts) without protecting anything. Data routes stay gated (RED forge test still
403s /v1/o11y/api/v1/query_range).
go.mod: o11y v1.4.1 -> v1.5.0 (identical go.mod hash — no new transitive deps).
Build-gate: CGO_ENABLED=0 go build ./... = 0, go test . = ok, go test ./clients/o11y = ok.
The #133 embed pinned o11y v1.3.13, whose runtime authenticates via o11y-native
JWT (tokenizer.GetIdentity on the Authorization bearer). The live gateway-header
traffic the standalone o11y:0.2.0 pod serves — identity injected as X-Org-Id/
X-User-Id/X-User-Email by the gateway — would 401 against that. So activating the
v1.3.13 embed could not replace the pod.
This repoints the embed to the MAIN o11y line (v1.4.1), which resolves identity
through the IdentN resolver's iamidentn provider (default-enabled) from those
gateway session headers, with iamauthz (Hanzo IAM Casbin) for authorization —
the SAME auth model as the running pod. Gateway-header traffic authenticates
(200), not 401.
- clients/o11y/embed.go: build the runtime via pkg/signoz.New with the SAME
provider factories the standalone cmd/community server uses (noop zeus,
licensing, gateway, auditor, meterreporter; iamauthz; ClickHouse
telemetrystore; sqlite sqlstore; IdentN to iamidentn), then app.NewServer to
server.PublicHandler (new accessor, o11y v1.4.1). runtime.Start runs the
registry background services (incl. the ruler/alert-rule-manager)
non-blocking; we never call server.Start (cloud owns its HTTP listeners; OpAMP
stays out-of-process). Gate/proxy-fallback structure (clients/o11y/o11y.go) is
unchanged: still O11Y_TELEMETRYSTORE_DATASTORE_DSN-gated, still fail-soft to
the reverse proxy.
- go.mod: hanzoai/o11y v1.3.13 to v1.4.1 (main line, iamidentn). Drop the stale
replace prometheus/alertmanager to hanzoai/alertmanager v0.28.2 — it forced
o11y's code onto the old fork whose api/v2 returns hanzoai/common types that
clash with o11y v1.4.1's upstream prometheus/common structs. o11y v1.4.1 (and
the pod) build against upstream prometheus/alertmanager v0.31.1; cloud has no
direct alertmanager import, so it now matches.
Telemetry backend (ClickHouse datastore StatefulSet, cluster insights) is
untouched — the embedded runtime queries it over ClickHouse-native :9000.
Build-gate: CGO_ENABLED=0 go build ./... OK; go test ./clients/o11y/... OK; vet OK.
v1.49.0 adds the durable workflow primitives social-orchestrator needs to run
on cloud's embedded gated engine (ServeGated :9999): signal-to-running-workflow
re-dispatch, continueAsNew, startChild, typed search attributes, workflowId
conflict policy, and the signalWithStart wire fix. No cloud code change — the
embedded engine + gated listener pick up the fixes on rebuild.
Constructs the ONE hanzoai/o11y runtime IN-PROCESS (clients/o11y/embed.go) —
the SAME bootstrap the standalone cmd/server runs (o11y.New with its provider
factories -> app.NewServer -> server.PublicHandler) — and installs it via
o11y.SetHandler, so /v1/o11y/* is served by THIS binary against the ClickHouse
`datastore` (StatefulSet, cluster insights) instead of reverse-proxying a
standalone o11y Deployment. The standalone o11y pod can now retire; the
ClickHouse datastore stays as the telemetry backend.
- clients/o11y/embed.go: buildEmbeddedHandler wires telemetrystore (ClickHouse/
datastore), sqlstore (sqlite under cloud's data root), querier, dashboards,
alerts; starts the registry services + the alert rule manager (StartBackground).
Enabled by O11Y_TELEMETRYSTORE_DATASTORE_DSN (the DSN is the one knob).
- o11y.go Register callback: prefer the in-process runtime; fall back to the
reverse proxy when the embed is disabled (no DSN) or fails to init — fail-soft,
zero downtime. Proxy handler + gate + tests retained for the fallback path.
- Bump hanzoai/o11y v1.3.12 -> v1.3.13 (adds Server.PublicHandler + StartBackground).
- Drop the stale `replace gorilla/mux => containous/mux`: it was a copied Traefik
replace block; Traefik is not in the graph and nothing calls the containous API,
but the fork lacks mux.MiddlewareFunc that o11y's otelmux needs. Standard
gorilla/mux v1.8.1 satisfies every consumer.
Deferred (reported, not faked): OpAMP collector management (a second websocket
listener) is not started in-process — telemetry ingest continues on the existing
collector->datastore path. Build/test gate is CGO_ENABLED=0 (as prod ships): o11y
+ hanzoai/sqlite resolve to a single modernc sqlite driver registration.
Consolidation: kill the standalone tasksd pod by running its consumers on cloud's
in-process embedded engine. After Embed wires the loopback (ungated, in-process
ai-ingest) listener, call emb.ServeGated(ctx, 9999, validator) to expose the SAME
engine cluster-wide under mandatory identity gating.
RequireIdentity: every request on :9999 must carry an IAM auth_token, validated
against {IAMIssuer}/v1/iam/.well-known/jwks (HIP-0111) and org-scoped to its owner --
the same trust anchor as the HTTP SanitizeIdentity boundary. The loopback dialer for
ai-ingest is untouched (127.0.0.1:19999, ungated, cloud's own trust boundary).
Fail-soft: a missing IAMIssuer or a bind failure logs and leaves the gated surface
down without disturbing ai-ingest. 9999 mirrors the port the retired tasksd exposed,
so a consumer repoint changes only the host (tasks.hanzo.svc -> cloud.hanzo.svc).
Depends on hanzoai/tasks#8 (ServeGated + identity over ZAP). Pinned here to that
branch's commit; repin to the tagged release once #8 merges. universe adds the :9999
Service port.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Consolidation: kill the standalone tasksd pod by running its consumers on cloud's
in-process embedded engine. After Embed wires the loopback (ungated, in-process
ai-ingest) listener, call emb.ServeGated(ctx, 9999, validator) to expose the SAME
engine cluster-wide under mandatory identity gating.
RequireIdentity: every request on :9999 must carry an IAM auth_token, validated
against {IAMIssuer}/v1/iam/.well-known/jwks (HIP-0111) and org-scoped to its owner --
the same trust anchor as the HTTP SanitizeIdentity boundary. The loopback dialer for
ai-ingest is untouched (127.0.0.1:19999, ungated, cloud's own trust boundary).
Fail-soft: a missing IAMIssuer or a bind failure logs and leaves the gated surface
down without disturbing ai-ingest. 9999 mirrors the port the retired tasksd exposed,
so a consumer repoint changes only the host (tasks.hanzo.svc -> cloud.hanzo.svc).
Depends on hanzoai/tasks#8 (ServeGated + identity over ZAP). Pinned here to that
branch's commit; repin to the tagged release once #8 merges. universe adds the :9999
Service port.
HIGH-2: principal.ValidatedProject(c) (project, validated) — returns false
today (X-Project-Id is a caller-chosen label, not claim-bound), so the edge
gate + resource meter pass ProjectValidated=false and commerce degrades
project-scoped hard caps to soft. ONE lever to harden when IAM mints a
project claim. MED-4: DenyResource renders ErrSpendCapExceeded -> 402
spend_cap_exceeded (was 503). INFO-7: canonicalService unifies the edge
service label with the resource provider (ml/visor->compute, agents->agent,
security->security.scan) so a cap binds on both surfaces. Bump metering
v0.1.3 -> v0.1.4. Tests green (incl DenyResource spend_cap).
Adds TEAM_PUBLIC_URL / PUBLIC_ORIGIN config: callbackOrigin() returns the
configured public origin (e.g. https://hanzo.team) for the OAuth redirect_uri
instead of the request Host, so cloud emits the registered public callback even
behind the gateway (where the request Host is the internal cluster service).
Unset = unchanged (falls back to originOf). Lets hanzo.team route through the
gateway UNIFORMLY like api.hanzo.ai — removes the need for the temporary
direct-to-cloud edge route.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
BillingGate uses metering.AuthorizeVerdict (funds+cap, one round trip):
renders a distinct 402 spend_cap_exceeded (scope/cap/spent) and sets
X-Spend-Warn at the soft threshold; gates on the request price. New ONE
ScopeRateLimit middleware composes zip/middleware.RateLimit per-scope
(org/project/service), dynamic rpm from commerce (short-TTL cache,
fail-open), 429 + X-RateLimit-* — wired after identity, before billing.
ResourceMeter threads project + service(=provider) so resource creation
is scope-gated too. Scope always from the validated principal. Bump
metering v0.1.2 -> v0.1.3. Money-path tests green (402/warn/429/isolation).
pickVFSClient returns an S3-backed types.VFSClient (clients/s3vfs.go) when
S3_ADMIN_ACCESS_KEY/SECRET_KEY are set, else DisabledVFS (R-7 fail-closed
preserved). Reuses the SAME s3admin.Admin construction as clients/s3 (DRY, one
credential path). Put/Get/Delete over the shared 'team-blobs' bucket, per-tenant
key prefix (files.go builds team/blobs/<verified-org>/<ws>/<blobId>). S3 NoSuchKey
maps to types.ErrBlobNotFound (honest 404/idempotent-204); any other S3 error →
502 fail-closed (never a dishonest 404). Bucket create-if-absent self-heals a boot
blip. Red-reviewed SHIP. This is the repoint gate: hanzo.team avatars/attachments
now work off cloud.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Ports hanzoai/team-go into the unified cloud binary as a zip-native subsystem
(order 138, /v1/team/*): account (IAM OAuth bridge + workspaces/members on SQLite),
transactor (Huly wire over wsx, serverVersion 0.6.0 preserved), bots-as-members
(in-process agents.ListForOrg → Employees, removal-reconcile), files (FrontStorage
contract, org+workspace-membership scoped, byte-derived content-type allow-list).
Security (Red-reviewed, all closed): fail-closed SERVER_SECRET degrade-health-only
(never crashes the binary/CI smoke-boot), token exp/nbf, seg() traversal guard,
setCookie verify, cross-tenant blob isolation, VFSClient.Delete fail-closed (deps.VFS
never nil, R-7). Supersedes the stale clients/team a parallel branch swept onto main.
Real deps.VFS wiring (avatars) follows in the next patch (.97) before the hanzo.team
front repoint, so nothing regresses. Migration + repoint + rip of the standalone
team-go Deployment are the remaining cutover steps.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
- POST /v1/admin/customers/:org/credit gains `source` (trial|prepaid). A staff
comp defaults to TRIAL (non-cash Credit bucket, grant:admin tag → billing/bucket
DepositKind Credit); only explicit "prepaid" mints real money (admin-grant tag →
Prepaid). Fail-closed: unknown→trial, so a comp never silently becomes payout-able
cash. source recorded in the audit before/after + response.
- grantCredit refactored to a shared applyGrant core (ONE credit-write path).
- NEW GET /v1/admin/grants — the credit-grant ledger across all orgs, projected
from the tamper-evident audit trail (action admin.customer.credit): org, amount,
source, reason, staff actor, date, txid, result. Honest-empty without a local
audit store.
- NEW POST /v1/admin/grants — issue a grant to any org from the operator Grants
view (org in body), funneled through the SAME applyGrant core.
- Both global-admin gated (s.guard). grantTag unit-tested.
go build/vet/test ./clients/admin green.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
- POST /v1/admin/customers/:org/credit gains `source` (trial|prepaid). A staff
comp defaults to TRIAL (non-cash Credit bucket, grant:admin tag → billing/bucket
DepositKind Credit); only explicit "prepaid" mints real money (admin-grant tag →
Prepaid). Fail-closed: unknown→trial, so a comp never silently becomes payout-able
cash. source recorded in the audit before/after + response.
- grantCredit refactored to a shared applyGrant core (ONE credit-write path).
- NEW GET /v1/admin/grants — the credit-grant ledger across all orgs, projected
from the tamper-evident audit trail (action admin.customer.credit): org, amount,
source, reason, staff actor, date, txid, result. Honest-empty without a local
audit store.
- NEW POST /v1/admin/grants — issue a grant to any org from the operator Grants
view (org in body), funneled through the SAME applyGrant core.
- Both global-admin gated (s.guard). grantTag unit-tested.
go build/vet/test ./clients/admin green.
Align cloud to the target: KMS is the embedded luxfi/kms (clients/kms) alongside
embedded IAM; the last external hanzoai/kms dependency is removed.
- go.mod: bump github.com/luxfi/kms v1.11.6 → v1.11.8 (match the deployed image);
remove github.com/hanzoai/kms/sdk/go v1.1.1; go mod tidy.
- clients/mpcseal (NEW): the minimal client-side-CEK sealing client for the
SEPARATE luxfi/mpc node ring — a faithful, behavior-identical inline of the
subset of the former hanzoai/kms/sdk/go that clients/fleet + clients/provisioning
use (NewClient/Unlock/Set/Get/Delete + Argon2id→HKDF→AES-256-GCM). luxfi/kms has
no drop-in equivalent (its pkg/ is server/ZAP/store, not a Vault client), so
inlining the used subset is the minimal correct change that removes the external
dep without altering the wire protocol or trust model. Drops the HPKE Wrap/Unwrap
the callers never used.
- clients/{fleet,provisioning}: swap import path only; call sites untouched.
- clients/kmssvc/login.go + docs/consolidation.md: comment/table refs → luxfi.
Verified: go build ./... = 0, go vet = 0, gofmt clean, clients/provisioning tests
pass. go.mod + go.sum carry zero hanzoai/kms references.
Follow-up (separate, tested change): fold fleet/provisioning sealing into cloud's
embedded deps.KMS once types.KMSClient gains Delete + verified against the live MPC
ring — one KMS surface. Once the universe KMS-collapse PR merges, the deprecated
hanzoai/kms repo/package can be archived.
The gateway now mints X-Project-Id (an org SUB-SCOPE) alongside X-Org-Id.
Thread it through the keyed surfaces, backward-compatibly — the default
project ("default", or an absent header) resolves to today's exact keys,
so existing single-project tenants are byte-identical.
- principal.Project(c): the ONE read accessor, mirroring c.Org() (zero-copy
header read, cloned on retain). Defaults to DefaultProject when the header
is empty. principal.DefaultProject / IsDefaultProject own the default-scope
semantics in one place (shared contract value with iamauth.DefaultProject).
- fleet: registry refs shard by project via the ONE scopeRef seam —
"<org>/fleet/clusters" for the default project, "<org>/<project>/fleet/
clusters" for a non-default one (index, sealed kubeconfig, cache key).
- ml: tenant namespace is "ml-<org>" for the default project and
"ml-<org>-<project>" for a non-default one; both org and project are
validated against strict DNS-label regexes (no lossy fold) and the composed
label is length-checked against the 63-char ceiling, keeping the
(org, project) -> namespace map injective. A hanzo.ai/project attribution
label is stamped for non-default projects.
- visor BYO fleet + ml federation resolve project via principal.Project.
Billing stays keyed on the paying org (a project has no separate prepaid
balance); project is isolation + attribution, not a billing key.
luxfi/age v1.5.0 was upstream-retagged (transient files GC'd from the tag
tree), so the tag's zip content on the origin no longer matches the h1 hash
recorded in go.sum. Cold-cache builds (fresh CI, empty GOMODCACHE) fail with:
verifying github.com/luxfi/age@v1.5.0: checksum mismatch
SECURITY ERROR
v1.5.1 dereferences the same commit, is immutable, and is sum.golang.org
verified (h1:Gj8iHMMi0lGkKT/mlXV2HVBr2m3vt2v0eKVsTMTtAQM=). Surgical: age
require + go.sum only. go mod verify clean.
* feat(crm): startup-program applications resource (intake + AI screen + pipeline)
Public unauthenticated intake POST /v1/crm/applications (rate-limited + honeypot)
writes a dedicated crm_applications record (all fields in metadata JSON), a
best-effort CRM Company+Contact projection, and kicks off an AI screen via the
gateway (score / tier1 / suggested credits / summary / draft reply) that
auto-advances applied->screened. Staff GET/PATCH drive a stage machine
(applied->screened->qualified->credits-offered->onboarded, +rejected w/ reason).
Non-fatal if the LLM is unavailable.
* test(crm): startup applications — intake, honeypot, idempotency, AI screen, stage machine
10 tests: public intake creates application+CRM projection with all fields in
metadata; honeypot drop; validation; idempotent resubmit; end-to-end AI screen
with a fake gateway (score/tier1/credits/reply + auto-advance applied->screened);
non-fatal screen on gateway error; staff PATCH stage machine (advance/skip-block/
reject-requires-reason); pure canTransition + parseScreen + detectTier1.
---------
Co-authored-by: hanzo-dev <dev@hanzo.ai>
PR #124 embedded the last-known-good admin-tasks build as a de-risking
fallback. This replaces it with a FRESH build of current admin-tasks HEAD,
built from source in a combined gui+admin workspace (hanzogui@7.3.x +
@hanzogui/admin@7.3.0 workspace-linked; @hanzogui@7.3.x is unpublished, so it
must build inside that workspace — see clients/tasksvc/ui/README.md).
Same contract (base=/_/tasks/, api=/v1/tasks); full component parity
(namespaces/workflows/schedules/batches/deployments/activities/nexus/history).
Embed tests still assert the real bundle (not the placeholder).
One and one way: BYO k8s / BYO-GPU / bare-metal attach now lives on the SAME fleet
surface as managed clusters (visor /v1/clusters), not a parallel /v1/ml/clusters.
- clients/fleet: the shared per-org BYO-cluster Registry — kubeconfig sealed in the
org's KMS, validated by reaching the cluster (node + nvidia/amd GPU inventory),
tenant-scoped by the ZAP-propagated X-Org-Id. ONE source of truth.
- visor: POST /v1/clusters (attach) + DELETE /v1/clusters/:id (detach), and BYO
clusters MERGE into GET /v1/clusters beside managed ones. Nominal management fee
(rides the compute-fee config — no bespoke env var; customer brings the compute).
- ml: deleted the parallel /v1/ml/clusters; dynForOrg federates ML serving onto the
org's registered cluster via the shared registry (home client when none).
Builds + vet + tests green.
clients/tasksvc served /_/tasks from github.com/hanzoai/tasks/ui, whose
ui/dist is an empty 'No UI build present' placeholder — so tasks.hanzo.ai
still routed to the standalone tasks-ui pod (a Temporal-Web-UI fork).
cloud is the ONE process that serves tasks.hanzo.ai (durable.go's embedded
engine + /v1/tasks surface), so cloud now owns the UI embed too: a local
clients/tasksvc/ui package bakes the real admin-tasks SPA build (base=/_/tasks/,
api=/v1/tasks) into the binary via //go:embed. One binary, one origin, the
real UI — which lets the tasks-ui Deployment/Service/CR be retired.
Tests prove the embedded bundle is the real SPA (not the placeholder), the
SPA deep-link fallback, immutable asset caching, and GET-only.
Companion security test to the observe subsystem: proves a /v1/o11y/logs
request lands on the org-scoped handler (order 44), never falling through to
the unscoped hanzoai/o11y reverse-proxy wildcard (order 70) that would bypass
tenant scoping (attack #4).
Validated the handler SQL against the live signoz_traces schema: response_status_code
is LowCardinality(String), so a raw >= 500 raises NO_COMMON_TYPE and asInt64 on it
yields 0. Wrap with toInt32OrZero() in the RED errs count and the request-log status,
matching the verified live query (real per-org buckets returned).
The fe3a5fd agents-metering refactor split MeterUsage out of Meter and
dropped the Model:kind write, so EVERY per-product debit (functions/invoke,
s3/op, provisioning, ml, tracker, automations, security) recorded an empty
model — losing per-item revenue attribution in the commerce ledger. Restore
the one-place mapping in Meter (all 8 resource callers flow through it).
Also fix two stale test doubles that read the retired X-IAM-Org-Id header;
commerce reads X-Org-Id only (same $0-revenue class as the admin.go fix), so
they saw an empty org. Full suite: 58 ok, 0 fail.
Register a machine's GPU into the cloud fleet with one command and see it on
the console's existing Machines + GPUs pages, tagged provider=byo.
- clients/visor: a BYO worker is a heartbeating presence activity in the org's
`fleet` tasks namespace (cloud.EmbeddedTasks). fleet.go reads it and folds it
into the SAME machineView/gpuView the console renders (provider=byo,
location=on-prem, gpu model + VRAM, online/offline by heartbeat), plus a raw
GET /v1/fleet/workers. /v1/machines and /v1/gpus union Visor's inventory with
the BYO workers and degrade gracefully (BYO stays visible if Visor is down).
- cli: `hanzo gpu connect|status|disconnect` — reuses the `hanzo login`
IAM token (org from its claims; token auto-refresh), detects GPUs via
nvidia-smi, registers + heartbeats the fleet presence record, and runs an
outbound worker loop claiming from `gpu-jobs` (pluggable handlers: echo,
studio.render→local ComfyUI). --daemon installs a systemd --user unit.
- go.mod: hanzoai/tasks v1.46.0 → v1.47.0 (the claim + lease-reaper surface).
E2E (z@hanzo.ai): connect → GB10 'spark' shows provider=byo on
/v1/fleet/workers + /v1/machines + /v1/gpus → echo job claimed + completed.
launchBot bound an agent *name* but never created that agent, so
messageBot's in-process run (/v1/agents/:agent/run -> Resolve) 404'd
"agent not found" — a launched bot could not be messaged.
launchBot now create-if-absent's the bound agent via the SAME
POST /v1/agents the console uses (one create path, forwarding the
caller's validated identity -> org-scoped, IDOR-safe), BEFORE launching
the metered machine so a bad request (e.g. a non-catalog model) 400s
before anything is provisioned. Idempotent: an existing agent (409) is
reused. An omitted model takes the deployment default
(deps.AIDefaultModel, a valid catalog model) threaded from config — no
hardcoded model id.
Also add create/update-time model validation: a client-supplied model
outside the gateway's served catalog is a clean 400 (via the optional
types.ModelLister the real gateway client implements) instead of a
confusing run-time 502; fail-open when the catalog can't be enumerated.
Tests: agents model-validation + default (real store); httpAI.Models
against a fake gateway; visor launch->auto-create->message->resolve E2E
incl. the before/after 404->200 gap proof, idempotency, and bad-model
fail-fast (no machine provisioned).
notify's send surface now reads provider credentials EXCLUSIVELY from cloud's
embedded KMS (cloud.Deps.KMS) at the org-scoped, rotatable ref
orgs/<org>/notify/<svc>/<key> — the same /orgs/<org> namespace
clients/integrations uses, so a cred is seedable + rotatable via
POST /v1/kms/orgs/:org/secrets with no operator-injected env Secret and no
restart. The org is the VALIDATED principal's tenant, never a client header.
Removes the env-first fallback (envCreds/envFirst + the os import): no secret
is ever read from the environment, hard-coded, or logged. A missing key leaves
the value empty and constructProvider fails closed.
Tests rewritten to inject a fake KMS (no env), plus a per-org isolation test
and a regression that creds() ignores the legacy TWILIO_* env entirely.
Pulls hanzoai/ai#71: the RAG default embedder (object/init.go seed) now points at
the Hanzo gateway (CLOUD_AI_BASE_URL / CLOUD_AI_API_KEY, model text-embedding-qwen3)
instead of an empty ProviderUrl that hit api.openai.com directly. A server-side
embed to api.openai.com from in-cluster crawled ~180-210s and then failed, so RAG
ingest (/v1/rag/embed) hung AND the Qdrant vector collection was never created
(writeDocsToVector sample embed timed out before ensureVectorCollection ran).
Gateway embeddings are <1s (proven live). The seed self-heals an existing
api.openai.com-direct default to the gateway on boot, so this deploy converges the
live default-embed provider with no manual console repoint.
The console (console2 WebSearch module) reaches search through the /cloud proxy
with a signed-in USER BEARER, not the shared X-API-Key. searchGuard required the
key on every call (F2 hardening), so the console got 503/401 — "backend not
initialized" — even though searxng+crawl are deployed and the upstream defaults
are correct.
Reconcile to the ONE-WAY gate the rest of the /v1 data plane uses: at the zip
layer, a request with a validated principal (principal.Validated — X-User-Id
minted by the identity middleware from a verified JWT) proxies straight to
SearXNG; a request with NO principal falls to the unchanged key-based searchGuard
(the hanzo.chat server path). A caller with neither is still refused, so F2 (no
open metasearch proxy) holds — proven by TestSearchNoPrincipalNoKeyRefused.
searchGuard (net/http) is untouched; its 503/401 tests stay green. New coverage:
TestSearchValidatedPrincipalBypassesKey (console bearer, key unset -> 200),
TestSearchNoPrincipalNoKeyRefused (anonymous, key unset -> 503).
agent-runner mints M2M inference token from in-cluster IAM (fixes /v1/agents/:ref/run 502). Forward-integrated with main (#118 admin-guard audience). Reconciles live sha-b9639df onto a semver release.
cloud-api SanitizeIdentity validates the forwarded IAM bearer against
defaultJWTAudiences before granting global-admin (owner==adminOrg). The
admin.hanzo.ai guard is client hanzo-admin-guard, so its tokens carry
aud=hanzo-admin-guard, which was missing from the allowlist -> the bearer
failed validation, resolved anonymous, and the SuperAdmin gate read false
-> 403, even though the token owner IS admin.
Append the guard client_id (forwards-only, mirrors gateway iamauth). Admin
authority still requires owner==adminOrg, so no widening. Pairs with
hanzoai/gateway audience fix.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
pickAIClient built the M2M token URL from cfg.IAMIssuer (=https://hanzo.id), a
Cloudflare-fronted host. In-cluster the runner's server-side POST to
https://hanzo.id/v1/iam/oauth/token 403s with CF edge error 1006, so the
oauth2 client-credentials fetch fails and EVERY POST /v1/agents/:ref/run 502s:
'cloud: chat completion: oauth2: cannot fetch token: 403 Forbidden / 1006'.
New aiM2MTokenURL resolves the token endpoint split-horizon, mirroring the KMS
login-broker (clients/kmssvc) exactly — one policy, no drift:
1. CLOUD_AI_IAM_TOKEN_URL override
2. in-cluster IAM_URL (already wired to http://iam.hanzo.svc for JWKS)
3. public IAMIssuer fallback (single-process deploys)
IAMIssuer stays https://hanzo.id for JWT iss-validation (untouched). The chat
base URL is pointed in-cluster via CR env CLOUD_AI_BASE_URL (universe).
Proven in-cluster: mint token from iam.hanzo.svc + chat to gateway.hanzo.svc
both 200 (real completion). Unit test pins the 3-branch resolution order.
Pulls in the ai fix that closes the hz_ widget-key free-inference hole: widget
keys now bill the OWNER ORG (object.WidgetKeyOwner), so reserveBudget +
recordUsage + the balance gate all engage instead of running free/unmetered.
Bounded to the restricted widget model set + token cap; fail-secure when a widget
key is unattributable.
The per-tenant KMS secret-sync login broker derived its IAM token-exchange URL from the public issuer (hanzo.id), which Cloudflare 403s for in-cluster server-side POSTs → the sync could never authenticate. Prefer in-cluster IAM_URL (+ CLOUD_KMS_IAM_TOKEN_URL override), fall back to issuer. Unblocks PaaS per-tenant secret env (proven: git-built ai-demo app deployed on maxpower).
Mounts /v1/notify/{send,send/sms,send/email,health} natively in-process as the
cloud subsystem "notify" (order 139) — the native, in-process replacement for
the standalone notifyd (github.com/hanzoai/notify) Deployment.
notifyd's ONLY production consumer is Hanzo IAM's OTP send
(POST /v1/notify/send?sync=true, event=iam.otp_sent), and the live tenant's
template/provider/event tables are empty, so this folds exactly that contract
and nothing more. It reuses notifyd's OWN public provider packages
(service/{twilio,twilioemail,plivo,mail}) and wire types (pkg/types) — no
duplication of provider plumbing; only the internal-only cred->constructor glue
is mirrored.
Security: unlike the ClusterIP-internal notifyd (which trusted a raw X-Org-Id),
/v1/notify/send is reachable via the public gateway here, so it gates on a
VALIDATED principal and derives the org from principal.Tenant — the same
trust-boundary move clients/auto makes. Credentials come from env (the
KMS-synced notify-twilio Secret) and KMS via cloud.Deps.KMS; none is hard-coded
or logged. Ships a built-in iam.otp_sent template so the fold is strictly more
available than notifyd is today (whose empty store would 400 an OTP send).
Sync-only: the Temporal notify-send async plane is intentionally NOT folded;
async (no ?sync=true) returns 503, exactly as notifyd does without a worker.
Build-gated: go build ./... green; go test ./clients/notify/... green; gofmt/vet
clean. go.mod adds only hanzoai/notify + its provider transitive deps.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The fail-closed 503 body no longer names ZT_CLIENT_ID/ZT_CLIENT_SECRET;
it now reads 'networking is not configured on this deployment' — a
customer-facing string the console renders as a clean 'not available yet'
state. Ops still see the env names in the Warn log at Mount. No behavior
change; the gate() fail-closed contract is identical.
RED M-2 [MED] — contain panics in the async agent-turn goroutine. The dispatched
turn (handleSlack* → slackAgentReply → agents.RunOnBehalf, a large surface over
UNTRUSTED Slack input) ran UNRECOVERED — middleware.Recover() only wraps the sync
request goroutine — so a panic would crash the ENTIRE shared multi-tenant cloud
binary (every tenant, every subsystem). Introduced slackSpawn: runs an
already-slotted turn in a recovered goroutine (recover defer registered LAST so it
runs FIRST; the slot release still runs after it, so a panicking turn frees its
slot). Test TestSlackTurnPanicRecoveredAndSlotReleased.
RED M-1 [MED] — shed BEFORE burning the dedupe key. The old order was
mark-then-dispatch: MarkSlackEvent recorded the event_id, then a pool-full drop
silently 2xx-acked → Slack never retried and a later retry was deduped away, so
the @mention vanished. New order (events + slash): verify HMAC → resolve org →
TRY-ACQUIRE a pool slot; on shed record NOTHING and return a retriable 429 (Slack
re-delivers when a slot frees — the turn never ran, so no double-run and no burned
key); on acquire → MarkSlackEvent (release the slot on duplicate/error) → spawn.
The fast empty-2xx ack is kept for the normal path. Test
TestSlackShedReturnsNon2xxAndDoesNotRecord.
Also corrected the slack_dedupe.go replica note (we no longer "always 2xx-ack" —
a capacity shed returns a retriable non-2xx and records nothing, so it is not a
double-process).
RED M-3 [MED] deploy single-replica (Recreate + persistent CLOUD_DATA_DIR) — the
universe cloud manifest, owned by the deploy lane; the in-code single-writer
invariant comment is kept accurate here.
go build ./... = 0, go vet ./... = 0, go test ./clients/integrations/
./clients/agents/ -race green (25 tests).
RED H1 [HIGH] — wire the bridge into integrations.Mount (was only in a test
helper → the front-door didn't exist in prod). The 5 literal routes are
registered BEFORE the /:provider wildcards (registration-order precedence, same
discipline as clients/agents' static-before-:ref) and are PUBLIC at the JWT
layer: IdentityMiddleware only POPULATES a principal (never rejects) and
DefaultPrice returns 0 for /v1/integrations/* so BillingGate passes through —
reached exactly like /:provider/callback. Auth is HMAC (events/commands) /
signed __Host- cookie (link legs) INSIDE the handler. New test
TestSlackRoutePrecedence proves GET /slack/link hits slackLink (not :provider),
/v1/integrations/slack still resolves the provider view, and the webhook is
reachable with no principal.
RED M1 [MED] — per-org concurrency sub-limit. The agent-turn pool was one
process-global semaphore; one org bursting @hanzo could starve every tenant.
Replaced with orgLimiter (global cap + per-org cap, SLACK_AGENT_ORG_CONCURRENCY
default 8). The org is now resolved SYNC in the webhook path so the pool keys on
the RESOLVED tenant before a slot is taken. New test TestOrgLimiter.
RED M2 [MED] — corrected the false "holds across replicas" dedupe claim: the
table is per-process embedded SQLite (single-writer per HIP-0302), so the billed
webhook path MUST run single-replica (stated as the shipping invariant); a
shared SETNX store is the multi-replica follow-up.
RED L1 [LOW] — moved the dedupe-table DDL into the store's migrate() (store.go)
— fail-loud at Mount, one place — and removed the lazy first-use ensure whose
LoadOrStore-before-run could permanently disable the path on a transient DDL
error. slackBridgeReady now only inits the process pool + link seen-set.
Deferred (flagged for clients/integrations owner): L2 UNIQUE(provider,
external_id)+first-org-wins refusal on duplicate team connect; L3 purge
user:<slackUser>:refresh secrets on disconnect (currently inert after
disconnect, no leak).
go build ./... = 0, go vet ./... = 0, go test ./clients/integrations/
./clients/agents/ -race green (18 + 5 tests).
Port the hardened @hanzo Slack agent front-door from team-go/pkg/slack into
the unified Hanzo Cloud integrations plane, so Slack is ONE connector aligned
with the one-binary north star. It CONSUMES the existing Slack OAuth provider
(the per-org bot token it seals) and the framework seams
(OrgForExternalID / TokenFor / ConnectionFor); it adds no new custody path and
edits no existing file.
clients/agents:
- onbehalf.go: exported in-process RunOnBehalf(ctx, org, userSub, ref, input) —
the clean in-process twin of the HTTP run handler (no gateway hop, no
Cloudflare/IPv6 exposure). Resolves the agent org-scoped, runs it through the
SAME runAgent -> executeRun -> meter path, bills billingActor(org, userSub)
against org's ledger. Takes org+userSub DIRECTLY (caller pre-authenticated).
clients/integrations:
- slack_events.go: Slack Events webhook + slash command. HMAC-verified over the
EXACT raw body with a 5-min replay window; url_verification challenge; routes
@mention + DM to an on-behalf-of run; durable dedupe on event_id; fast empty
ack + bounded async worker pool. Posts the reply into the thread with the
org's bot token, or the link prompt EPHEMERALLY.
- slack_link.go: transplant-safe 3-leg per-user link (__Host- init/link cookies,
leg1<->leg2 nonce continuity checked BEFORE any exchange, single-use). Binds
Slack<->Hanzo via hanzo.id OIDC (hanzo-slack client) and seals the refresh
token per (org, "slack", "user:<slackUser>:refresh").
- slack_verify.go: Slack signature verify + single-use link-state crypto
(constant-time HMAC over s.stateKey; orthogonal to the OAuth-connect state).
- slack_dedupe.go: durable event-dedupe table as Store methods (no store.go edit).
PER-ORG ISOLATION (ship bar): an event's org comes ONLY from
OrgForExternalID(team_id) — never the payload; the reply uses THAT org's bot
token (TokenFor); the run is THAT org's agent (RunOnBehalf org-scoped). Tests
prove team A's event never resolves/tokens/runs as org B.
Mount wiring (5 routes) is handed to the clients/integrations owner — this
change adds NO Mount edit (clean separation); handlers are (s *svc) methods.
Tests (go test -race, green): HMAC reject (bad/missing/stale), dedupe
idempotency, per-org isolation (end-to-end bot-token capture proves the reply
used the connecting org's token), link transplant-rejected (no/mismatched init
cookie refused before exchange), RunOnBehalf bills the right actor.
The recorded zip h1 for github.com/luxfi/age v1.5.0 (zC/Fw…) did not match
the immutable Go checksum transparency log (sum.golang.org), which records
G69Hb… — the same bits the module proxy and local cache serve. The stale
hash made the ENTIRE module unbuildable: every `go build` failed with a
checksum mismatch / SECURITY ERROR. The /go.mod hash already matched sumdb;
only the zip h1 was wrong. Aligning it to the transparency-log-verified
value unblocks the repo (`go mod verify` -> all modules verified). age is an
indirect dependency; no version bump.
Cross-org fleet o11y for admin.hanzo.ai (global-admin only, s.guard fail-closed):
fleet totals (requests/tokens/cost/errors/orgs/models from hanzo.cloud_usage;
latency p50/p95/p99 + error-rate + services from signoz_traces; log volume from
signoz_logs), usage + log-volume timeseries, and top-N orgs/models/services
leaderboards, plus the fleet Langfuse generation rollup. Un-org-scoped by design
— the one place a fleet operator crosses tenants; a non-admin bearer is refused
403 before a row is read. Reuses the shared aiobject.DatastoreQuery transport
(no second connection) and the compute/analytics honest-empty pattern; admin
reads only, owns no table. Time bounds are positional params, bucket interval a
server-side constant — injection-safe. Pure builders + parsers unit-tested.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
luxfi/age@v1.5.0 was force-retagged upstream: proxy.golang.org now serves
content hashing to G69HbSV… while go.sum pinned the stale zC/Fw… → every
cloud release fails at `go mod download` with a SECURITY ERROR (checksum
mismatch), blocking the whole pipeline. Dockerfile already sets GOSUMDB=off,
so the mismatch is against the committed go.sum, not the sumdb. Realign to the
exact hash CI computes from the proxy (same fix IAM shipped as fee44857).
Addresses CTO's post-RED fix set (isolation boundary already approved airtight).
MED-1 (exactly-once metering/audit/persistence across ALL entrypoints): the
durable path is now the SINGLE owner of run bookkeeping. FlowRunWorkflow runs a
RecordRunStartActivity keyed on the workflow id (workflow.GetInfo — a scheduled
cron mints a fresh id per tick, so each tick is its own metered run despite the
schedule embedding one fixed FlowRunInput.RunID). Store CreateRunIfAbsent (row
idempotency) + ClaimMeter (atomic metered-flag 0->1) meter+audit only the winner,
so manual /run, MCP, and cron never double-bill. Manual /run no longer meters/
audits — it only CreateRunIfAbsent for immediate visibility. RecordRunEndActivity
records terminal status. Proof: TestScheduledRunMeteredExactlyOnce (a tick meters
once + shows in listRuns; two ticks = two distinct runs) + TestRunStartBookkeeping-
Idempotent (recordRunStart twice = one meter, one row).
MED-2 (honest SSRF blocklist): isPublicIP now rejects the IANA special-use ranges
Go's net helpers miss — 100.64/10 CGNAT (Alibaba metadata 100.100.100.200),
0/8, 192.0.0/24, 192.0.2/24, 192.88.99/24, 198.18/15, 198.51.100/24, 203.0.113/24,
240/4, 64:ff9b::/96 NAT64 — plus v4-mapped-v6 normalization. Comment no longer
overclaims a complete cloud-metadata blocklist. TestIsPublicIP covers each range +
public IPs still allowed.
MED-3 + LOW-4: step-count (<=256) + serialized-tree (<=512KB) caps at create /
version / operation time -> honest 422; resume payload bounded (<=64KB) -> 413.
LOW-2: per-org concurrency limiter (429) on run-starts + synchronous MCP tool calls
(bounds the core.delay goroutine lever). TestConcurrencyLimiter + TestFlowStepCap +
TestResumePayloadBounded.
LOW-1: MCP meters/audits AFTER Run, outcome derived from the real result — a failed
/ SSRF-blocked / not-connected call audits as error and is NOT billed. TestMCPAuditOutcome.
LOW-3: updateFlow validates publishedVersionId names an existing version OF THIS
FLOW in-org (else 422). TestUpdateFlowPublishedVersionValidated.
INF-1: register() panics at init on a <connector>_<action> tool-name collision so a
future connector can't silently make MCP dispatch ambiguous. TestToolNameCollisionPanics.
Tests: 25/25 green (CGO=0 build/vet/test; -race clean under cgo). Full module builds;
cmd/cloud links. catalog.json untouched.
Authorize needs only SLACK_CLIENT_ID (a public value in every consent URL);
the SECRET is required only at the callback token exchange. Gate available/
connect on client_id so an org reaches Slack's Allow screen as soon as the
public id is set, while a deployment still missing SLACK_CLIENT_SECRET fails
the exchange with an honest ?error=slack (never a dead-end).
Pulls the cors_filter static-allowlist fix so console.lux.cloud (and
zoo/pars brand consoles) stop getting 403 "origin is not allowed" on
/v1/signin. Cleared stale sum.golang.org-poisoned go.sum entries for
re-tagged luxfi/{age,precompile,keys} (GOPRIVATE direct re-records the
current content hashes; matches the repo's GOSUMDB-off CI recipe).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Native-Go /v1/automations/* subsystem in the unified cloud binary. Composes
three existing seams, never reinvents them:
- clients/integrations — per-org connector creds via integrations.TokenFor
(KMS-sealed, fail-closed); connectors never touch KMS directly.
- cloud.EmbeddedTasks — the ONE shared in-process durable engine; a flow
runs as a durable workflow in the OWNER's namespace (per-org lazy worker,
mirroring ai/object/ingest_tasks.go).
- clients/principal — the ONE tenant gate on every data handler.
Isolation is physical: ONE SQLite file, org column + org-led index on every
table; the durable activity's SOLE credential scope is FlowRunInput.Owner —
the VALIDATED org set at flow-start, never a client-supplied field.
Surface: pieces catalogue (go:embed), flows CRUD + versions + FlowOperation
apply, durable runs (start/list/get/resume via SignalWorkflow), enable/disable
(POLLING -> CreateSchedule), and a HIP-0300 MCP JSON-RPC tool surface
(/v1/automations/mcp) exposing every connector action as <connector>_<action>.
Connectors (Tier-A, self-registering): core (http_request SSRF-guarded via a
dialer Control hook, delay, code data-mapper, wait_for_approval signal
waitpoint), slack (send_message), github + google_sheets/drive (fail closed
until integrations custodies their tokens).
Metering + audit on flow-run start and MCP tool call. Order 148 (after
integrations 137, before ai's /v1/* catch-all 150).
Tests (16, all green with -race): store org-isolation, HTTP org-gating (403),
durable flow run reaching SUCCEEDED with threaded step outputs on an embedded
tasks engine, connector token isolation (in.Owner is the sole cred scope),
MCP tools/list + gated tools/call dispatch, pieces catalogue.
* feat(o11y): emit an OTel SERVER span per /v1/* request over the ZAP wire
Cloud installed a ZAP tracer provider (cmd/cloud initTelemetry) but nothing in
the handler chain opened a span, so no request ever flowed through it — the o11y
Monitoring tab saw zero hanzo-cloud request traces (receiver="zap" span count
was flat-zero while logs streamed over ZAP).
TracingMiddleware (middleware_tracing.go) opens one SERVER span per /v1/*
request off the GLOBAL tracer (= the ZAP provider), records the OTel HTTP
semantic-convention attributes (method, route, status) + request_id/org, maps
error/5xx to an error span status, and writes the span context back onto the
request via SetContext so every downstream span (agent.run -> agent.step -> the
chat client span in clients/aihttp) parents under it: one trace tree per
request. Health/readiness/metrics + non-/v1 paths are skipped so probes never
flood the trace store. Wired right after RequestID in the canonical pipeline
(serve.go) so the whole authenticated chain nests under it.
c.Path()/c.Method()/headers are zero-copy views over the fasthttp request
buffer, which is recycled for the next request BEFORE the batch span processor
serializes the span asynchronously — so retained views corrupt (live: a
GET /v1/models span exported with http.route="/v1/chat/c..."). strings.Clone
pins our own copy for every retained attribute. Tests cover emission, attribute
mapping, error status, parent/child propagation, the skip set, and an env-gated
on-wire live test (CLOUD_ZAP_LIVE_ENDPOINT) that ships real spans to a ZAP
receiver — the async-export + ctx-reuse path the in-memory recorder can't model
(and the one that surfaced the corruption).
* feat(o11y): make cloud the single tracer-provider owner — one wire (ZAP)
The fused cloud binary set the ZAP provider first, then ai.Bootstrap (during
MountAll) called hanzoai/ai object.InitTelemetry which, seeing the CR's
OTEL_EXPORTER_OTLP_ENDPOINT, installed a SECOND, competing OTLP provider. OTel
global delegation is first-writer-wins for handles created before the first
SetTracerProvider (cloud's package-level tracers keep ZAP), but the ai GenAI
tracer is resolved lazily AFTER the second Set, so its spans stranded on
OTLP(:4318) while ZAP owned the rest — the split that left receiver="zap" span
count at zero for hanzo-cloud (verified live: spans arrived only via
receiver="otlp").
Composition-root fix: once cloud installs the ZAP provider, clear the
OTLP-exporter env (OTEL_EXPORTER_OTLP_ENDPOINT / _TRACES_ENDPOINT) so no embedded
subsystem installs a competing OTLP provider. Exactly one provider (ZAP), one
wire, deterministic regardless of CR env drift. In the fused binary OTLP is only
ever the collector's interop RECEIVER, never cloud's exporter; standalone
cmd/aid (no ZAP endpoint) is unaffected and keeps its OTLP path.
---------
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The auto engine now gates /v1/auto/pieces/{piece}/run on a shared secret (it
trusts X-Org-Id absolutely, so the write+SSRF surface needs an in-band caller
proof). cloud is the ONLY legitimate caller — it resolves each org's real token
and pins the provider URL — so it presents the secret (from KMS, the same
PIECES_RUNNER_SECRET) as X-Piece-Run-Secret. pieceSync fails closed if the
secret is unset (a doomed call the engine would 403 anyway).
Mounts workflow automation + the ~280-app activepieces long tail in-platform,
per-org, through the ONE knowledge store. Go core + JS on-demand.
clients/auto: /v1/auto/* per-org REVERSE PROXY to the standalone Hanzo Auto
engine (one engine, one store — not re-embedded). The auto engine trusts
X-Org-Id absolutely, so this proxy IS the trust boundary: it GATES on a
validated principal (refuses the anon-forge X-Org-Id-with-no-credential path)
and re-stamps outbound identity from validated values only (strips every
smuggled authority alias). Pure gate+proxy in clients/auto/proxy (5 isolation
tests: anon-forge 403, per-org forward, smuggled-header strip, path preserved).
clients/kb: the first LONG-TAIL connector (notion). Identical OAuth lifecycle
(HMAC-org-bound state, KMS token path) but its PULL runs the activepieces JS
piece through the auto engine's on-demand runner (sync_piece.go) instead of
native Go — then files each record via the SAME framework.Ingest path. One
ingestion path; a JS-sourced doc lands in the same per-org store+index as a
Go-sourced one. clients/kb/notion is the pure record-shaper (6 tests).
ONE catalog: /v1/kb/connectors/catalog lists native Go + long-tail piece
connectors in one list, each badged kind native|piece (3 tests).
RED LOW-1: collection() + kmsRef() now route org through provisioning.SanitizeOrg
(the codebase's ONE normalizer) so the physical Qdrant namespace + KMS path are
injective in the owner ("a b" != "a_b") — defense in depth under the payload.org
filter. Injectivity tests + KB integration tests updated to derive the collection
through the helper (robust to the normalizer).
All tests green under CGO=0 (production config). Full binary boots; /v1/auto
mounted, anon-forge 403, catalog gated, spine (kb/framework health) 200.
Address review: proxy the customer card list to commerce's admin-group PORTAL
endpoint (GET /v1/billing/portal/payment-methods), not the user-group
/payment-methods. PortalPaymentMethods 400s without a ?customerId=, so pinning
only ?user= would break it — generalize the proxy's subject pinning to the FULL
commerce edge-auth key set {user,userId,customerId} (now a shared
billingSubjectKeys var, identical to clients/console + commerce), pinned to the
caller's OWN org on every request. This leaves NO billing endpoint unfiltered
regardless of which param it reads (usage/balance/gpu-eligibility read user;
portal/payment-methods requires customerId) and is strictly more tenant-safe.
pinSubjectBody reuses the same var. The console keeps requesting the same-origin
/v1/billing/payment-methods (mounted here); the portal hop is server-side only.
Tests updated: widen-scope now asserts every subject key is pinned (org dropped);
payment-methods asserts the portal path + customerId pin. 14/14 pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The customer billing proxy (clients/billing) exposed only usage+balance, so the
console GPU launch gate — card-on-file check + prepaid eligibility read + the
prepay-only charge — had no org-scoped cloud route and fell through to the
console pkg's /v1/billing/* wildcard (admin-shaped 403). Extend the SAME
commerce-proxy helper (no new HTTP/auth machinery) with the three enforcement
routes commerce 1.46.28 serves (api/billing/gpu_charge.go + portal):
GET /v1/billing/gpu-eligibility -> commerce GET (read-only launch gate:
{eligible,reason,prepaidAvailable,cardOnFile,...}; amountCents +
minPrepaidCents + currency pass through)
POST /v1/billing/gpu-charge -> commerce POST (prepay-only, card-required
debit; commerce enforces both gates + gpu-tagging server-side; status
forwarded verbatim: 201 ok / 402 card_required|insufficient_prepaid)
GET /v1/billing/payment-methods -> commerce GET (masked brand+last4 cards
for the card-on-file check; type passes through)
All org-scoped to the caller's OWN org from the VALIDATED IAM owner claim
(principal.Tenant), identical to usage/balance — a client can never widen scope:
the GET subject is pinned to ?user=<org> (commerce's privileged payment-methods
branch filters CustomerId on it), and the POST body subject is pinned to the
{user,userId,customerId} set (mirrors clients/console + commerce edge-auth), so
a forged body can never charge another tenant. New commerceProxy.post + the
pinSubjectBody helper; no principal -> 401, unconfigured -> 501.
Tests: gpu-eligibility scope+passthrough+forged-subject overwrite;
payment-methods scope+type; gpu-charge body-subject pin + 402 verbatim + 401
no-principal + 501 unconfigured; pinSubjectBody unit. 14/14 pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On console.hanzo.ai the ingress routes /v1/* straight to cloud-api:8000 (the
console Next BFF is only at "/"), so the console's /v1/billing/usage +
/v1/billing/balance calls land on cloud-api — NOT the console's per-tenant
commerce proxy. cloud-api wired commerce billing ONLY under the admin-gated
aggregate (/v1/admin/*), so a normal org owner (davelorenzini/maxpower) hitting
/v1/billing/usage had no customer route and was denied -> 403 -> the "Access
required" wall on EVERY product overview + o11y usage panel.
Add a customer-facing, org-scoped billing READ surface (clients/billing):
GET /v1/billing/usage and /v1/billing/balance. Org = the VALIDATED IAM owner
claim (principal.Tenant — the trusted X-Org-Id the identity middleware minted
from the caller's verified session; never a client header), so a customer reads
ONLY their OWN org. Proxies commerce with COMMERCE_SERVICE_TOKEN + X-Org-Id=<org>
and the per-org billing subject pinned to user=<org> (admin.orgSubject /
metering identityFromCtx — verified live: user=<org> returns the real wallet);
returns commerce's raw body + status verbatim (the console parses the raw ledger).
Tenant isolation: no client-supplied subject/org query is ever forwarded, so
scope can never be widened. The all-orgs god view stays admin-only (clients/admin).
Tests: subject pinned to caller org, forged user/userId/customerId/org dropped,
start/end/currency pass through, no-principal -> 401 (commerce untouched),
unconfigured -> 501, commerce status forwarded verbatim.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pulls ai#68: async OpenAI Sora-style video API onto the cloud router.
POST /v1/videos/generations returns a video_<uuid> job immediately (was
sync ~104s → console /ai proxy 502); GET /v1/videos/{id} polls; GET
/v1/videos/{id}/content streams the MP4. Metering exactly-once (hold on
create, settle on completion, reaper releases abandoned), ownership-secured.
ai v1.800.0 go.mod is identical to v1.799.3 — no dependency-graph change;
only the hanzoai/ai hash lines move. Verified: cloud binary builds clean
(CGO_ENABLED=0) and the router now serves /v1/videos/generations,
/v1/videos/:id, /v1/videos/:id/content (spark-video backend).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Durable replacement for the Huly/Svelte hanzo.team tracker whose upstream
each-block reactive-batching render race left issue lists rendering zero
rows. Native Go over one org-scoped SQLite store: rows return as plain JSON
and render deterministically (no Svelte reactivity in the path).
clients/tracker/store.go — projects + issues on {DataDir}/tracker.db, the
same modernc/SQLCipher driver + MaxOpenConns(1)+WAL pattern as projectsvc/
crm. Per-project monotonic issue numbering allocated inside one tx (never
races under the single-writer conn). Cascade delete in a tx. org column is
the tenancy key; every query filters WHERE org=?.
clients/tracker/tracker.go — /v1/tracker/projects[/:key][/issues[/:num]] CRUD.
org = principal.Tenant (validated IAM owner claim, HIP-0026), 403 otherwise.
Status/priority closed sets; board/list via ?status=. Create wired to the
shared per-org billing seam (free by default; ops prices via
CLOUD_TRACKER_FEE_CENTS). Registered order 129, before the AI /v1/* catch-all.
subsystems/subsystems.go — one blank import links it into the binary.
Store CRUD/numbering/status-filter/cascade/tenant-isolation proven green on
real SQLite (clients/tracker/tracker_test.go).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
ROOT CAUSE of durable ingest silently running inline (round-trip: github ingest blocked
25s+, no workflow in Tasks): the embedded engine only registers 'default' at boot and does
NOT lazily create namespaces on ExecuteWorkflow, so dialing Namespace:<org> made the worker
poll a non-existent namespace and BLOCK → EnqueueIngest hung → handler fell back to inline.
Fix: dial 'default' (always registered). Data isolation is unchanged — it's in the workflow
INPUT (IngestSource is owner-scoped), never the namespace. Now github/crawl enqueue a real
durable workflow that appears in Tasks (under default).
V6: accept the owner-bound per-tenant machine audience (<owner>-platform-kms) so a real client_credentials sync token clears SanitizeIdentity + the /v1/kms guard; decoupled from global-admin (isKMSMachinePrincipal). V1: per-IP rate limit + MaxConnsPerHost(h2-off) on the public login broker. Blue→Red→Blue→Red: Red SHIP (0 crit/high/medium). Activation runbook in the PR body + EnsureOrgIdentity doc.
Fourth app lane (after cms/erp/help): a Notion-like KB + agent memory + app
connectors as a 'kb' module on clients/framework — no new Base, no new database.
Fixtures (module kb): kb-page (wiki tree via a self-Link parent + Lexical
RichText body), kb-memory (agent memory: note/fact/observation), kb-source
(connector-ingested docs), kb-connector (connection metadata; OAuth token in KMS,
never in the doc/logs). All CRUD/permissions/tenant-isolation/install are the
framework's generic /v1/framework/* surface + the generic @hanzo/ui renderer.
Indexing (index.go, the ONE vector-write path): an after_save hook embeds every
knowledge write (page/memory/source) via the gateway and upserts it into the org's
OWN Qdrant collection (kb_<org>) with an org-pinned payload; on_trash removes it.
Human wiki + AI memory are ONE per-org knowledge store, indexed once. Fail-open at
index time (a vector outage never blocks a knowledge write); fail-honest at query.
Retrieval (subsystem.go): POST /v1/kb/search is the org-scoped RAG entry point —
collection AND payload filter both pinned to principal.Tenant, so a caller can only
retrieve its OWN knowledge. Degrades to an honest empty result when the index is
down.
Connectors (connectors.go, sync.go): per-org OAuth to GitHub/Slack/Google that
ingest external docs INTO the same store + index (via framework.Ingest → same
after_save hook — one ingestion path, never forked). OAuth state is HMAC-bound to
the validated org (defeats login-CSRF/mix-up); tokens live in KMS at a per-org path.
GitHub is end-to-end (repo READMEs + issues); Slack/Google share the OAuth
lifecycle + normalizer with an honest 'listing not yet implemented' depth marker.
framework.Ingest/UpdateData/FindByField/Search: the in-process create-with-hooks
API first-party producers (the connector sync) use so off-request writes run the
exact validate + lifecycle pipeline and stay physically org-scoped.
Tests (16, all green): engine-Validate on every fixture; page self-Link + Lexical
body; connector-has-no-token; per-org pointID/collection/kmsRef isolation; payload
org-pin; OAuth-state org-binding + tamper/cross-provider/wrong-key rejection;
Ingest validates+fires-hooks+org-scoped. Integration (real SQLite + mock
Qdrant/embeddings over the real HTTP surface): install → create kb-page in org A →
after_save indexes into kb_A → search as A retrieves it → search as B sees NOTHING
(no cross-org leak); forged-org search → 403; kb-memory lands in the same kb_A
namespace.
Adversarial review of the connector framework (state HMAC, nonce custody,
per-org KMS token custody, console redirect). Core design held: state forgery,
nonce replay/race, cross-tenant KMS pathing, principal-forge, and the seam
fail-closed contract were already sound. Fixes are defense-in-depth + red-style
proving tests; the contract is unchanged (no /api/, no /v1/slack route).
Fixes
- Ingest sanitization (vector 7): provider-supplied NON-secret metadata
(account label / external id / bot user id / scopes) is now stripped of C0
control chars + DEL and length-bounded at the ONE framework ingest point in
callback, before it is logged, stored, or reflected. Kills log-line/separator
injection via a crafted Slack workspace name and bounds per-org row growth.
Secret token VALUES bypass this and go straight to the KMS seal.
- Open-redirect hardening (vector 4): success/failRedirect fold into one
query-escaping consoleRedirectURL builder (DRY + unit-testable). Confirms the
Location host is always the env-fixed console origin; hostile provider detail
can't break out of the query into host/scheme/path or inject CRLF.
- Request bounds (vector 10): callback rejects an oversized OAuth `code`
(maxCodeLen, also covers the in-process ZAP plane); verify() rejects an
oversized state token before any base64 work (maxStateLen).
- kmsDelete uses errors.Is(kms.ErrSecretNotFound) for wrap-safe idempotency.
Tests (all real, -race green; 39 pass)
- state: dot-injection/degenerate split, MAC-checked-before-parse,
validly-signed-but-hostile-org rejected, overlong rejected.
- store: concurrent single-winner nonce consume (race proof).
- http: no-open-redirect property, end-to-end metadata sanitization,
disconnect anonymous-forge -> 403 (secret+row survive), github scaffold
callback fails closed at the Configured gate before any exchange,
oversized-code rejected.
Provider-agnostic /v1/integrations plane: one registry, N providers.
Slack = full reference impl (OAuth v2 bot-token); GitHub = scaffold + #51 seam.
Per-org token custody in KMS (sealed); state-authed HMAC callback with
single-use nonce; org derived ONLY from signed state on the public callback.
Generic /v1/integrations/{provider}/callback — no /v1/slack/* (team-go owns it).
Wired at order 137 (after security 136, before AI 150).
release.yml — invert the tag/build order so a git tag can NEVER exist without a
pushed, boot-verified image (the phantom v1.786.42/43 → ImagePullBackOff cause):
main push → compute next version → build → SMOKE (boot to "listening") →
push image → git tag (receipt) → notify universe
- Tag is minted only AFTER the push step succeeds; any build/smoke/push failure
fails the run before the tag step → fail-run-no-tag.
- concurrency group `release-cloud` (cancel-in-progress:false) serializes runs so
two main pushes can't collide on a number; the queued run re-reads tags and
lands on the next patch → monotonic.
- next version = max(highest git tag, highest pushed ghcr container tag) + 1,
folding in container tags so a pushed-but-untagged number is never reused.
- removed the `tags: v*` trigger (this workflow now OWNS tags — a hand-cut tag has
no image behind it and won't build); notify-universe fires only on a successful
build+tag, so universe is never told about a phantom.
sqlite — fix `panic: sql: Register called twice for driver sqlite` under
CGO_ENABLED=1 (blocks `go test ./...` + clean cgo rebuilds). #96 moved cloud's
stores to github.com/hanzoai/sqlite (mattn under cgo) while several embedded deps
still import modernc.org/sqlite directly (ai, tasks, base, commerce, o11y, orm) →
two packages register "sqlite" under cgo. Prod is CGO_ENABLED=0 (all one modernc
package, deduped) so prod never paniced; the panic is cgo-only.
- bump github.com/hanzoai/sqlite v0.1.4 → v0.1.5: adds the `sqlite_purego` opt-out
build tag that forces the fork's pure-Go (modernc) backend under cgo. Default
cgo path is unchanged (mattn/SQLCipher) so IAM/commerce encryption is untouched.
- bump github.com/hanzoai/ai → the commit that routes object/adapter.go + cmd
tools through hanzoai/sqlite instead of modernc (never modernc directly).
- Makefile: CGO_ENABLED?=0 default (matches the shipped Dockerfile) so `make
build`/`make test` register "sqlite" once and exactly mirror prod; new `test-cgo`
target proves the cgo path via `-tags sqlite_purego`.
Verified: CGO_ENABLED=0 `go build/test ./...` and CGO_ENABLED=1 `-tags
sqlite_purego go build/test ./...` both pass with NO panic (the eval package that
panicked now passes in both modes). Pre-existing clients/s3 + clients/functions
billing-attribution test failures are unrelated (present on clean main, both
modes) and out of scope.
billing.go's isSafeSegment left percent-escape (`%2f`/`%2e`) and matrix-param
(`;`) segments undecoded, so `/v1/billing/x/..%2fadmin` forwarded
`x/..%2fadmin` verbatim; the Go http client + commerce's own router
decode+normalize it downstream into a path that tunnels PAST /v1/billing into
another surface. commerce.go had already patched its call site with an inline
`%;` check — braiding the policy across call sites.
Harden the ONE shared segment guard instead: isSafeSegment now rejects empty,
`.`/`..`, slash, backslash, percent-escape, matrix-param, and any control char.
Both bridges (billing + commerce) get the complete guard from one place, and
commerce.go's call site drops the now-redundant inline check.
Regression test: `..`, `%2f`, `%2e%2e`, and `;` all 400 and never reach
upstream (proven to fail against the pre-fix guard).
The store twin of the just-merged /v1/billing/* bridge (#102). #81 namespaced
the console's commerce store calls to the canonical same-origin /v1/commerce/*
(SPA->server->/commerce proxy); the statically-exported console now terminates
every dynamic call at the unified cloud binary's /v1, so the binary must
reverse-proxy /v1/commerce/* to the commerce service. Without it the commerce
embed was incomplete.
clients/console/commerce.go serves GET|POST|PUT|PATCH|DELETE /v1/commerce/<path>
-> commerce's BARE store surface /v1/<path> (the console-side 'commerce'
namespace is stripped: the deployed commerce cmd/commerced mounts
api.Route(Group('/v1')), so products/orders/customers/... live at /v1/<kind>
while money lives at /v1/billing/*). Exactly the mapping console2's next.config
rewrite proved live (/v1/commerce/:path* -> /commerce/v1/:path* ->
commerce.svc/v1/:path*).
IDOR-safe: the org is the VALIDATED caller's own (resolveCaller ->
principal.Validated / c.Org()), never a client value; a bearer-less forged
X-Org-Id has no validated principal and is refused 403 before any commerce call.
Reuses the commerceDo(base,token) S2S transport billing.go/topup.go share
(admin COMMERCE_SERVICE_TOKEN + X-Org-Id, which commerce's EdgeAuth trusts only
behind the service token). Least privilege: a store-head allow-list (identical
to console2 proxy-allow.ts COMMERCE_HEADS) so the bridge can never tunnel to
/v1/billing (its own subject-scoped bridge), /v1/checkout, or tenant admin.
Hardened over a naive port: rejects percent-encoded path segments (%2f/%2e),
which the router leaves undecoded in the wildcard param but the Go http client +
commerce's router normalize downstream -- 'product/..%2fbilling' would otherwise
tunnel to /v1/billing past the allow-list (RED). Mirrors console2 pathIsClean.
/v1 only. CGO_ENABLED=0 go build ./... ok; go test ./clients/console/ ok.
Companion to #89 (KMS-sealed PaaS secret env) + universe #321. The sync was
INERT for two structural reasons this closes, and the per-tenant scoping the
task requires is now enforced at cloud's ONE auth boundary — proven by test.
WHY IT WAS INERT (coordinate drift + wrong CR shape):
- Seal path ≠ read path. cloud sealed at /platform/tenant-<org>/<app> but the
kms-operator reads through cloud's org-scoped surface /v1/kms/orgs/<org>/
secrets/... which folds to /orgs/<org>/... — a DIFFERENT record, never found.
- The CR set projectSlug="platform" (a literal) and omitted secretsScope.keys,
which the CRD REQUIRES (MinItems=1; luxfi/kms has no list endpoint). Either
alone starves the sync.
- hostAPI carried a /v1/kms suffix; the operator appends /v1/kms/... itself, so
login + read URLs doubled the prefix.
THE FIX (secrets.go):
- Seal at the org-scoped coordinate orgs/<org>/platform/<app>/<KEY> — the EXACT
store path cloud's org-scoped read surface addresses. Seal and read are now one
coordinate (proven: TestPaaSSecretSealReadAlignment).
- CR carries projectSlug=<org>, secretsPath=platform/<app>, envSlug=default, and
the explicit sorted key roster. hostAPI = KMS root.
PER-TENANT SCOPING (the NON-NEGOTIABLE) — enforced, not hoped:
- The operator authenticates as a per-tenant IAM machine identity (owner=<org>)
via the NEW /v1/kms/auth/login broker (kmssvc/login.go): it exchanges the
caller's clientId/clientSecret at IAM's client_credentials endpoint and returns
IAM's owner-scoped token verbatim. cloud is a relay, not an issuer.
- cloud's org-scope guard admits /orgs/<org>/... ONLY when the VALIDATED owner ==
that org (SanitizeIdentity derives owner from the token, ignoring client
X-Org-Id). So tenant-A's credential can NEVER read tenant-B's path — 403 before
the store is touched (proven: TestPaaSSecretCrossTenantDenied).
- credsSecret is a PER-TENANT name in the tenant namespace — never a shared
platform-wide reader (that would be a cross-tenant hole = NO-SHIP).
PROVISIONING (ensureTenantKMSAuth) — fail-closed, one privileged seam:
- On app-create/deploy cloud ensures the tenant's owner=<org> credential is
projected into tenant-<org> as the creds Secret the CR references, via an
injected tenantKMSIdentity provider. nil provider (default) ⇒ honest "pending"
(operator can't log in ⇒ reads nothing) — NEVER a shared or wrong-org identity.
- FLAGGED: the concrete provider needs a scoped IAM admin credential cloud does
not yet hold (clients/admin/iam.go replays the caller's cred, no service
identity). Until wired/verified, sync stays safely pending. Flip ON = provision
the per-tenant identity; the security invariant holds regardless of how it is
minted (the guard is the enforcement point).
Tests (CGO_ENABLED=0 — prod/CI build mode; CGO test builds double-register sqlite,
a pre-existing module issue): alignment round-trip, cross-tenant 403 (+ A→A 200,
B→B 404, unauth 403), login broker happy/bad-cred/malformed, per-tenant provisioning
(fail-closed when unprovisioned, org-bound projection, no cross-tenant ask). Existing
kmssvc red-team guard vectors unchanged and passing.
The BFF catch-all sweep's one real "server work -> Go handler" case. console2's
app/billing/v1/[...path]/route.ts injected the commerce SERVICE token and pinned the
caller's billing subject server-side (work a static export cannot do). Ported to
clients/console/billing.go: GET|POST /v1/billing/* forwards to commerce with the admin
COMMERCE_SERVICE_TOKEN, scoping every request to the VALIDATED caller's own subject
(billingSubject + scopedBillingSearch + scopedBillingBody, the Go port of console2's
billing-scope.ts), so a tenant can only read/act on its OWN ledger.
IDOR-safe: the subject is the validated principal (resolveCaller: principal.Validated /
c.Org() / c.User()), never a client userId/org. A forged X-Org-Id with no validated
X-User-Id is refused (403). Unset COMMERCE_SERVICE_TOKEN -> honest 501.
DRY: commerceDo (topup.go) refactored to take (base, token) so the wallet top-up AND
this bridge share one S2S transport. No behavior change to topup (its tests pass).
Tests (billing_test.go): billingSubject personal/dedicated, subject pin + org drop +
passthrough (query & write body), forged-value overwrite, 403 no-principal, 501 no-token,
and the end-to-end scoped forward to a fake commerce. go build ./clients/console/ = 0;
go test (CGO_ENABLED=0) ok. The default-CGO modernc-vs-CGO sqlite double-register that
panics the package is a pre-existing repo-wide issue (separate sqlite-one-driver lane).
Co-authored-by: Hanzo AI <ai@hanzo.ai>
SanitizeIdentity minted an un-forgeable X-Org-Id but passed the org sub-scopes
X-Project-Id / X-App-Id through verbatim, so a caller could assert ANOTHER org's
project as a compute_usage attribution key or per-project sub-scope. Sanitize the
sub-scopes in the ONE trust boundary:
- Delete every X-Project-Id/X-App-Id on ingress (no raw client copy survives),
then re-inject only for a validated principal, against the acted-as org.
- Refuse a cross-org X-Project-Id: a project REGISTERED to a DIFFERENT org than
the validated org is dropped; the caller's own registered project and
unregistered free-form within-org labels survive (projectIsForeign; fail-closed
on a registry error).
- Drop both sub-scopes entirely on the anonymous path.
- X-App-Id is a caller label, not an isolation boundary (no cloud subsystem
scopes access by it; the un-forgeable org bounds any mislabel) - forwarded on
the validated path, dropped when anonymous.
Dependency-inverted like sites.SetResolver: projectsvc registers a
TenantScopeResolver at Mount; cloud never imports the project registries. The
visor proxy forwards the now-validated sub-scopes so compute attribution lands on
the caller's own project. X-Org-Id anti-forgery is unchanged.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Add a RichText fieldtype to the DocType engine so a content field can hold a
Lexical EditorState JSON string (the console renders it with a native WYSIWYG).
The value is opaque text — validate.go coerces it verbatim (clipped to the scalar
bound), stored in the schemaless doc blob, round-tripping through create→get with
no shape enforcement (that's a UI concern). Minimal + fail-closed: the fieldtype
is added to the validated allow-set, so an unknown type is still rejected.
CMS: the seeded content body (Article/Page/Post) becomes RichText, and a
Data field is added for optional per-project scoping (the console's org→project
switcher filters content by it; empty = org-level). One CMS engine; project is a
filter.
Tests: TestDocTypeValidate accepts a RichText field; TestFieldTypeValidation
proves a Lexical JSON string round-trips verbatim through validation.
The ML control plane (clients/ml, /v1/ml) ran under the pod SA (cloud-api),
braiding KServe/Kubeflow cluster reach onto the product-API identity. newDynamic
now re-scopes the in-cluster client to a mounted cloud-ml ServiceAccount token
when HANZO_ML_TOKEN_FILE is set (keeps in-cluster host+CA, swaps only identity),
fail-closed if the configured token is unreadable. Unset -> unchanged behaviour
(pod SA) for local/dev and pre-cutover. Pairs with universe ml-rbac.yaml
(cloud-mlsvc ClusterRoleBinding -> cloud-ml). go build + go vet clean.
The execution queue for "all Hanzo Go services merge into the one cloud binary"
(HIP-0106): wave 0 already-merged (8 embedded modules + 37 native clients),
wave 1 tasks+visor (this build), wave 2+ mount queue (notify2, extract-svc,
playground, ...), and keep-standalone with reasons (iam/gateway/kms-MPC/registry
/s3/docdb/chain daemons). Waves are sequential through go.mod to avoid the
in-flight collision this wave hit.
Consolidates the Tasks product surface into cloud — the follow-up durable.go
named ("consolidating that surface into cloud is the follow-up"). durable.go
already embeds the ONE tasks engine in-process (loopback ZAP :19999) for ai's
durable ingest; this mounts THAT SAME engine's HTTP handlers, so the Tasks
UI/API reads the same durable state as ingest. One engine, one binary, one way —
no second Embed.
- clients/tasksvc (order 147, before ai's /v1/* catch-all): adapts the shared
engine's HTTP surface onto the zip mux at /v1/tasks/* + the embedded React UI
at /_/tasks/*. The engine is created after MountAll, so the surface resolves
cloud.EmbeddedTasks() lazily per request (503 fail-soft until wired).
- gate: settings/cluster/health stay open (no per-org data); the data surface
(namespaces/workflows/mcp/events) refuses an unvalidated principal (403, never
the unscoped store) and threads the gateway-validated org into the engine via
tasks/pkg/auth.WithIdentity — per-(org,ns) shard isolation, matching the rest
of the cloud data plane (clients/principal).
- durable.go: export EmbeddedTasks() — the single shared-engine accessor.
- bump hanzoai/tasks v1.43.0 → v1.46.0 (in-proc identity seam + one sqlite
driver). No local replace directives.
Proof: /v1/tasks/cluster returns nodeId "cloud-tasks" (durable.go's engine),
settings 200, data routes 403 without a principal, /_/tasks UI 200.
ERP (clients/erp): ERPNext-core DocType fixtures + native-Go GL/stock hooks —
idempotent deterministic-leg postings (exactly-once under concurrent submit),
on_cancel reversal, finite-guarded totals, double-entry submit gates.
Help (clients/help): Frappe Helpdesk-core fixtures, pure DocTypes, no hooks.
Both register on the framework engine at init; installed per-org via
/v1/framework/modules/{erp,help}/install. No new HTTP surface — ERP/Help ARE
documents on /v1/framework/*, drawn by the same generic DocType renderer as CMS.
Verified CGO=0 (production Dockerfile config): binary boots clean (no double
sqlite driver register), /v1/framework/health 200, /v1/models unshadowed (503
real AI handler), erp+help tests green under -race.
Mounts /v1/bots in cloud as the sibling of /v1/machines — a Bot is an
Agent(cloud /v1/agents) + a kind=bot Machine(vm) + their AgentBinding,
composed as a thin proxy over the SAME Visor client the machines routes use:
GET /v1/bots list (vm /v1/machines?kind=bot + bindings join)
POST /v1/bots/launch machine launch{kind:bot} THEN bind-agent
GET /v1/bots/:id machine + its binding (404 if not a bot)
DELETE /v1/bots/:id unbind THEN terminate the machine
POST /v1/bots/:id/:action message=agent run | stop|pause=unbind
Plus the machine agent-binding proxies cloud lacked (vm already serves them):
POST /v1/machines/:id/bind-agent
GET /v1/machines/:id/agent-binding
DELETE /v1/machines/:id/agent-binding
GET /v1/agent-bindings
Every route org-gated by the validated principal (principal.Tenant), forwarded
to vm as ?owner=<org> — 403 without a valid IAM owner, exactly like machines.
No vm change: kind=bot launch + bind-agent are already live at visor:19000.
message runs the bot's bound agent via the ONE agent runner (/v1/agents/:agent/run).
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Embed's default os.MkdirTemp("") resolves to /tmp, absent in the distroless cloud image
→ 'tasks.Embed: tempdir: stat /tmp: no such file or directory' → fail-soft to inline.
Pin DataDir to {deps.DataDir}/tasks (MkdirAll first) so the in-process engine actually
boots. Verified: without this the warn+inline-fallback fired cleanly in prod (v1.786.48).
cloud embeds hanzoai/tasks IN-PROCESS (loopback ZAP, durable.go) and injects a per-org
dialer into ai's ingest — there is no external tasks service to auth to, no per-org token
minting, no HTTP inner-cloud hop. Long ingests (github/crawl/s3) run as durable workflows
in the owner's namespace (CONTRACT §6); upload stays inline. Fail-soft: embed error →
ai dialer unset → inline fallback. Bumps ai → v1.796.4 (per-org ingest dialer). One engine,
one binary, one way. NOTE: embedded store is memdb today (survives worker crash via retry,
not process restart); console /tasksd still points at the cluster tasks Service for the UI
(consolidating that surface into cloud is the follow-up).
Red verdict FIX-THEN-SHIP (0 critical; all findings within-tenant integrity —
isolation, forge-proofing, gates, ledger perms, console deletions all refuted/solid).
- HIGH (TOCTOU double-post): on_submit postings ran before the atomic docstatus flip
with hash-named legs, so N concurrent submits over-posted the ledger 4-6x. Every
GL/stock leg now has a DETERMINISTIC name (voucher-<kind>-<index>, via prompt
autoname) and postLeg is idempotent (pre-read + re-check on create-conflict), so
posting is exactly-once under any concurrency AND replayable after a partial
failure — no engine change, uses the existing store API. Balances stay SUM(ledger).
- MED (cancel did not reverse): on_cancel hooks append reversing ledger rows (swap
debit/credit; negate qty), sharing the SAME leg computation as submit.
- LOW (non-finite total -> 500): finite guards in the totals hooks -> clean 422.
- LOW (comment): tightened the ledger-immutability doc — manager bypass is
within-tenant authority only (Red confirmed no cross-org escalation).
Regressions: 8 concurrent submits -> exactly 1x200 + GL posted once (2 legs,
race-clean); cancel -> net GL zero-sum + net stock zero; overflow qty*rate -> 422.
go test -race 9/9 (CGO=1) + CGO=0 + vet clean + full cmd/cloud binary.
Every AI call already writes the proven hanzo.cloud_usage ledger (model,
provider, tokens, cost_cents, org, user, status) via ai/object's zapWriteUsage
— the same recordTrace funnel that emits the OTel GenAI span. This mounts the
missing native route the console Observe > Observations surface calls, reading
that ledger as Langfuse-v3 GENERATION observations (org-scoped by the validated
principal, bound positional params, bounded LIMIT). No new emission path: one
recordTrace, fanned to o11y (span) + Langfuse (span) + this ledger (read).
- telemetry.go: Observation model + ObservationFilter; ListObservations on the
Telemetry interface, dsTelemetry (cloud_usage query), memTelemetry (honest
empty); asInt64 coercer (cloud_usage UInt32 tokens / UInt64 cost — asFloat
only handles float types).
- eval.go: listObservations handler + observationView/toObservationView mapping
to the console Observation shape; GET /v1/evals/observations route.
- observations_test.go: view mapping (success/error), asInt64 coercion, mem
telemetry empty + org-required.
Requires a cloud rebuild + deploy-by-sha to go live (route is code, not env).
Co-authored-by: hanzo <a@hanzo.ai>
- zaptrace: otlptrace.Client over the ZAP wire (github.com/zap-proto/http) —
spans marshaled as OTLP protobuf, shipped over ZAP frames, NEVER OTLP-HTTP
(:4318)/gRPC(:4317). Target = collector zapreceiver (:4319).
- cmd/cloud/telemetry.go: initTelemetry uses the ZAP exporter; enable via
OTEL_EXPORTER_ZAP_ENDPOINT (default otel-collector.hanzo.svc:4319); keeps the
no-op-when-unset posture.
- clients/aihttp.go ChatCompletion: OTel GenAI client span (gen_ai.system/
operation.name/request.model + response.model + usage.{input,output}_tokens),
RecordError on failure. Captures the previously-discarded resp.Usage.
- clients/agents/agents.go: runAgent opens a per-run root span (agent.run),
executeRun a child agent.step span; the LLM client span nests under them —
one trace per run: run -> step -> chat.
Test: zaptrace TestUploadTracesOverZAP green (span over the real ZAP wire).
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Supersede the Base swap (67bb0bc). Hanzo Base has a REST/realtime document
API but NOT the MongoDB wire protocol, so it is not a drop-in docdb — a
customer's mongodb:// driver cannot speak to it. The managed "document
database" must accept existing MongoDB drivers unchanged.
The per-org dedicated docdb instance is now a FerretDB v1.24 instance
speaking the MongoDB wire protocol on :27017, backed by Hanzo SQL (SQLite,
pure-Go) — Mongo databases map to SQLite files under /state, collections to
tables, documents to JSON1 rows. ZERO raw mongod (no WiredTiger), ZERO
Postgres, ZERO go.mongodb.org driver in cloud (FerretDB is a deployed pod,
not a Go import). Per-instance SCRAM auth via FerretDB's SQLite-backend
new-auth (FERRETDB_TEST_ENABLE_NEW_AUTH + FERRETDB_SETUP_*); the returned
password is the instance admin credential, sealed in KMS.
engine fields:
- image ghcr.io/hanzoai/docdb-sqlite:1.24.0 — pinned FerretDB v1 with the
SQLite backend handler, mirrored from upstream by hanzoai/docdb CI (v2 and
the hanzoai/docdb Postgres/DocumentDB fork both dropped SQLite; v1 is the
last line that carries it). Distinct package from the Postgres-backed
ghcr.io/hanzoai/docdb that backs shared chat-docdb.
- fsGroup 1000: FerretDB is distroless and runs as UID:GID 1000 (no
entrypoint can chown), so a fresh block PVC must be group-writable via the
pod securityContext.fsGroup the operator stamps from spec.fsGroup — else
the instance CrashLoops on "permission denied" writing /state.
- FERRETDB_STATE_DIR + FERRETDB_SQLITE_URL pin both process state and the
SQLite files onto the mounted /state PVC (persist across restarts).
Verified end-to-end against the FerretDB v1.24 SQLite image with this exact
env: mongosh Insert/Find/Update/Delete over the wire protocol, and /state
held per-db admin.sqlite + events.sqlite with "SQLite format 3" magic — no
WiredTiger datadir, no Postgres PG_VERSION (both locally and in hanzoai/docdb
CI on the mirrored image).
TestDedicated_DocdbIsFerretOnSQL asserts the FerretDB image, SQLite backend
env, mongodb:// connString, per-instance SCRAM credential, fsGroup 1000, and
the absence of any Postgres/IAM/Base env. unavailableKinds stays empty.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Second and third app lanes on clients/framework, reusing the generic DocType
engine + install path + generic renderer (like clients/cms) — zero forked
engine, zero HTTP surface of their own, per-org on Base/SQLite.
- clients/erp: 20 ERPNext-core DocTypes (module "erp") — masters (item/
warehouse/customer/supplier/account/department/employee), submittable
transactions with child Tables (sales-order/-invoice/purchase-order/
stock-entry/journal-entry/payment-entry), and two read-only hook-posted
ledgers (gl-entry, stock-ledger-entry). Business logic as native-Go hooks:
line/document totals (before_save), balanced GL on invoice/journal/payment
submit, append-only stock ledger on stock-entry submit, and double-entry /
non-empty submit gates. Posting is org-scoped via ev.Org + ev.Store.
- clients/help: 5 Helpdesk DocTypes (module "help") — hd-ticket (status
workflow) + hd-agent/hd-team/hd-sla/hd-canned-response. Pure fixtures, no
hooks — the purest DRY proof; self-contained (no cross-lane Link).
- SLUG names (erp-*/hd-*), series naming for transactions, field naming for
masters (console slugifies on write), hash for ledgers — all reachable via
the generic renderer; no collision with CMS (Author/Media/Page/...) or CRM.
- subsystems: blank-import erp + help so their init() registers the lanes.
Tests: fixtures-valid/model-spec/install-transact-roundtrip/submit-gates/
tenant-isolation/ledger-read-only — go test -race green (CGO=1 and CGO=0),
go vet clean, full cmd/cloud binary builds.
The pure detection engine moves to clients/security/detect (a stdlib-only
LEAF): one engine, two surfaces — the /v1/security HTTP subsystem and the
new local CLI both consume it, neither drags the other in. The subsystem
now calls detect.ScanContent/Rules/SeverityRank; behavior is unchanged.
hanzo security scan [path...] walks a tree, runs the engine, and exits
non-zero when a finding at/above --fail-on (default low; 'none' = report
only) is present — a pre-commit/CI/agent guardrail with no server, auth,
or network. Skips vendored/binary files; never prints a raw secret (masked
preview only). hanzo security rules lists the catalog. -o json supported.
Tests: 9 CLI (find+fail, clean-pass, fail-on threshold/none, json, vendor+
binary skip, bad flag, rules, control-verb), engine+subsystem unchanged.
Extends the white-label issuer-set validation (this branch) to the AUDIENCE half:
a lux/zoo/pars session token carries aud=<brand>-cloud (HIP-0111: client_id == app
== aud), so the audience allowlist must include each or the cloud-native identity
sanitizer 401s a valid lux token even after the issuer gate passes.
- brand.go BrandAudiences() derives <brand>-cloud for every registry brand
(hanzo-cloud, lux-cloud, zoo-cloud, pars-cloud, bootnode-cloud) — one source of
truth, mirroring BrandIssuers(); no hand-listed audience.
- config.go jwtAudiencesFromEnv() now ALWAYS unions BrandAudiences() into the
resolved allowlist (baked like the brand issuers). A legacy hanzo-only
GATEWAY_ALLOWED_AUDIENCES env override still accepts lux-cloud — the brand auds
don't depend on getting the deploy env perfectly right. Fail-secure: only ADDS
the known-good <brand>-cloud client_ids, never an arbitrary aud. unionStrings
dedupes so an env-supplied entry is never duplicated.
Tests: BrandAudiences (registry-derived, covers every brand), jwtAudiencesFromEnv
brand-union (baked default AND a hanzo-only env override both accept lux-cloud, no
duplicate). Paired with hanzoai/ai#64 (the per-brand signin code exchange).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Widen identityValidator to a trusted issuer SET (primary UNION BrandIssuers) so one
cloud binary validates hanzo AND lux/zoo/pars tokens off the one shared IAM JWKS.
Fail-secure: only known-good brand issuers added. NOT built (phantom go-sqlite3
v2.0.3 dep blocks) / NOT gated / NOT deployed — needs: per-brand EXCHANGE client
verification, build-dep fix, hanzo-login no-regression gate, red review.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
github.com/hanzoai/sqlite is the ONE Hanzo SQLite driver: it registers the
"sqlite" database/sql name under BOTH build tags (cgo → mattn+SQLCipher,
encrypted at rest; !cgo → pure-Go modernc, wrapped internally). Fourteen
cloud stores plus the pg→sqlite migration blank-imported modernc.org/sqlite
DIRECTLY, so a CGO_ENABLED=1 build registered "sqlite" twice — the fork's
mattn registration AND the direct modernc one — and panicked at init
("sql: Register called twice for driver sqlite"), taking down the whole
fused hanzo/cloud binary.
Swept every direct `_ "modernc.org/sqlite"` → `_ "github.com/hanzoai/sqlite"`
(sql.Open("sqlite", …) calls unchanged — same driver name). go.mod promotes
the fork to a direct require and demotes modernc to indirect (it survives
only as the fork's !cgo backend). Stale comments calling modernc the
"primary" driver corrected. Production already builds CGO_ENABLED=0 (one
modernc package, no collision); this makes the driver choice consistent and
unblocks a CGO_ENABLED=1 + libsqlcipher encrypted build.
NOTE: five UPSTREAM modules (base/core, ai/object, o11y sqlstore,
commerce/db, orm/db) still import modernc directly; a CGO_ENABLED=1 fused
build stays collision-prone until they adopt the fork too. Out of scope for
this repo; tracked as the cross-repo follow-up.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
First business app lane native on the Hanzo Framework DocType engine:
generic module-install path (/v1/framework/modules[/:module[/install]]) +
the CMS content model as fixtures (Page/Post/Article/Media/Navigation/Author,
module cms). Additive — no change to the proven /v1/framework/* isolation.
RED verdict: SHIP (0 crit/high/med).
The first Semgrep-class capability shipped natively in the cloud binary,
per hanzoai/security POSTURE.md's plan of record. One subsystem, the
established clients/ pattern (self-registering Mount, org-scoped store
under DataDir, audit + metering wired), zero external tools.
- engine.go — the reusable detection core: pure (path,content)→findings,
no I/O. Pattern rules (AWS/GCP/GitHub/Stripe/Slack/npm keys, private-key
blocks, JWTs) + a Shannon-entropy-gated generic-assignment rule so
`secret = "changeme"` is not flagged but a real high-entropy token is.
THE INVARIANT: a finding never carries the raw secret — only a masked
preview (4+4 ends, middle starred; short secrets fully starred) and the
SHA-256 fingerprint (dedupe + rotation tracking). Persisting plaintext
would make the findings DB the very thing we scan to prevent.
- store.go — per-tenant SQLite ({DataDir}/security.db), scans + findings
tables, org column is the isolation boundary on every query; SaveScan is
one transaction so a scan is never half-written. Mirrors clients/git.
- security.go — mounts /v1/security/{health,rules,scans,scans/:id,
findings,findings/:id}. submitScan runs the engine, persists redacted
findings, meters one unit, emits a tamper-evident audit record (the
tally, never the secrets). Registered cloud.RegisterWithShutdown(
"security", 136, …) + one blank-import line in subsystems.
Tests (14, no skips, no fakes): engine_test proves each rule fires, the
entropy gate, line mapping, dedupe, severity ordering, and that no field
ever echoes the raw secret; security_test proves the HTTP surface,
cross-tenant isolation (evil sees 0 of acme's scans/findings, 404 on id),
the no-principal 403, the severity filter, and that a clean scan persists
a real zero-findings record. go build ./... + go vet + -race all clean.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Framework: a generic, DRY module-install path so an app lane (CMS/ERP/Helpdesk)
declares its DocTypes as fixtures (framework.RegisterModule, sibling to
RegisterHook) and installs them per-org via the engine's own gate:
GET /v1/framework/modules list registered lanes
GET /v1/framework/modules/:module lane fixtures + which are installed in-org
POST /v1/framework/modules/:module/install ensure fixtures exist (managerOnly, idempotent)
'modules' reserved so the static routes are never shadowed by a document route.
CMS (clients/cms): the first lane — the content model as fixtures only, NO HTTP
surface of its own. Page/Post/Article (slug-named, status Draft/Published,
author Link), Media (Attach-backed DAM), Navigation (JSON menu), Author. A CMS
collection IS a framework DocType (module 'cms'); content IS documents;
publishing IS a status field. Registered at init; installed per-org.
Secure by default: install is managerOnly (owner seeded trust-on-first-use),
create-if-absent (never clobbers a customised DocType), stamps the module tag,
and every op stays per-org via principal.Tenant.
Tests: install/idempotency/unknown-404/tenant-isolation/forged-principal-403/
non-owner-403/module-tag; CMS fixture validity + content-model spec + a full
HTTP install->create Author->create Page(link)->publish->filter round-trip.
The router (zip over fasthttp) runs with Fiber's default UnescapePath:false, so
c.Param() returns path segments verbatim. A DocType/document/role name that is
legal per docTypeNameRe but contains a space ('Sales Invoice', 'System Manager')
arrives percent-encoded ('%20') and never matched its stored value: GET/PUT/
DELETE /v1/framework/:doctype/:name, submit/cancel, and revokeRole (:user/:role)
all 404'd, while create+list (no name in the path) worked — records could be made
yet be unreachable, and a granted System Manager could not be revoked.
Fix scoped to the ONE seam: pathParam() percent-decodes every framework path
param (getDocType/replaceDocType/deleteDocType, access(:doctype), docName,
revokeRole). NOT a global fiber.Config UnescapePath flip — that would change
segment splitting on the KMS secret-path, model-catalog, git and s3 wildcards
(c.Params("*")) that legitimately carry encoded slashes; the framework-local
decode is orthogonal and zero-blast-radius. Malformed escapes fall through to an
honest 404, never a panic.
Unblocks space-named DocTypes for the CMS/ERP/Help app lanes. Tests: red→green
round-trip (create -> GET/PUT/submit/cancel/DELETE by name) + space-named role
revoke; full framework suite green under -race.
Reconciles task #52 (eliminate Mongo) onto main's dedicated-per-org instance
model. The managed "document database" (docdb) is now a dedicated per-org Hanzo
Base instance — JSON document collections on per-tenant SQLite with native
realtime (SSE /v1/realtime), IAM-native — NOT a per-org FerretDB/Mongo instance.
- dedicated.go: docdb engine swapped ghcr.io/hanzoai/docdb:0.1.0 (mongodb://,:27017,
POSTGRES_*) -> ghcr.io/hanzoai/base (http://.../v1, :8090, /data), dsType "base".
New engine fields: dataMount (emits spec.volumeMounts so the data PVC actually
mounts — the operator does NOT auto-mount) and iamAuth (IAM-native: no per-
resource password generated/sealed/returned; admin Secret carries IAM_URL/
KMS_URL/IAM_CLIENT_* from the cloud binary's own IAM identity). baseInstanceEnv
helper. createDedicated honors iamAuth (no pw path). datastoreCR emits
volumeMounts. Runs on the operator's GENERIC Datastore controller — spec.type is
free-form, image/ports/volumeMounts drive the StatefulSet verbatim, NO operator
Rust change (verified). datastore engine stays ClickHouse (not Mongo).
- provisioning.go: header + sanitizeIdent comments de-Mongo'd.
- go.mod: mongo-driver demoted direct -> // indirect (zero Go imports of
go.mongodb.org remain; it survives only as a transitive requirement).
- test: TestDedicated_DocdbIsBase asserts the docdb CR is the base image on :8090
with /data mounted, connString http://.../v1 (no mongodb://), no credential
(IAM-native), IAM env in the admin Secret. PASS.
Audit (unchanged): zero customer docdb data, so the swap is clean.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Red found the self-service enablement write path (POST /v1/enablement/optin|optout)
+ the view keyed on raw c.Org() instead of principal.Tenant(c). On the bearer-less
direct-to-pod path SanitizeIdentity restores a client X-Org-Id with no validated
principal, so an off-gateway caller could opt an org it does not own into/out of a
beta (cross-tenant enablement write; bounded — betaOrgs membership of already-beta
items only, no money/data/global-state, not reachable through the gateway).
Fix (DRY — pricing already imports principal, the READ catalog gate already uses it):
- enablementOpt: resolve subject via principal.Tenant(c), 401 on !ok (was raw c.Org()).
- enablementView: resolve via the existing trustedOrg(c) (validated-principal gate).
- Corrected the docstrings (subject is the VALIDATED tenant, not 'never client-supplied').
Red's two PoC attack tests (enablement_attack_test.go) now GREEN; full enablement
suite unchanged + green + -race. Closes the one MEDIUM from Red's cockpit review.
Live-verified: /v1/billing/transactions returns {count, transactions:[...]}, not a bare array.
My client decoded a bare [] (test fake also returned bare — mock hid the bug), so the analytics
retention/churn/active/usage ledger read got ZERO rows despite real usage (maxpower: 268 txns).
Now decodes the wrapped shape (bare-array fallback for robustness); test fake mirrors the live shape.
User app secret env is no longer refused — it is sealed into cloud's embedded
KMS and wired to the pod via an operator-materialized k8s Secret, never
plaintext, never logged.
The path a secret takes (secrets.go):
1. SEAL — createApp / PUT .../env seal every secret:true value into deps.KMS
at a per-tenant/app coordinate (platform/<tenant-ns>/<app>/<KEY>);
the persisted env_json value is blanked. Fails CLOSED if KMS is
unavailable — a plaintext secret never lands in the DB as a fallback.
2. DECLARE — on deploy (applyLive, the ONE shared choke point) cloud writes a
canonical KMSSecret CR (secrets.lux.network/v1alpha1) into
tenant-<org> declaring a managed Secret <app>-env sourced from that
KMS scope. Best-effort: a missing CRD/RBAC degrades to an honest
'pending' status, never a failed deploy.
3. MOUNT — the Service CR renders each secret env as valueFrom.secretKeyRef →
that Secret (optional:true so the pod boots pre-sync); the hanzo
operator (which already supports secretKeyRef) mounts it into the
Deployment env. cloud is never in the plaintext path at runtime.
Also: PUT /v1/platform/projects/:p/apps/:a/env to set/rotate env post-create;
honest secretSync status (pending|syncing|ready|failed) from the KMSSecret CR
conditions on the app view; KMSSecret teardown on app/project delete.
Tests: seal blanks+seals+fails-closed; injective KMS refs (no cross-tenant
collision); canonical KMSSecret CR shape; apply/patch/delete; sync-status
mapping; and an end-to-end deploy asserting the Service CR carries secretKeyRef
(never plaintext) and the KMSSecret CR is authored. go build ./...=0, vet clean.
datastore (ClickHouse) + docdb (FerretDB) were honest-gated (unavailableKinds)
because a SHARED backend can't scope a per-tenant role. Replace the gate with a
DEDICATED-instance strategy: each create launches the org's OWN instance via an
operator Datastore CR + admin Secret in tenant-<org> (derived from the VALIDATED
org, never a request field). Isolation is BY INSTANCE — a cross-tenant grant is
impossible, there being one tenant on the instance — which un-gates both kinds.
- dedicated.go: engine table (image/ports/admin-env/DSN per kind), instanceName
(<prefix>-<orgHash10>-<name>, DNS-1123), the k8s orchestrator (ensure tenant
ns + RBAC wait, apply/observe/delete the Datastore CR + admin Secret + reap the
retained PVC) behind an interface a fake stands in for, createDedicated,
reconcileDedicated (provisioning->ready off the operator's status.phase), and
dropDedicated.
- Billing (first-class): a provision debit carrying the size dimension
(Model=<kind>:<size>) lands on the CALLER's org via the ONE commerce meter, and
a recurring GB-day footprint sweep charges every running instance's own org —
the reserved hook, now unblocked by the instance's declared size. Drop removes
the row, stopping the meter, and reaps the PVC so no storage leaks.
- unavailableKinds now empty (mechanism kept); shared datastore/docdb
provisioners deleted (one way only); the 5 shared kinds untouched.
- Datastore CR (not DocDB) is used because only its controller writes
status.phase, the readiness signal; type forced per engine.
Tests: hermetic dedicated suite (fake orch + mock commerce) proves CR/Secret
shape, two-org isolation, ready reconcile, drop+PVC reap, and per-org billing
attribution; a build-tagged livecluster test proves the whole path against the
real operator. go build ./cmd/cloud + go test ./clients/provisioning/... green.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Completes the console subsystem's standalone-route port for the True 1-binary
FE. The three remaining console2 Next server routes that do REAL server work
(not vanishing BFF proxies) now terminate natively in the unified binary, so
console2 can drop them from its static export:
POST /v1/console/waitlist waitlist.go — session-gated join to the Base
waitlist plugin; the recorded email is BOUND to
the gateway-verified X-User-Email (a signed-in
user can't enroll a third party), honest 501
when WAITLIST_URL is unset.
GET /v1/console/embed-status embed.go — server-authoritative entitlement
(owning brand org / global admin only) + a
time-boxed reachability probe. SSRF-free: the
target is <app>.<brand-domain> for the FIXED
deployment brand (deps.Brand) — no client host
in the target at all.
POST /v1/console/topup/wallet topup.go — verify an HUSD transfer on-chain
(plain eth JSON-RPC, no EVM dep) and credit the
VALIDATED caller's own org for the ON-CHAIN
amount via the S2S commerce billing API. IDOR-
safe (ignores any client userId); honest 501
greenfield gate until HUSD is deployed.
All three resolve the caller from the VALIDATED principal only (same trust
boundary as keys/onboard); a forged X-User-Id/X-Org-Id is refused. docs is a
pure host->URL redirect with no server work, so it stays client-side in console2
(no handler here). Registered in the ONE routes() place; full unit coverage
(fake IAM/waitlist/RPC/commerce), go build ./... clean, binary boot-proven to
serve the console SPA at / with every /v1/console/* route resolving (403/501/503,
never 404).
v1.786.33 set zip.Config.ReadBufferSize=32768 (GATEWAY_READ_BUFFER_SIZE) but
zip v1.2.0's HTTP transport built a bare fasthttp.Server and dropped it — the
edge still 431'd at 4 KiB. zip v1.2.1 propagates the App Config onto the
transport's fasthttp.Server, so the 32 KiB header ceiling now takes effect on
cloud:8000 (the api.hanzo.ai/v1/* backend).
The console E2E saw /v1/crm/summary miss a just-created record. Root cause was a
stale/eventually-consistent read; the current handler already counts LIVE
(s.Counts -> SELECT COUNT(*) per table on the same store the writes hit), so a
create/delete is reflected with ZERO lag — verified live (create company ->
summary companies +1 immediately).
Add TestSummaryReflectsCreateImmediately: create -> Counts shows +1, delete ->
Counts shows -1, all in one synchronous flow. This guards against any regression
to a materialized/async rollup. No production code change needed — the fix is the
regression lock.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Gap 2 (analytics 502 on ClickHouse i/o timeout): /v1/analytics/{overview,timeseries,top}
returned a raw 502 when a DatastoreQuery hit a connectivity failure even though
requireDatastore()'s not-connected path already 503s. warehouseErr() now maps a
transport/connectivity error -> 503 'warehouse unavailable' (retryable); a REACHABLE
warehouse that rejected the query (bad SQL/protocol) stays 502. Table-driven test.
Gap 3 (docdb/datastore provisioning) — SAFETY REWORK (replaces the earlier
readWriteAnyDatabase change, which was cluster-wide = a cross-tenant hole):
- datastore + docdb are honest-GATED (unavailableKinds -> 503 'not yet available',
refused BEFORE billing or any backend write) because their backends cannot mint a
per-tenant-SAFE credential:
* datastore (ClickHouse): no grant-capable per-tenant admin (GRANT ALL -> Code 497);
unblocking is backend-side (StatefulSet grant-capable admin).
* docdb (FerretDB/DocumentDB): engine implements ONLY cluster-wide roles
(clusterAdmin -> Postgres SUPERUSER, readWriteAnyDatabase) — no per-db role.
- docdbProvisioner.Create keeps requesting the CORRECT per-db 'readWrite' role (the
tenant-safe target); when FerretDB supports it, drop the gate and it works as-is.
- Tests: gated kinds -> 503 with the provisioner never run; the 5 guaranteed kinds
(sql/vector/kv/search/s3) are asserted NOT gated.
Bar met: 5/7 data kinds fully work; datastore+docdb show an honest 'coming soon',
never a security hole.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
The public HTTP edge (zip/fiber) uses fasthttp's default 4 KiB per-conn
read buffer, which caps total request-header size and returns 431 (Request
Header Fields Too Large) above it. Once an admin-guard Domain=.hanzo.ai SSO
cookie is set on every subdomain, a browser's request headers cross ~4 KiB
and every request to api.hanzo.ai/v1/* (gateway -> cloud passthrough) 431s.
Raise the edge ceiling to a sane 32 KiB (nginx large_client_header_buffers
parity) via zip.Config.ReadBufferSize, env GATEWAY_READ_BUFFER_SIZE (shared
with the gateway edge so both trust boundaries agree on ONE value; tunable
down if the per-conn memory budget demands). Internal zip services keep the
4 KiB framework default — only the browser-facing edge opts up.
Repro (pre-fix): POST cloud:8000/v1/agents with a 9 KiB Cookie -> 431
Server: fasthttp. Post-fix: same request -> 403 (auth), no 431.
create and list return an agent's public id (agent_<hex>), but get/run/update/
delete/runs resolved the URL path segment ONLY against the name column — so a
client that used the id create returned got 404 "agent not found". A created
agent was listed but neither gettable nor runnable by the identifier the API
handed back.
One-way fix: Store.Resolve(org, ref) matches either the public id or the
org-unique name (id wins on the astronomically-unlikely in-org collision),
org-scoped fail-closed so a cross-tenant ref is 404, never a leak. Every
path-addressed handler (get/update/delete/run/runs) resolves through it and keys
every downstream store op on the resolved a.Name. Route param :name -> :ref to
say what it accepts. The run path's validated-principal gate and single
product=agent debit are unchanged.
Tests (go test -race): Resolve by id and by name return the SAME agent; full
create -> get-by-returned-id -> run-by-returned-id all 200 the same agent with
real output; cross-org ref denied 404; a run addressed by the returned id meters
exactly once (product=agent).
The 1-binary console (HIP-0106) shipped as a 3-file STUB: nothing ran
console2's static export before `go build`/the image build, so `//go:embed
all:webui/dist` baked only the fallback shell.
- `make webui` (new): runs hanzoai/console2 `npm run build:embed` and overlays
the static export into webui/dist so a plain `go build` embeds the full
@hanzo/gui console. `make build-standalone` = webui → build. CONSOLE2_DIR
points at a console2 checkout (default ../console2).
- Dockerfile console stage + webui.go + the stub shell: docs corrected — the
pipeline is `build:embed` (a static export), not `next build`; the export now
prerenders clean (console2 build-embed.mjs neutralizes the root layout's
request-time headers() read), so the image embeds the real console instead of
silently degrading to the shell. Bumped the export heap to 8192 for headroom.
Verified: build:embed → webui/dist (index.html 368 KB, /_next/static assets) →
CGO_ENABLED=0 go build ./cmd/cloud → the running binary serves the real console
at / (200, references /_next/, not the stub), the SPA shell for deep links
(/orgs), fingerprinted assets immutable-cached, and the /v1 API on the SAME
origin ({"service":"base","status":"ok"}); unmatched /v1/* is a real 404, not
HTML. webui_test.go (7 tests) green against the real bundle.
Red measured 3-6 System Managers seeded when concurrent role-less members first
administered a fresh org: managerOnly did a check-then-insert (OrgHasRoles then
AssignRole) with a TOCTOU window. Fix: store.SeedOwnerIfUnowned is a SINGLE
conditional INSERT ... SELECT ... WHERE NOT EXISTS(SELECT 1 FROM fw_roles WHERE
org=?), so the unowned-check and the insert are one atomic statement — exactly
one concurrent first-caller's row lands. RowsAffected==1 => this caller is the
seeded owner; ==0 => re-resolve (a concurrent grant may have made them a
manager) else 403. No UNIQUE index (multiple SMs are legit later via AssignRole;
only the AUTO first-seed must be singular). Removed the now-dead OrgHasRoles.
Test: TestAtomicOwnerSeed — 8 concurrent role-less first-callers → exactly 1
seeded winner + exactly 1 System Manager row. 22 tests total, race-clean.
searchGuard treated the searxng X-API-Key as OPTIONAL — a MISSING key passed —
so GET /v1/websearch/search was an open proxy to the Hanzo-operated metasearch
instance (unauthenticated request-forgery + cost surface). Its scrape sibling
(scrapeHandler) already fails closed; this brings search to parity:
- key unset → 503 (surface not configured, never open-to-all)
- X-API-Key missing → 401 (constant-time compare of "" vs want fails)
- X-API-Key mismatch→ 401
Safe for the real caller: the LibreChat searxng client sends the configured
searxngApiKey (universe chat configmap wires searxngApiKey=${WEBSEARCH_API_KEY})
as X-API-Key, so only anonymous callers are turned away.
Tests: TestSearchMissingKeyRejected (was ...Allowed) → 401; new
TestSearchUnsetKeyFailsClosed → 503; TestSearchProxyRewritesToSearchPath and
TestMountRoutesThroughRouter now present the key. go build/vet/test green.
LOW-1 — Single submit-immutability: updateDocument/createDocument for a Single
now route through writeSingle, which enforces the SAME draft-only guard as the
non-Single path (a submitted/cancelled Single → 409, not a silent mutation) and
preserves a redacted Password across an unchanged update.
LOW-2 — secure-by-default permissions (no open-to-all footgun):
- permission.can() is now DEFAULT-CLOSED: removed the 'empty perms => open to
every org member' branch. A permless doctype is manager-only; a role-less
member is denied.
- DocType.normalize() seeds a System Manager perm at define time, so a stored
doctype is never silently permless (explicit in UI + audit).
- Owner seeding moved from resolveAccess (any member is SM until a role exists)
to managerOnly as trust-on-first-use: the FIRST validated principal to
administer an org with no roles becomes its persisted System Manager (the
owner) — exactly one member, deterministically, never cross-tenant.
Tests: +2 (TestSingleSubmitImmutability, TestPermlessDefaultClosed); 21 total
race-clean. go build ./... CGO=1 & =0 green, vet + gofmt clean. Fixed binary
boot-verified (framework health 200, gate 403, forge 403).
The prior fix re-fetched luxfi/age with checksum-checking off, which recorded the
DIRECT-vcs hash. luxfi/age@v1.5.0 was force-re-tagged, so the direct tree hash
differs from the immutable proxy zip hash — the container build (GOPROXY=proxy +
GOSUMDB=sum.golang.org, GOPRIVATE dropped for luxfi/* on purpose) verifies against
the sumdb and hit "checksum mismatch / SECURITY ERROR" on `go mod download`.
Replaced the h1: zip hash with the canonical value from
sum.golang.org/lookup/github.com/luxfi/age@v1.5.0 (the /go.mod hash already
matched). Now go.sum == what the proxy+sumdb serve → container verification passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The console-embed stage's contract is "a missing static target is a degrade, not
an error" — but the guard only handled the target being ABSENT. When console2
exposes build:embed AND it CRASHES (currently: /signin Server-Components
prerender error kills `next build`), the `&&` chain failed the whole cloud image,
so a frontend prerender bug took down the entire Go backend build (release runs
for projectsvc S3 fix + kms refactors all failed here, not on Go).
Complete the stated contract: wrap build:embed so a build FAILURE also degrades to
the committed fallback shell (/out stays empty → Go embeds webui/dist/index.html).
The standalone console2 Deployment is the primary console; this embed is a
same-origin convenience and must never gate the backend image.
(console2 /signin static-export prerender crash tracked separately for the
console track — this makes cloud CI robust to it either way.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two dep-rot issues blocking the cloud (Go backend) build:
1. A transitive dep requires the non-existent mattn/go-sqlite3 v2.0.3+incompatible.
The unversioned replace didn't stop Go reading v2.0.3's go.mod during graph
load. Fixed with a VERSIONED replace (v2.0.3+incompatible => v1.14.16, the last
real go-sqlite3, drop-in package sqlite3). cloud's primary sqlite is
modernc.org/sqlite (pure-Go); hanzoai/sqlite (encrypted, package `sqlite`) is a
separate driver, adopting it is a real migration not this phantom fix.
2. luxfi/age@v1.5.0 go.sum checksum mismatch → removed stale lines + re-fetched.
go build ./internal/org/ (the sqlite consumer) now clean; module graph resolves.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Hanzo Framework: Frappe's DocType/metadata core rebuilt native in Go on
Base/SQLite, mounted at /v1/framework/* (subsystem order 129). ONE engine +
ONE generic UI renders every business app — CMS content-types, ERPNext
DocTypes, Helpdesk all become just DocTypes on this engine. No Frappe/Python
runtime dependency; the engine is pure Go.
- DocType registry: define/list/get/replace/delete metadata per-org
- Generic metadata-driven document CRUD with ?filters=/fields=/order_by=/limit=
- Fieldtypes: Data/Int/Float/Currency/Check/Date/Datetime/Text/SmallText/
LongText/Select/Link/Table/Attach/JSON/Password (all validated)
- Naming: hash / field: / prompt / series patterns (INV-.YYYY.-.#####)
- Link relations (in-org ref check + fetch_from), Table child rows
- docstatus 0/1/2 with submit/cancel for submittable doctypes
- Per-org permissions (DocType perms by role) + per-org role store
- Go lifecycle hook interface (before_insert/before_save/after_save/
on_submit/on_cancel/on_trash) — gpython/goja runner is a later add to the
SAME interface
- Password fields: argon2id hash on write, redact on read (fail-secure)
Security: org derived ONCE via clients/principal.Tenant (validated principal
only; forged X-Org-Id refused 403). Every table + query is org-scoped. 19
tests (race-clean) prove cross-org isolation, forged-principal refusal,
permission enforcement, field-type validation, and the docstatus lifecycle.
Boot-verified locally (health 200, doctypes 403, forge refused).
The embedded luxfi/kms core (cloud/types-only leaf, built by build.go before the
app exists to break the import cycle) is now just 'kms'; the Fiber /v1/kms/* mount
subsystem (imports cloud) is 'kmssvc' (its existing internal name). One clean name
each, no unnecessary compound. build+vet+tests green.
publicReadPolicy used Principal {"AWS":["*"]} + array Resource, which
SeaweedFS's S3 policy engine rejects with 'Policy has invalid resource' —
aborting ensureBucket (SetBucketPolicy) BEFORE any files upload, so every
projectsvc deploy failed ('object storage'/'invalid resource') and no site
was ever served. Use scalar Principal "*" + scalar Resource, which SeaweedFS
accepts and is equally valid on AWS S3 / MinIO. Verified: mc anonymous
set-json with this exact shape succeeds against the s3.hanzo.ai SeaweedFS
gateway; a site uploaded to the now-public hanzo-sites bucket serves 200 at
https://s3.hanzo.ai/hanzo-sites/<org>/<slug>/index.html.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Found by testing the ACTUALLY-deployed hanzoai/crawl:0.0.1 (= Crawl4AI
0.8.6) against clients/websearch's crawl adapter: 0.8.x/0.9.x return the
/crawl result's `markdown` as an OBJECT
{raw_markdown, fit_markdown, markdown_with_citations, ...}, and signal the
batch with a boolean `success` (no `status`). crawlResult.Markdown was
typed `string`, so json.Decode errored on the object form → crawl()
returned an error → EVERY scrape returned {success:false} with empty
content. hanzo.chat Web Search's scrape half was therefore dead even once
crawl is running.
Fix: markdownField.UnmarshalJSON accepts either a bare string OR the
object (preferring the cleaned fit_markdown, raw_markdown fallback); the
crawlResponse envelope now also accepts boolean `success` alongside the
legacy `status`. Neither envelope field is required — Results[0].Success
is authoritative.
Tests (real 0.8.6 response shape):
TestScrapeHandlesCrawl4AIObjectMarkdown — object markdown + bool success
→ success:true, returns fit_markdown (was: {success:false}).
TestMarkdownFieldAcceptsBareString — bare-string form still works.
All 12 clients/websearch tests pass; go build + go vet clean.
Contract verified live: crawl4ai 0.8.6 POST /crawl {urls:[...]} returns
synchronously (no task_id polling) with url/markdown/success/metadata —
matches the adapter otherwise.
deploymentLogs returned only the recorded timeline + a Job reference, ending
with a '(live BuildKit Job logs stream in phase 2)' placeholder. This closes
that phase-2 gap: it streams the ACTUAL pod logs from the cluster — the BuildKit
Job's pod while a git build runs, and the running app's pod once deployed — so
the console's per-deployment Logs pane shows real output, operator-consistent.
- logs.go: buildLogs (job-name=<jobName> pod in the build ns), appLogs
(app.kubernetes.io/instance=<slug> pod in tenant-<org>), one podLogsBySelector
path (newest pod, tail-bounded 400 lines, byte-capped 256 KiB keeping the tail,
8s time-boxed). Every read is org-scoped and time-boxed.
- k8s.go: add a typed kubernetes.Interface clientset (from the SAME rest.Config)
held ONLY for the Pods().GetLogs subresource the dynamic client cannot express;
nil-safe — a construct failure leaves logs degrading to the timeline and never
disables the CR control plane (all on dyn).
- deploy.go: deploymentLogs now appends real build + app logs and stamps a
(build|app|none) so the console can label the pane. HONEST DEGRADE:
an unreachable cluster / absent pod yields the recorded timeline + a stated
'not available' note — source stays reflecting real streamed content, never a
fabrication.
- 8 tests over a fake typed clientset: newest-pod selection, tenant-namespace
scoping (acme never reads victim's pod), no-pod/no-clientset honest degrade,
and the handler surfacing live build logs (source=build) vs degrading
(source=none) — asserting the phase-2 placeholder is gone.
The console can be a static export only once its two NON-proxy Next server
routes (app/keys, app/onboard) — which mint/revoke the user's hk- Cloud API
key and create the user's org as the confidential hanzo-console IAM client —
have a native home. Port them to a clients/console subsystem mounted at
/v1/console/* in the one binary (task #41, True 1-binary FE): the embedded SPA
calls /v1/console/* on its own origin, and the last stateful Node handlers go.
- clients/console/iam.go: confidential-client (client_secret_basic) IAM caller
for mint/revoke/get the hk- key + create/read/update an org. Honest 501 when
IAM_MINT_CLIENT_ID/SECRET are unset (mirrors identity.ts mintConfigured()).
- clients/console/console.go: /v1/console/{keys(GET/POST/DELETE),onboard(POST),
health}. Every route requires a VALIDATED principal; the IAM id is DERIVED as
<owner>/<name> from the gateway-minted X-User-Id/X-Org-Id, never a request
value — a caller can only ever act on their OWN key/org (red-bar structural).
- clients/console/onboarding.go: faithful Go port of console2 onboarding.ts —
pure slug + reserved-name policy (admin/built-in/app + hanzo/lux/zoo/pars).
- Registered as consolesvc (order 122) so /v1/consolesvc/health does not shadow
the real fail-closed /v1/console/health probe.
- 16 tests: unauth 403 (forged X-Org-Id refused, IAM never touched), mint/get/
revoke scoped to the derived id + show-once + no secret leak on GET, 501/502
honesty, onboard first-run(create+move)/additional(create-only)/reserved 400/
taken 409/personal auto-suffix, + pure-policy unit tests.
Close the two PaaS domain gaps so a customer can put their app on their own
domain, operator-native, from the console.
- Seed a canonical default host <slug>.<org>.<sitesHost> on app create, so
every app has a working HTTPS URL the moment it deploys (operator issues the
cert). Never removable.
- New /v1/platform/.../domains surface (list/add/verify/remove). An org-subtree
host is active immediately; a BYO custom host (yourco.com) is claimed PENDING
and returns the exact DNS records to publish (TXT ownership token at
_hanzo-challenge.<host> + CNAME to the app host).
- Verify resolves DNS: a matching TXT token proves control (DNS-01 model), then
the host is rendered into the app's operator Service CR ingress via applyIngress
(cert-manager TLS comes for free). Honest still-pending on not-yet, never fake.
- platform_domains table: host PRIMARY KEY = global uniqueness (one org per host,
like site_hosts); pending→verified lifecycle. Cascade-deleted with app/project.
- validateOrgDomains extended: a non-subtree host renders ONLY when this org owns
a VERIFIED claim; unverified/foreign/apex hosts still refused (RED hardening kept).
- ingressSpec extracted (one TLS shape shared by serviceCR + applyIngress);
observeDomains surfaces operator status.endpoints/phase for honest live state.
- Tests: verified-custom accept + pending/foreign refuse; full add→verify→remove
HTTP flow with fake DNS; global uniqueness (two orgs/two apps); apex refusal;
default-host seeding; CR ingress render. go build + go test green.
Found by testing the ACTUALLY-deployed hanzoai/crawl:0.0.1 (= Crawl4AI
0.8.6) against clients/websearch's crawl adapter: 0.8.x/0.9.x return the
/crawl result's `markdown` as an OBJECT
{raw_markdown, fit_markdown, markdown_with_citations, ...}, and signal the
batch with a boolean `success` (no `status`). crawlResult.Markdown was
typed `string`, so json.Decode errored on the object form → crawl()
returned an error → EVERY scrape returned {success:false} with empty
content. hanzo.chat Web Search's scrape half was therefore dead even once
crawl is running.
Fix: markdownField.UnmarshalJSON accepts either a bare string OR the
object (preferring the cleaned fit_markdown, raw_markdown fallback); the
crawlResponse envelope now also accepts boolean `success` alongside the
legacy `status`. Neither envelope field is required — Results[0].Success
is authoritative.
Tests (real 0.8.6 response shape):
TestScrapeHandlesCrawl4AIObjectMarkdown — object markdown + bool success
→ success:true, returns fit_markdown (was: {success:false}).
TestMarkdownFieldAcceptsBareString — bare-string form still works.
All 12 clients/websearch tests pass; go build + go vet clean.
Contract verified live: crawl4ai 0.8.6 POST /crawl {urls:[...]} returns
synchronously (no task_id polling) with url/markdown/success/metadata —
matches the adapter otherwise.
Front the Lux chain-data plane over HTTP so the console's Indexer and
Oracles pages read REAL chain state from api.hanzo.ai/v1/* instead of
rendering "not connected":
- GET /v1/indexers -> luxfi/indexer explorer REST (/health + latest
block): per-network chain/network/height/health. lag honestly omitted
(the indexer REST exposes indexed height, not the chain HEAD).
- GET /v1/oracles -> luxfi/graph GraphQL priceFeeds (O-Chain PriceFeed
registry): real on-chain price feeds; honest-empty when none.
Principal-gated (403 without a validated IAM principal); brand-scoped
(each brand's cloud is wired to its own indexer/graph, a ledger is public
within a brand). Honest 502 on unreachable upstream, never a fabricated
row. Mirrors clients/visor + clients/zt structure; interface-seam tests
against a fake upstream. Registered order 135.
Env-gated on OTEL_EXPORTER_OTLP_ENDPOINT; non-fatal; clean no-op when unset (safe to ship before the collector is live). Installs the global tracer provider with a service.name resource so the console Monitoring tab filters this product. Mirrors ai/object/telemetry.go. Traces-only; metrics/logs are a tracked follow-up.
RED review fixes on the sites router:
1) [HIGH] ONE reserved-subdomain source (clients/sites/reserved.go:
baseReserved baked-in + operator SetReservedExtra, never subtractable),
consulted at THREE points that can no longer drift: serve (siteSlug),
project-create (createProject -> 400), and host-bind (Store.BindHost ->
errReservedHost). site_hosts can now NEVER physically hold a reserved host,
so a reserved subdomain never resolves even if the ingress regex drifts —
the serve gate is a backstop, not the sole guard. Widened the set to app/
auth/payment/brand labels (console, sites, internal, gateway, login, secure,
account, signin, auth, pay, wallet, admin, brand terms, ...).
2) [MED DoS] Serve now STREAMS objects (Fiber SendStream, Content-Length from
info.Size, fasthttp closes the reader) instead of io.ReadAll-buffering up to
64 MiB per request on the unauthenticated edge — removes the OOM vector.
Same for the 404.html path.
3) [LOW] Non-GET/HEAD on a site host → 405 + Allow: GET, HEAD.
Tests: IsReserved, reserved-host-never-serves backstop, 405, BindHost-rejects-
reserved (even with a forced project row), create-rejects-reserved-slug via the
real handler. All green; no regressions.
Close the two LOW follow-ups on the cold-start tenant-RBAC fix, plus confirm
the fresh-org fail-closed status. All on top of v1.786.23 (already SHIP).
L2 — git reconciler no longer treats a transient tenant-RBAC delay as TERMINAL
and no longer head-of-line-blocks other orgs. reconcileBuild now does ONE
non-blocking readiness probe (ensureTenantReady: create-namespace-if-absent +
single SelfSubjectAccessReview) instead of the synchronous image path's ~45s
in-line waitForTenantRBAC. If the operator's RoleBinding has not landed, the
deployment stays 'building' and re-drives on the next 10s tick — never a
permanent fail (there is no client to retry a git build) and never a 45s stall
of the shared sequential reconciler. Only the elapsed build deadline fails it
honestly. errTenantProvisioning from applyLive is also caught as transient
(defense in depth). Namespace-create is decomplected into one shared
ensureNamespaceExists; ensureNamespace (sync, blocking) and ensureTenantReady
(async, probing) compose it.
L1 — image deploy path gains a per-org in-flight-deploy cap (inflightGate,
maxConcurrentDeploys, default 8 via CLOUD_PLATFORM_MAX_CONCURRENT_DEPLOYS),
mirroring the git build cap. deployImage acquires a slot before applyLive's
~45s RBAC wait and releases on any return; over-cap is a retryable 429 refused
BEFORE recording an attempt. Bounds request-goroutine pile-up on a wedged
operator; per-org (one org's saturation never throttles another); fail-closed.
I2 — confirmed the truly-fresh-org path fails closed on 503 (RBAC pending),
NOT a raw 502: the namespace IS created (the trigger for the operator's
RoleBinding), then the bounded RBAC wait yields errTenantProvisioning -> 503.
Tests (all -race green): reconciler stays 'building' then goes live on a later
tick once RBAC lands (not 'failed'); over-cap image deploy -> 429 with per-org
isolation + slot-release re-admit; fresh-org deploy -> 503 with namespace
created + no Service CR + honest 'error' deployment recorded.
The /v1/admin money panels (finance/orgs/overview) read $0 for every org despite
real balances (lux $10,000, maxpower $20,498) because the commerce client used
the wrong org selector on BOTH axes:
- commerce.go get(): sent X-IAM-Org-Id, which commerce does NOT read. Commerce
EdgeAuth resolves the per-org billing namespace from the TRUSTED X-Org-Id header
(trusted only with the COMMERCE_SERVICE_TOKEN bearer). X-IAM-Org-Id silently
fell back to the default (COMMERCE_SERVICE_ORG) namespace.
- admin.go orgSubject(): keyed the wallet subject as "org/org"; commerce keys the
per-org wallet under the BARE org slug (user=<org>) within the X-Org-Id namespace
(the 2026-07 commerce durability rework, commerce >=1.46.8).
Either alone zeroed the reconciliation; both were present. The prior comments
encoded the wrong model ("commerce resolves from COMMERCE_SERVICE_ORG, header
advisory") — corrected to the verified contract.
Verified LIVE against commerce /v1/billing/{balance,usage-rollup}:
user=lux + X-Org-Id: lux -> $10,000.00 (1,000,000c)
user=maxpower + X-Org-Id: maxpower -> $20,498.13 (2,049,813c)
user=lux/lux OR X-IAM-Org-Id -> $0 (the bug)
The fleet-wide /v1/costs COGS god-view is org-independent and correctly sends no
org (unchanged).
Regression guard: TestCommerce_ReconcilesWithXOrgIdBareSlug — a contract-accurate
fake commerce that returns money ONLY for X-Org-Id + bare-slug user; proven
red->green (fails on org/org, passes on the fix). Full admin suite green.
Co-authored-by: blue <blue@hanzo.ai>
Red review of the live agent-session control plane.
FIX (MEDIUM, systems lens): sessionsStream retained root := c.Query("root")
verbatim. c.Query is a zero-copy view into the fasthttp request buffer, and the
SendStreamWriter loop OUTLIVES the handler (runs after the Ctx is recycled), so
the long-lived root filter raced a reused buffer — within-org stream-filter
corruption / UB. tenant() already clones org for this exact reason; clone root
the same way. Not cross-tenant (org is cloned + bus-filtered); fixes the race
and honors the file's own 'never touch Ctx after return' invariant.
TEST (vector #4): add TestSessionEventSeqConcurrent — 64 parallel AppendEvents
to one session must yield seqs exactly {1..N}, no gaps (no lost write) no dupes
(no raced MAX+1). Proves the single-conn + UNIQUE(session_id,seq) guarantee
under -race instead of only asserting it.
The canonical cloud registry every surface hangs off: live agent SESSIONS +
the subagent tree, streamed over ZAP, remote-controllable. This is the
view/control/stream layer; durable execution rides hanzoai/tasks, not a
bespoke scheduler.
Model + store (agents.db, same tenancy pattern as agents/runs):
- Session{id,agent,org,actor,status,parentSessionId,rootSessionId,title,
startedAt,endedAt,taskWorkflowId,taskRunId,events[]}. Subagent tree =
sessions linked by parentSessionId; the outer agent is the root, each
spawned subagent a child, all sharing rootSessionId. Parent must exist
in the SAME org (TOCTOU-checked in the write path) so a tree can never
cross tenants. Per-session monotonic event Seq.
REST (org-scoped via principal.Tenant, fail-closed):
- POST /v1/agents/sessions register (opt parentSessionId)
- GET /v1/agents/sessions list (filter root/parent/status)
- GET /v1/agents/sessions/:id detail + children + recent events
- GET /v1/agents/sessions/:id/tree full subagent-flow graph (1 query)
- PATCH /v1/agents/sessions/:id status/title (terminal is monotonic)
- POST /v1/agents/sessions/:id/events append message/tool-call/spawn/log
- POST /v1/agents/sessions/:id/{pause,resume,stop,message} control
Routes register BEFORE /v1/agents/:name (Fiber matches in registration
order); /stream precedes /:id for the same reason.
ZAP live stream:
- GET /v1/agents/sessions/stream (SSE) rides the ZAP machine transport
natively (zip SendStreamWriter streams through ListenZAP — proven by
zip stream_test). In-process bus is the single fan-out seam a direct
ZAP push subscription attaches to. Org-filtered, non-blocking, laggard-
drop; GET endpoints are the source of truth.
Durable execution = hanzoai/tasks (architecture alignment):
- Root session -> a tasks workflow; subagent -> child workflow (same
rootSessionId). TaskController seam mirrors the tasks SDK Client
(Signal/Cancel); control forwards to it when a session is task-backed,
else records the command as a durable control event for stream-
consuming surfaces. Default is the disabled (record-only) controller;
the live client.Dial(TASKS_URL) plug-in point is marked in Mount.
Run integration (#5): the ONE runAgent path (HTTP + scheduler) opens a
root session per run (best-effort, never fails the run), so every run is
visible in the same registry.
Tests (real): store tree-linking + cross-tenant/dangling parent deny +
event seq/counts; HTTP tree assembly; cross-tenant read/tree/control/
append/parent deny; control authz (no validated principal -> 403) +
tasks forward (signal/cancel, forward-failure 502, record-only fallback);
event append/seq + status monotonicity; run-opens-session; bus fan-out/
org-filter/overrun/close. go build+vet+test clean; -race clean.
Add clients/sites: a HOST-routed public site server that turns
<slug>.hanzo.app into the static site a project deployed to OUR S3
(<org>/<slug>/ in CLOUD_PROJECTS_BUCKET). Installed as the FIRST middleware
in the compose root, ahead of identity/billing, so a published site is a
public artifact served straight from S3 — never a tenant API call.
Tenant isolation (RED-focus): the org + S3 prefix come ONLY from the store
lookup keyed by the validated subdomain slug, never from the request path or
a client header. Object keys are rooted-clean (path.Clean under '/') so no
../ or encoded traversal can escape the <org>/<slug>/ prefix into another
project or org. A globally-unique site_hosts binding table makes a bare
subdomain resolve deterministically to exactly one tenant (project slugs are
only org-unique); binding is first-come and cannot be hijacked.
Cache: one canonical policy (sites.CacheControlFor) applied both when writing
objects at deploy and when serving them — HTML public,max-age=60,s-maxage=86400;
content-hashed assets immutable 1y; middle TTL otherwise; per-project
cacheControl override on the document TTL. Cloudflare purge-by-cache-tag
(site-<org>-<slug>) on redeploy AND delete; creds from KMS/env
(CF_API_TOKEN/CF_ZONE_ID), honest no-op when unset. Cache state (TTL +
lastPurgeAt) exposed on the project API.
Tests: traversal/cross-tenant isolation proof, host-routing + reserved-host
exclusions, first-come/no-hijack subdomain binding, CF purge client.
Add clients/zt — a thin, org-scoped facade over the Hanzo Zero Trust
controller's OpenZiti Edge Management API (/edge/management/v1), backing
the console's Networks, Service Mesh and Edge pages (which render
"not connected" today).
Surface (all org-scoped by the validated principal):
GET /v1/networks[/:id] the org's ZT overlay, projected from its edge-routers
GET /v1/mesh/services ZT edge services
GET /v1/edge/nodes ZT edge-routers + real online/disabled/offline status
- client.go: one HTTP path — Ziti password-auth (KMS-injected
ZT_CLIENT_ID/ZT_CLIENT_SECRET, zt-session header), cached session with
re-auth-and-retry-once on 401, generic {data,meta} pager, honest error
mapping, TLS trust via ZT_CA_PEM. Fails closed 503 when unconfigured.
- types.go: ZT wire structs + console view structs + PURE mapping.
Tenant isolation is the "org-<org>" role attribute (the ONE tenancy
convention ZT expresses natively); list/get filter to the caller's org.
- zt.go: routes/handlers, registered as subsystem "zt" (order 134).
- http_test.go: fake controller (interface seam) — asserts 200, tenant
isolation, shape, health mapping, 401 re-auth retry, fail-closed 503.
Honest-empty over fabrication throughout: no org tag -> invisible; no
routers -> no network; no metrics -> omitted (UI renders em dash).
New global-admin read GET /v1/admin/compute powering the console Bots + Machines
operator boards. Aggregates hanzo.compute_usage(org, app, project, kind, event,
machine_id, size, price_cents, ts) grouped by (org, app, project, kind) over the
shared datastore client (aiobject.DatastoreQuery — the clients/analytics transport,
no second conn). `kind` is an OPEN LowCardinality spectrum (bot|machine|cluster|
nodepool|container|function|…) matched as a PLAIN STRING — ?kind= narrows to any
kind (Bots=bot, Machines=machine; future Clusters/Functions reuse this endpoint),
?org filters, ?range=24h|7d|30d bounds. Two-level roll-up: inner argMax(event,ts)
per machine -> outer counts machines, active (latest non-terminal), sum(price_cents),
max(ts). Honest-empty when the warehouse/table isn't wired yet (visor/commerce
emitter pending) — never a fabricated fleet. Global-admin only (s.guard); stays v1.x.x.
New clients/visor subsystem fronts Visor (the cloud OS at visor.hanzo.svc) and
serves the console's Machines/GPUs/Clusters pages as clean, tenant-scoped REST off
the unified cloud binary — replacing the god-mode /paas admin proxy that 501s.
Routes (every route org-scoped by the validated principal → Visor ?owner):
GET/POST /v1/machines, GET/DELETE /v1/machines/:id -> get-machines / machines/launch / delete-machine
GET /v1/gpus (+ /v1/gpus/alerts) -> per-accelerator inventory derived from GPU machines
GET /v1/clusters, node-pool create/scale/delete -> get-node-pools / *-node-pool
View JSON mirrors the console normalizers exactly (visor.ts/compute.ts/platform.ts)
so the FE renders with no change. No fabrication: GPU rows are real accelerators of
real GPU machines, clusters are real node pools, and telemetry Visor lacks is
omitted (renders — not 0). Auth: KMS service credential (Basic) or forwarded bearer.
Tests: tenant scoping/isolation, machine/gpu/cluster shape, GPU slug derivation,
launch quote+real+delete. go build ./... + go vet + go test all green.
The console Environments/Pipelines/Builds/Releases pages rendered "not
connected" because they call top-level REST that no cloud subsystem served.
Serve them natively from the platform control plane, DERIVED from the SAME
per-org project/app/deployment/build records (no new data model, no fabrication):
- GET /v1/environments — distinct Application.Environment targets across the
org's apps, each aggregating its apps (services), with a derived
type/status. List-only: an environment is a scope on apps, not a record.
- GET /v1/pipelines — one per app: its build/deploy config (repo|image) plus
the status/timing of its latest deployment. List-only: a pipeline is an app.
- GET /v1/builds — the REAL arcd BuildKit build records (platform_builds),
joined to app repo + deployment commit. List-only: builds are triggered by
the app deploy path (git source) — one trigger, not a duplicate here.
- GET /v1/releases — deployments actually applied to the cluster
(status deploying|live): a released image tag on an app/environment.
Every route is org-scoped through the same validated-principal gate (s.tenant →
requires c.User()); the response is the exact `{ "<plural>": [...] }` wrapper the
console FE normalizers read. Three org-wide store aggregates back them
(ListAllApplications / ListDeploymentsByOrg / ListBuildsByOrg), org the only
tenancy predicate. Real records or an honest empty — never fabricated history.
Tests: shape (200 + wrapper + derived fields), org isolation (second org sees
empty, no cross-tenant leak), forgeable-org refusal (no X-User-Id → 403).
Add clients/do: an org-scoped facade over digitalocean/godo's native VPCs
and LoadBalancers services, backing the console's VPC + Load Balancers pages
(which render 'not connected' today because nothing serves them).
- Routes: GET/POST /v1/vpcs, GET/DELETE /v1/vpcs/:id and the same for
/v1/load-balancers. Real godo calls, honest empty/error states, never
fabricated.
- Tenant isolation: DO is a single account, so a resource's physical DO name
is 'o'<orgHash>-<friendly> via provisioning.BucketName (the SAME org-hash
convention clients/s3 uses). List filters the account inventory to the
caller's prefix; get/delete confirm prefix ownership before acting; a
cross-tenant id reads 404 (existence-oracle guard).
- Fail-closed: absent DO_API_TOKEN every op is an honest 503.
- FE shape matches console2 VpcModule/LoadBalancerModule verbatim (vpcs[],
loadBalancers[] with the exact field names).
- Registered as subsystem 'do' (order 123). godo v1.197.0 added to go.mod.
- Tests: per-org VPC + LB isolation, forge-path 403, fail-closed 503.
Same class as c2a7534: upstream force-re-tagged luxfi/age v1.5.0 and
luxfi/pq v1.0.3 (content moved, /go.mod hashes unchanged), so the committed
zip h1: sums no longer match the served bits and `go build ./...` fails
verification. These deps entered the graph via hanzoai/commerce/metering
v0.1.2 (the agents scheduler/billing). Re-record the current zip hashes.
The console2 Agents dashboard calls two org-wide routes that were never
reachable: the bare /v1/agents/:name wildcard captured "metrics"/"activity"
as an agent name (Fiber matches in registration order), so both 404'd and the
dashboard rendered a permanent "not connected" state.
- Register the two static routes BEFORE :name so they win the match.
- GET /v1/agents/metrics?range=24H|7D|30D -> a per-agent invocations-over-time
histogram bucketed from REAL agent_runs rows; the Resource Usage rollup is
all-null because this store meters no CPU/mem/storage/cost (honest em-dash,
never a fabricated trend). Shape mirrors console2 normalizeMetrics exactly
({range,series:[{key,points:[{t,v}]}],resource:{...}}).
- GET /v1/agents/activity -> org-wide recent-activity feed: each recorded run
is an invoked/failed event, each agent's own create/update timestamps are
created/updated events; merged newest-first, capped 50. Shape mirrors
normalizeActivity ({activity:[{id,kind,agent,message,at}]}).
- Store: add RunsSince(org,since,limit) — org-wide runs across all agents,
tenancy on the org column; powers both surfaces.
- Tests prove the surfaces are not shadowed (200, not 404), reflect only real
runs, and stay org-isolated.
deps.AI was permanently nil under the default all-enabled config
(pickAIClient returned nil for cfg.Enabled("ai") and no in-process ai
subsystem ever filled it), so every agent run 503'd "inference is not
configured on this deployment" before the already-live per-org metering.
New AI client (clients/aihttp.go), two credential modes:
- AIHTTPAt: static-key OpenAI-compatible client (CLOUD_AI_API_KEY) — an
operator/pre-provisioned-key override.
- AIHTTPM2M: durable default — mints+auto-refreshes an IAM client-
credentials token from the binary's OWN identity (IAM_CLIENT_ID/SECRET)
via x/oauth2/clientcredentials. No static key to rotate, no expiry cliff,
no new secret to store. On Hanzo the identity resolves to
admin/hanzo-cloud, which the gateway treats as balance-exempt, so cloud's
per-org ResourceMeter stays the single revenue debit (no double-bill).
- build.go pickAIClient: static key -> M2M -> ZAP RPC -> fail-closed stub.
Never returns nil (the live bug). Secret never logged.
- config.go: CLOUD_AI_BASE_URL (default https://api.hanzo.ai/v1),
CLOUD_AI_API_KEY (optional), CLOUD_AI_DEFAULT_MODEL (default
deepseek-v4-flash), AIAuthClientID/Secret from IAM_CLIENT_ID/SECRET.
- clients/aihttp_test.go: httptest OpenAI emulation — default-model
substitution, content parse, 4xx/5xx mapping, empty-choices error, and
M2M token mint+use+cache.
Composes with the live metering (Gate/MeterUsage) untouched. Model routing
is the gateway's job; empty model -> cheap default is the only cloud-side
fallback (no in-code model aliasing).
Addresses Red MED-1/MED-2/INFO-3:
- MED-1: revenue.Configured now means the source was actually READ (listOrgs
succeeded), not merely wired. A transient IAM failure → configured:false, never a
fabricated zero that flips margin negative into a false 'burning' alarm. Per-org
read failures mark the commerce source not-ok (partial), never presented as whole.
- MED-2: /v1/costs is a fleet-wide god-view — commerce resolves the namespace from
COMMERCE_SERVICE_ORG, NOT from a request header (it never reads X-IAM-Org-Id).
Dropped the no-op org arg from costs(); corrected the false 'resolves namespace
from X-IAM-Org-Id' claims in commerceClient + orgSubject docs.
- Test: TestFinance_RevenueSourceDown_NoFabrication proves no fake revenue/margin
when the IAM org list is unreadable while COGS still flows.
(INFO-3 false-green fix lands console-side in financeHealth.) 10 admin tests green.
The finance board's cost side now CONSUMES commerce /v1/costs (the single
vendor-COGS source of truth: DigitalOcean compute + the LLM providers we resell)
instead of re-reading DigitalOcean's billing API to derive a DO-only cost. This
removes cloud's duplicate DO COGS read and gives the board the multi-vendor
per-vendor breakdown for free.
- commerceClient.costs() reads GET /v1/costs over the admin S2S service token
(COMMERCE_SERVICE_TOKEN, no IAM user → commerce requireCostsAdmin M2M path).
- financeCost carries {configured,totalCents,vendors[],period}; margin cost is
now the multi-vendor TotalCents, not DO month-to-date spend.
- DigitalOcean stays ONLY as the orthogonal promo-credit/runway treasury view
(commerce does not track our prepaid credit); its MTD spend feeds runway alone.
- /v1/admin/finance shape is additive (cost.digitalocean preserved) and the
global-admin guard is unchanged.
- tests: fake commerce now serves /v1/costs; margin = revenue - COGS; DO-off path
proves COGS still flows from commerce (decoupled).
A brand-new tenant's namespace is created by ensureNamespace, but the operator's
tenant-RBAC controller projects cloud-api's `cloud-api-platform` RoleBinding
(get/create resourcequotas/limitranges/services.hanzo.ai in tenant-<org>)
ASYNCHRONOUSLY. The first-ever deploy raced ahead of that RoleBinding and failed
with `resourcequotas ... is forbidden`, self-healing only on a manual retry.
Gate ensureNamespace on a SelfSubjectAccessReview readiness poll
(waitForTenantRBAC): before touching the quota objects, ask the apiserver — as
cloud-api's OWN identity — "can I get resourcequotas in tenant-<org>?" and wait,
with bounded exponential back-off (~45s ceiling), for the operator's RoleBinding
to land. An already-onboarded tenant is confirmed by a single fast probe (no
sleep), so existing deploys are not slowed. On timeout, fail CLOSED with a
retryable errTenantProvisioning (deployErrStatus -> HTTP 503, honest
"provisioning, retry") — never a fabricated success. Creating a SSAR needs no
tenant RBAC (system:basic-user), so the probe is itself immune to the window it
closes. No RBAC or namespace derivation is loosened.
Tests (clients/platform/tenant_rbac_test.go): retry-until-RoleBinding-lands
succeeds end-to-end (quota+Service CR written); bounded timeout fails closed
(503, no quota/CR written); ctx-cancel aborts promptly; ready tenant resolves in
one probe (auto-glue fast path unchanged).
Lands the agent-backend metering feature onto the zip->zap-proto-migrated main
(cloud is already fully migrated on main; 8 subsystem pins + MountAll clean).
- /v1/agents/* self-meters a per-run fee to commerce: pre-authorize the org's
prepaid credit balance fail-closed (402 insufficient_balance), debit on success,
attributed to product "agent". Added to selfMeteredPrefixes so the edge gate
never double-bills. Run path (money-moving) requires a VALIDATED principal
(c.User() non-empty), refusing the no-bearer forge path; scheduled runs carry an
unforgeable 'scheduler'-prefixed actor.
- ResourceMeter.Gate now forwards costCents as AuthInput.AmountCents so the gate
enforces available >= fee (not merely > 0) — a 1-cent balance can no longer
authorize a run that takes the ledger negative. MeterUsage generalizes the
per-org debit (Actor/Model/token attribution) while Meter keeps its signature.
- Long-running agents: cron scheduler scans once a minute over a partial index
(ix_agents_scheduled), bounded per-org cap (CLOUD_AGENT_MAX_LONG_RUNNING);
scheduler.stop drains in-flight runs inside the SIGTERM budget via ShutdownAll.
Deps: commerce/metering v0.1.0 -> v0.1.2 (Actor field + AuthInput.AmountCents).
All other pins inherited from migrated main (ai v1.789.1, authz v1.10.3,
base v1.4.6, commerce v1.42.29, licensing v0.1.1, metrics v0.4.1, o11y v1.3.12,
vfs v0.4.4). hanzoai/zip stays out of the graph; zip == zap-proto/zip v1.2.0.
go mod verify clean; go build ./... EXIT 0; MountAll boot-smoke clean (no
want *zip.App); agents -race + root/ml/provisioning billing tests green.
Verified the /api/ -> /v1/ rip is COMPLETE on origin/main: zero owned /api/
route registrations, zero owned /api/ client strings. The rip landed in
cf58f8d (productsvc: drop residual /api/ prefix) plus the o11y and eval
cleanups. Every remaining /api/ reference is a non-owned external contract:
- clients/o11y/o11y.go, o11y_test.go: doc comments ("no /api/, no rewrite")
documenting the now-removed upstream rewrite.
- clients/pricing/pricing.go: https://openrouter.ai/api/v1/models — a
third-party vendor URL.
- clients/platform/*_test.go: "apps/api/deploy" where `api` is a user's
APP NAME inside /v1/platform/... paths (not an API prefix).
- zapface/dispatch.go: comment already says "the /v1 convention".
- clients/prompts/catalog.json: a prompt-catalog data blob describing a
different project (prompts.chat), not this repo's routes.
The IAM /api/add-usage-record callout referenced in older cloud docs is a
Casdoor/casibase-lineage endpoint the LEGACY Node cloud-api used (documented
in hanzoai/commerce auth/iam_admin.go). The current Go cloud-api does NOT
call it: billing meters to commerce via hanzoai/commerce/metering. No
cross-service IAM usage-record dependency exists in this repo.
The only change here is a deps-hygiene fix so the build verifies clean:
luxfi force-re-tagged age@v1.5.0 and pq@v1.0.3, drifting their module-zip
h1: hashes vs the recorded go.sum (go.mod hashes unchanged). Realigned to
the upstream hashes at the SAME versions — no major/minor bump.
CGO_ENABLED=0 GOWORK=off go build -mod=readonly ./... -> exit 0
go test ./clients/{o11y,eval,pricing,ml,admin} ./zapface . ./clients/platform -> all ok
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
applyService (operator Service CR write) was not jointly ordered with
FinalizeLive (live-pointer DB write): under concurrent same-app deploys an
OLDER deploy's CR write could land AFTER a NEWER one already went live, leaving
the live Service CR image lagging the recorded live version. deployImage had no
supersede gate at all.
Introduce applyLive — the ONE deploy mechanic shared by the image-source path
and the git build reconciler — running supersede-check → applyService →
FinalizeLive as one per-app-serialized critical section (appMutex, fixed-shard,
O(1) memory). An older deploy that loses the race is superseded and never writes
its CR. FinalizeLive's monotonic CAS still backstops DB monotonicity.
go vet + full clients/platform test suite green.
HIGH-2 — cloud-api touches NO K8s Secret:
- delete ensurePullSecret + secretsGVR + CLOUD_PLATFORM_PULL_DOCKERCONFIG;
the per-tenant ghcr-pull Secret is provisioned by the OPERATOR's tenant-RBAC
controller (from a KMS-synced source). serviceCR only REFERENCES it by name.
cloud-api's ServiceAccount holds no secrets grant and issues no Secrets call.
MED-1 — monotonic live-version finalize (no build-time inversion):
- Store.FinalizeLive: ONE atomic conditional UPDATE that advances an app to live
ONLY when its version >= the currently-live version (no read-then-write TOCTOU).
- reconcileBuild gates on buildSuperseded before applying the (older) CR, and
records a late/older build 'superseded' (image still succeeded) instead of
regressing the running workload. deployImage shares the same FinalizeLive.
- tests: TestFinalizeLiveIsMonotonic (store CAS) + TestBuildReconcilerVersionMonotonic
(e2e inversion: newer-first-live, older-late-superseded, CR never downgraded).
The git deploy path launched a BuildKit Job fire-and-forget and left the
deployment stuck 'building' — deploy.go documented the build watcher as
'phase 2'. Implement it as the ONE owner of the handoff:
- reconcile.go: a restart-safe reconciler (state in the store, not a
goroutine) that scans 'building' deployments, checks each build Job, and on
success applies the operator Service CR with the built image (the SAME
applyService the image path uses) → deployment 'deploying', app 'live'. On
failure/deadline it records the honest error. Started from Mount, stopped on
Shutdown. Org-scoped: every write targets tenant-<row.Org>.
- store.ListBuildingDeployments: cross-org 'building' query (reconciler input).
- k8s.jobOutcome/jobResult: ONE Job terminal-state classifier shared by the
concurrent-build cap and the reconciler.
- k8s.serviceCR imagePullSecrets + ensurePullSecret: the built image is PRIVATE
(ghcr.io/hanzoai/tenant-<org>/*); provision the tenant GHCR pull secret from
CLOUD_PLATFORM_PULL_DOCKERCONFIG (KMS-synced; no-op when unset) and reference
it so the operator's pod can pull.
Tests: jobOutcome classifier + ListBuildingDeployments (oldest-first, cross-org,
building-only). Full platform suite green (RED authz/cmd-injection incl.).
clients/eval/telemetry.go opened a SECOND direct clickhouse.Open with a
parallel CLOUD_EVALS_CLICKHOUSE_* cred namespace, bypassing the shared ZAP
datastore mesh that clients/analytics + ai/object already use. Consolidate:
- eval telemetry now routes every write/read over ai/object's shared client
(aiobject.DatastoreExec / DatastoreQuery / DatastoreEnabled), the same peer
the o11y ledger + /v1/analytics use. One connection, one pool, one
retry/backoff, one KMS-injected cred namespace (DATASTORE_*).
- DELETE the CLOUD_EVALS_CLICKHOUSE_* namespace and the private clickhouse.Open.
- Ownership stays clean: eval owns only its two tables (hanzo.eval_traces,
hanzo.eval_scores); ai/object owns hanzo.cloud_usage / hanzo.observations.
- No batch primitive needed — eval Records are single-row, mapping to the
shared DatastoreExec INSERT ... VALUES (?) the o11y write path already uses.
- Async-connect aware: readiness gates per-op on DatastoreEnabled() (honest
'unavailable' in the boot window), tables ensured idempotently, latched once.
- Reads bind org + narrowers positionally (?) — no interpolation; LIMIT always
applied. Tenant isolation + score finiteness invariants unchanged.
provisioning's direct CH client is a distinct control-plane concern — untouched.
go build + go test ./clients/eval/... green.
A green go build/vet/test does not catch a binary that PANICS at startup.
v1.786.14/.15/.16 compiled clean but crashed at boot with
cloud: mount metrics: metrics.Mount: app is *zip.App, want *zip.App
(a runtime type-assert from an incomplete hanzoai/zip -> zap-proto/zip
migration), published green, and CrashLooped in prod. The only gate that
catches this class is running the binary.
Restructure the single build-push into build(load) -> smoke -> push:
1. Build once to a local cloud:smoke tag (push:false, load:true), warming
the BuildKit builder cache.
2. Boot that exact image with a minimal, prod-representative env (writable
ephemeral /data + a throwaway 32-byte KMS master key so the KMS plane
mounts on its normal ready path) and assert it reaches "listening" with
NO startup-crash signature (metrics.Mount / mount metrics / panic /
want *zip.App), else exit 1 BEFORE any push. Container always rm -f.
3. Re-run build with push:true and the real tags: identical context /
platform / secrets, so every layer is a cache hit from step 1 and it
only publishes the already-tested image.
Stays on the self-hosted arcd amd64 scale set + GH_PAT; notify-universe
unchanged.
Proven to DISCRIMINATE against the published images:
ghcr.io/hanzoai/cloud:v1.786.18 (known-good) -> SMOKE PASS (exit 0)
ghcr.io/hanzoai/cloud:v1.786.15 (known-bad) -> SMOKE FAIL (exit 1)
SECURITY: cloud origin/main (55b073e7) had DIVERGED from the LIVE image
v1.786.18 (faef12aa) at merge-base 862d4623 — v1.786.18 was tagged+deployed
from fix/cloud-1.786.18-zip-and-gates but NEVER merged back to main. It carries
the F1 forged-X-Org-Id cross-tenant gate that main LACKED:
- clients/principal (validated-principal helper) — new package
- bot + o11y reverse-proxy forge gates (bot.go, o11y.go + red_forge_test.go)
- whole-data-plane principal gating across agents/crm/eval/functions/git/kms/
ml/plan/pricing/projectsvc/prompts/provisioning/s3
Building .20 from main+platform ALONE would have REGRESSED this live fix
(reopened the forged-X-Org-Id hole — the ".17 insecure, do NOT deploy" hole).
This merge makes the artifact a true SUPERSET of the security floor.
Only conflict: clients/provisioning/provisioning.go import block — resolved to
keep BOTH the F1 principal.Validated(c) gate AND the platform sanitizeOrg
injectivity crit-fix (OrgHasUnsafeRune + raw-byte hash). go.mod/go.sum merged
clean (main and .18 converged on identical dep floors: ai 1.789.1, authz
1.10.3, base 1.4.6, commerce 1.42.29, o11y 1.3.12, vfs 0.4.4, zap-proto/zip).
Result = main (analytics/crm/templates/git + zip migration) + /v1/platform mount
+ v1.786.18 F1 gate. All 24 subsystems registered. go build ./... + go vet green.
Tests green together: platform (TestRED_CommandInjectionBlocked, cross-tenant,
injective), F1 (TestRed_BotProxyForwardsForgedOrgNoPrincipal,
TestRed_O11yProxyGatesForgedOrgNoPrincipal), provisioning, principal, root cloud.
Merge RED-PASSED blue/paas-v1platform@86cb4c15 onto main@55b073e7, mounting the
per-org container-app PaaS control plane at /v1/platform (HIP-0106) — the deploy
engine behind one-click app deploys (ERP/Helpdesk ride on it).
Conflicts resolved (5 files, all combine-both-sides — no logic dropped):
- subsystems/subsystems.go: KEEP every registration — platform (order 124) +
analytics/crm/git/templates/prompts/agents/functions + all pre-existing.
- clients/provisioning/provisioning.go: main's zip canonical import +
branch's sanitizeOrg injectivity crit-fix (OrgHasUnsafeRune reject,
raw-byte SHA-256, no TrimSpace) — both preserved.
- middleware_identity.go: main's zap-proto/zip + doc; branch's OrgHasUnsafeRune
(root cloud pkg) + SanitizeIdentity handler hardening — both preserved.
- provisioning_test.go / middleware_identity_test.go: both test sets kept.
Integration: main migrated the repo hanzoai/zip -> zap-proto/zip; the branch
predated it, so the new clients/platform/{deploy,platform,http_test}.go were
rewritten to the canonical github.com/zap-proto/zip (incompatible zip.Ctx types
otherwise). go.mod unchanged from main (zap-proto/zip v1.2.0 direct); no new dep
(k8s.io/{apimachinery,client-go} v0.35.0 already present via ml/paassvc).
Crit-fixes preserved EXACTLY (RED-PASSED, proven green):
argv build (TestRED_CommandInjectionBlocked), org-slug injective
(TestSanitizeOrg{Injective,WhitespaceInjective}, TestBuildImageRefIsInjective),
ResourceQuota (TestEnsureNamespaceAppliesQuota), no cross-tenant
(TestHTTPCrossTenantIsolation, TestNamespaceIsDerivedFromOrgNotInput,
TestServiceCRAlwaysPinnedToTenantNamespace).
go build ./... + go vet + go test (platform/provisioning/root cloud) all green.
Startup crash on v1.786.15/.16: `cloud: mount metrics: metrics.Mount: app is
*zip.App, want *zip.App`. Cloud's core migrated to github.com/zap-proto/zip
(v1.786.13→.15) so app is *zap-proto/zip.App, but eight hanzo modules cloud
imports still pinned the OLD github.com/hanzoai/zip and registered subsystems
that type-assert app.(*hanzoai/zip.App). Distinct import paths = distinct Go
types, so MountAll's first such subsystem ("metrics", hanzoai/metrics@v0.4.0)
failed the assert at runtime (build/vet/test stayed green — the mismatch is
runtime-only). authz/base/o11y were the same latent break behind it.
Forward fix (the canonical-home migration was already released upstream; cloud
merely lagged): bump every lagging module to its migrated tag —
ai v1.789.1-… → v1.789.1 authz v1.10.1 → v1.10.3
base v1.4.1 → v1.4.6 commerce v1.42.27 → v1.42.29
licensing v0.1.0 → v0.1.1 metrics v0.4.0 → v0.4.1
o11y v1.3.7 → v1.3.12 vfs v0.4.1 → v0.4.4
authz pinned to v1.10.3 specifically: v1.10.2/v1.10.4 carry an unrelated
GetPolicy 2-value change that breaks the pinned hanzoai/iam; v1.10.3 has the zip
migration AND the iam-compatible 1-value signature. commerce pinned to v1.42.29
(v1.43.0 regressed back to hanzoai/zip). clients/analytics (from the merged .16
work) is cloud's own code and is migrated in-place. Result: hanzoai/zip is gone
from go.mod/go.sum and the whole module graph; the compiled binary mounts
metrics+o11y+all subsystems and reaches "listening" with no panic.
Also folds in the COMPLETE F1 close (RED found the gate was partial — two
reverse-proxy paths still forwarded a forged X-Org-Id):
- clients/bot: gate proxy() on principal.Validated before forwarding X-Org-Id to
bot-gateway (RED PoC red_forge_test.go now passes: no-principal forge → 403).
- clients/o11y: wrap the installed reverse-proxy handler in gate() — refuse any
request with no X-User-Id before it reaches the o11y runtime (forge twin test).
- clients/crm: extend the forge guard to WRITE+DELETE verbs (belt-and-suspenders).
- middleware_identity: refresh the stale FAIL-MODE comment — post-F1 the DATA
plane also fails secure on a cold-cache JWKS failure (bounded by stale-on-error).
Base = F1 (fix/cloud-data-plane-principal-gate, bff2a688) + origin/main (analytics
.16, 862d4623). One healthy image: F1 + analytics + zip-fix + bot/o11y gates.
Aligns clients/analytics with serve.go + every other in-tree clients/* subsystem,
which import github.com/zap-proto/zip. The original commit imported the OLD
github.com/hanzoai/zip, making analytics.Mount assert *hanzoai/zip.App while serve
passes *zap-proto/zip.App — a boot-time mount type mismatch. Removes the last in-tree
hanzoai/zip importer so once the migration lane re-releases the external subsystem
modules on zap-proto/zip, main boots with ONE zip.App type. (The DEPLOYED v1.786.17
is built off v1.786.13, which is all-hanzoai/zip, and is unaffected.)
RED found the cloud data plane trusted a bare X-Org-Id with no validated
principal. Off-gateway (direct api.cloud.hanzo.ai / in-cluster), a request with
`X-Org-Id: victim` and NO credential read/wrote/deleted another tenant's data:
CRM PII, KMS secrets, prompts, agents, evals, ML namespaces, projects, git repos.
SanitizeIdentity RESTORES a client X-Org-Id on the bearer-less "Phase-1 data"
path but leaves X-User-Id empty, so c.Org() alone is forgeable; c.User() (set
ONLY from a verified bearer/cookie) is the authentic signal.
ONE canonical gate — new clients/principal:
- Validated(c): request carries a validated principal (X-User-Id set).
- Tenant(c): verbatim org, gated + bounded + cloned; ("",false) => caller 403s.
The S3/eval fix (already shipped) is now the single source of truth, used
everywhere instead of six drifting hand-rolled copies.
Data-plane resolvers gated (were bare c.Org()):
crm, prompts, agents, functions, git, eval -> principal.Tenant (verbatim)
kms.guard (closes forged X-Org-Id==:org bypass), ml (per-org k8s ns),
projectsvc, provisioning, s3 -> principal.Validated + own normalize/admin-bucket
plan, pricing (public catalog) -> validated principal selects its overlay, else
the public "hanzo" default (never a forged org's overlay)
middleware_billing (no victim-ledger drain/probe), audit_middleware (aligned)
Breaks no real client: the console BFF always mints a user-bound bearer
(X-User-Id set); opaque-API-key callers hit /v1/ai/*, not these subsystems.
Tests (F4): clients/principal unit suite (forged-org-no-principal refused,
verbatim no-fold, empty/overlong refused, Validated only from X-User-Id); crm
forged-org 403 on every collection (PII); kms Vector8 forged X-Org-Id==:org with
no principal -> 403 (secrets). Existing per-subsystem tests send X-User-Id on the
legit path as the BFF does.
go build ./... green; go vet green; go test ./clients/... . green.
Adds clients/analytics (order 132, registered as analyticssvc so the real
/v1/analytics/health owns the probe, not serve.go's generic liveness route) —
the backend for the console Native Analytics module (unified-analytics.md §5).
Two read lenses over the ONE hanzo warehouse, reusing the SAME clickhouse-go/v2
client the ai o11y ledger opens (ai/object DatastoreQuery/DatastoreEnabled/
EnsureCloudUsageTable/ResolveCloudUsageWindow) — no second CH client, DRY:
- LLM lens (REAL): hanzo.cloud_usage — requests/tokens/spend/models/errorRate
- web+commerce lens: hanzo.events — honest-empty until the collector emits
Surface (read-only, org-scoped, /v1):
GET /v1/analytics/overview per-org KPIs (llm real; web/commerce honest-empty)
GET /v1/analytics/timeseries requests/tokens/spend over hour|day buckets
GET /v1/analytics/top top models (real) + top products (honest-empty)
GET /v1/analytics/health datastore connectivity + lens-table availability
Tenant isolation is the security bar: tenant() requires a VALIDATED principal
(c.User(), set by SanitizeIdentity only for a verified bearer) AND a valid org
(c.Org(), the minted owner claim) — closing the Phase-1 no-bearer forged-X-Org-Id
data path exactly as clients/s3 does. Every query binds the org POSITIONALLY
(query.go llmWhere/eventsWhere), so a maxpower token can never read another org.
ClickHouse creds are KMS-injected env (DATASTORE_*), never hardcoded.
Tests: query-boundary isolation (org bound, never interpolated, incl SQLi slug),
honest-empty, real-number KPIs, errorRate, gap-filled series, top-models pct;
HTTP: no-principal->403, forged-org-no-bearer->403, datastore-down->honest 503,
bad-range->400, health owned-by-analytics honest 503 when down.
Integrates the fail-closed per-org credit-drawdown gate into cloud's last two
free data-plane subsystems (functions invoke, s3 op) via the ONE shared
cloud.ResourceMeter, plus the per-org billing-key fix (gate keys on the org slug).
HELD — NOT DEPLOYED. cloud origin/main is non-bootable until the zip migration
(#25 / o11y) lands; this ships with the cloud unfreeze. No dependency changes
(go.mod identical to main), so it adds no new build risk beyond the pre-existing
#25 o11y blocker. Verified: clients/functions 5/5, clients/s3 6/6, root billing
tests green; go build + vet clean on all changed packages.
The prepaid credit balance is PER-ORG (one credit pool covers the whole org),
so the metering gate must query the ledger by the org slug. identityFromCtx keyed
User on "{org}/{sub}", which queries an empty per-user ledger and 402s a fully
funded org — a revenue-blocking false-decline for every real request. Key User
on the org slug (bare sub only when org is absent), mirroring
metering.IdentityFromGatewayHeaders so cloud and every product key the SAME
ledger entry. The {org}/{sub} actor identity belongs on the usage audit trail,
not the gate; metering v0.1.0 carries no Actor field, so it is omitted until the
module ships the User/Actor split.
Covered by resource_billing_test.go (per-org debit asserts user==org). Root +
functions + s3 packages: go build + vet clean, tests green.
Wire fail-closed credit-drawdown metering into the two remaining cloud data-plane
subsystems that were free, reusing the ONE shared cloud.ResourceMeter (the same
Gate+Meter provisioning and ml already use). No free tier: pre-authorize the
org balance (insufficient -> 402, unreachable -> 503, nothing runs), then debit
on success. No second metering path.
- functions: POST /v1/functions/:name/invoke gates before sandbox compute and
debits the caller org on a real execution (product "functions", unit
"invoke", fee CLOUD_FUNCTION_FEE_CENTS, $1.00 default). A sandbox transport
failure ran no billable compute -> not charged.
- s3: the data-plane guard is the ONE place it meters -- every guarded op gates
before S3 is touched and debits on handler success (product "s3", unit
"op", fee CLOUD_S3_FEE_CENTS). A handler error is not billed.
- resource_billing: Meter now records the billed unit as Usage.Model (kind), so
provisioning kinds (sql/vector/kv/...), functions "invoke", s3 "op" and ml
kinds all get per-item attribution in the ledger under their product label.
- middleware_billing: add /v1/functions/ and /v1/s3/ to selfMeteredPrefixes so
the edge gate never double-bills the subsystem's own charge.
Tests (real commerce + sandbox doubles): functions 5/5, s3 guard 6/6 -- 402 on
empty balance with no compute run, debit-on-success to the CALLER org (never the
client default), no-bill-on-failure, free-fee ungated, unconfigured no-op,
tenant isolation. go build + vet clean; affected packages green.
Add a global-admin-only finance panel to the /v1/admin/* surface for
admin.hanzo.ai: DigitalOcean credit burn-down (our primary ~$40k credit
venue), month-to-date spend, revenue, MRR, gross margin, and runway.
- digitalocean.go: DO billing client (GET /v2/customers/my/balance +
/billing_history). Authoritative DO sign convention (from DO's public
OpenAPI spec): the three money fields are decimal-DOLLAR strings; a
NEGATIVE account_balance = credit we hold, so creditRemaining =
-account_balance (clamped at 0). dollars parsed to int64 cents at the
edge. Token DO_API_TOKEN from env (KMSSecret); unset => {configured:false},
never a fabricated balance.
- commerce.go: mrrCents reader — sums active/trialing subscriptions'
monthly-normalized plan price (yearly => /12) for fleet MRR.
- finance.go: financeData shape + computeFinance, a PURE derivation
(grossMargin = revenue - cost, marginPct, runwayDays = credit/burn or
null, profitable). Handler fans out to DO + commerce, honest-empty on
every unconfigured/unreachable path. Mounted under s.guard (global-admin
only; no-principal/tenant-admin/forged => 403).
- Tests: computeFinance math (profitable + burning-faster + null-runway +
unconfigured), DO sign/parse, MRR normalization, full-pipe aggregation
with fake DO+commerce, honest-unconfigured-DO path; /v1/admin/finance
added to the gate test so 403-for-non-admin is proven.
CGO_ENABLED=0 go build ./cmd/cloud: exit 0. go test ./clients/admin/: ok.
Bump to the streaming transport so cloud's SSE endpoints (o11y traces, MCP
notifications, chunked bodies) push over ZAP as live streams, not buffered blobs.
Additive — existing buffered responses unchanged. Build + tests green.
The template-fork merge (#56) left clients/projectsvc/fork.go +
fork_test.go importing github.com/hanzoai/zip while their package sibling
projectsvc.go already moved to github.com/zap-proto/zip (9872d1d), so
`go build ./cmd/cloud` failed with a Ctx/Handler type mismatch. One import
line each — one and one way, the whole package on zap-proto/zip.
Adds clients/git: org+project-scoped Git hosting inside the unified cloud
binary — the "internal Gitea, native" foundation agents push code into.
Control plane (X-Org-Id tenancy, HIP-0026; optional X-Project-Id sub-scope):
POST /v1/git/repos create a bare repo -> repoView (201)
GET /v1/git/repos list the tenant's repos
GET /v1/git/repos/:name repo detail (branches, HEAD, sizeBytes)
DELETE /v1/git/repos/:name delete + purge storage (204)
GET /v1/git/usage per-repo + total bytes for the tenant
Smart-HTTP git protocol (real `git clone`/`git push` work natively):
GET /v1/git/:org/:repo/info/refs?service=git-upload-pack|git-receive-pack
POST /v1/git/:org/:repo/git-upload-pack (clone/fetch)
POST /v1/git/:org/:repo/git-receive-pack (push)
Storage: bare go-git repos on a go-billy filesystem; go-git's server
transport (plumbing/transport/server) reads/writes it for clone AND push.
MVP backs billy with osfs under {DataDir}/git/<org>/<project>/<repo>.git;
a documented TODO(vfs) seam swaps in hanzoai/vfs (S3/SeaweedFS) — vfs.FS
does not yet implement the go-billy surface go-git's dotgit requires.
Billing: every repo tracks sizeBytes, re-measured on create and after each
push; each measurement emits a meterable `git.usage org=.. repo=.. bytes=..`
log line. TODO(billing) seam for a commerce metering event.
Tenant isolation on every query (org empty -> 403; path :org must match the
authenticated tenant). Registered as subsystem "git" order 132.
go-git/v5 v5.19.1 + go-billy/v5 promoted indirect -> direct (no version bump).
Verified: CGO_ENABLED=0 go build ./cmd/cloud (exit 0); go test ./clients/git
(CRUD+isolation, info/refs advertisement, in-process go-git clone/commit/
push/re-clone round-trip, cross-tenant 403) all pass; real `git` CLI
clone/push/re-clone round-trip confirmed against a live server.
Move to the ZAP-family canonical framework (zap-proto/zip) and its final API:
app.Listen(cfg.ZAPListenAddr, "http://"+cfg.ListenAddr) — one verb, transport is
the address scheme (ZAP primary + HTTP extra from one call). Free MCP tool surface
rides along at /mcp over both transports. go.sum re-recorded vs the immutable proxy.
Whole tree builds; cloud/zapface/storagelock tests pass.
Add POST /v1/projects/fork — the ONE way to start a project from the Hanzo
starter-kit gallery in-console. The handler reads the ONE embedded templates
catalog (templates.Get; no catalog copy), maps the template's freeform
framework label to the projectsvc build-hint enum, and funnels through the SAME
createProject path POST /v1/projects uses, so slug validation, org scoping, ID
minting, and conflict handling are not duplicated.
- clients/templates: export List()/Get(slug) so projectsvc reads the catalog
through one door; the HTTP GET handler now uses Get too (DRY).
- clients/projectsvc: extract create -> createProject (the shared internal
create path); fork.go seeds a CreateProject from the template (name=title or
override, slug=target or template slug, framework mapped, repo=gallery source)
and calls createProject; org-scoped (X-Org-Id) exactly like the other routes.
- Framework mapping (mapFramework): Vite wins -> vite; Next.js -> next;
React -> react; enum names pass through; bare HTML/* -> static.
Tests: end-to-end wire tests over the real route (template->project mapping,
org scoping/isolation, dup 409, missing-slug 400, unknown-template 404) plus a
mapFramework table pinned to the real gallery labels.
zip v0.5.0 un-stubs the ZAP transport, so cloud now serves BOTH transports from
the ONE app: app.Serve(cfg.ZAPListenAddr, cfg.ListenAddr) binds ZAP (primary,
:9653 via CLOUD_ZAP_LISTEN — already in config) alongside HTTP (:8000). The log
already advertised the zap addr; now it is actually bound. Every /v1 route answers
identically over either transport (no RPC registry, routes ARE the ZAP surface).
- go.mod: hanzoai/zip v0.2.1 -> v0.5.0 (+ zap-proto/http v0.1.0 transitively).
- serve.go: app.Listen(http) -> app.Serve(zap, http).
Verified: go build ./... green (whole 54-file zip surface); go vet clean;
cloud + zapface + storagelock + admin tests pass. Prod ZAP port :9653 will be
open (was closed — the stub never bound).
cloud is greenfield original (not a casibase fork); the residual casibase/casdoor
names in comments + one type + one error string were off-brand AND contradicted
that provenance (e.g. storagelock's 'casibase-derived cloud-api' lineage story).
De-branded to Hanzo-referential throughout — the {status,msg,data,data2} WIRE shape
is unchanged (console2 depends on it); it is simply OUR /v1 envelope now.
- zapface: casibaseEnvelope type -> envelope; all 'casibase /v1' -> '/v1'.
- storagelock: dropped the casibase-lineage narrative; 'casibase's XORM knob' ->
'the storage driver knob'; classify string -> 'driverName=postgres'. SQLite is
the only backend, full stop (no transitional-config language).
- subsystems: iam '(Casdoor)' -> '(Hanzo IAM)'.
- clients/admin: 'casibase envelope' -> '/v1 envelope' throughout.
- tests: de-branded; the integration test's simulated 'casdoor_session_id' cookie
-> the REAL 'iam_access_token' contract (cookieTokenNames), so it's more faithful.
Verified: go build + go vet clean; storagelock/zapface/clients-admin tests pass.
The org identifier was TrimSpace'd at both trust-boundary sites
(middleware_identity.go on claims.Owner + client X-Org-Id, and
provisioning.sanitizeOrg before hashing), so two DISTINCT IAM orgs differing
only by edge/internal/unicode whitespace ('acme' vs 'acme ' vs 'ac me' vs an
NBSP/ZWSP variant) collapsed onto ONE tenant-<slug> namespace / image ref /
bucket / DB — a cross-tenant fold (IAM org name is an unvalidated varchar, so a
fold-sibling is registrable and mints a valid token).
FIX — normalize+VALIDATE at the trust boundary, reject rather than fold:
- cloud.OrgHasUnsafeRune: refuse any org bearing a whitespace / control /
zero-width-format (Cf) rune. fasthttp OWS-trims header values, so folding
such an org could never round-trip through transport — rejection (fail
secure) is the only injective option. Visible case/'.'/'-' still fold
injectively via the org-slug hash.
- middleware_identity.go: owner is taken verbatim from the validated principal
(no TrimSpace) and refused if unsafe -> request resolves org-less, every
tenant() gate fails closed 403. Client X-Org-Id refused likewise.
- provisioning.sanitizeOrg: reject unsafe-rune inputs (defense-in-depth for
non-header callers e.g. clients/s3) and hash the RAW bytes, never a trimmed
copy. c.Org() is now the sole tenancy source and injective end-to-end.
Regression tests: {acme, 'acme ', 'ac me', NBSP/ZWSP/BOM/tab variants} ->
distinct-or-rejected, never colliding (provisioning + platform + middleware
JWT-owner path). go build ./... + go vet + affected suites green.
First slice of the unified-backend-go program: a native-Go port of the Twenty
CRM core model (company/person/opportunity standard objects, composites
flattened to scalar columns) mounted at order 131 in the one cloud binary.
- Base/SQLite store ({DataDir}/crm.db), tenant isolation = org column on every
query (c.Org() from the validated IAM owner claim, HIP-0026). Mirrors
clients/prompts + clients/eval exactly (the ONE storage pattern).
- Full CRUD for all three entities + per-org summary counts; in-org referential
integrity (a relation can never point across tenants; errBadRef -> 422).
- /v1/crm/{summary,companies,contacts,opportunities} — /v1 only, no /api.
- Tests: per-org isolation, CRUD round-trip, referential integrity,
delete-clears-refs, list filters/counts, HTTP round-trip + validation. 7/7 pass.
No proxy to a NestJS backend; this is the thesis (business apps as native-Go
/v1 subsystems on Base) embodied as one working brick.
/v1/platform (PaaS) — RED do-not-ship findings. Tenancy core untouched.
CRIT-1 — OS command injection in the privileged BuildKit Job:
launchBuildJob now emits buildctl as EXEC-FORM argv ([]string, no `sh -c`),
so no shell parses any input. repo.url / dockerfile / git-ref are validated
(validate.go): https-only URL to an allowlisted git host, no shell/flag
metachars; safe relative dockerfile (no `..`); safe branch/tag/commit ref
(no `#`, no metachars). Output image ref is forced server-side — a client
cannot override --output/--opt. Validation runs at the build choke point AND
early at createApp (400). red_cmdinj_poc_test.go flipped to a passing guard.
CRIT-2 — sanitizeOrg collision (non-injective) → cross-tenant takeover:
deleted the lossy platform.sanitizeOrg; tenant()/tenantNamespace()/
buildImageRef() now use the ONE injective provisioning.SanitizeOrg (DRY,
reused — commit 4e77020c). Image ref made injective too: org+app are now
separate '/'-joined path components (ghcr.io/hanzoai/tenant-<org>/<app>),
neither slug can contain '/', so (a-b,c) vs (a,b-c) no longer collide.
MED-3 — quotas / replica bounds / shared-build DoS:
clampReplicas caps replicas to [1,20] (env CLOUD_PLATFORM_MAX_REPLICAS) at
createApp, applyService, and scaleService (fail-secure default). ensureNamespace
applies a ResourceQuota + LimitRange per tenant namespace (idempotent).
launchBuildJob caps concurrent builds per org (default 3, errTooManyBuilds→429).
Tests: cmd-injection blocked (5 vectors), org-slug + image-ref injectivity,
replica clamp (unit+HTTP), namespace quota/limitrange, concurrent-build cap.
go build ./... green; clients/platform + provisioning + s3 tests green.
A synchronous run drives up to maxRunItems paired LLM calls against the SHARED
in-process gateway. Two bounds stop one org from degrading every tenant:
- Per-org concurrency cap (maxConcurrentRunsPerOrg=4): a run acquires a per-org
slot after passing validation; excess concurrent runs fail fast with 429 (never
queued — queuing just relocates the exhaustion). Slot released on every return.
- Total wall-clock deadline (maxRunDuration=10m): the item loop runs under a
context.WithTimeout; a run that exceeds it is cancelled, remaining items are
recorded as honest errors, and the partial summary returns (Scored counts only
real successes → 502 when nothing scored). A runaway can never pin a request +
a gateway slot indefinitely.
Also (RED LOW, verified NOT present at 249a136e despite belief): getDataset now
sizes its collection via store.CountItems (SELECT COUNT(*)) instead of loading up
to maxListLimit full item bodies just to len() them (~96MB amplification on a
large dataset).
Tests: TestRunConcurrencyCap (semaphore fill/refuse/release), TestRunDeadline
Bounded (blocking runner + tiny deadline → cancels, 502, honest item errors, no
hang). go build ./clients/eval/ . green, go vet clean, all eval tests pass.
Two cross-tenant fixes on the /v1/evals API layer (RED review):
1. Principal gate (HIGH): tenant() now requires a non-empty c.User() (X-User-Id,
set ONLY by SanitizeIdentity from a verified token/session and stripped from
client input). Its Phase-1 residual RESTORES a client X-Org-Id on the
NO-principal path (bearer-less / opaque hk-/sk- key / invalid bearer), so
without this gate a direct-to-pod request 'X-Org-Id: victim' with no auth read
/wrote/deleted the victim org's datasets (PII golden outputs), scores, traces
and runs. This is the same trust signal the audit layer uses (actorFromCtx).
2. Buffer-aliasing (correctness on the isolation key): c.Org() is a zero-copy
view into the fasthttp request buffer, reused after the request ends. The org
is our tenant KEY and is retained (telemetry events, run records); tenant()
now strings.Clone()s it so a stored org can never silently mutate into another
value once the buffer is recycled (was manifesting as run scores landing under
a corrupted org id).
red_cap_test.go TestRed_ForgedHeaderCannotCrossTenant now sends a no-principal
X-Org-Id:victim request asserting 403; eval_test.go asserts tenant()='' without a
validated principal. go build ./clients/eval/ . green, go vet clean, all eval
tests pass.
Replace the Langfuse-fork proxy (crash-looping console P1012) with a native,
org-scoped evals system. Storage split per CTO directive:
- metastore (store.go): Base/SQLite, per-org config — datasets, dataset-items,
evaluators, score-configs, dataset-run defs. Composite (org,id) keys so an id
is never a cross-org global key (no existence oracle; two orgs may reuse ids).
- telemetry (telemetry.go): datastore/ClickHouse MergeTree, append-only traces +
scores-as-events; every read binds org as a named param + LIMIT; behind an
interface with an in-memory impl for tests. Reuses the Langfuse v3 CH shapes.
- runner (runner.go): pluggable EvalRunner (Complete + Judge); gateway runner is
the spine (any model/judge, no token cap), DO can drop in as an adapter later.
Tenant isolation is c.Org() (validated bearer owner) ONLY — never client
X-Project-Id/X-Org-Id (the cross-tenant break the old proxy shipped). Score
integrity: NaN/Inf rejected, values validated against the org's score-config,
categorical labels checked against the allowed set. Content caps + name regex
guard injection/traversal/amplification.
24 tests pass: metastore + HTTP cross-tenant isolation, forged-header guard,
score-integrity, content caps, run orchestration over a stub runner.
/v1 only. TDD (go test green). gofmt+vet clean. go build ./... green.
One clean env family for the ONE shared S3 access path (clients/s3admin)
and the provisioning control plane. Rename across s3admin, clients/s3,
projectsvc, and provisioning — code, comments, log/error strings, tests:
CLOUD_S3_ADMIN_ENDPOINT -> S3_ADMIN_ENDPOINT
CLOUD_S3_ADMIN_ACCESS_KEY -> S3_ADMIN_ACCESS_KEY
CLOUD_S3_ADMIN_SECRET_KEY -> S3_ADMIN_SECRET_KEY
CLOUD_S3_SECURE -> S3_SECURE
CLOUD_S3_REGION -> S3_REGION
CLOUD_S3_PUBLIC_ENDPOINT -> S3_PUBLIC_ENDPOINT
CLOUD_S3_PUBLIC_SECURE -> S3_PUBLIC_SECURE
The cloud CR (universe 9e57d71d) already stamps BOTH the old and new
ACCESS_KEY/SECRET_KEY spellings from the s3-credentials secret, so this
image roll is drop-in: old names stay populated until the new image
lands, then the S3_* names take over. Everything else resolves from code
defaults (s3.hanzo.svc:9000 / us-east-1 / s3.hanzo.ai). go build + vet +
tests green across all four packages.
No TODO, no placeholder: the "full customer encryption" brick is real and wired
into the Replicator. Each org's SQLite snapshot is sealed with a distinct
AES-256-GCM key DERIVED from the KMS master via HKDF(master, label, orgID) — the
master never leaves the process, per-org keys are in-memory only. The SeaweedFS
object is ciphertext; orgs are cryptographically isolated (orgID bound as GCM
AAD, so a blob can't be replayed under another org); rotating the master re-keys
everything. Nonce is derived from (key, plaintext) so identical content seals
identically — the Replicator's version-skip keeps working under encryption.
Wired: NewReplicator(..., WithEncryption(cipher, orgID)) → Push seals, Pull opens.
Omit the option and the DB is stored in the clear (local dev only).
Pure Go stdlib (crypto/aes, crypto/cipher, crypto/hmac, crypto/sha256, crypto/subtle) —
package org stays dependency-free + testable. Tests: round-trip, wrong-master reject,
cross-org isolation, tamper detection, deterministic-per-content, master-rotation
rekey, and an end-to-end encrypted Replicator (stored bytes are ciphertext, reader
with the key restores plaintext, no plaintext leak without the key).
A custom domains[] entry was rendered straight into the operator Service CR
ingress.hosts, so a tenant could claim another org's host or a Hanzo apex
(api.hanzo.ai) and the operator would serve an Ingress for it. Require every
custom host to be under the caller's OWN '<org>.<sitesHost>' subtree (e.g.
maxpower may only claim *.maxpower.hanzo.app); anything else is refused 501
(verified arbitrary custom domains are phase-2 domain CRUD). Closes the
cross-tenant/apex domain-hijack vector reachable in the first slice. +2 tests
(unit + HTTP); 23 tests green.
/v1/platform mutates cluster state (operator Service CRs + BuildKit Jobs in
tenant-<org>) — more consequential than a data read — so trusting X-Org-Id alone
would let a direct-to-pod caller forge X-Org-Id:victim with NO bearer and
deploy/read into another tenant (SanitizeIdentity's documented Phase-1 residual).
tenant() now gates on c.User() (X-User-Id, set ONLY for a validated principal),
mirroring the Red-hardened clients/s3.tenant. Proven live: forged X-Org-Id with
no token -> 403 (was 200); real JWT -> 200; forged header + real token ->
validated owner wins. Every legitimate caller (gateway/console BFF) carries a
user-bound bearer, so no real client breaks.
RED MED-1 (cross-tenant eval-score read). eval.tenant() scopes the console API
key pair (console-pk-{org}/console-sk-{org} in KMS) and was PREFERRING the raw
`X-Project-Id` request header over the bearer-pinned org, then feeding it to
resolveKeys(). X-Project-Id is a project sub-scope WITHIN an org and is
DELIBERATELY excluded from SanitizeIdentity.authorityHeaders (client-controllable,
per middleware_identity.go). So a caller who set `X-Project-Id: victim-org` made
resolveKeys fetch ANOTHER org's console-pk/console-sk from KMS and read that org's
eval scores/datasets — a cross-tenant break. Reading a raw X-Org-Id header was
equally unsafe (SanitizeIdentity strips a client copy and re-mints c.Org() from
the token).
Fix: tenant() returns ONLY c.Org() — the org SanitizeIdentity pinned from the
validated bearer owner (HIP-0026) — the same authoritative selector
agents/prompts/provisioning use. X-Project-Id no longer influences KMS key
selection anywhere in the evals facade (it was used nowhere else). Per-PROJECT
key scoping, if ever needed, must derive the project from a membership check UNDER
c.Org(), never a raw sub-scope header (Phase-2; keys stay org-scoped today).
The console2 BFF (app/cloud) was the sole compensating control (it drops
X-Project-Id without forwardScope); this closes the defect AT THE SOURCE so
isolation no longer hangs on one omitted proxy header. New test
TestTenantIgnoresClientProjectID pins that a forged X-Project-Id never becomes the
tenant. go build ./clients/... + go test ./clients/eval/... green.
Port the standalone Dokploy (platform.hanzo.ai) tRPC backend into the unified
cloud binary as clients/platform, mounted at /v1/platform (HIP-0106). Per-org,
IAM-validated, Base/SQLite store; the deploy path writes an operator hanzo.ai/v1
Service CR into the caller's OWN tenant-<org> namespace (derived from the
validated X-Org-Id, never a request input) and the operator reconciles it. Git
apps build via an in-cluster BuildKit Job (arcd model); image apps deploy
directly. Complements clients/paassvc (admin fleet board) and clients/projectsvc
(static sites) with the container-app PaaS.
Goa is the design-first contract (clients/platform/design, goa gen -> OpenAPI 3);
the runtime is native zip handlers (one router, behind SanitizeIdentity) — the
generated net/http server is deliberately NOT mounted so the identity trust
boundary is not routed through the fiber<->net/http adaptor.
- store.go: projects/applications/deployments/builds, org column tenancy
- k8s.go: tenant-<org> namespace derivation + Service CR apply/scale/delete + BuildKit Job
- platform.go: Mount + project/app CRUD + tenant() gate + health
- deploy.go: deploy/start/stop + deployment history/logs, fail-closed (no fabricated success)
- 20 tests: store CRUD, cross-tenant isolation (RED bar), fail-closed deploy,
fake-cluster deploy-into-tenant-ns success, secret-env rejection
go build ./... green; go test ./clients/platform/ green.
Red re-review (0 critical, 1 high, 1 med, 1 low): the s3 data-plane HIGH was
confirmed CLOSED, but Red found the same forge open on the provisioning control
plane (worse: destroy DB + credential exfil), proved my org-fold dispute WRONG
with a reachable cross-tenant collision, and asked to lock the fix's dependency.
- [HIGH] provisioning.tenant() now requires ctx.User() (provisioning.go:454) —
same gate as the s3 fix. Without it, an in-cluster caller could forge
'X-Org-Id: victim' with NO bearer and POST /v1/sql (allocate a DB in the
victim's namespace + receive its connection string + password), DELETE
/v1/sql/:name (destroy the victim's DB), or enumerate resources. SanitizeIdentity
restores a forged X-Org-Id on the no-principal Phase-1 path but strips X-User-Id;
gating on it refuses only the anonymous forge. Test:
TestForgedOrgWithoutPrincipalRefused (forged org + no principal -> 403 across
POST/DELETE/GET, provisioner never runs).
- [MED, dispute WITHDRAWN — Red was right] provisioning.sanitizeOrg is now
INJECTIVE (provisioning.go:468): identity on a clean [a-z0-9-] slug, else the
fold + '-'+16hex SHA-256(raw owner) — mirroring iam/object/orgdb.go:orgSlug.
The old lossy fold collapsed 'Acme'/'acme' and 'team.a'/'team-a' onto one slug,
and since the whole tenant->bucket/DB namespace hashes THAT slug, two distinct
orgs shared one physical namespace (reachable: the IAM org name is a varchar
with no shape validator, so a fold-sibling is registerable + mints a valid
token). Tests: TestSanitizeOrgInjective (the exact collisions no longer collide,
incl. derived orgHash) + updated TestSanitizeOrg.
- [LOW] locked the s3/provisioning fixes' cross-file dependency:
TestSanitizeIdentity_AnonPathHasNoUserId asserts a client-forged X-User-Id does
NOT survive the anon path (ctx.User()=="") while X-Org-Id does — so a future
refactor that restored X-User-Id fails this test first.
All fold consumers (orgHash -> SQL/KV/CH/S3 physical namespaces) inherit the
injective slug through the ONE sanitizeOrg. go test green (provisioning + s3 +
s3admin + root identity); cmd/cloud builds; gofmt/vet clean; zero go.sum drift.
Red adversarial review (0 critical, 1 high, 3 med, 4 low). Fixes:
- [HIGH] tenant() now REQUIRES a validated principal (ctx.User()/X-User-Id).
SanitizeIdentity restores the client's raw X-Org-Id on the no-principal
'Phase-1 data path' but leaves X-User-Id empty; a pure data plane trusting
X-Org-Id alone let an in-cluster caller (co-namespace pod) forge
'X-Org-Id: victim' with NO bearer and get cross-tenant object CRUD. Gating on
X-User-Id refuses ONLY that anonymous forge path — every legitimate caller
reaches s3 through the console BFF /cloud proxy which mints a user bearer, so
no real client breaks. Object storage never serves an unauthenticated
principal. Test: TestForgedOrgWithoutPrincipalRefused (forged org + no
principal -> 403 across the full route surface).
- [MED] presign TTL 15m -> 5m: bounds a minted capability's post-revocation
lifetime (presigned URLs have no server-side revocation; the TTL IS the
window). Documented the unwired-rate-limiter platform gap.
- [LOW] cleanKey rejects control bytes (\x00-\x1f) + backslash: a null byte
serializes as %00 (C-string truncation risk for a downstream consumer) and
'\' is a non-Go path separator. Tests extended.
- [LOW] friendlyBucket re-validates the recovered name against bucketNameRE:
a prefixed-but-non-conforming bucket (only reachable out-of-band, never via
createBucket) is treated as not-owned, so listBuckets never echoes an
unaddressable name. Test added.
- Documented the ESCALATED residuals (not subsystem-fixable): single omnipotent
SeaweedFS identity (isolation is app-layer only until STS/scoped creds), and
the intentional S3-vs-KMS org-normalization divergence (S3 must fold to match
provisioning's bucket naming; KMS keys secrets by exact owner).
Verdict was fix-then-ship; no critical, HIGH is bounded (not internet-reachable,
gateway strips X-Org-Id at the edge). go test ./clients/s3/... green (20 tests);
cmd/cloud builds; gofmt clean; zero go.sum drift.
Adds the DATA plane over the shared SeaweedFS S3 gateway as /v1/s3/* on the
unified cloud binary (HIP-0106), the companion to clients/provisioning's s3
CONTROL plane. One console, one backend — no external s3.hanzo.ai UI.
- clients/s3admin: the ONE shared S3 access path. Both projectsvc/blob (deploy
blob store) and the new s3 subsystem build their minio client here from the
SAME CLOUD_S3_ADMIN_* creds (DRY — no second S3 client anywhere). Leaf pkg
(minio-go only), so no import cycle. Separate public-host client mints
presigned URLs a browser can follow.
- clients/projectsvc/blob.go: refactored to build its minio client via s3admin
(was inline minio.New). Existing projectsvc tests unchanged + green.
- clients/s3: /v1/s3/{health,buckets,buckets/:bucket,buckets/:bucket/objects,
buckets/:bucket/objects/*}. Org-scoped bucket-per-org via the EXACT
provisioning.BucketName scheme (exported) = bucketName(physicalName(org,name)):
org-hash prefixed AND '_'->'-' folded to a DNS-safe S3 name — so a bucket
provisioned via POST /v1/s3 {name} is browsable here AND a bucket created here
is a valid S3 name (the raw physicalName has underscores S3 rejects). Upload/
download = presigned PUT/GET URLs (browser goes direct to S3; admin cred never
leaves the server; object key path-clean-guarded; time-boxed 15m). Fail-closed
503 without creds. Registered 's3svc' order 118 (< provisioning 120) so the
static /v1/s3/buckets + /v1/s3/health win Fiber's first-match scan ahead of
/v1/s3/:name and the generic-health route does not shadow the real probe.
- subsystems.go: one-line blank import.
Tests: go test ./clients/s3/... ./clients/s3admin/... green — fail-closed 503,
org 403, route-ordering (s3 owns /v1/s3/buckets + /health, not provisioning
:name), bucket-name + object-key traversal 400, cross-tenant physical-name
isolation, AND bucket-name consistency with provisioning + DNS-safety (no '_').
Zero go.sum drift (minio-go already a dep). cmd/cloud builds.
Pulls hanzoai/ai#a5a199e9: the cloud_usage/observations ledger now uses a direct
clickhouse-go/v2 client (object.InitDatastore in the shared Bootstrap) instead of
the dead ZAP 'datastore peer'. Fixes GET /v1/get-cloud-usages 'datastore peer not
connected'. Requires DATASTORE_ADDR/USER/PASSWORD env (wired on the cloud CR).
Tenants were shown the internal admin address (e.g. vector.hanzo.svc:6333) in
create/get/list responses + the connectionString — unusable from an app and a
leak. Add publicEndpoint(kind): HTTP kinds (vector/search/docdb/s3) → the unified
api.hanzo.ai gateway (/v1/<kind>/*); native-wire DBs → <kind>.hanzo.ai on the
native port (sql.hanzo.ai:5432, kv.hanzo.ai:6379, datastore.hanzo.ai:8123). So a
customer gets a real, routable endpoint for their app + hanzo.app. Per-kind
override via PUBLIC_<KIND>_HOST/_PORT. The DSN host:port is remapped too.
Co-authored-by: zeekay <z@zeekay.io>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live 1.786.5 returned 500 "no such column: id" on GET /v1/prompts. The removed
clients/prompt facade (shipped in 1.786.4) had created a `prompts` table in the
SAME {DataDir}/prompts.db with a versioned-rows layout that has NO `id` column.
clients/prompts' `CREATE TABLE IF NOT EXISTS prompts` no-ops on that existing
table, so every `SELECT id,... FROM prompts` failed against the persisted PVC.
migrate() now inspects the `prompts` table via PRAGMA table_info; when it exists
but lacks `id` (the legacy signature) it drops the legacy pair and rebuilds the
forward schema. The legacy rows carry no durable product data (the facade shipped
one release; the console Compute pages never wrote through it), so this is
forward-only and idempotent — once `id` exists it never fires again.
TDD: TestMigrateSelfHealsFromLegacyPromptSchema seeds the exact legacy schema +
a row, opens the store over the same file, and asserts List/Upsert succeed. Green.
Mounts the red-approved (3 rounds) per-org product control planes natively
in the unified cloud binary (the "all products in the cloud binary" thesis):
- clients/prompts order 126 /v1/prompts/* versioned prompt library
- clients/agents order 127 /v1/agents/* autonomous agents + runs
- clients/functions order 128 /v1/functions/* serverless fns + invoke
All three are org-scoped by the gateway-minted X-Org-Id (HIP-0026), fail
closed when it is absent, and persist to a per-tenant modernc.org/sqlite
store under CLOUD_DATA_DIR (no CGO). Each ships adversarial red_*_test.go
coverage (all green).
DRY reconciliation with main: main had independently added clients/prompt
(singular) also registering "prompts" for /v1/prompts. Two subsystems both
named "prompts" would double-append to cloud.Registry and double-mount the
same routes. blue's clients/prompts is the red-approved superset (adds
/v1/prompts/metrics — which console2's PromptsModule consumes — plus DELETE,
strict fail-closed X-Org-Id, and a reserved-name guard), so it becomes the
ONE owner of /v1/prompts/*; the earlier clients/prompt facade is removed.
Build green (CGO_ENABLED=0 go build ./...); prompts/agents/functions +
root + cmd/cloud tests all pass.
The console (Langfuse image 3.159.55) 307-redirects /api/public/v2/prompts →
/v1/prompts, delegating prompts TO cloud. promptsvc used to proxy the other
way → redirect loop → 500. Port prompts NATIVE: an org-scoped, versioned
prompt registry in a single SQLite file (base pattern, modernc driver, WAL),
list/get/create. No proxy, no loop, all-SQLite by construction. Verified:
GET /v1/prompts → {data:[]}, POST creates v1, GET lists it, GET /:name
returns versions. Honest-empty when none.
HIP-0106 'all Go embeds in cloud': the KMS secrets plane runs in-process
in the cloud binary instead of the standalone Infisical fork.
- clients/kmsembed: cloud-free Client (implements types.KMSClient) over a
luxfi/zapdb SecretStore. AES-256-GCM envelope (per-secret DEK sealed
under a 32-byte master KEK); plaintext never touches disk. New() uses a
fail-SECURE 3-way store-open (keyed→encrypted on-disk; no-key+no-store→
ephemeral in-memory, no plaintext registry; no-key+existing-store→fail
loud, never silently shadow encrypted data). Sign fails closed when no
MPC backend is co-hosted. One place for key-shape validation.
- clients/kms: Fiber subsystem mounting /v1/kms/* (order 10), org-scoped
CRUD via cloud's auth; /v1/kms/health + /v1/kms/config.
- build.go pickKMSClient: Enabled('kmssvc') → in-process kmsembed.New,
fail-closed to DisabledKMS on error (never nil).
- config: CLOUD_KMS_MASTER_KEY_REF / CLOUD_KMS_MPC_ADDR / _VAULT_ID.
Reviewed blue+red: 40 tests (7 build + ~33 adversarial), gofmt/vet clean,
go.sum zero-diff. Red verdict: ship (fail-closed + confidentiality
invariants hold across the full key/store matrix).
Brings the released ai v1.789.0 onto main: it has BOTH the V1IamRewriteFilter
(the /v1/iam/* account-surface fix shipped off-main as the 1.785.35 hotfix) AND
the canonical X-Project-Id/X-Environment/X-User-Id header sweep. Resolves the
divergence — main is now the one lineage (prompts + bot + headers + org refactor
+ SeaweedFS replication + iam fix). Build + embed/identity tests green.
Drop the confusing "tenant" concept: in Hanzo the ORGANIZATION is the tenant
boundary (identity, billing, per-org SQLite are all org-scoped). Renamed
internal/tenant → internal/org and made ownership explicitly PER-ORG: one replica
writes ALL of an org's databases (root + per-project + per-user), for locality +
intra-org consistency; the org moves as a unit on failover.
- owner.go: Owner/IsOwner/Replicas over Rendezvous (HRW) hashing of orgID — every
replica computes the same writer-owner from the same membership, NO coordinator.
- membership.go: live replica set via a pluggable Source (StaticSource/CLOUD_REPLICAS
now; K8s Endpoints / zapd gossip later); lock-free AmOwner hot-path.
- replica.go: Replicator Push (owner → SeaweedFS) / Pull (reader ← SeaweedFS,
version-skip) + DBPath (orgs/<org>[/<scope>]/<service>.db, HIP-0302).
- vfsstore.go: object store bound to hanzoai/vfs (SeaweedFS) — NO minio, NO external
S3 SDK. hanzoai/sqlite + hanzoai/base back the local DB handle (the DB interface).
Package is PURE stdlib (local vfsClient interface, zero cloud deps) so it builds +
tests without the dep tree. 16 tests pass: determinism, single-owner, even
distribution (±35%/50k), exact minimal-reshuffle, ordered failover, push/pull
round-trip, skip-unchanged, ownership handover, DBPath layout, vfs adapter. gofmt clean.
- botsvc (order 143): reverse-proxies /v1/bot/* to the in-cluster bot-gateway,
stripping the /v1/bot prefix (the gateway serves bare paths: /v1/bot/health →
bot-gateway /health) and forwarding the gateway-minted identity headers. The
console2 Bot module's /v1/bot/health probe now resolves instead of 404.
Verified e2e against the real bot-gateway: /v1/bot/health → 200
{"service":"bot","status":"ok"}.
- provisioner: rename the datastore/docdb provisioners by the HANZO PRIMITIVE
(datastoreProvisioner/newDatastore, docdbProvisioner/newDocdb) not the backing
tech — the ClickHouse/MongoDB driver imports + wire-protocol schemes stay
(functionally required), but the type names read as the primitive.
Note: /v1/s3, /v1/datastore, /v1/docdb are ALREADY mounted (provisioning loops
its 7 kinds); /v1/memory is served by the ai monolith. functions has no backend
— left honest (not fabricated).
The console2 Prompts module hit GET /v1/prompts → 404 "not routed on this
host" because no subsystem owned the route. Add promptsvc: a thin facade
(order 144, mirrors evalsvc) proxying list / get-by-name / create to the
console public prompts API (/api/public/v2/prompts) under the project-scoped
console key pair (HTTP Basic) — the same console + auth the eval facade
already composes. No prompt logic reimplemented; the console owns storage,
versioning, and labels.
Verified locally: the binary boots with "prompts surface mounted", GET
/v1/prompts routes to promptsvc (503 honest "no console API key" in the
keyless test env, not a 404), /v1/prompts/health → 200 (no route shadow).
In prod the console-keys secret is already wired (evalsvc), so prompts
resolve real data. eval tests + embed tests stay green.
The load-bearing primitive of the horizontally-scalable OSS cloud (Hanzo V8).
Every replica of the unified binary computes the SAME writer-owner for a tenant's
per-tenant SQLite from the SAME membership set — via Rendezvous (HRW) hashing —
so there is NO election, NO lock service, NO discovery. This removes the whole
"who finds/owns/reaches service X" plumbing class for tenant state:
- Owner(tenant, members): deterministic, order-independent, exactly one owner.
- IsOwner(tenant, self, members): the per-write hot-path check.
- Replicas(tenant, members, n): owner + ordered failover successors (pre-warm S3).
HRW gives minimal reshuffle on membership change (only a departed replica's ~1/N
tenants migrate; the rest stay put) — cheap rolling deploys + scale-out. Pure Go
(crypto/sha256, no deps). 6 tests: determinism, single-owner, even distribution
(±35%/50k), minimal-reshuffle (exact), ordered failover. All pass; gofmt clean.
Wires into: owner holds the SQLite WAL (1 writer + N readers), streams WAL to
SeaweedFS/S3 (HIP-0107); non-owners read the S3 copy or forward strong writes.
Per-tenant envelope encryption (DEK wrapped by KMS master) makes the S3 file
ciphertext — tenants crypto-isolated. Membership feeds from K8s Endpoints / zapd.
Pulls hanzoai/ai 56a55d1c so the unified binary's monolith reads the
canonical X-User-Id (cloud middleware_identity already injects it) instead
of the never-sent X-IAM-User-Id. Completes the X-IAM-* → canonical sweep
(org/project/env/user) in the compiled-in monolith. Builds + tests pass.
Pulls hanzoai/ai 7aab19aa into the unified binary so the monolith's
tenant-context filter reads the canonical X-Project-Id / X-Environment
(was X-IAM-*, never sent → project+env scoping was empty across ~200 /v1
routes) + the CORS allow-list accepts them. Pseudo-version pending the ai
v1.788.1 release tag; re-pin to the clean tag on CI cut. Full binary
builds, embed + identity tests pass.
evalsvc.tenant() read `X-IAM-Project-Id`, which nothing sends — console2
stamps the canonical `X-Project-Id` — so a selected project always fell
through to org-level and project selection scoped ZERO backend calls.
Switch the reader to `X-Project-Id` (one canonical header, per the
one-way rule); resolveKeys already accepts the project slug. Update the
identity-sanitizer comment to name the canonical sub-scope.
Build green, eval + identity tests pass.
The repo carried TWO console embeds from parallel work: webui.go (wired in
Serve via mountConsole, no build tag — the working one) and clients/console
(a second take gated by //go:build cloud, imported by nobody after the earlier
build fix). The tag hid it from a normal build and the stale bundle doc comment
made it look like the whole binary needed -tags cloud. It never did.
- Delete clients/console/ — dead duplicate (unimported, tag-excluded, would
double-mount "/"). Zero //go:build cloud tags remain in the repo.
- subsystems.go: correct the doc comment — subsystems register unconditionally;
plain `go build ./cmd/cloud` (no tags) links and mounts the full set. Drop the
now-stale clients/console note.
One console path (webui.go), builds normally. Verified: go build ./cmd/cloud
(no tags) → 427MB binary, go vet clean.
The audit-trail commit 587952be rewrote 5 luxfi h1: module-zip hashes
(age, keys, pq, precompile, zap) to non-canonical values recorded under a
local GOPRIVATE/GONOSUMDB env, breaking 'go mod download' in the release
Docker build (checksum-DB SECURITY ERROR). go.mod is byte-identical to the
last-green build fbb76912, so restore its go.sum. All 5 now match
sum.golang.org. No code change.
* fix(deps): reconcile 5 drifted luxfi go.sum hashes (age keys precompile pq zap)
These 5 luxfi modules were re-published under the same version tags (monorepo
re-tag); every local + CI module cache and the proxy agree on the new zip
hashes while the committed go.sum still pinned the old ones, so ALL builds fail
with a checksum-mismatch SECURITY ERROR. Reconcile go.sum to the hashes every
source agrees on (what go mod tidy would write). GONOSUMDB already trusts luxfi
(fetch direct). Pre-existing drift, orthogonal to the audit feature; needed to
build.
* feat(cloud): compliance-grade audit trail — tamper-evident, append-only, live
FedRAMP AU-* / SOC 2 CC-* audit control for the unified cloud binary. Every
security-relevant request against this binary is captured as a structured,
hash-chained record in an append-only store the app can only INSERT into, and
queryable through the global-admin-gated /v1/admin/audit surface.
WHAT
- audit/ package (record + chain + store + redact + query/verify), no route
knowledge, pure security logic:
* record.go — the AU-3 event model (actor/action/resource/auth/outcome/
source-ip/ua/request-id/before-after) + the hash-chain math:
hash = SHA256(canonical(record, hash+prevhash zeroed) || prevHash),
genesis-anchored, DRY (add a field → covered by the hash automatically).
* store.go — a single serialized Recorder (mutex + SQLite MaxOpenConns(1))
owning the chain head; INSERT-only SQLite primary ({DataDir}/audit.db,
zero-loss, synchronous) + optional best-effort ClickHouse mirror. Restart
recovers the head so the chain continues (never forks).
* query.go — filtered Query (parameterized; org/actor/action/resource/result/
time) + Verify (walks the chain, recomputes every hash, reports the exact
seq where a tamper/delete/reorder first breaks it).
* redact.go — secret-key denylist (deny-by-key-name, recursive, fail-closed)
for the before/after an explicit emit point supplies.
- audit_middleware.go (cloud pkg) — the ONE place every security-relevant
request is recorded (decomplected: one predicate, every route). Sits AFTER
SanitizeIdentity (validated, unforgeable actor/isAdmin) and BEFORE BillingGate
(so billing 402/503 + admin 403 denials are audited too). Captures METADATA
ONLY — never request/response bodies — so a secret in a body can't leak.
Records mutations + all /v1/admin/* + all 401/403. Fails the request CLOSED
(503) if the trail write fails (AU-5). Resolves the effective status from a
returned *zip.HTTPError so error-returning denials are audited.
- audit_mirror.go — ClickHouse MergeTree OLAP mirror (insert-only by engine),
best-effort projection for fleet retention/query. Driver already in go.mod.
- clients/admin/audit.go — rewires GET /v1/admin/audit to cloud's REAL store
(was an IAM get-records proxy; kept as a federated fallback) + adds
GET /v1/admin/audit/verify. Both behind the existing global-admin s.guard.
TESTS (all real, on-disk SQLite, no mocks)
- audit/: chain seals+links, verify passes clean, DETECTS field-tamper /
deletion / reorder (out-of-band UPDATE/DELETE on a 2nd connection), restart
continues the chain, concurrent appends stay gapless+verified (-race), redact
strips secrets + fails closed, SQL-injection filter is inert.
- cloud/: middleware records a mutation with validated identity, audits a 403
denial, SKIPS safe reads, audits admin reads, NEVER captures a secret-bearing
body, no-op when unconfigured, fails closed on write error.
- clients/admin/: /v1/admin/audit returns real records + integrity summary,
filters, verify endpoint, 403 without global-admin (no data leak), nil-store
fallback.
THREAT MODEL
- Forge actor/admin: impossible at request level (SanitizeIdentity strips
X-User-IsAdmin, actor from validated JWT).
- Forge the chain: an out-of-band edit re-hashes differently; keeping the chain
valid requires recomputing the whole suffix — bounded by an externally-pinned
head (Head()) for AU-9 (tail-truncation detection). Documented.
- Skip the middleware: mounted at the compose root before MountAll; the /zap
plane replays through the same Fiber app (all middleware), so no bypass.
- Fail-closed is POST-RESPONSE: prevention is the AC layer (runs before the
action); the trail is detection/accountability. Documented precisely.
Store is a compliance control: empty DataDir is a hard boot error unless
CLOUD_AUDIT_DISABLED=true (explicit opt-out). Secrets from env/KMS only; no
plaintext credential ever enters a record.
* harden(audit): scrub credential-shaped path segments + expand redaction denylist
Defense-in-depth from self-review before adversarial handoff:
- scrubCredentialSegments/scrubToken: a token that ever rides in a URL PATH
(an hk-/sk-/pk-/fw_/hz_ key, reusing isAPIKey) is replaced with a marker in
both Record.Path and resource.ID, so a secret in the path is never recorded
verbatim. Normal identifiers (:name/:slug/:id/uuid/numeric) pass through.
Proven by TestAudit_ScrubsCredentialInPath.
- Redaction denylist gains passphrase, privkey, social_security, phrase (covers
seedPhrase/recoveryPhrase) — closing the key-name gaps found by enumerating
real credential field names. TestRedact_StripsSecrets now asserts them.
* harden(audit): close raw-secret leak in path/resource-id/user-agent
Self-review PoC found a real residual leak beyond prefixed keys: a raw
high-entropy secret (64-hex, or a JWT) in the URL path — and a bearer/key in
the client User-Agent — were recorded verbatim (isAPIKey only matched
hk-/sk-/pk-/fw_/hz_ prefixes). Closed:
- scrubToken now also catches JWTs (eyJ + two dots) and long unbroken
high-entropy alphanumeric runs (>=32, mixed, no separators) — a raw API
key/hex secret. UUIDs (hyphens), slugs, names, emails, numeric ids pass
through (TestScrubToken_NoFalsePositives).
- scrubFreeText scrubs credential-shaped words from the User-Agent (splits on
space/=/;/,) and caps length at 512. Normal UA prefix preserved.
Proven: TestAudit_ScrubsCredentialInPath (prefixed+raw-hex+JWT),
TestAudit_ScrubsSecretInUserAgent. Query-string secrets already safe (c.Path()
excludes the query string).
* fix(audit): close audit-evasion via /health suffix on mutations (SECURITY)
Self-review PoC found a real evasion: isSecurityRelevant skipped ANY path
ending in /health, so a mutating POST /v1/admin/orgs/x/health (wildcard route
or an attacker-named segment) slipped past the audit trail entirely — the
worst class of bug for a compliance control (silent bypass).
Fix: check the unconditional security signals FIRST and without exception — a
401/403 denial, any /v1/admin/* call, and any POST/PUT/PATCH/DELETE are ALWAYS
audited whatever the path. Only after that is a safe read dropped (all safe
reads, incl. liveness probes, are request-log noise → not recorded). The
suffix-based /health exemption is gone; it can no longer suppress a mutation.
Proven by TestAudit_HealthSuffixCannotEvadeAudit (POST .../health IS audited;
GET /v1/kms/health is not) and the unchanged TestAudit_SkipsSafeReads.
* harden(audit): no false-attribution — anonymous request records no org/sub
Self-review: SanitizeIdentity's Phase-1 residual restores a client-supplied
X-Org-Id for the data path, so an UNAUTHENTICATED attacker sending
X-Org-Id: victim-org could stamp an audit event with a victim's org (false
attribution), even though X-User-Id/IsAdmin are correctly stripped.
Fix: actorFromCtx gates the recorded actor on a VALIDATED principal — a
non-empty c.User() (X-User-Id, which SanitizeIdentity sets only from a verified
JWT). With no validated sub (anonymous, or an invalid/garbage bearer that failed
validation), the actor is recorded EMPTY: the event stands as an honest
anonymous mutation identified by SourceIP, never mis-attributed to a claimed
org. With a validated sub, org/sub/email are authoritative.
Proven by TestAudit_AnonRequestNotAttributedToForgedOrg (runs the real
SanitizeIdentity ahead of AuditTrail).
* fix(audit): close 4 scrub-bypass classes from Red review (MEDIUM)
Red's adversarial review found the path/UA credential scrub (5dfdf0e4) had
4 bypass classes that let a secret reach the immutable trail:
1. base64url with -/_ (looksLikeHighEntropyToken rejected any non-alnum)
2. all-alpha opaque >=len (required digits>0)
3. percent-encoded prefix (hk%2D… defeated the isAPIKey match)
4. UA glued by :/()[] (scrubFreeText split on too few delimiters)
Fixes:
- looksLikeHighEntropyToken now accepts the FULL base64url alphabet
[A-Za-z0-9_-] (RFC 4648 §5), drops the digit requirement, exempts dotted
values + canonical UUIDs, threshold lowered to 24 (128-bit base64 / 24-hex).
- scrubToken percent-decodes before every credential test (url.PathUnescape),
so %2D/%5F can't hide structure.
- scrubFreeText tokenizes on a broad delimiter superset (= ; , : / \ ( ) [ ]
{ } " ' < > | & ?) and rebuilds in one pass preserving delimiters —
replacing the fragile strings.ReplaceAll.
Proven by TestScrubToken_RedReviewBypassClasses (all 4 classes + UA glue) and
the expanded TestScrubToken_NoFalsePositives (uuids, model names like
claude-opus-4-20250514/text-embedding-3-large, slugs, normal UAs unchanged).
* feat(audit): operationalize AU-9 tail-truncation anchor (Red LOW #2)
Red: the Head() pin is inert unless operationalized — a hash chain can't detect
that the last K records were deleted (the surviving prefix self-verifies); only
an independent, durable head-digest series catches the count regression.
Adds a checkpoint emitter to the Recorder:
- StartCheckpoints(interval, logFn): a periodic goroutine emits the head digest
{count, head, ts} to the append-only observability log (o11y) every interval
(CLOUD_AUDIT_CHECKPOINT_INTERVAL, default 5m), plus a FINAL checkpoint on
Close so the shutdown head is anchored.
- CheckpointSink: when the ClickHouse mirror implements it, the digest is ALSO
persisted to an INDEPENDENT audit_log_checkpoints table (MergeTree) — so
truncating the local SQLite chain cannot rewrite the anchor history.
- Detection (compare consecutive checkpoints, alert on count regression) lives
in o11y where alert rules belong; the binary emits the tamper-evident anchor
to an independent sink. /v1/admin/audit/verify already returns (count,head) as
the pollable anchor too.
Proven: TestCheckpoint_EmitsHeadDigest (log + independent sink get count=7 head
on Close), TestCheckpoint_CountMonotonicDetectsTruncation (delete tail → prefix
still self-verifies, but count regresses 10→6 = the o11y alert signal).
Race-clean.
* harden(audit): close dotted-exemption + standard-b64 + nested-encoding scrub gaps
Self re-review (pre-empting Red's scoped re-review) found the round-1 scrub fix
still had gaps: the dotted-exemption let a raw secret bypass by appending '.x',
standard-base64 tokens (with +/) slipped, and nested percent-encoding (%252D)
survived a single decode.
Rewrote looksLikeHighEntropyToken to SCAN for the longest UNBROKEN base64-ish
run (>=24) anywhere in the value, over BOTH url-safe (-/_) and standard (+//)
alphabets — so '<rawsecret>.x' still trips (pre-dot run >= 24) and a standard-
base64 secret is caught. UUIDs stay exempt; real slugs/model-names/filenames
(report.pdf, text-embedding-3-large) have no 24-char run so they pass.
percentDecode now iterates (bounded x3) to normalize nested encodings.
All bypass classes closed (Red's 4 + dotted/standard/double-encoded), zero
false positives — TestScrubToken_RedReviewBypassClasses + NoFalsePositives
extended. The 20-char short-secret is a deliberate non-match (lowering below 24
would over-scrub legit hex-ish ids).
* fix(audit): address Red re-review — model-id over-scrub, UA glue, checkpoint (2 MED + 1 LOW)
Red re-review of the scrub/checkpoint code found 2 MEDIUM + 1 LOW:
MEDIUM 1 — model-id over-scrub (AU-3 regression): the round-3 run-scanner
counted '-' as a token char, so hyphenated model ids (claude-3-5-sonnet-
20241022, 26-char run) were redacted on audited routes (PATCH/DELETE
/v1/ml/models/:name, /v1/admin/catalog/models/*) — an auditor lost WHICH model
changed. Fix: isHighEntropyRunChar EXCLUDES '-' (kept +/_ for base64). Model
ids break into short runs (max ~9, far under 24); real secrets stay unbroken
>=24 runs — even a url-safe token using '-' as a separator has a >=24 run on one
side (AbCdEf-GhIjKl_MnOpQrStUvWxYz012345 -> 27). 6 model ids added to
TestScrubToken_NoFalsePositives.
MEDIUM 2 — UA free-text bypass: isFreeTextDelimiter omitted . @ # ~, so a
prefixed key glued by them (client@sk-live-KEY) stayed one token whose prefix
was no longer sk-/hk-. Fix: add . @ # ~ to the delimiter set; also flag a lone
eyJ-prefixed JWT header segment regardless of length (a JWT header is never a
legit id). Real UA dots are version separators (<24, safe). Fixed the stale
isFreeTextDelimiter docstring that referenced a nonexistent exemption. Proven by
4 glue-char probes in TestScrubToken_RedReviewBypassClasses.
LOW — checkpoint robustness: (a) StartCheckpoints now guards double-start with a
flag (the field/WaitGroup write was -race-flagged on a 2nd call) and
the docstring is corrected; (b) the on-Close final checkpoint to the independent
sink is now SYNCHRONOUS with a bounded 5s ctx (was fire-and-forget — the AU-9
independent anchor could be stale exactly at shutdown when an attacker truncates).
Proven by TestCheckpoint_DoubleStartIsSafe (-race) + TestCheckpoint_CloseSyncsToSink.
34 tests green, race-clean, -tags cloud.
* harden(audit): structured-id exemption beats hyphen-exclusion (11%->0.09% token bypass)
The prior fix (exclude '-' from the entropy run to protect model ids) opened an
~11% bypass for 32-byte url-safe-base64 secrets whose '-' happened to break
every 24-run (measured over 10k random tokens). Excluding '-' was too blunt.
Better construction: INCLUDE '-' in the run alphabet again (so a base64url token
embedding '-' is caught by its run), but exempt STRUCTURED IDs up-front via
isStructuredID — a value with >=3 hyphen groups where every part is <=12 chars
(dictionary words / short numbers: claude-3-5-sonnet-20241022). A raw secret
does not decompose that way. Measured: 0 model over-scrub, 0.09% residual on
32-byte base64 tokens that randomly resemble an id AND are prefixless AND sit in
a URL path (real keys carry hk-/sk- prefixes caught by isAPIKey; JWTs by
looksLikeJWT). An interior-hyphen raw secret with LONG parts still redacts.
Proven: TestScrubToken_NoFalsePositives (12 model ids/slugs pass) +
TestScrubToken_RedReviewBypassClasses (interior-hyphen long-part secret redacts).
34 tests green, -tags cloud.
* fix(audit): lexical structured-id test closes Red MEDIUM + guard started race (LOW)
Red final re-review: isStructuredID was SHAPE-only (>=3 hyphen groups, parts
<=12) — attacker-satisfiable. A secret chunked to that shape
(AbCdEfGhIjKl-MnOpQrStUvWx-YzAbCdEfGhIj, or deadbeef-cafebabe-01234567-89abcdef)
was exempted; ~2.15% of random 128-bit tokens leaked by chance. Entropy-count
alone can't separate them (deepseek-r1-distill-qwen-32b has 24 non-hyphen chars,
same as a 128-bit secret).
Fix: isStructuredID now requires every group to be WORD-LIKE (isWordLikeGroup) —
lexical content, not shape. A group is rejected if it is dense MIXED-CASE (base64
chunk) or a long ALL-HEX-WITH-LETTERS run >=8 (hex chunk like deadbeef); an
all-digit version date (20241022) stays word-like. Measured: 0 model over-scrub,
0 crafted-attacker bypass, natural random-token leak 0.0004% (128-bit) / 0%
(192-bit+) — down from 2.15%. Real keys (hk-/sk- prefix) and JWTs are caught
regardless.
LOW: StartCheckpoints' started check-and-set now under r.mu (was -race-dirty on
a concurrent 2nd call; prod-unreachable but now clean).
Proven: TestScrubToken_RedReviewBypassClasses (4 chunked-secret classes redact) +
TestScrubToken_NoFalsePositives (17 model ids/slugs pass). 34 tests green, -race,
-tags cloud.
* harden(audit): two-stage detection + document single-case-chunk residual bound
Restructured looksLikeHighEntropyToken into two stages after tracing the
fundamental limit Red is probing:
- Stage 1 (UNCONDITIONAL): a >=24 UNBROKEN run over [A-Za-z0-9_+/] (hasHighEntropyRun,
'-' and '.' are separators). Catches every raw secret WITHOUT internal separators
(hex, base64) at 100% — the realistic 'client bug put a raw key in the URL' case.
Never over-scrubs a hyphenated id (runs are short).
- Stage 2 (separated values): exempt a lexical structured-id (isStructuredID: >=3
word-like hyphen groups); otherwise redact. Catches mixed-case/hex chunks.
ACCEPTED RESIDUAL BOUND (documented in code): a secret deliberately chunked into
>=3 single-case-ALPHABETIC groups <=12 chars is lexically indistinguishable from
a hyphenated model id (deepseek-r1-distill-qwen-32b carries the SAME 24-char
entropy budget) — NOT closable by any length/case/count rule without a word
dictionary (over-engineering for a defense-in-depth URL/UA backstop). It does not
widen exposure for any REAL credential: Hanzo keys are prefixed (isAPIKey, any
length), JWTs are eyJ-prefixed (looksLikeJWT), bodies are never read. The residual
is an adversary chunk-encoding their OWN secret into a URL to seed an admin-only
audit row — contrived, low-value. The realistic accidental leak (unbroken raw key)
is caught by stage 1.
Removed dead isHighEntropyRunChar. 34 tests green, -race, -tags cloud.
* feat(paassvc): native in-process PaaS deploy control plane (/v1/paas/*)
Port the standalone Dokploy platform's observe + deploy halves into the unified
cloud binary as clients/paassvc — the 'one and only one way to deploy' made
native. Follows the clients/ml pattern exactly: a self-contained dynamic k8s
client, cloud.Register'd from init() (order 128), global-admin-gated, fail-closed
when no cluster.
Surface (global-admin only; user-facing view lives in console2):
GET /v1/paas/apps fleet drift board (declared/running/latest/drift+health)
GET /v1/paas/apps/:app one service row by CR name (main->test->dev)
POST /v1/paas/apps/:app/deploy deploy a tag by merge-patching Service CR .spec.image
GET /v1/paas/health real k8s reachability + Service CRD probe
- drift.go: 1:1 port of apps-drift.ts (computeDrift/isSemverTag + 6 DriftKinds,
identical severities). Pure, zero IO.
- paas.go: observeFleet lists hanzo.ai/v1 services across hanzo/-testnet/-devnet
(inventory.ts DEFAULT_TARGETS), joins the live Deployment for the running tag
(the operator Service CR status does NOT surface the running image — confirmed
against the live CRD), health/phase/endpoints from the reconciled CR status.
deploy merge-patches .spec.image (deploy-executor.ts parity) -> operator rolls it.
- Stateless: reads the cluster live (CRs are the source of truth); no apps-table
copy, no cron readers (dropped vs the Node platform).
Tested green:
- 30+ unit cases incl. the 9 drift-contract cases ported verbatim; go test green,
gofmt clean, go vet clean, full cloud binary builds (-tags cloud).
- Live-cluster probe (paasintegration tag, PAAS_IT-gated): observeFleet returned
82 real rows matching kubectl; an idempotent same-image patch on pricing
round-tripped through the operator with generation unchanged (6->6) = write
path proven WITHOUT triggering a rollout, zero disturbance to live state.
Design + full port map: universe/docs/architecture/paas-in-cloud.md.
RBAC (cloud-paassvc -> cloud-api SA): universe infra/k8s/cloud/paassvc-rbac.yaml.
Additive: platform.hanzo.ai stays as the internal-admin console; this is the
native backend + (next) the console2 user UI. No forced retirement.
* polish(audit): close hex-chunk residual (hex rule 8->4) + correct residual doc
Red final review (SHIP verdict) flagged 2 non-blocking polish items:
1. Lower isWordLikeGroup hex rule from len>=8 to len>=4 — Red verified across 19
real model ids that NONE has an all-hex-with-letters group of len>=4, so this
closes the small-hex-chunk leak (md5/sha in 'xxxx-xxxx' display grouping:
abcd-ef01-2345-6789-…) at 100% with ZERO model-id over-scrub. hexChunkMinLen=4.
2. Correct the residual doc: the accepted bound is now ONLY single-case-ALPHABETIC
base32 chunks (lowercase/uppercase-only, no hex-letter runs >=4) — mixed-case
base64 AND all hex-chunk sizes are now caught. The remaining case is genuinely
unclosable without a word dictionary and exposes no real credential.
Proven: TestScrubToken_RedReviewBypassClasses now includes 4-char hex groups
(abcd-ef01-…) which redact; TestScrubToken_NoFalsePositives (19 model ids) still
pass. 36 tests green, -race, -tags cloud. Red re-review: none needed.
Port the standalone Dokploy platform's observe + deploy halves into the unified
cloud binary as clients/paassvc — the 'one and only one way to deploy' made
native. Follows the clients/ml pattern exactly: a self-contained dynamic k8s
client, cloud.Register'd from init() (order 128), global-admin-gated, fail-closed
when no cluster.
Surface (global-admin only; user-facing view lives in console2):
GET /v1/paas/apps fleet drift board (declared/running/latest/drift+health)
GET /v1/paas/apps/:app one service row by CR name (main->test->dev)
POST /v1/paas/apps/:app/deploy deploy a tag by merge-patching Service CR .spec.image
GET /v1/paas/health real k8s reachability + Service CRD probe
- drift.go: 1:1 port of apps-drift.ts (computeDrift/isSemverTag + 6 DriftKinds,
identical severities). Pure, zero IO.
- paas.go: observeFleet lists hanzo.ai/v1 services across hanzo/-testnet/-devnet
(inventory.ts DEFAULT_TARGETS), joins the live Deployment for the running tag
(the operator Service CR status does NOT surface the running image — confirmed
against the live CRD), health/phase/endpoints from the reconciled CR status.
deploy merge-patches .spec.image (deploy-executor.ts parity) -> operator rolls it.
- Stateless: reads the cluster live (CRs are the source of truth); no apps-table
copy, no cron readers (dropped vs the Node platform).
Tested green:
- 30+ unit cases incl. the 9 drift-contract cases ported verbatim; go test green,
gofmt clean, go vet clean, full cloud binary builds (-tags cloud).
- Live-cluster probe (paasintegration tag, PAAS_IT-gated): observeFleet returned
82 real rows matching kubectl; an idempotent same-image patch on pricing
round-tripped through the operator with generation unchanged (6->6) = write
path proven WITHOUT triggering a rollout, zero disturbance to live state.
Design + full port map: universe/docs/architecture/paas-in-cloud.md.
RBAC (cloud-paassvc -> cloud-api SA): universe infra/k8s/cloud/paassvc-rbac.yaml.
Additive: platform.hanzo.ai stays as the internal-admin console; this is the
native backend + (next) the console2 user UI. No forced retirement.
Red HIGH-1 (cross-tenant CRUD, PROVEN): the isolation key used a lossy
sanitizeOrg (lowercase/punct->'-'/32-char truncate) so distinct IAM owners
(acme/ACME/acme!/32-char-prefix) collapsed into one storage bucket. Fixed:
tenant() now keys on the EXACT validated org from SanitizeIdentity — never
normalized. Removed the magic 'admin' bucket (empty org -> 403, even for
admins). sanitizeOrg deleted from the key path (kept only as a cosmetic,
clearly-labeled namespace normalizer in functions).
Red MED-1 (46MB response amplification): prompt content now capped at 64KiB;
version history is bounded, metadata-only (no per-version content echo);
metrics uses a true COUNT.
Red INFO: >900s timeout now clamps to 900 (was reset to 30) in create+invoke.
Red's three adversarial tests INVERTED into regression guards that assert
isolation HOLDS: TestRed_OrgKeyExactIsolation (8 collision classes, all
isolated), TestRed_NoAdminBucketConfusion, TestRed_PromptContentCapped.
All suites green under -race; fused subsystems graph links.
Commit 7bd5c77 added clients/console (a second, //go:build cloud-gated take on
the go:embed console) and imported it unconditionally in the subsystems bundle.
The default build (go build ./cmd/cloud, no -tags) excludes those files, so the
whole binary failed: 'build constraints exclude all Go files in clients/console'
— which also fails the CI image build, blocking every deploy.
The working, wired console embed is webui.go's mountConsole (called from Serve
after all /v1 routes). clients/console is redundant with it and would double-mount
'/'. Drop the broken import so main builds; consolidating onto ONE console path
is a clean follow-up.
Verified: go build ./cmd/cloud → 426MB binary; booted it and the ONE process
serves console '/' (200 HTML), SPA fallback /gpus (200 HTML), /v1/metrics/health
(200), /v1/nope (503 non-HTML — decline-list holds), /healthz (200).
Three native HIP-0106 subsystems, each org-scoped by the gateway-minted
X-Org-Id (SanitizeIdentity trust boundary), Base/SQLite in DataDir, secrets
by KMS reference only. Follows the projectsvc template; wired via one
additive block in subsystems/subsystems.go (orders 126/127/128).
- prompts /v1/prompts/* versioned prompt library (create=new version)
- agents /v1/agents/* agent defs + real run via deps.AI, recorded runs
- functions /v1/functions/* serverless registry + invocations/metrics;
invoke delegates to the code-exec sandbox and
fails closed (503) when unconfigured — never
runs tenant code in-process, never fabricates.
Tenant isolation proven at the store AND HTTP layers (Fiber app.Test):
no-org -> 403, cross-org list empty, cross-org get/delete/run -> 404.
All suites green (CGO=1).
Hanzo V8: Open Edition. cloud/clients/console go:embeds dist/ (the console2 static
export) and mounts the SPA at "/" with SPA-fallback, order 990 — the last-resort
catch-all AFTER every /v1/* route (isAPIPath refuses to HTML-fallback /v1,/zap,/_,
/healthz so JSON clients get honest 404s). Registered in subsystems.go.
This is the seam that makes ONE Go binary the whole cloud — edge + gateway + every
subsystem + the frontend. Placeholder dist/index.html is overwritten by the
console2 static-export bundle at image-build time. Build verified: go build -tags
cloud ./clients/console/ clean.
One artifact, one origin: the same hanzoai/cloud binary now serves the
console (@hanzo/gui, from hanzoai/console2) at the web root AND the /v1 API
from one process — no separate console Service, no second origin. Flagship
OSS-cloud consolidation (HIP-0106).
Serve (webui.go)
- The console is compiled in via `//go:embed all:webui/dist` and mounted as
the app's TERMINAL catch-all in Serve — LAST, after every /v1 subsystem
route, the /zap plane, and the health contract. Fiber v3 matches in
registration order, so real API routes always win; only paths that match
nothing else reach the SPA.
- SPA fallback: `/` and any client-side route (`/orgs`, `/models`, …) serve
index.html (Cache-Control: no-cache) so deep links / reloads work.
Fingerprinted assets (assets/, _next/) are served immutable for a year,
with brotli/gzip precompressed-sibling negotiation when the build emits
.br/.gz. Served through a stdlib http.Handler (correct Content-Type,
conditional GET) adapted onto zip via zip.AdaptNetHTTP.
- API precedence + namespace safety: an UNMATCHED path under an API/ops
prefix (/v1/, /zap, /healthz, /readyz, /metrics) returns a real 404 — never
the SPA shell — so clients calling a mistyped /v1/… never get HTML 200.
- Same-origin: the embedded console calls /v1 on its own host; the session
cookie is first-party — no CORS, no second-origin token dance.
- Reuses the hanzoai/static plugin's SPAMode semantics; implemented in-binary
because static.Handler is disk/S3-only today (its New() takes a Root/S3
bucket, not an fs.FS) so it can't serve an embed.FS — teaching it fs.FS is
the clean follow-up to collapse onto the shared plugin.
Build pipeline (Dockerfile)
- New `console` stage builds the console2 static bundle → /out; the Go build
overlays it into webui/dist BEFORE `go build` so go:embed bakes it in.
- webui/dist/index.html is a committed fallback shell (a real same-origin /v1
bootstrap) so `go build` always compiles and the binary always serves a UI
even without the Node toolchain; the image build overwrites it with the real
console. Built assets are .gitignore'd — generated at build time, never
committed as source.
Tests (webui_test.go) — boot the app + assert, end-to-end via app.Fiber().Test:
GET / → shell; deep links → shell 200 (not 404); /v1/models → API (not SPA);
unmatched /v1/… → 404 (not HTML); assets served directly; HEAD; and path
traversal (../, %2e%2e) cannot escape the embed FS. 7/7 green.
Honest current state: console2 ships 15 Next server route handlers
(app/**/route.ts, KMS-token proxies) so it emits a Node server bundle, not a
static export — the image embeds the fallback shell until console2 exposes a
build:embed static target or those routes land here as native /v1 endpoints.
The Go embed/serve plumbing is complete and needs no change to light up the
full console the moment the static bundle exists.
Drive-by: brand_test.go asserted the pre-pin hanzo issuer (iam.hanzo.ai);
brand.go was pinned to hanzo.id in 991e7bef, so the test was stale — aligned
to the shipped behavior (brand.go unchanged). Root package: 34/34 green.
The o11y fork now registers its routes at their exact public path (/v1/o11y/*),
so the reverse proxy forwards unchanged — removed rewritePath (/v1/o11y→/api) and
the TestRewritePath test. One and one way: the route IS the path on both sides.
Cloud o11y tests pass.
* feat(cloud): /v1/exec (Code Interpreter → sandbox) + /v1/websearch (SearXNG+Firecrawl-compat over Hanzo search+crawl)
hanzo.chat's Run Code and Web Search agent tools speak fixed LibreChat
provider contracts. cloud-api is the single /v1 edge, so it owns those
surfaces and routes them to Hanzo's own infra — never an external SaaS.
- clients/exec (order 140): mounts /v1/exec, /v1/exec/*, /v1/upload,
/v1/download/*, /v1/files/* — the @librechat/agents CodeExecutor contract
(POST /exec {lang,code} X-API-Key -> {stdout,stderr,files}). Transparent
reverse proxy to a SANDBOXED executor (CODE_EXEC_UPSTREAM). NO os/exec here;
the executor is the isolation boundary. X-API-Key (CODE_EXEC_API_KEY, KMS)
enforced constant-time, fail-closed.
- clients/websearch (order 141): mounts /v1/websearch/search (SearXNG JSON,
proxied to a Hanzo-operated metasearch WEBSEARCH_UPSTREAM) and
/v1/websearch/v1/scrape (Firecrawl shape, backed by Hanzo Crawl/Crawl4AI —
{url}->{success,data:{markdown,metadata}}). WEBSEARCH_API_KEY (KMS).
- Both register before ai (150) so their specific paths win over ai's /v1/*
catch-all. Mirrors the clients/o11y reverse-proxy pattern.
Tests: proxy verbatim-forward, path rewrite, auth fail-closed/reject,
crawl->firecrawl shape adaptation. All green.
* test(cloud): mount-through-Fiber integration tests for exec + websearch
Prove Mount() registers the overlapping static+wildcard routes (/v1/exec &
/v1/exec/*, /v1/websearch/*) on a real zip/Fiber router without panicking,
and that requests route end-to-end through the router to the guarded
handlers (proxy forward, firecrawl-shaped scrape, auth reject). Closes the
gap where direct-handler tests bypassed route registration.
o11ysvc→o11y, evalsvc→eval, mlsvc→ml, plansvc→plan, pluginsvc→plugin,
pricingsvc→pricing, productsvc→product, provisioningsvc→provisioning. The suffix
was stutter (svc = service). Package name == dir == the bare noun now.
o11y is `package o11y` importing `github.com/hanzoai/o11y` with a PLAIN import (no
alias): the import name is file-scoped and you never qualify your own package, so
`o11y.SetHandler` resolves to the upstream — the local `upstream()` URL func is
untouched. subsystems.go import paths + gojahost comment updated. Renamed packages
+ subsystems build clean.
Aggregator facade mounting the /v1/admin/* surface the Hanzo Admin Console
(admin.hanzo.ai, apps/operator) calls, matching its api.ts contract
field-for-field. Fans out over HTTP to the real upstreams — IAM (orgs, users,
roles, applications, audit, me), commerce (spend, credits), o11y (health) —
exactly like the o11ysvc/productsvc read facades; holds no store of its own.
Every route is GLOBAL-ADMIN ONLY, fail-closed: the guard reuses c.IsAdmin(),
which after SanitizeIdentity is true only for a JWT-validated principal whose
org is the admin org (IAM's IsGlobalAdmin), matching the gateway's admin-guard.
Anonymous and tenant-admin callers are denied 403 on every route (regression
locked in TestGate_DeniesEveryRoute). The IAM fan-out replays the caller's own
cookie/bearer — no adminsvc service credential — so it never reads more than the
caller could, and IAM re-checks IsGlobalAdmin. Commerce uses the existing
KMS-synced COMMERCE_SERVICE_TOKEN; no secret is hard-coded or logged.
Panels with no in-binary feed yet return the honest empty state, never a
fabricated number: the usage timeseries + per-product breakdown (insights/
datastore) and the product/workload registry + infra tiles (platform apps
table). The operator renders these as empty/em-dash by design.
Endpoints: overview, orgs, users, roles, applications, audit, usage, products,
me, sync. Registered order 146; blank-imported in subsystems.go.
Tests: gate denial across all routes x anonymous/tenant-admin/tenant-user, gate
allow for global admin, real aggregation (orgs/users/overview/usage) against
mock IAM+commerce, credential-replay assertion, IAM-error-surfaced (not
fabricated), honest-empty series/products. go build ./... + go test green.
`dbName=hanzo_cloud` alongside driverName=sqlite crash-looped cloud-api (fail-closed
on a benign leftover). Decomplect: the guard's one job is "reject Postgres" =
driverName=postgres OR a postgres:// DSN. A database name selects nothing, so drop
`dbName` from forbiddenEnvs entirely (the Go binary never reads it). Also correct the
lineage label: the legacy cloud-api is casibase (Go) — it lives on as hanzoai/ai,
which mounts INTO this hanzoai/cloud orchestrator — NOT "Python/TS". Tests updated:
dbName is never a violation; driverName=postgres + postgres DSN still are. Forwards
perfection, no backwards-compat leftover.
Rename cloud-api's own public product routes from /api/<route> to
top-level /v1/<route>, per the openapi v1.0.0 lock-in (no /api/ prefix;
the subdomain is api.* so /api/ double-prefixes):
/api/search-docs/indexes -> /v1/search-docs/indexes
/api/search-docs/stats -> /v1/search-docs/stats
/api/vector/collections -> /v1/vector/collections
/api/vector/stats -> /v1/vector/stats
These are the only /api/ paths cloud-api REGISTERS (serves). The remaining
/api/ literals are upstream calls cloud-api MAKES to other services that
genuinely serve /api/ — left untouched:
- evalsvc: /api/public/* (Langfuse console API proxy targets)
- o11ysvc: /v1/o11y/* -> /api/* runtime rewrite (destination)
- pricingsvc: openrouter.ai/api/v1/models (external)
Hard cutover (no dual-serving, per no-backwards-compat). Coordinated with:
universe cloud-api-v1 AUTH_PUBLIC_PATHS, python-sdk + hanzo-docs RAG
clients, and the openapi cloud spec — all moving to /v1 together.
FIX1 (red, HIGH — the deploy landmine): brand.go defaulted the `hanzo` brand
IAMIssuer to https://iam.hanzo.ai, but the live .well-known/openid-configuration
on BOTH hanzo.id and iam.hanzo.ai reports issuer=https://hanzo.id +
jwks_uri=https://hanzo.id/v1/iam/.well-known/jwks (iam.hanzo.ai is a routing
alias, not the token issuer). With the baked default, SanitizeIdentity's issuer
check would fail on every real token -> every principal anonymized -> ALL global
admin gets 403 (fail-secure, no forgery opened, but admin broken platform-wide).
The cloud CLI already defaults to hanzo.id; lux/zoo/pars already point at their
own .id issuers. Pin hanzo -> https://hanzo.id so the correct config is
default-by-default. (JWKS derivation then yields the correct hanzo.id JWKS.)
INFO (red): go-jose ValidateWithLeeway only enforces exp when present
(`if c.Expiry != nil`), so a token with NO exp would never expire. Reject a
missing exp explicitly, exactly like a missing iss. +1 test (22 green).
No subsystem reads cfg.IAMIssuer except SanitizeIdentity + a log line, so the
brand default change is contained.
Non-LLM resources were free: anyone could provision sql/vector/kv/s3/datastore/
docdb/search (provisioningsvc) and ml models/train jobs/experiments (mlsvc, GPU)
for $0 — only LLM calls were metered. This adds the same per-org commerce gate
the LLM edge gate uses, in-handler, so every create is paid for.
ONE shared primitive (no copy-paste per kind), reusing Deps.Metering (the single
commerce client) — the in-handler analogue of BillingGate:
cloud.ResourceMeter (resource_billing.go)
Gate(ctx, org, kind, costCents) pre-create balance gate, fail-CLOSED
Meter(org, kind, amountCents, …) post-success debit, per-org, async
DenyResource(c, err) 402 insufficient_balance / 503 unavailable
ResourceFeeCents(prefix, kind) configurable flat fee, $1.00 default
Wired into BOTH create paths (provisioningsvc + mlsvc) via the shared type:
gate runs after request validation and BEFORE any backend/k8s object is created
(no free provisioning, even on a commerce outage); meter runs only after the
resource is persisted/created.
Multitenancy (the whole point): org is the caller's resolved slug from tenant(c)
— the SAME value that namespaces the resource, now JWT-derived by the #66
identity sanitizer (not a spoofable header). It is sent to commerce as BOTH the
user identity AND X-IAM-Org-Id, OVERRIDING the client default org, so the balance
checked and the ledger debited are always the caller's own — never a default,
never another tenant. Proven by tests asserting commerce sees X-IAM-Org-Id:<caller>
(not the client default "hanzo") on both the balance check and the debit.
Env-aware (3-env split): the gate fires in EVERY env; test/dev are sandbox-but-
billed against their own per-env commerce/Square (structural, not a code branch).
Env is threaded config→deps as an attribution label and is NEVER a billing
bypass — proven by a test that testnet/devnet still refuse at zero balance.
Cost model: real, configurable per-kind flat fee (CLOUD_PROVISION_FEE_CENTS[_KIND],
CLOUD_COMPUTE_FEE_CENTS[_KIND]); 0 makes a kind free and un-gated; invalid/negative
is ignored so a typo can't silently free a paid resource. Ongoing storage GB-month
and GPU-hour reuse the SAME Meter primitive with a usage-derived amount from a
future runtime watcher — no live-size source here, so no size is fabricated.
Tests: resource_billing_test.go (gate allow/refuse/free/fail-closed/fail-open,
caller-org-not-default for both balance and debit, tenant isolation, env-never-
bypasses, unconfigured/nil no-op, fee resolution, deny shapes) + per-subsystem
integration tests proving the gate is wired into the real create path (402 before
backend on zero balance, 201 + caller-org debit when funded, free-kind un-gated).
go build ./... clean, go test ./... green, gofmt + vet clean.
zip.Ctx.Org()/IsAdmin() read X-Org-Id / X-User-IsAdmin verbatim, trusting the
gateway to be their sole minter. But cloud-api is reachable WITHOUT the gateway
in front (in-cluster cloud-api.hanzo.svc:8000, and historically the public
cloud-api.hanzo.ai), so a direct caller could forge `X-User-IsAdmin: true` and
pass every admin gate: the new /v1/admin/catalog writes, /v1/pricing/sync, and
the provisioningsvc/mlsvc literal "admin" tenant bucket.
Add SanitizeIdentity, an early middleware (before BillingGate + every subsystem)
that strips every client-supplied authority header and re-derives identity ONLY
from a validated IAM JWT. Validation is a tiny go-jose JWKS validator
(auth_identity.go) that mirrors gateway/v2/iamauth — deliberately NOT imported
to avoid a module cycle (gateway/v2 already imports hanzoai/cloud) and pulling
the gateway's KrakenD/gin/traefik tree for ~150 lines. Admin authority is
granted ONLY to a verified GLOBAL admin (owner == AdminOrg), so an org-admin
(IAM also sets isAdmin=true for org owners) can't escalate. Non-admins are
pinned to their own org; a verified global admin's org-switch is honored. One
middleware makes every existing c.IsAdmin()/c.Org() reader trustworthy with no
handler changes.
Phase-1 residual (documented in middleware_identity.go): with no validatable
bearer the client X-Org-Id is passed through for DATA scoping (the console
browser data path depends on it) — closing that is Phase-2; the ADMIN boundary
is closed on every path because X-User-IsAdmin is never restored from a header.
Fail-secure: a validator misconfig (issuer/JWKS) makes admin 403, never opens.
Tests (middleware_identity_test.go): 18 subtests — forged header grants nothing,
org-admin can't escalate or cross-tenant, global-admin org-switch honored,
expired/wrong-key/wrong-audience/api-key/missing-issuer all anonymous, cookie +
HTTP-Basic paths. go-jose promoted to a direct require (already in the graph).
Red review of feat/catalog-enablement found two in-branch leaks; fixed:
FIX#2 (HIGH): the root GET /v1/pricing returned the WHOLE bundle blob
(hanzoModels, thirdPartyModels, providers, freeModels, families) un-gated
via the `fixed` passthrough — an un-gated second source for everything the
leaf routes hide. New GateRootData() (catalog.go) gates the root in place:
hanzoModels+thirdPartyModels via VisibleCatalog (hanzoModels tagged "Hanzo"
so a disabled Hanzo provider cascades), providers via VisibleProviders, and
the id-reference lists freeModels + families[].models kept only if the
referenced model survived (admins keep all). Route moved out of `fixed` to
app.Get("/v1/pricing", gatedRoot). Audited the rest of `fixed`
(subscriptions/blockchain/iam/base/paas/tools/gpu/policy/cloud/compute):
all draw from the plans catalog with ZERO model/provider identity keys —
no gating needed; summary stays gated (providers sub-dict) with counts as
aggregate stats.
FIX#3 (fail-closed): empty DataDir was a Warn + :memory: fallback — a
security control that silently fails OPEN (admin-hidden models re-expose on
pod restart). Now a hard boot error (prod sets CLOUD_DATA_DIR;
provisioningsvc already requires it, so the unified binary always has one).
FIX#5 (DoS guard): overrides now bounded at 64 KiB + depth 32
(checkOverride) — bounds the recursive merge under a forged-admin write.
Tests: TestGateRootData (root gated identically to leaves: disabled/beta/
admin across hanzoModels/thirdPartyModels/freeModels/families/providers,
summary counts untouched), TestCheckOverride (object|null, size+depth caps),
TestMount_EmptyDataDir_FailsClosed, + e2e GET /v1/pricing gating and an
over-deep override PATCH->400. go build ./... + go test ./... green, gofmt.
NOT fixed here (infra, tracked separately): forgeable X-User-IsAdmin via
direct-to-pod cloud-api route — pre-existing, shared by every cloud IsAdmin
route; needs gateway routing + NetworkPolicy restriction in universe/operator.
Decision (b): the cloud repo is the ONE authoritative builder of ghcr.io/hanzoai/cloud
(it assembles every subsystem incl. o11ysvc). But cloud pinned ai v1.785.14 while the
prod AI-built image (1.785.26) embeds ai code through the blue-money P0 security wave.
Bump ai to v1.786.1 — which contains ALL of it (balance ledger + overdraft gate, JWT
iss/aud validation, secret redaction, single-pod ledger invariant, aud env-keys,
redact allowlist, global-admin {admin,built-in}) — so the cloud-repo image is an
UPGRADE, never a regression below 1.785.26. luxfi/zap v0.8.8 -> v0.8.11 (tidy).
Build verified: go build ./cmd/cloud clean (453MB binary). Unblocks shipping o11ysvc.
The last 5 release builds failed at `FROM golang:1.26-alpine` with
"toomanyrequests: unauthenticated pull rate limit" (429) from Docker Hub on the
shared runner, so no new cloud image has shipped — the deployed image predates the
o11ysvc mount (o11y /v1/o11y/* still 503) and the commerce mount. Switch the build
base to public.ecr.aws/docker/library/golang:1.26-alpine (immutable ECR Public
mirror, no rate limit) — the same fix already shipped in hanzoai/console2. Build
logic unchanged. Unblocks shipping o11ysvc → o11y live → retire old Langfuse console.
Add the backend admin layer that makes "admin enables -> customer sees"
real for the model/provider catalog, without forking the static
@hanzo/pricing bundle (still the sole source of truth for catalog
content/shape).
One overlay store, one gate:
- catalog.go: SQLite/Base overlay (table catalog_overlay, PK (kind,id);
default = enabled, so an empty store is a no-op). Pure gate
VisibleCatalog/VisibleProviders applies {enabled,betaOrgs,overrides}
onto the bundle output: visible iff own AND provider overlay admit the
org (enabled || org in betaOrgs); overrides merge via RFC 7386. Admins
see every entry, annotated under _overlay.
- admin.go: global-admin (c.IsAdmin) write surface — GET /v1/admin/catalog
(full catalog + state), PATCH /v1/admin/catalog/models/* (slashed ids via
greedy wildcard) and /providers/:name. Partial-update PATCH; override
validated as JSON object|null.
- pricingsvc.go: gate wired into the catalog read path (models, free,
featured, providers, summary, model/:name); non-catalog routes unchanged.
Overlay opened at {DataDir}/catalog.db (in-memory fallback), closed in
Shutdown.
Default behavior unchanged for live customers (all enabled). Tests:
pure-gate units (default-all-visible, disabled-hidden-except-beta,
override-merged deep, admin-sees-all, provider cascade), store round-trip,
and an end-to-end HTTP test (wildcard routing, IsAdmin 403, enable->see flow).
console2 admin UI is a separate agent's job; it consumes these endpoints.
The committed go.sum pinned hanzoai/base@v1.3.2, whose tag was force-re-tagged
upstream (live content hash drifted from the recorded hash). The release
Dockerfile verifies modules against the committed go.sum with GOSUMDB=off, so
`go mod download` hit "checksum mismatch / SECURITY ERROR" and every release
build failed.
v1.4.1 is the latest base tag that (a) has a stable, immutable hash and (b)
still registers as a cloud subsystem (v1.4.2+ dropped cloud.Register and would
break subsystem assembly — 13 vs 14 subsystems, /v1/base/health 404). Pin v1.4.1:
fresh-cache go mod download is clean, registry assembles 14 subsystems, all
health endpoints 200, full suite green.
New projectsvc subsystem (HIP-0106) — the ONE org-scoped store of
buildable/deployable sites, shared by hanzo.app (builder) and
console.hanzo.ai (Projects module). Both read/write the same records
through the gateway (X-Org-Id from the IAM JWT); no second copy of state.
- CRUD: POST/GET/PATCH/DELETE /v1/projects (+ /:slug)
- Deploy: POST /v1/projects/:slug/deploy
- artifact mode: tar(.gz) of built site -> OUR S3 (s3.hanzo.ai,
CLOUD_PROJECTS_BUCKET) under <org>/<slug>/, public-read, live URL
- git mode: queue + CI completion hook (/deployments/:id/complete)
- Deploy history: GET /v1/projects/:slug/deployments(/:id)
- SQLite store (modernc), versioned deployments, tenant isolation by org
- Reuses CLOUD_S3_ADMIN_* creds (one S3 path, like provisioningsvc)
- Path-traversal + size/file guards on artifacts; index.html required
- Published contract in CONTRACT.md for console2 to consume
Tests: store CRUD/isolation/ordering, deployment versioning, slugify,
provider detection, safeRel traversal guard, tar/tar.gz walker. All pass.
The build routed luxfi/* + hanzoai/* DIRECT via git insteadOf, which re-fetches a
re-pointed tag's tree (luxfi/age@v1.5.0) whose hash differs from go.sum's proxy
hash → 'verifying github.com/luxfi/age@v1.5.0: checksum mismatch / SECURITY
ERROR'. luxfi/hanzoai are PUBLIC: resolve them via the IMMUTABLE public proxy +
the committed go.sum (which already pins the proxy hashes). Only zap-proto/*
stays first-party-direct. Matches the drop-GOPRIVATE fix in hanzoai/iam +
luxfi/kms.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
cloud becomes a thin runtime host: alongside the compiled-in application
subsystems it mounts services from a runtime manifest (CLOUD_PLUGINS) with no
rebuild. Two plugin kinds, both reduced to app.Mount(prefix, http.Handler):
wasm — a polyglot module (Rust/WASM, Python, TypeScript) loaded in-process
via github.com/hanzoai/goa (wazero/gpython/goja; pure Go, stays
CGO_ENABLED=0). Drop a .wasm + manifest entry → mounted.
proxy — a standalone server (e.g. the beego apps ai, vm) reached over a
pluggable transport. The "zap" transport registers via
pluginsvc.RegisterTransport; proxying defaults to HTTP until then.
Adding/updating a service = edit the manifest + drop a .wasm or redeploy the
standalone — the cloud binary is unchanged unless its own core changes.
Registered at order 900. Static binary preserved; package + full suite green.
v2.1.2 verifies JWTs via the published JWKS instead of parsing the configured
cert PEM. cloud-api's /v1/signin (the ai dep's code->session exchange) configured
an unparseable Certificate (IAM returns the cert NAME 'cert-built-in' to
global-admin callers, not a PEM) -> 'iamsdk: not valid PEM' -> every
hanzo-cloud/hanzo-console SPA login failed (console + admin). go.sum realigned
(first-party re-tag dep-rot). cmd/cloud builds clean.
Pulls hanzoai/ai @069b83ce into cloud:1.785.23: residual 200-leak fix for
/v1/chat/completions, /v1/embeddings, /v1/rerank, /v1/messages — an invalid
credential with a malformed/incomplete body now returns 401 (was 200/400),
authenticating before the body is parsed. ai go.mod unchanged, so only the ai
require + its go.sum zip hash move.
The base/pq 'realign' in the prior commit adopted anomalous bits from a local
direct-fetch; the build container (and the working cloud:1.785.20 build) resolve
the ORIGINAL stable hashes via the module proxy/cache. Net go.sum change is now
exactly the hanzoai/ai bump (c08be563). luxfi re-tag churn
(fix/threshold-*-checksum-convergence) does NOT touch these pinned hashes.
Pull in the ai auth-status fix: invalid/unknown hk- key -> 401, insufficient
balance -> 402, bad model -> 400 (was HTTP 200 with an error body on all of
them). Cloud-api is the single backend validating hk- keys for both
api.cloud.hanzo.ai and the gateway (api.hanzo.ai), which proxies the status.
Realign go.sum zip hashes for hanzoai/base v1.3.2 and luxfi/pq v1.0.3 to the
current origin (upstream re-tags; /go.mod hashes unchanged) so the build
resolves in a fresh container. Verified: CGO_ENABLED=0 go build ./cmd/cloud
links clean.
CI fetches via proxy.golang.org,direct; both tags were force-re-pushed so
the committed zip hashes no longer matched what the proxy serves:
luxfi/pq v1.0.3 pFlQm1... -> ksw1dm... (proxy commit 90d2223)
hanzoai/base v1.3.2 BdTNDNe... -> 7GcHpg... (proxy commit 33d12949)
Minimal go mod tidy fix (2 lines); go.mod unchanged; cmd/cloud builds clean.
Unblocks the /v1/memory embed (hanzoai/ai fe516793).
The o11y runtime (SigNoz query server) serves its API under /api; the registered
handler owns the documented /v1/o11y/* -> /api/* rewrite. The proxy now strips the
public prefix and prepends /api so /v1/o11y/v3/query_range reaches /api/v3/query_range
(verbatim forwarding hit the SPA fallback instead of the API).
The o11y subsystem (hanzoai/o11y, order 70) mounts /v1/o11y/* but delegates to a
handler installed via o11y.SetHandler — never called in the unified cloud binary,
so the surface 503'd 'o11y runtime not initialized'. The heavy o11y runtime runs
as a dedicated Deployment; cloud now installs a reverse proxy to it (O11Y_UPSTREAM,
default o11y.hanzo.svc:80) so /v1/o11y/* serves real telemetry. Path preserved
verbatim; gateway-terminated identity forwarded.
luxfi/* and hanzoai/* tags were re-pointed upstream (base v1.3.2, pq v1.0.3,
zap v0.8.8, et al.); the committed go.sum went stale and a clean image build
failed 'go mod download' verification. Re-record the current direct-fetch
hashes (proven non-first-party set untouched). No go.mod change.
base@v1.3.2 was re-pointed after cloud's go.sum was recorded; the committed
zip hash (BdTNDNe3…) is the stale public-proxy first-seen content, while the
live tag (direct git, the path CI's GOPRIVATE takes) hashes to 7GcHpg…. CI
fetches base DIRECT and fast-fails the build at the checksum mismatch — the
last stale hash blocking a green main (pq was realigned in e4d68b36; this is
the same upstream-re-tag fix, mirroring 43a2ac57 for luxfi age/keys/zap).
go.mod /go.mod hash unchanged (graph-load verified it); only the module zip
hash needed realigning. Verified: clean-cache readonly linux/amd64 -mod=mod
direct build is green.
The committed zip hash went stale after luxfi/pq@v1.0.3 was re-tagged
upstream; cloud's clean image build failed go mod download verification.
Update to the current origin hash.
Pulls the per-user billing-subject fix into cloud-api: the gateway now keys the
balance gate + usage debit on object.BillingSubject(owner,name), so individuals
in the shared 'hanzo' org are billed independently (own balance, own $5) instead
of sharing+draining the single (hanzo,hanzo) balance.
New cloud subsystem (order 130) fronting the kubeflow forks via the k8s
dynamic client, scoped per-org by namespace (ml-<org>):
- /v1/ml/models CRUD + PATCH + /predict (kserve InferenceService;
predict proxies to the model's v2 data plane /infer)
- /v1/train/jobs CRUD (trainer TrainJob)
- /v1/train/experiments CRUD + /trials (katib Experiment/Trial)
- /v1/ml/health, /v1/train/health real probes: k8s reachability + CRD presence
(200 ok / 503 degraded with the real reason; never status-theater)
Tenant boundary is the per-org Kubernetes namespace; the org->namespace map is
injective (strict slug regex, no lossy fold) so two tenants can never share a
namespace. User-supplied labels can't override the tenant org marker. The k8s
client is built in-process from the in-cluster service account with a KUBECONFIG
fallback (self-contained like provisioningsvc's backends, not on shared
cloud.Deps); it fails closed (503 / degraded health) when unconfigured.
Promotes k8s.io/apimachinery + client-go to direct requires. Registered via
blank import in subsystems.go. Unit tests cover the security-critical pure
helpers (tenant injectivity, name validation, label-override guard, GVRs).
Picks up the streaming fix: first delta carries role:assistant and the
empty-choices usage chunk is gated behind stream_options.include_usage —
resolves hanzo.chat 'reading role' no-reply (separate from the dbx fix).
Picks up JSONList[T] (sql.Scanner + driver.Valuer over JSON) for
Message.VectorScores/Suggestions/ToolCalls/SearchResults, fixing the dbx
'unsupported type []model.SearchResult, a slice of struct' 500 that killed
console2 sign-in (welcome-message insert) and every hanzo.chat AI message
save. No new deps; transitive hunyuan -> v1.3.48 (already required by ai main).
luxfi re-tagged age v1.5.0, keys v1.1.0 and zap v0.8.8 in place. The public
module proxy serves each tag's first-seen (now stale) content, while the
Dockerfile fetches first-party DIRECT via GOPRIVATE — so `go mod download`
hit SECURITY ERROR (checksum mismatch) against the stale proxy zip hashes
committed in go.sum. Record the live DIRECT zip hashes for all three
(the /go.mod hashes are unchanged). Verified clean in the exact build env
(golang:1.26-alpine, GOPRIVATE=hanzoai/luxfi/zap-proto, GOPROXY=proxy,direct):
go mod download + CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build ./cmd/cloud ./cmd/hanzo.
The cloud merge pulled in tencentcloud-sdk-go monorepo modules whose tags are
nested paths (tencentcloud/hunyuan/v1.0.1074). GOPROXY=direct forced ALL modules
through direct VCS, which CANNOT resolve those nested tags ('unknown revision') —
go mod download failed in-build (worked locally only via cache). Route public
deps through the module proxy; first-party (hanzoai/luxfi/zap-proto) still go
direct+token via GOPRIVATE, so no private leak and re-pointed tags still resolve.
Union merge — keep BOTH main's and the feature branch's work:
- subsystems/subsystems.go: register main's provisioningsvc (order 120)
AND the feature's productsvc (order 145) alongside evalsvc/plansvc/pricingsvc.
- serve.go: collapse the two parallel :9090 listeners into ONE. Keep the
feature's robust healthSrv/healthMux lifecycle (ReadHeaderTimeout, graceful
Shutdown, fatal-on-bind) and FOLD main's HIP-0113 /metrics into healthMux,
so the single ops port serves /healthz /readyz /health /metrics. Keep the
feature's /zap zapface WebSocket plane. Drop the duplicate inline ops
goroutine (and its now-unused io import) to avoid a double-bind CrashLoop.
- config.go: union HIP-0111 IAM-issuer-from-brand AND the ZAP web-origin allowlist.
- go.mod/go.sum: union both dep sets; take the feature's newer hanzoai/ai
(2962d31c, /v1/embeddings + /v1/rerank) and main's newer luxfi
(database v1.19.3, threshold v1.9.9, age v1.5.0). RESTORE the feature's
'replace sashabaranov/go-openai => hanzoai/go-openai v1.40.0' that the merge
dropped — ai's reasoning path needs Delta.ReasoningContent (fork-only).
go.sum regenerated via go mod tidy honoring 02f56364 (first-party sumdb skip).
Build: go build ./... green (CGO=1; only luxfi/accel ld warnings). go vet clean.
evalsvc is a thin facade: proxies /v1/evals/{datasets,dataset-items,evaluators,
scores} → console's public REST API (Langfuse v3 fork, owns the eval/observability
data model) and orchestrates POST /v1/evals/runs against the in-process model
gateway (run model over dataset → score → trace). No eval logic reimplemented —
evals ARE LLM observability; cloud unifies the surface, console owns the logic.
Registered at order 145, before the AI /v1/* catch-all.
Extend cmd/hanzo with a gcloud/doctl-class control plane (client mode),
selected by the first token alongside the existing server-mode subsystem
dispatch. Thin client over IAM (hanzo.id), the platform REST control plane
(platform.hanzo.ai/v1), and the cloud /v1 API — no parallel API.
- login/logout/whoami/auth: IAM password grant, token in ~/.hanzo (0600)
- apps list|get|sync: platform apps board (declared/running/latest/drift)
- deploy: rolling zero-downtime redeploy via the container redeploy surface
- clusters list|get|create|select|install-baseline|target: dedicated DOKS
- build: platform-native (arcd) build enqueue
- k8s target; config get|set|list|path
- global --org/--output/--platform-url/--iam-issuer/--platform-token
- secrets only via env/~/.hanzo, never hardcoded; platform kubeconfig never fetched
- stdout kept machine-readable (server-graph init chatter redirected to stderr)
44 unit tests (client + command wiring via httptest). Verified live:
login -> whoami -> apps list (79 apps) -> deploy pricing (gen 4->5, zero-downtime).
Pulls hanzoai/ai feat/embeddings-rerank, which adds the OpenAI-compatible
POST /v1/embeddings and Cohere/Jina-compatible POST /v1/rerank endpoints on the
same auth + provider-routing path as /v1/chat/completions. Embeddings reuse the
already-configured OpenAI Direct key (kms://OPENAI_API_KEY); rerank needs no new
key (bi-encoder cosine over the resolved embedding model, native Jina/Cohere
proxy when keyed).
* feat(cloud): derive IAM issuer from brand, plumb into Deps (HIP-0111)
The unified cloud binary is one artifact serving every brand's API host
(api.hanzo.ai, api.lux.cloud, api.zoo.cloud, api.cloud.pars.network). Brand
is a per-deployment value, but the IAM issuer was hardcoded to iam.hanzo.ai
for every brand and Config.IAMIssuer was never plumbed into Deps — so a lux
or zoo deployment would validate JWTs against the wrong issuer (or not at all,
subsystems having no issuer to use).
White-label the token-validation issuer by brand:
- brand.go: PUBLIC brand→IAM registry (issuer + domain). hanzo→iam.hanzo.ai,
lux→lux.id, zoo→zoo.id, pars→pars.id, bootnode→id.bootno.de. One source of
truth; public values live in code, not KMS.
- config: when CLOUD_IAM_ISSUER/--iam-issuer is unset, derive it from the
brand via the registry (no longer silently iam.hanzo.ai for all brands).
- deps + build: add Deps.IAMIssuer, set from cfg in BuildDeps, so subsystems
validate against {issuer}/v1/iam/.well-known/jwks per HIP-0111.
Root package: go build + go vet + go test green (registry + issuer-derivation
tests). The fused -tags cloud binary build is blocked by a pre-existing
workspace go.sum mismatch (luxfi/age via go.work), orthogonal to this change.
* deps: converge luxfi/threshold -> v1.9.9, fix luxfi/age v1.5.0 re-tag hash
Both threshold and age were re-tagged upstream, leaving cloud/go.sum with
hashes that no longer match what the proxy serves. threshold@v1.9.4's
recorded sum broke `go work sync` and the fused -tags cloud build:
verifying github.com/luxfi/threshold@v1.9.4/go.mod: checksum mismatch
Converge on the single workspace-wide versions:
- threshold v1.9.4 -> v1.9.9 (latest 1.9.x; matches base + mpc)
go.mod require bumped; go.sum gains v1.9.9 zip+go.mod sums; drops the
unused v1.9.4 zip sum; keeps v1.9.4/go.mod (consensus@v1.25.0 still
requires it in MVS) with the correct post-retag hash.
- age v1.5.0: corrected the stale zip hash
(zC/Fw/ptZwAXr9nqrxmrcf8752EIl1Lq9RECp9OmCO0= ->
G69HbSV4R3vKEH9B0CulnRaMdSdf4RalMgP8xKmxHeI=) to match the re-tagged
module; go.mod sum was already correct. Version unchanged (v1.5.0,
consistent with base/kms/mpc).
All hashes authoritative (match proxy + mpc/base go.sum). No suppression
flags, no downgrade.
* feat(provisioning): control plane that creates logical resources in live shared backends
Adds clients/provisioningsvc, registered at order 120 and linked by one blank
import in subsystems/subsystems.go. It turns "create a database" into a real
logical resource inside the already-live shared product backends, scoped to the
gateway-minted org (X-Org-Id / c.Org()).
Surface (kind in databases|vector|datastore|kv|search|storage|docdb):
POST /v1/<kind> {"name":"<slug>"} -> 201 {id,kind,name,status,host,
port,username,database,connectionString,password?}
GET /v1/<kind> -> 200 [{id,name,kind,status,host,port,createdAt}]
GET /v1/<kind>/<name> -> 200 {id,name,kind,status,host,port,username,database}
DELETE /v1/<kind>/<name> -> 204
(GET /v1/provisioning/health is auto-registered by Serve.)
Backends (admin creds + in-cluster .svc defaults via env):
databases -> Postgres (pgx) CREATE ROLE + CREATE DATABASE
vector -> Qdrant (net/http) PUT /collections/{name}
datastore -> ClickHouse (clickhouse-go) CREATE DATABASE + USER + GRANT
kv -> Redis (go-redis) ACL SETUSER (keyspace-scoped)
search -> Meilisearch (net/http) POST /indexes
storage -> S3/MinIO (minio-go) MakeBucket
docdb -> MongoDB (mongo-driver) createCollection + createUser
Physical resources are namespaced org_<org>_<name> so tenants never collide;
the name is validated to a slug at the boundary and all SQL identifiers are
quoted — injection-safe.
Secrets: per-resource passwords (databases/kv/datastore/docdb) are sealed in
Hanzo KMS (github.com/hanzoai/kms/sdk/go, client-side encrypted); only a
secret_ref is persisted in SQLite. When KMS is unconfigured the service
degrades safely — the password is returned once in the create response and
nothing is written in plaintext. vector/search/storage have no per-resource
password (shared key auth out of band).
Metadata lives in ONE pure-Go SQLite DB ({DataDir}/provisioning.db, modernc),
UNIQUE(org,kind,name); multi-step writes run in a transaction.
Drivers were already indirect deps; importing them promotes them with no
version bumps. Tests cover the store (insert/get/list/delete, org isolation,
duplicate->conflict), name validation, org sanitization, identifier safety,
and token generation.
* fix(provisioning): close cross-tenant physical-name collision + native kind naming
BLOCKING SECURITY FIX. physicalName folded the org→name boundary by joining
hyphen-underscored org and name, so two distinct tenants could map to ONE
physical backend resource: physicalName("acme","my-db") ==
physicalName("acme-my","db") == "org_acme_my_db" (bucketName collided too).
On KV that is a cross-tenant credential takeover (idempotent ACL SETUSER
overwrites tenant A's user/keyspace); on SQL/datastore/S3 a cross-tenant DoS
and existence oracle. UNIQUE(org,kind,name) did not protect the physical layer.
- physicalName(org,name) = "o" + hex(sha256(org))[:16] + "_" + sanitizeIdent(name):
a FIXED-WIDTH org hash makes the boundary unambiguous, so cross-org folds are
cryptographically negligible. bucketName derives from the (now injective)
physical via the '_'→'-' bijection, so one guard covers every backend. Both
stay backend-valid (Postgres 63-char identifier limit, S3 3-63 char bucket).
- store: add global UNIQUE(physical_name) index + PhysicalExists pre-check; the
create handler now FAILS CLOSED with 409 BEFORE touching a backend on any
residual name-fold, never silently sharing a physical resource. Row maps
physical_name -> (org,kind,name) so names stay traceable.
- tests: injectivity (physicalName + bucketName), handler org-gate (empty
X-Org-Id -> 403 for non-admin), KMS safe-degrade (password returned once,
nothing persisted in plaintext, secret_ref empty).
NATIVE NAMING (Hanzo brand: product name, never upstream OSS name):
kind "databases"->"sql", "storage"->"s3"; final set = sql, vector, datastore,
kv, search, s3, docdb. env CLOUD_STORAGE_*->CLOUD_S3_*. Wire connection schemes
(postgres://, redis://, mongodb://) unchanged — protocol, not branding.
Minor: package-level Shutdown closes the store (mirrors plansvc); comment that
trusting X-User-IsAdmin is acceptable (blast radius = literal "admin" bucket).
---------
Co-authored-by: zeekay <z@zeekay.io>
Picks up the account-backed user-preferences endpoint so console2 (and any
product) can persist cross-product, cross-device customizations onto the IAM
user account.
The console Search/Indexes and Vector panels are hardcoded to call
api.cloud.hanzo.ai/api/search-docs/* and /api/vector/* with a bearer
service key. cloud-api owns those paths now: productsvc proxies them to
the in-cluster Meilisearch (search.hanzo.svc) and Qdrant (vector.hanzo.svc)
and translates each upstream response into the exact JSON the console's
tRPC routers decode (SearchIndex/SearchStats, VectorCollection/VectorStats).
Read-only, shape-translating glue — no search/vector logic reimplemented.
Bearer key enforced with a constant-time compare (gateway bypasses these
paths via AUTH_PUBLIC_PATHS since the key is opaque, not a JWT). Endpoints
degrade to an honest empty body when the upstream is unreachable, matching
the console panels' graceful-empty contract.
Verified locally against the live search/vector services: 6 real indexes
(43,162 docs), 2 real Qdrant collections; wrong/absent key -> 401.
* feat(cloud): derive IAM issuer from brand, plumb into Deps (HIP-0111)
The unified cloud binary is one artifact serving every brand's API host
(api.hanzo.ai, api.lux.cloud, api.zoo.cloud, api.cloud.pars.network). Brand
is a per-deployment value, but the IAM issuer was hardcoded to iam.hanzo.ai
for every brand and Config.IAMIssuer was never plumbed into Deps — so a lux
or zoo deployment would validate JWTs against the wrong issuer (or not at all,
subsystems having no issuer to use).
White-label the token-validation issuer by brand:
- brand.go: PUBLIC brand→IAM registry (issuer + domain). hanzo→iam.hanzo.ai,
lux→lux.id, zoo→zoo.id, pars→pars.id, bootnode→id.bootno.de. One source of
truth; public values live in code, not KMS.
- config: when CLOUD_IAM_ISSUER/--iam-issuer is unset, derive it from the
brand via the registry (no longer silently iam.hanzo.ai for all brands).
- deps + build: add Deps.IAMIssuer, set from cfg in BuildDeps, so subsystems
validate against {issuer}/v1/iam/.well-known/jwks per HIP-0111.
Root package: go build + go vet + go test green (registry + issuer-derivation
tests). The fused -tags cloud binary build is blocked by a pre-existing
workspace go.sum mismatch (luxfi/age via go.work), orthogonal to this change.
* deps: converge luxfi/threshold -> v1.9.9, fix luxfi/age v1.5.0 re-tag hash
Both threshold and age were re-tagged upstream, leaving cloud/go.sum with
hashes that no longer match what the proxy serves. threshold@v1.9.4's
recorded sum broke `go work sync` and the fused -tags cloud build:
verifying github.com/luxfi/threshold@v1.9.4/go.mod: checksum mismatch
Converge on the single workspace-wide versions:
- threshold v1.9.4 -> v1.9.9 (latest 1.9.x; matches base + mpc)
go.mod require bumped; go.sum gains v1.9.9 zip+go.mod sums; drops the
unused v1.9.4 zip sum; keeps v1.9.4/go.mod (consensus@v1.25.0 still
requires it in MVS) with the correct post-retag hash.
- age v1.5.0: corrected the stale zip hash
(zC/Fw/ptZwAXr9nqrxmrcf8752EIl1Lq9RECp9OmCO0= ->
G69HbSV4R3vKEH9B0CulnRaMdSdf4RalMgP8xKmxHeI=) to match the re-tagged
module; go.mod sum was already correct. Version unchanged (v1.5.0,
consistent with base/kms/mpc).
All hashes authoritative (match proxy + mpc/base go.sum). No suppression
flags, no downgrade.
---------
Co-authored-by: zeekay <z@zeekay.io>
The unified cloud binary is one artifact serving every brand's API host
(api.hanzo.ai, api.lux.cloud, api.zoo.cloud, api.cloud.pars.network). Brand
is a per-deployment value, but the IAM issuer was hardcoded to iam.hanzo.ai
for every brand and Config.IAMIssuer was never plumbed into Deps — so a lux
or zoo deployment would validate JWTs against the wrong issuer (or not at all,
subsystems having no issuer to use).
White-label the token-validation issuer by brand:
- brand.go: PUBLIC brand→IAM registry (issuer + domain). hanzo→iam.hanzo.ai,
lux→lux.id, zoo→zoo.id, pars→pars.id, bootnode→id.bootno.de. One source of
truth; public values live in code, not KMS.
- config: when CLOUD_IAM_ISSUER/--iam-issuer is unset, derive it from the
brand via the registry (no longer silently iam.hanzo.ai for all brands).
- deps + build: add Deps.IAMIssuer, set from cfg in BuildDeps, so subsystems
validate against {issuer}/v1/iam/.well-known/jwks per HIP-0111.
Root package: go build + go vet + go test green (registry + issuer-derivation
tests). The fused -tags cloud binary build is blocked by a pre-existing
workspace go.sum mismatch (luxfi/age via go.work), orthogonal to this change.
The net/http adaptor path 404'd: fasthttp's synthetic ResponseWriter can't be
hijacked, so coder/websocket.Accept failed and Fiber returned 404 for /zap
(confirmed live: 'GET /zap status 404'). Switch to zip/wsx (fasthttp/websocket)
which upgrades natively. Handler now returns a zip.Handler that mints the auth
slot BEFORE upgrade (401 fail-closed), captures the cookie/bearer in the
per-connection closure, and runs the binary-ZAP read loop with ws.ReadMessage/
WriteMessage. serve.go mounts app.Get("/zap", ...). Adds fasthttp/websocket
to go.sum (zip/wsx dep). End-to-end WS integration test rewritten against a
real zip app + native upgrade — green.
HealthListenAddr was declared but never bound — the operator's liveness
probe targets :9090/healthz and readiness :9090/readyz, so the pod failed
liveness and got SIGTERM'd in a ~90s CrashLoop (clean exit-0 'shutdown
requested'). Bind a stdlib health server on the health port serving
/healthz + /readyz (+ /health), separate from the :8000 API so the health
surface never shares failure modes with the API stack. Graceful-shutdown it
alongside the app.
Drive a real ZAP binary frame through the full server path (WS upgrade ->
mintCap -> rpc.ParseRequest -> dispatch -> Fiber /v1/* mount -> casibase
envelope -> rpc.BuildResponse -> WS reply), asserting: real provider data
round-trips, the session cookie is replayed to the /v1 handler, query/body
mapping works, unknown method surfaces !ok, and an unauthenticated upgrade
fails closed with HTTP 401.
Decomplect health/ops from the product API: the ops endpoints move off the
app listener (:8000, /v1/*) onto cfg.HealthListenAddr (:9090), unauthenticated
and unversioned. Liveness (/healthz) and readiness (/readyz) are now distinct.
stdlib-only, zero new deps. Makes cloud the reference impl for HIP-0113 and
unbreaks the cloud-api probe (was /v1/health → 404 on the unified binary).
luxfi/age v1.5.0 was re-pointed to a newer commit; sum.golang.org pins
the first-seen hash immutably, so a fresh build fetching our own module
hit `verifying github.com/luxfi/age@v1.5.0: checksum mismatch · SECURITY
ERROR`.
- go.sum: re-record age v1.5.0 zip h1: to live content (G69Hb… → zC/Fw…);
/go.mod hash was unchanged.
- Dockerfile: add explicit GONOSUMDB scope (first-party only) and drop the
fragile `rm -f go.sum && go mod download` self-heal — it masked the stale
go.sum and re-recorded unverified hashes on any transient error. Correct
committed go.sum + GOPROXY=direct is the one durable way.
Never global GONOSUMDB=* / GOINSECURE. Root cause is the upstream
force-re-tag practice, which must stop.
kms history was rewritten to remove white-label brand leaks
(Liquidity/redacted); the v0.159.1 tag now points at a rewritten commit
with a new tree hash. go.mod content unchanged (only h1: tree hash
changes). cloud builds green (go build ./... exit 0). go mod verify: all
modules verified.
Pulls hanzoai/ai#fix(ratelimit): tier lookup queries commerce by org slug
(?user=) instead of ?apiKey=, fixing the 400 that starved paid orgs of
their rate limits. No other dep changes (age/threshold lines reordered,
same immutable bits).
The prior commit refreshed these via local direct fetch (GOPRIVATE), recording
the GitHub-rewritten bits. The cloud Dockerfile fetches via proxy.golang.org
first (GONOPROXY=hanzoai only), so the build downloaded the proxy bits and
failed go.sum verification on threshold@v1.9.4/go.mod. Restored to the exact
proxy hashes from v1.785.13 (which built clean). Only the ai bump remains the
real go.mod/go.sum delta.
Pulls the ai fix where the controller hk- key lookup hit the legacy
/api/get-user (served as @hanzo/id SPA HTML, breaking API-key auth on
/v1/chat/completions). Refreshes go.sum for force-retagged luxfi/age@v1.5.0
and luxfi/threshold@v1.9.4 (upstream re-tag drift, GOPRIVATE — not caused by
this change). cloud (CGO_ENABLED=0) builds clean.
Folds in the zap-native + scraper per-org balance fixes so EVERY balance
check (gate, controller backstop, ZAP premium gate, zap balance query,
scraper preflight) reads the one per-org balance. Final image for the
per-org billing unification.
Completes the per-org billing unification: both the BalanceGateFilter AND the
resolveProviderForUser backstop now key by org slug + stamp X-Hanzo-Org, so the
single per-org credit is the balance every LLM call checks.
Makes the LLM layer real:
- zen3/zen4/aliases re-pointed from dead Fireworks serverless to DO-AI
- do-ai provider key unified to kms://DO_AI_API_KEY (env-first resolution)
- provider re-seed self-heals ClientSecret/ProviderUrl/State on boot
Several luxfi modules were force-rewritten upstream so go.sum captured the
rewritten direct-fetch bits, which conflict with the proxy's checksum-DB
artifacts: luxfi/age v1.5.0, luxfi/threshold v1.9.4, luxfi/zap v0.8.8.
Combined with the GOPROXY split (luxfi via proxy, hanzoai/zap-proto direct),
go.sum now pins the proxy/sumdb-authoritative hashes. luxfi stays in GONOSUMDB
so the few proxy-absent versions (e.g. luxfi/constants@v1.5.8 → 404) fall to
direct without a sumdb-lookup error while still pinned by go.sum.
Validated locally with the exact Dockerfile env: full 'go mod download' +
'CGO_ENABLED=0 go build ./cmd/cloud' succeed, 'go mod verify' = all modules
verified, binary embeds ai v1.785.9-...-44cd5f9a (per-org balance gate).
GOPRIVATE forces BOTH direct-fetch and sumdb-bypass for every match, so
luxfi/age went direct to GitHub and hit the force-rewritten v1.5.0 tag
(h1:KEjq... != go.sum/sum.golang.org h1:G69H...), failing go mod download.
All luxfi/* modules we use are on the public proxy, so drop luxfi/* from
GONOPROXY (keep only hanzoai/* + zap-proto/*, whose just-pushed pseudo-
versions the proxy 404s). luxfi/* now resolves via proxy.golang.org =
immutable checksum-DB bits matching go.sum. Validated locally with the exact
Dockerfile env: luxfi/age proxy-clean, hanzoai/ai direct-clean.
luxfi/age v1.5.0 was re-pushed on GitHub with content differing from the bits
sum.golang.org recorded (h1:G69H... original vs h1:KEjq... rewritten), so the
GOPRIVATE-forced direct fetch failed go.sum verification. Public luxfi/* are all
on the proxy; resolve through proxy.golang.org first (immutable, checksum-DB
artifacts) and fall back to direct only for repos the proxy 404s (private). Pins
public deps to verified bits; private resolution unchanged.
Unifies the LLM balance gate with commerce's per-org credit: the gate now
keys billing by org slug and stamps X-Hanzo-Org so a single per-org credit
(X-Org-Id=<org>) is the balance the gate checks and usage debits. Fixes
insufficient_balance on funded orgs (gate previously queried per-user in the
default 'hanzo' namespace).
Fixes console2/cloud login end-to-end: the IAM SDK now loads the app cert from
/v1/iam/* so Signin's ParseJwtToken succeeds (was 'iamsdk: not valid PEM'),
establishing a real session that admin endpoints accept.
Fixes get-account 'unsupported type []model.SearchResult, a slice of struct'
and the matching scan errors — Message.SearchResults/VectorScores/Suggestions/
ToolCalls and all slice/map model fields now round-trip via JSON in the data
layer. Completes SQLite-native cloud-api login.
luxfi/threshold@v1.9.4 is being re-tagged upstream, so its checksum drifts
from go.sum and 'go mod download' fails in CI. Record private-module hashes at
build time (-mod=mod; GOPRIVATE keeps them off the public sumdb) and drop the
stale threshold entry so it re-records cleanly.
Unblocks the Base/SQLite cloud-api: the ai subsystem now creates its tables
from the Go structs on a fresh embedded SQLite store (no external migrations).
Pins the merged ai main commit that adds StringList (sql.Scanner/Valuer over
JSON) for list columns — fixes the casibase data-layer panic
'unsupported Scan ... string into *[]string' that broke OAuth sign-in.
v1.785.8 sets beego CopyRequestBody in Bootstrap so the unified binary's AI
controllers can read POST bodies (json.Unmarshal of c.Ctx.Input.RequestBody);
without it /v1/chat/completions returned 'unexpected end of JSON input'. Completes
the unified-AI serve path: routing (bare /v1/*) + no-panic (session mgr) +
scratch-safe (memory sessions) + body parsing (CopyRequestBody).
v1.785.7 adds the scratch-safe memory session provider on top of the bare /v1/*
mount + session-manager build. Without it the unified binary 503'd on every
request (file session provider can't write in the read-only scratch root). With
all three fixes, /v1/chat/completions and the other OpenAI routes serve through
the unified binary.
In the unified binary the pricing subsystem (order 112) mounted a bare /v1/models
alias that shadowed the AI subsystem's (order 150) OpenAI-compatible /v1/models —
the {data:[{id,…}]} model list the api.hanzo.ai gateway forwards to cloud-api and
clients (cowork model picker) consume. Pricing's annotated catalog already lives
at /v1/pricing/models, so the bare alias only introduced a shape regression.
Remove it; pricing stays strictly under /v1/pricing/*. Now /v1/models, like the
other OpenAI routes, resolves to AI's beego handler via its /v1/* catch-all.
v1.785.6 carries BOTH unified-binary fixes:
1. AI mounts casibase routes at bare /v1/* (not /v1/ai/*) so the api.hanzo.ai
gateway, which forwards /v1/chat/completions etc. unchanged, resolves.
2. beego session manager built in Bootstrap so forwarded requests don't panic.
Together these make /v1/chat/completions, /v1/chat, /v1/models, /v1/messages
serve through the unified binary exactly as the gateway sends them.
ai v1.785.5 fixes the embedded /v1/ai/* HTTP 500: the unified binary never
calls beego.Run(), so beego.GlobalSessions was nil and every forwarded request
panicked in SessionStart. v1.785.5 builds the session manager in the shared
Bootstrap, so /v1/ai/chat/completions, /v1/ai/models and all nested routes
serve. Cloud binary builds green against it.
The ghcr.io/hanzoai/cloud package is linked to hanzoai/ai (cloud->ai rename),
so this repo GITHUB_TOKEN is denied write (permission_denied: write_package) via
the shared workflow. Build self-contained on the hanzo-build-linux-amd64 scale
set and log into GHCR with GH_PAT (admin:org+write:packages). gh_token still
feeds the Dockerfile private-module fetch. Dropped GHA cache (same denial +
artifact quota).
gateways v2 tags were invalid Go modules (go.mod lacked the /v2 path), so
gateway v2.9.7+incompatible could not resolve on a clean fetch and cloud failed
to build. gateway v2.14.8 fixes the module path; import the /v2 path in the
subsystems bundle and pin v2.14.8. Build + go mod verify clean; binary boots
with no init panic.
gateway v2.9.7 introduced a go.mod still declaring module path
github.com/hanzoai/gateway at a v2 tag, which makes v2.9.7+incompatible an
invalid version (a module with a go.mod at major>=2 must use a /vN path).
v2.9.6 is the last go.mod-free v2 tag, so +incompatible is valid there; API is
identical for the subsystem blank-import. One patch down, no code change.
Several private module tags were re-tagged after the committed go.sum was
generated, so a clean CI fetch failed go.sum verification (SECURITY ERROR:
checksum mismatch). Regenerated the affected private entries from current
remote content via go build -mod=mod (versions unchanged). Build is green and
boots without panic; go mod verify passes.
ARC ephemeral runners only match jobs targeting the scale-set name as a label.
The shared workflows default [self-hosted,linux,amd64] matches only classic
static runners (evo pool, offline) so the job sat queued with the listener
reporting assigned-job=0. gateways successful builds use runs-on:
hanzo-build-linux-amd64 (runner hanzo-build-linux-amd64-cvs28-runner-*); pass
that as runner-amd64.
GitHub-hosted runners for this org are billing-frozen (jobs fail in ~5s:
"recent account payments have failed"), so the bespoke ubuntu-latest release
workflow can never start. Use the canonical hanzoai/.github docker-build.yml
reusable workflow, which runs on the self-hosted arcd pools and injects GH_PAT
as the gh_token BuildKit secret the Dockerfile needs for private cross-org Go
modules. amd64-only (cluster arch) to complete without the arm64 pool.
The unified binary pulls private hanzoai/* AND luxfi/* modules; the public
proxy 404s on them and the default GITHUB_TOKEN cannot read cross-org repos, so
the Docker build failed at go mod download.
- Dockerfile: split deps layer; GOPRIVATE + BuildKit gh_token secret +
git insteadOf to fetch private modules over authenticated git (mirrors the
proven hanzoai/ai Dockerfile). COPY --chmod=0755 the binary so the scratch
image can never ship a non-executable /cloud (the 0644 CrashLoop class).
- release.yml: source the gh_token from the org GH_PAT (cross-org RO PAT that
actually exists), not the never-configured HANZO_GH_RO_TOKEN.
Fixes user-token AI on api.hanzo.ai:
- ai v1.785.4: AI runtime initializes in Mount() (#29) so /v1/ai/* serves real
completions (no more 503 "ai runtime not initialized"); ai self-meters
Commerce on the correct /v1/billing/* path (1.784.2 casibase used the dead
/api/v1/billing/* -> 404 for real user JWTs, blocking the prepaid balance
gate + usage auto-debit).
- beego v2.3.10: grace flag-registration guard so the unified binary (ai beego
v1 + iam beego v2) does not panic ("flag redefined: graceful") at init.
- ai v1.785.4 also swaps deprecated denisenkom/go-mssqldb -> maintained
microsoft/go-mssqldb (one mssql driver registration; no "sql.Register called
twice" panic).
- replace sashabaranov/go-openai => hanzoai/go-openai v1.40.0 (ReasoningContent
field) — mirrors ai own replace, which does not transit to this main module.
Verified: CGO_ENABLED=0 go build ./cmd/cloud produces a 316MB executable that
boots clean (base + full subsystem set) with no init panic; full boot stops
only on expected in-cluster config (IAM_KEYS_URL/IAM_AUDIENCE), supplied by the
cloud-api CR env.
cmd/cloud and cmd/hanzo each blank-imported the same 16-subsystem list, so
adding/removing a subsystem meant editing two files (repeat-yourself). Move
the list into one package, github.com/hanzoai/cloud/subsystems; both
entrypoints blank-import only that. One source of truth for what's linked into
a Hanzo binary — dispatcher and full-surface binary mount an identical set by
construction.
(Bundle is a sibling subpackage, not the root cloud package: subsystems import
cloud for Deps+Register, so a root bundle would cycle.)
Verified: go build -tags 'cloud cloud_mount' . ./subsystems ./cmd/cloud
./cmd/hanzo green (-mod=readonly); hanzo --help still lists 18 subcommands
(16 subsystems + cloud + datastore); go test ./... green.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
cloud pinned commerce v1.39.1, which predates commerce's cloud-mount
integration: cloud.Register("commerce", 100, ...) in init() behind
//go:build cloud, added at v1.42.5. So commerce silently was NOT in the fused
surface or hanzo's subcommands. v1.42.5 (latest tag) registers correctly —
`hanzo --help` now lists commerce; the unified binary composes 16 subsystems.
iam stays v1.19.4 (no cascade). go build -tags 'cloud cloud_mount' + full
go test ./... green.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Adds cmd/hanzo — one binary dispatched by subcommand: `hanzo <svc>` serves one
subsystem, `hanzo cloud` the full fused surface, `hanzo iam` the standalone
Beego IdP (iamserver.Run), `hanzo datastore` documents the ClickHouse boundary.
DRY: the cloud-server body (compose root + HIP-0106 /v1/<name>/health contract
+ graceful shutdown) is extracted from cmd/cloud's main() into cloud.Serve(enable)
— the ONE shared place. cmd/cloud now calls cloud.Serve(nil), gaining graceful
shutdown + health endpoints (strict superset of its prior body, no regression).
cmd/hanzo dispatches through the same cloud.Serve.
Beego non-collision holds: iam registers routes inside iamserver.Init(), not
package init(); one Beego v2 path; visor (Beego v1) intentionally unlinked.
iam v1.19.4 moves indirect->direct (cmd/hanzo imports iam/iamserver).
Verified: go build -tags 'cloud cloud_mount' . ./cmd/cloud ./cmd/hanzo green
(-mod=readonly), go vet clean, hanzo --help lists 17 subcommands.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
iam v1.19.0–v1.19.3 panic at boot: routers/router.go registered
GET /v1/iam/run-authz-command → ApiController.RunAuthzCommand, a method never
added, so Beego panics at route registration when iamserver.Init() runs (this
hit cloud-api too). v1.19.4 (cut from iam main: dangling route removed +
initAdminUser seeds via conf.AdminOrg) fixes it. Transitive zap-proto/go
v0.3.0→v1.1.0 required by pkg/iam v1.18.4. go build -tags 'cloud cloud_mount'
./cmd/cloud green; -mod=readonly verified.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
github.com/luxfi/threshold v1.9.4 was re-pushed at the same version
tag without updating go.sum in this repo. The recorded h1: digest no
longer matches the upstream zip on proxy.golang.org, so cold clones
fail at `go mod verify` / `go build`:
verifying github.com/luxfi/threshold@v1.9.4: checksum mismatch
downloaded: h1:H69e9QkvygDtWkni1FkD5ztLdiTasmpJXcuVzgebdxo=
go.sum: h1:/TsgIzo/e/DIx++J0+9eNuS7HkpSaXbVk+HvhlUOsmE=
SECURITY ERROR
Regenerating go.sum (single line update) restores parity with
upstream and the build succeeds.
Reproduction (pre-fix):
git clone https://github.com/hanzoai/cloud
cd cloud && GOPRIVATE='github.com/hanzoai/*,github.com/luxfi/*' \
go build ./cmd/cloud
# verifying github.com/luxfi/threshold@v1.9.4: checksum mismatch
No code changes outside go.sum.
The README advertises `docker run -p 8080:8080 ghcr.io/hanzoai/cloud:latest`
but there's no release pipeline in the tree to produce that image.
This workflow publishes the Docker image on git-tag push (matching
v*), on default-branch push (as :latest and :sha-<short>), and on
manual dispatch.
Includes a Buildx secret pattern for fetching private upstream
modules (hanzoai/iam, hanzoai/commerce, hanzoai/gateway, ...) using
either the default GITHUB_TOKEN or an org-level HANZO_GH_RO_TOKEN
when cross-org private reads are required.
Reference Helm chart that renders a single Deployment + Service +
optional PVC for the unified cloud binary. Mirrors the binary's
CLI flags via .Values.hanzo (brand, domain, dataDir, iamIssuer,
enable). Pod runs as nonroot UID 65532 matching the Dockerfile.
Not a substitute for luxfi/operator + Service CRD — this chart is
for users who want raw k8s manifests without installing the
operator first.
Reference Docker Compose manifest for the unified cloud binary, with
matching .env.example and a deploy/README.md that documents required
vs optional environment variables (the binary refuses to mount IAM
without HANZO_IAM_ISSUER, so make that explicit).
No code or config changes outside deploy/.
`cmd/cloud-smoke` (existing) exercises the in-process mount path on a
mock zip.App with two health endpoints. It does not boot the actual
`cmd/cloud` binary or hit the HTTP surface customers will use.
This script closes that gap: it builds `./cmd/cloud`, boots it under
the same default-safe `--enable` list used in deployments (omits the
`iam` subsystem until the v1.19.2 boot panic is patched), waits for
the listener to bind, then probes the five endpoints whose expected
status is fixed by the HIP-0106 contract:
/healthz 200 process health probe
/v1/models 200 model catalog (no auth)
/v1/plans 200 plansvc (goja-hosted)
/v1/pricing 200 pricingsvc (goja-hosted)
/v1/base/collections 401 base alive, auth-gated
If any probe regresses, the script dumps the tail of the boot log and
exits non-zero — making it usable as a CI gate and as a local "does my
clone actually serve?" check.
Env knobs (`PORT`, `LISTEN`, `BIN`, `DATA_DIR`, `ENABLE`,
`KEEP_RUNNING`, `BOOT_TIMEOUT`) let it drop into different
environments without a Makefile change. The `IAM_*` env vars default
to the production hanzo.id JWKS — required by `kms` for inbound JWT
validation even with `iam` disabled — and can be overridden per
deployment.
Pairs with the Makefile in #5: `make smoke` already exists for the
mount-time path; this is the runtime counterpart and can be wired as a
sibling target (`make smoke-runtime`) in a follow-up once #5 lands.
Minimal developer ergonomics for the unified cloud binary. Targets
wrap go build / go test / docker build for the existing Dockerfile,
plus a `make run` shortcut that matches the README quickstart
(--enable=iam,base,kms,gateway,o11y).
No code changes outside the new Makefile.
Repo currently has neither file. Two practical consequences:
1. `docker build .` copies the entire context including `.git/`, IDE
metadata, OS detritus (`.DS_Store`), and any local `.env` — bloats
the build context and risks baking secrets into image layers.
2. Without a `.gitignore`, the build output binary (`/cloud` per the
`Dockerfile` final stage), local `.env` files, and editor leftovers
are easy to commit by accident.
Both files cover the standard Go-project surface (binary at `/cloud`,
test outputs, coverage, env files), plus IDE/OS noise. The
`.dockerignore` additionally drops docs and tests so they don't enter
the runtime image — the binary is what ships, the README lives on
GitHub.
No behavior change; the binary the Dockerfile builds is bit-identical.
What changes is build-context size and the safety margin around
accidental commits.
IAM issues JWTs with iss=https://iam.hanzo.ai, but cloud-api defaulted the
expected issuer to https://iam.hanzo.id — so EVERY hanzo.id-login JWT was
rejected with 'invalid issuer claim (iss)' and the AI gateway's JWT auth path
was dead for all users (only hk-*/sk-* keys worked). One-char .id->.ai fix.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The cloud-smoke command was a throwaway harness that hand-mounted fake health
routes and avoided the real subsystem matrix (citing build issues in cmd/cloud
that are now fixed — the full binary builds). Replaced with a proper go test in
package main that exercises the actual path:
- TestRegistryAssemblesSubsystems: every subsystem main.go imports self-registers
via init() into cloud.Registry (proves the unified binary wires the matrix).
- TestMountAllAndServeHealth: BuildDeps -> MountAll -> serve; the self-contained
subsystems (base, authz, amqp, metrics, plans, pricing) mount in-process and
serve /v1/<name>/health = 200 via the real zip/fiber + jsonenc stack
(app.Fiber().Test, no listener / external services).
- TestDepGatedSubsystemsFailClosed: ai, o11y mount and return >=500 from the
disabled-dep stub — proving the BuildDeps three-mode contract end to end.
Discovered (not fixed here — separate subsystem bug): enabling iam panics with
"'RunAuthzCommand' method doesn't exist in the controller ApiController".
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
hanzoai/base@v1.3.2 and hanzoai/kms/sdk/go@v1.0.0 were re-tagged upstream, so
the recorded go.sum hashes no longer matched — `go build`/`go test` failed with
checksum mismatch (SECURITY ERROR) for anyone fetching fresh. Refreshed the
private-module hashes against the current tags.
Verified: go build ./... and go test ./... pass in default -mod=readonly mode.
No local replace directives (the 9 replaces are all published version pins for
the krakend/traefik gateway stack). cmd/cloud links to a single 272M binary.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The unified binary now serves the full native observability stack at
/v1/{metrics,logs,traces}/* — WAL-durable (survives restart, verified), zero
prometheus, zero Grafana. This is the working replacement for the Loki/Tempo/
SigNoz vendoring.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The prometheus-free replacement for the Grafana/Prometheus observability
backends. Registers at order 40, serves /v1/metrics/{health,batch,write,query};
ingests luxfi/metric.MetricBatch (the ZAP MsgMetricBatch wire shape). Verified
live: write+query and batch+query roundtrips return correct series. Binary stays
at zero prometheus. (Also refreshed the stale kms/sdk/go go.sum hash from the
wave's re-tag.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the dev pseudo-versions + the local `replace github.com/hanzoai/licensing
=> ../licensing` directive with the published release tags now that the three
subsystems are merged + tagged:
- github.com/hanzoai/plans v1.2.0 (was pseudo @147eced7)
- github.com/hanzoai/pricing v1.3.0 (was pseudo @0c4b4c12)
- github.com/hanzoai/licensing v0.1.0 (was v0.0.0 + replace => ../licensing)
licensing@v0.1.0 requires github.com/hanzoai/cloud@v0.0.0-00010101...; that
self-reference resolves to this main module, so no replace is needed. go.mod/
go.sum are tidy and `go build -mod=readonly ./cmd/cloud` produces the 303M
unified binary (boots with --enable=plans,pricing,licensing; all health 200).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Combine the licensing Mount (PR #3) with the plans+pricing goja mounts
(PR #4) onto one branch for the unified-binary re-pin.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mount the Node @hanzo/plans (data) and @hanzo/pricing (Express) services
INTO the unified cloud binary under /v1/plans/* and /v1/pricing/*, running
their JS inside the dop251/goja engine (the same engine base/plugins/gojavm
uses) — per HIP-0106. No service rewrite; Ship-of-Theseus to pure Go later.
New packages (the glue — clean module boundaries, no service source copied):
- clients/gojahost: a reusable goja VM-pool host. Compiles a service repo's
goja/bundle.js once, pre-warms a runtime pool (mirrors gojavm's pool +
compile-once + per-runtime ensureLoaded discipline), injects the catalog
JSON as globals, and dispatches handle({route,params,tenant}) -> {status,
body} with ctx-cancel interrupt. Eager-loads one runtime so a bad bundle
fails at Mount, not first request.
- clients/plansvc: Mount(app, deps) for /v1/plans/*. Loads the @hanzo/plans
bundle + embedded catalog, registers zip routes (subscriptions, cloud,
blockchain, dns, gpu, regions, storage, tools, policy, schema, vocab,
resolve/:id, entitlements/:id), threads X-Org-Id as the tenant for
per-reseller (tenant_id,id) catalog scoping. The entitlements.mjs
transforms (fromLegacy/toLicenseFeatures/resolvePlan) run in goja.
- clients/pricingsvc: Mount(app, deps) for /v1/pricing/* + /v1/models.
Express does NOT run in goja, so the Express transport is dropped; the
server.mjs read handlers run in goja via the bundle. The sync.mjs markup
(toMTok/processOpenRouterModel/...) also runs in goja via applyMarkup();
the admin-gated POST /v1/pricing/sync does the live OpenRouter fetch in Go
(net/http) and feeds raw JSON into the goja markup. _internal (provider
costs/routing) is stripped from public responses.
Wiring: cmd/cloud/main.go blank-imports both wrappers; each init() calls
cloud.Register (plans order 111, pricing 112, after iam/commerce/licensing).
go.mod references hanzoai/plans + hanzoai/pricing as their own private Go
modules (the JS + data live there, embedded; nothing copied into cloud).
Tests: clients/{gojahost,plansvc,pricingsvc}/*_test.go exercise the real
embedded bundles (vocab, resolve+license_features, 404s, _internal strip,
exact markup math). Verified end-to-end: binary boots with --enable=plans,
pricing; all routes serve real data through goja over HTTP; X-Org-Id tenant
scoping confirmed (reseller override-wins, isolation holds).
Also corrects a stale go.sum entry for github.com/hanzoai/kms/sdk/go@v1.0.0
(the module was retagged; the recorded hash no longer matched the origin,
blocking any build that pulls base/iam/commerce -> kms). Updated to the
current origin hash.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire the private hanzoai/licensing Go subsystem into the unified cloud
binary per HIP-0106, following the iam/commerce/ai Mount(app, deps)
pattern. Clean module boundary: licensing is imported as its own private
module (its mount.go self-registers via init(); cmd/cloud blank-imports
it). No subsystem source is copied into cloud — the signer/fingerprint
secret logic stays in the licensing module.
- cmd/cloud: blank-import github.com/hanzoai/licensing (order 110, after
iam=50 and commerce=100, since the /v1/licensing/issue flow depends on
both identity and entitlements).
- go.mod: require licensing + local replace (co-developed private module;
production resolves via tag/pseudo-version, drop the replace).
- types.CommerceClient: add CheckEntitlement(ctx, orgID, productID) plus
the LicenseEntitlement transport type. This is the entitlement flow that
gates issuance: commerce answers "does this tenant own the licensed
product?" and returns the plan's FLAT license-features per the
@hanzo/plans toLicenseFeatures vocab contract; the licensing mount copies
them verbatim into the signed token's `features` so the engine enforces
exactly the plan that was bought.
- clients: implement CheckEntitlement on the disabled (fail-closed) and
ZAP-RPC commerce stubs; in-process pass-through already satisfies it.
Tenant-scoped via orgID (X-Org-Id). Real KMS stays a licensing follow-up
(scaffold TODO); the Mount + entitlement-copy are the deliverable here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gateway dropped the legacy opencensus SaaS exporters (stackdriver was the last
prometheus source). Combined with o11y v1.3.7 + alertmanager/krakend-otel forks +
base v1.3.2 + kms Corona, the binary now links zero real prometheus packages.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Real prometheus in the unified binary is now a single leaf package
(prometheus/prometheus/model/value via a gateway dep). Down from 16+.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
replace krakend-otel => hanzoai/krakend-otel v0.13.1 (prom-free fork); pin
gateway v2.9.5 (opencensus prometheus exporter removed, counters -> luxfi/metric).
With the alertmanager fork + o11y v1.3.7 + iam v1.18.1, the only prometheus left
is the hanzoai/common+alertmanager fork shim core (6 pkgs) — needs those forks to
shed prometheus/common internally.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pkg/iam v1.18.0 imported upstream github.com/beego/beego/v2 while the rest of the
binary uses the hanzoai/beego prom-free fork; both register a global 'graceful'
flag in init() -> panic at startup. v1.18.1 (already fixed on iam main, just
untagged) uses the fork only. The unified binary now boots and shows the
white-label -brand/-domain/-enable tenancy flags.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
o11y's prometheus->hanzoai/alertmanager replace must be carried by the main
module (cloud), since a dependency's replace is ignored by consumers. With
o11y v1.3.7 (no-prometheus) + kms v0.159.1 (Corona), the full HIP-0106 binary
(11 subsystems on zip+ZAP, /v1 routing, luxfi/log+metric) compiles end-to-end.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Build still blocked separately on kms SignWithRingtail (luxfi/kms API gap) and
o11y type-mixing — tracked upstream; this lands the no-local-replace requirement.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-08 12:11:00 -07:00
2467 changed files with 427078 additions and 1438 deletions
<textx="378"y="322"font-family="Inter,system-ui,sans-serif"font-size="30"fill="#ffffff"opacity=".66">Hanzo Cloud — unified Go binary that imports every Hanzo-native…</text>
| grep -v '.github/workflows/containment.yml:'; then
echo "::error::found a build/release invocation passing -tags controlplane — clients/controlplane's stub crypto must never enter a release/serve binary (see clients/controlplane/doc.go)"
| grep -v '.github/workflows/containment.yml:'; then
echo "::error::found a reference to testing.testBinary outside the Go toolchain itself — this is the linker var that spoofs testing.Testing() in a real (non go-test) binary; the containment.go runtime guard trusts that signal, so setting it anywhere in a real build path defeats it (see doc.go)"
hits=1
fi
if [ "$hits" -ne 0 ]; then exit 1; fi
echo "OK: no build/release path sets -tags controlplane or spoofs testing.Testing()"
- uses:actions/setup-go@v5
with:
go-version-file:go.mod
- name:go env for private modules (matches Dockerfile — zap-proto is direct+authenticated)
2. **Agent-NAME axis** — needs (1): the agent run debit records `provider=agent` +
`model=<llm>` but not the agent name, so `metadata.agent` stays honest-empty
until commerce persists an `agent` field the agents meter sets to `a.Name`.
3. **compute split** — `ml` (predict) and `visor` (GPU) both meter `provider=compute`;
read-side can't split `inference` vs `gpus`. Needs (1) so each sets its product id.
4. **exec / containers** (`clients/exec`, Code Interpreter) — authed by a shared
service key (X-API-Key), NO per-org identity, so it can't meter per-org; its
compute is billed upstream at the chat/agent layer that invokes it.
5. **playground** — routes to `/v1/ai/*`, already metered as AI inference.
## 5. Secrets
- Per-tenant, KMS-managed only (`kms.hanzo.ai`, KMSSecret CRDs). No shared
service key stands in for a user. The only service tokens that exist are
narrow, per-tenant, and never used to impersonate a user for LLM spend.
- A surface's own OIDC registration is a PUBLIC client — there is no client
secret to store.
## Surface conformance (as of this contract)
| Surface | Login (PKCE public) | Forwards user token | Org from `owner` | Billed via cloud (org,project) |
|---|---|---|---|---|
| **studio.hanzo.ai** | ✅ reference | ✅ (validates locally) | ✅ | ⚠️ renders run on studio's own GPU workers and self-report to commerce keyed by org via a per-tenant commerce token — org-keyed, but not the forward-bearer-to-gateway path (studio does not call the cloud LLM gateway for its core renders) |
| **console.hanzo.ai** | ✅ | ✅ same-origin `/v1` through the gateway | ✅ | ✅ (it IS the canonical consumer) |
| **hanzo.app** | ❌ confidential client (`IAM_CLIENT_SECRET`, userinfo/introspect) | ✅ to its own backend; org from token `owner` | ✅ | ❌ builder AI runs on OpenRouter with an apiKey (`lib/llm/generation-api.ts`), NOT the cloud gateway — off the unified meter |
hanzo.app is the remaining gap: it needs the same treatment chat just got — switch
its IAM registration to a PKCE public client, and route its builder AI generation
through api.hanzo.ai forwarding the user's IAM bearer so usage meters against the
echo">> FATAL: refusing to ship the placeholder console. Fix the console build:embed, or pass --build-arg ALLOW_PLACEHOLDER=1 for a pure-Go dev image.";\
exit 1;\
fi;\
fi
FROMscratch
# ── Go build stage (CGO=1 + SQLCipher — REAL at-rest encryption) ─────────────
# The unified binary embeds IAM (clients/iam) whose per-org store is SQLCipher-
# encrypted (orgIsolation=sqlite), and commerce's per-tenant money DBs likewise.
# A CGO=0 modernc build SILENTLY SHIPS PLAINTEXT. So this builds CGO=1 against
# system libsqlcipher — hanzoai/iam's proven recipe: the `libsqlite3` tag + a
# libsqlcipher symlink + -DSQLITE_HAS_CODEC, with the modernc double-registration
# guard, TestEncryptionProof, and the cek.go golden-vector KAT baked in — so a
# build that fails to link REAL SQLCipher, or that would decrypt existing stores
# differently, produces NO image. alpine3.22 MATCHES the runtime base so the
# libsqlcipher soname the binary links is the SAME one present at runtime. ECR
# Public mirror avoids Docker Hub's 429 rate-limit on shared CI runners.
# ---- agent-skills stage: regenerate the FULL /.well-known/agent-skills catalog
# from the hanzoai/openapi SOT (skills.py) and carry it into the Go embed path
# BEFORE `go build`, the SAME way the console bundle is produced. The committed
# catalog is only the tiny `ai` fallback; prod must embed the full set. FAIL-HARD:
# if the clone/generation can't produce the master index, the image is not built.
MODERNC="$(CGO_ENABLED=1 go list -tags "libsqlite3 sqlite_fts5" -deps ./cmd/cloud 2>/dev/null | grep -c 'modernc.org/sqlite'||true)";\
["$MODERNC"="0"]||{echo"SQLITE-GATE FAIL: cmd/cloud links modernc.org/sqlite ($MODERNC pkgs) under CGO=1 — double-registers \"sqlite\" with hanzoai/sqlite(mattn) and panics at init.";exit 1;}
# RED gate — ENCRYPTION PROOF + the cek.go GOLDEN-VECTOR KAT, under the SAME CGO +
# libsqlcipher build this image ships. TestEncryptionProof asserts real
# ciphertext-at-rest (SQLITE_REQUIRE_CODEC=1 makes a plaintext link FAIL → NO
# image). TestUnwrapGoldenFixture asserts a FROZEN pre-luxfi-swap 61-byte DEK
# sidecar still decrypts under the shipped luxfi/crypto-AEAD code — existing
# encrypted stores stay readable, or NO image.
RUN --mount=type=cache,target=/go/pkg/mod,sharing=locked \
@echo ">> embedded real console bundle into webui/dist (index.html $$(wc -c < webui/dist/index.html) bytes)"
agentskills:## Regenerate the FULL agent-skills catalog into clients/agentskills/catalog (go:embed source) from the openapi SOT. OPENAPI_DIR=<path to openapi>.
@test -f "$(OPENAPI_DIR)/skills.py"||{echo"openapi checkout not found at $(OPENAPI_DIR) — set OPENAPI_DIR=<path> or clone hanzoai/openapi";exit 1;}
# skills.py rewrites the whole catalog dir; the .gitignore keeps only the tiny
# `ai` fallback tracked, so the full set is embedded at build but never committed.
build-standalone:webuibuild## Build the REAL 1-binary console: console build:embed → webui/dist → go build.
hanzo:## Build the hanzo control-plane CLI into ./bin/hanzo (pure Go, same mode as cmd/cloud — registers the ONE "sqlite" driver exactly once; a plain CGO_ENABLED=1 `go build ./cmd/hanzo` links the fork's mattn backend alongside the embedded modernc importers and panics, see header).
smoke:## Build and run cmd/cloud-smoke (mount-time integration check).
$(GO) run ./cmd/cloud-smoke
test:## Run unit + integration tests (pure-Go, exactly as prod ships).
CGO_ENABLED=$(CGO_ENABLED)$(GO)test ./...
test-cgo:## Prove the cgo build works too — forces the fork's pure-Go backend via -tags sqlite_purego so the embedded modernc importers don't double-register "sqlite".
CGO_ENABLED=1$(GO)test -tags sqlite_purego ./...
vet:## go vet across the module.
CGO_ENABLED=$(CGO_ENABLED)$(GO) vet ./...
tidy:## go mod tidy + verify go.sum.
$(GO) mod tidy
$(GO) mod verify
docker:## Build the Docker image (uses repo Dockerfile, scratch final stage).
docker build -t $(DOCKER_IMAGE):$(DOCKER_TAG) .
docker-push:docker## Push the Docker image to ghcr.io. Requires docker login.
Unified Go control plane and binary for the Hanzo platform (HIP-0106).
@@ -13,7 +15,37 @@ docker run -p 8080:8080 ghcr.io/hanzoai/cloud:latest
## What this is
`hanzoai/cloud` is one Go binary that mounts every Hanzo subsystem (iam, kms, base, gateway, ai, commerce, vfs, mq, dns, amqp, mcp, o11y, ...) into a single multi-tenant process. Same artifact serves `api.hanzo.ai`, `api.osage.cloud`, `api.lux.cloud`, `api.zoo.cloud`, and every white-label reseller. Brand, enabled subsystems, and tenant scope are deployment configuration.
`hanzoai/cloud` is one Go binary that mounts every Hanzo subsystem (iam, kms, base, gateway, ai, commerce, vfs, mq, dns, amqp, mcp, o11y, ...) into a single multi-org process. Same artifact serves `api.hanzo.ai`, `api.osage.cloud`, `api.lux.cloud`, `api.zoo.cloud`, and every white-label reseller. Brand, enabled subsystems, and org scope are deployment configuration.
## `hanzo` — cloud control CLI
The same binary is also a gcloud/doctl-class CLI. The first token selects the mode:
logger.Warn("audit trail DISABLED by CLOUD_AUDIT_DISABLED — no tamper-evident record will be kept")
}
returnnil,nil
}
ifcfg.DataDir==""{
returnnil,fmt.Errorf("empty DataDir — the audit trail is a compliance control and requires a persistent data dir (set CLOUD_DATA_DIR); refusing to boot with a non-persistent trail that would lose all prior records on restart (or set CLOUD_AUDIT_DISABLED=true to opt out explicitly)")
}
iferr:=os.MkdirAll(cfg.DataDir,0o755);err!=nil{
returnnil,fmt.Errorf("data dir: %w",err)
}
// OLAP mirror is optional and best-effort. A mirror that cannot be reached at
// boot must NOT stop the binary — the local chain is the authority — so a
// mirror construction error is logged and the trail runs local-only.
// Validate the target is a REAL org (never mint an orphan wallet on a typo).
o,err:=FindOrg(s,ctx,cr,org)
iferr!=nil{
returnFail(c,err.Error())
}
ifo==nil{
returnc.JSON(404,map[string]any{"status":"error","msg":"customer not found","data":nil})
}
// FAIL-CLOSED durability (SOC2 AU-2/AU-5): a credit grant moves REAL money and MUST
// leave a durable, tamper-evident record. If this deployment has no audit store to
// record into, REFUSE the grant BEFORE any money moves — never an unaudited money
// move. A nil store is not a production state: cloud requires a persistent data dir
// for the trail (audit_serve.go), and nil arises only from the explicit
// CLOUD_AUDIT_DISABLED dev opt-out, on which moving money is not a supported op.
ifs.State.AuditStore==nil{
returnc.JSON(503,map[string]any{"status":"error","msg":"grant refused: no durable audit store is configured on this deployment; a credit grant must be recorded before money moves","data":nil})
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.