Fan POST /v1/analytics out to analytics.hanzo.ai (Umami /api/send) and
insights.hanzo.ai (PostHog /v1/e); ecommerce -> GA4/Meta conversion mapping.
Fail-soft fanout; adapters inert until per-site KMS keys are set (additive).
Two silent-drop contract bugs from adversarial (red) review of the Umami +
PostHog analytics sinks — both fail soft, so a paid product ingested zero.
- posthog [critical]: Send POSTed to <host>/batch, but Hanzo Insights
capture-rs registers ONLY /v1/e|/v1/s|/v1/ai (rust/capture/src/router.rs;
legacy /batch REMOVED, Django APPEND_SLASH=False) → every event 404'd and
was dropped (postJSON logs, never wedges). Repoint to /v1/e; the
{api_key,batch:[…]} body is unchanged — capture-rs's untagged
RawRequest::Batch variant accepts it verbatim.
- umami [medium]: umamiPayload.DistinctID serialized as `distinctId`, a field
absent from the fork's /api/send zod schema (src/app/api/send/route.ts) →
silently stripped, so the forwarded visitor id never keyed the session
(stitching fell back to IP+UA). Rename the JSON tag to `id`, the schema's
real session key: sessionId = id ? uuid(website,id) : uuid(website,ip,ua,salt).
Tests now assert the real WIRE contract (a struct-field assert cannot catch a
JSON-tag bug — the source of the false confidence red flagged):
TestPostHogSendEndToEnd → /v1/e; TestUmamiBuild + TestUmamiSendEndToEnd → the
visitor id marshals under `id`, never `distinctId`.
go build + go vet + go test ./clients/destinations/... green.
Three hardening fixes from live testing against the zrok controller:
- LOGIN-FIRST: token(org, create) tries login before create — an existing
account (the common path) costs ONE login and never touches admin; create
only on a real miss. Slims the controller interface to token()+overview()
(org-centric — the controller owns all credential derivation, the handler
just passes org; DRY).
- FRESH EMAIL: zrok SOFT-deletes accounts (a deleted email stays in the unique
index, un-recreatable forever), so a plain share-<org>@ that was ever touched
is burned. accountEmail now carries an HMAC freshness suffix
(share-<org>-<hash8>@hanzo.ai) — deterministic, unguessable, and distinct
from any prior scheme, so no burn can strand an org.
- ENV BASE: ZROK_API_BASE (default /api/v2) makes the /api/v2 -> /v1 cutover a
CR edit, not a rebuild — flip it the moment the /v1 zrok image deploys.
Builds + unit tests green.
share-<org>@hanzo.ai. A plus local part trips some validators, and zrok
soft-deletes accounts (its unique email index then blocks a recreate of
the same address) — a hyphenated, stable identity sidesteps both. Verified
live against the controller: create 201 + login 200 for share-hanzo.
Fan the /v1 analytics event stream out to two first-party analytics sinks
alongside the existing ad platforms, and map schema.org/GA4 ecommerce actions
onto GA4 and Meta's native conversion schemas.
- umami.go: POST /api/send (Hanzo Analytics fork). Credential-less public
beacon keyed by the non-secret website id; forwards the end-user UA+IP so
Umami attributes session/geo. Pageviews sent without a name (Umami's rule).
- posthog.go: POST /batch (Hanzo Insights fork). Project api_key is the KMS
secret, carried in the body (never URL/log); distinct_id keyed, empty dropped.
- fanout.go: resolveSecret treats a Secret-less, Fallback-less destination as
credential-less (public ingest) instead of failing closed.
- translate.go: map product_viewed/product_added/begin_checkout/purchase (+
schema.org aliases) onto the commerce standard events; lift ecommerce line
items (items/products array, or a first-class product id) into Conversion.Items.
- ga4.go: render GA4 items[] + purchase transaction_id.
- meta.go: render Meta content_ids/contents/content_type/num_items + order_id.
- destination.go: normalized Item type + Conversion.Items/Referrer.
- send.go: shared non-PII first-party analytics data helper.
Secrets stay in KMS; no credential in any endpoint, error, or log line.
Tests: adapter payload + ecommerce mapping (mock HTTP), translate vocabulary +
item lift, credential-less resolution. go test ./clients/destinations/... green.
Trust becomes exactly what IAM asserts: a valid SIGNATURE from a trusted ISSUER
(the brand set) plus EXPIRY. The audience (a minting app's client_id) is now
INFORMATIONAL, not an access gate — cloud no longer keeps a hand-maintained mirror
of IAM's app registry. That mirror was the lone non-IAM-native gate and it drifted:
every new first-party app (most recently hanzo-commerce, breaking the commerce-admin
AI assistant) silently 401'd until someone edited GATEWAY_ALLOWED_AUDIENCES. A new
first-party app now 'just works' with zero cloud change.
Removed: identityValidator.audiences, Config.JWTAudiences, jwtAudiencesFromEnv,
defaultJWTAudiences, BrandAudiences, unionStrings. validate() enforces issuer + expiry
via jwt.Expected{} (empty AnyAudience skips ONLY the audience match; go-jose still
checks exp/nbf against time.Now). Fail-secure on an empty ISSUER set is preserved.
PRESERVED (unchanged): owner-claim org scoping on every guard; SuperAdmin =
owner==adminOrg AND !isKMSMachinePrincipal; the KMS-machine SuperAdmin-denial
(isKMSMachinePrincipal reads claims.Audience directly, not the removed allowlist);
OrgHasUnsafeRune. Tests reframed to the new invariant and all green, incl. the KMS
red adversarial suite (multi-value aud, admin-slip, owner-bound machine-aud, trim/
unsafe owner). One new test proves any aud from a trusted issuer validates while a
wrong issuer / expired token still rejects.
The user-facing behavior change: a real admin (owner==adminOrg, isAdmin) is now
SuperAdmin from ANY first-party app, not only allowlisted ones — owner is the
authority, which is correct for an internal IAM where every app is first-party.
The controller's go-swagger API mounts at basePath /api/v2 and consumes/
produces ONLY application/zrok.v1+json (application/json → 415/500, and a
v1 path → the SPA's 202 HTML). ensureAccount/login/overview now hit the
right base with the right content type. Verified: /api/v2/login returns
the account token 200.
Provisions a per-org zrok account from the caller's validated IAM identity
so the Rust CLI's `hanzo share <port>` publishes a local service to a public
https://<token>.share.hanzo.ai URL with zero manual setup.
- POST /v1/share/enable: stateless, deterministic per-org account (email
share+<org>@hanzo.ai, password HMAC(secret,org)) — ensure-account + login
reconstruct the same credential every time; the zrok controller IS the
store, no local persistence. Returns {accountToken, controller, namespace,
urlTemplate} for the tunnel client. Idempotent.
- GET /v1/share: the org's active shares (CLI + console Shares view), honest-
empty when the controller is unreachable.
- Fail-closed 503 until ZROK_ADMIN_TOKEN; org from principal (never a client
field) so a caller can only ever provision/read its OWN org. .Group routes,
mockable controller client, unit-tested.
git.hanzo.ai (served by cloud's embedded git forge, clients/git) returned a raw
403 'sign in to view Hanzo Git' for signed-out visitors. Most Hanzo projects are
OSS, so the default face is now open, GitHub-style:
- uiExplore + /explore: lists every PUBLIC repo across ALL orgs, searchable (?q=),
no auth. Cross-org via {DataDir}/orgs enumeration (per-org stores have no global
index), capped at maxExploreScan.
- uiHome: signed out -> public explore/landing; signed in -> your org's repos.
- uiRepoAccess: repo/tree/blob/commits views serve PUBLIC repos anonymously;
private repos stay org-authed and answer the SAME 404 (no existence leak),
mirroring smart-HTTP's resolvePackRepo(allowPublic).
- Store.ListPublic: per-org public-repo query.
- state.dataDir threaded through Mount for the org enumeration.
Anonymous clone of public repos already worked (smart_http); this opens the
browser surface to match. Updated TestRootUI_HostGuard to the new public-landing
contract (/ -> 200 explore, missing repo -> native 404).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Seed the DB-backed blueprint with the complete corpus — the 64-principle
spine plus 888 modern + 114 heritage strategies (1002 total), each filed
under a spine principle and tagged by era. base.yaml is the assembled
genome, kept byte-synced with the GitOps origin.
Schema: Blueprint.Principles (the 64 archetypes); Strategy gains
principle/source/era/blog. Validate enforces referential integrity — every
strategy's principle resolves to a spine slug, no duplicate id across the
1002 — plus unique principle slugs.
Version-aware seed (SeedOrUpgrade): a monotonic seedVersion is stamped on
the seed row (source="seed"); an admin write flips it to source="admin".
On Mount a brand with no row is seeded, an UNEDITED seed at an older
generation is upgraded in place, and a brand ever admin-edited is never
touched again. A self-healing backfill marks legacy version>=2 rows as
admin.
Fix two review LOWs:
- authoringBlueprint returns a clone of the embedded fixture in its
pre-seed fallbacks, so an in-place PATCH can't corrupt the shared value.
- putBlueprint / listBlueprintVersions fetch the write key via
LatestResolved without parsing, so a corrupt stored row is still
replaceable by a valid PUT.
Tests: assembled seed parses+validates; /v1/guide/strategies surfaces the
888; a v2 seed upgrades an unedited v1 row and never clobbers an admin
edit; legacy backfill; the two LOW PoCs. Frozen wire green.
Make the whole Guide™ playbook DB-backed and SuperAdmin-editable — nothing static.
Schema: extend the engine into a full Blueprint (Section/Strategy/Template + an
Enabled lever on every item, default-on). The engine stays pure — Blueprint.Curriculum()
projects the ENABLED journey (disabled section drops its steps; disabled step drops and
its dependents lose the edge), so reconcile/Next/Available/Counts are unchanged. Parse is
fail-closed; Validate enforces DAG-acyclic + no-dangling over the authored graph AND the
enabled projection.
Seed: embed base.yaml (12 sections · 67 steps · 114 strategies · 6 templates, a
byte-identical synced copy of the universe fixture) as the blueprint; supersede
default.yaml. On Mount, seed-if-absent into a shared versioned store (guide_blueprint,
brand-keyed) — a redeploy never clobbers admin edits.
Resolve (three tiers): org override (per-org) → brand blueprint (shared DB, seeded,
SuperAdmin-authored) → embedded fixture (fail-safe). A disabled/unparseable tier is
skipped.
Author: /v1/guide/blueprint/* gated on IsSuperAdmin (owner==admin) — a normal org member
or per-org admin gets 403. GET/PUT the whole blueprint, GET versions (PITR), PATCH an
item (edit + the enable/disable lever). Every write is validated fail-closed and appended
as a new version; edits take effect on the next resolve.
Filter: GET /v1/guide/strategies?category=&stage=&workload= — the ENABLED corpus,
org-scoped, joined to the observe layer (stage:* is a monotone readiness floor,
research==formed; has:* maps to module:*/connected:*/analytics/revenue/deployed/
funnel:signups; extend standardConnectors so the join is real).
Green: go build ./clients/guide/... ./apps/...; go vet; go test ./clients/guide/... (50
tests incl SuperAdmin-gate 403, idempotent-seed no-clobber, org>brand>fixture resolution,
disabled section/step drop, corpus filter). apps TestWireOrderMatchesFrozen green (routes
added to the existing guide subsystem).
Add the Guide's OBSERVE layer so the Business AI grounds on an org's REAL
platform truth, org-scoped and honest-degrading.
Growth signals extend the Detector vocabulary (detect.go stays THE coupling
point): module:<name>, connected:<provider>, funnel:<stage>, deployed, revenue,
customers:>=<N>. Each reads a sibling subsystem through an injected Signals seam
bound at the composition root (apps/wire_seams.go) — guide imports none of them,
the coding-dispatcher injected-function pattern. A nil seam field honest-degrades
its signal to not-present; the vocabulary is the contract, a read fills in when a
provably org-scoped seam lands. lookupDetector resolves parameterized <kind>:<param>
signals so a curriculum step can auto-detect on any of them.
classifyStage folds a SignalSet into a first-principles growth stage
(formed->launched->activated->scaling) — pure, total, exhaustively tested; the
ladder is data, driven by OUTCOMES (presence, engagement, money), not setup.
GET /v1/guide/profile exposes {stage, signals, keyMetrics}, org-scoped on the
validated principal, READ-ONLY, recomputed each request (real-time by pull) via the
existing reconcile path + the growth probes.
Bound reads (provably org-scoped, nil-safe): framework.ModuleInstalled (new,
module-granularity sibling of Installed), integrations.Connected (new, boolean,
never the token), the in-package analytics funnel. deployed/revenue/customers stay
unbound seams (deploy is cluster-scoped; commerce/crm have no clean per-org
in-process read) — honest-degrading until a provably org-scoped read exists.
Tenant isolation: every signal read keys on the caller's own org; TestHTTPProfile
CrossTenantIsolation proves org A's profile never reflects org B's signals and each
request's probes read only the caller's org. framework/integrations seam reads have
their own org-scoping tests.
The unified store's durability plane auto-activated whenever S3_ADMIN_* was
configured (prod sets it), so deploying it would begin fencing real tenant
data on the object store's conditional-PUT (If-Match) atomicity before that
atomicity is validated against the deployed SeaweedFS version (Red's takeover-
fence staging gate H2). Gate it behind CLOUD_RESEARCH_DURABLE (off by default):
the store runs local-only until the flag is set, and the shard router already
pins each org to one writer (ha.Owner) — so this is NOT the rolling-deploy
outage, and the DDL-drift-proof + evidence-preserving record layer is fully
active. The object-store snapshot/fence turns on deliberately after the gate.
Complete the stranded Guide feature so a fresh org's journey works with no
per-org module install.
framework: MarkAlwaysOn/AlwaysOnModules mark a registered module always-on —
its DocType fixtures resolve for every org via GetDocType/ListDocTypes and the
Installed predicate, with no per-org install row. This defaults only the SCHEMA
on; every document row stays physically org-scoped. A per-org stored DocType
still overrides the fixture. always_on_isolation_test.go proves org A's records
are unreadable by org B even though the DocType resolves for both.
content: marketing is always-on (doctypes.go init), so content_generate
completes for a fresh org — the fix for "content: marketing module not installed
for org". Drop the now-unreachable not-installed guard in EnsureCatalogAsset (a
fresh org without a studio still quietly skips at Generate).
guide: builtin-2 journey rooted at "Form your company", then positioning →
launch steps and the daily agentic growth loop; positioning gates on company.
Add /v1/guide/suggest + /v1/guide/chat — read-only AI "what to do next", grounded
in the org's real progress + funnel, never running an action. Resolve the
white-label brand journey (brands/<brand>.yaml; zoo ships zoo-1) at Mount.
cek: TestMain seeds an ephemeral master key so the encrypted-at-rest stores open
on an encryption-capable test build (framework, content, guide) — mirrors
compliance/integrations/flags/venue.
The dashboard already POSTs pause/resume/stop/message, recorded as durable
KindControl events. A cloud-dispatched routed run gets them forwarded to the
tasks engine, but a LOCALLY-started `hanzo code` session is not task-backed —
so the running surface must pull them itself.
Add GET /v1/agents/sessions/:id/control?after=<seq>: an owner-scoped, cursor-
driven drain of a session's control commands (ListControlAfter filters to
control, oldest first). Read-only, org is the only tenant key, foreign id 404s.
Test covers filter/order/cursor/tenant-isolation.
Unify the research record layer (hanzoai/orm typed records, DDL-drift-proof
+ migrateLegacy evidence carry-forward) onto the HA file-durability layer
(ha-elected single-writer + fenced ship-before-ack). Reconcile the three
files both branches touch:
- store.go: ORM's typed-orm rewrite — supersedes the raw-SQL ALTER-migrate
path (schema migration is structurally impossible under orm).
- migrate_test.go: ORM's legacy-DATA migration test (ALTER-convergence is
obsolete under orm).
- research.go: both — HA's durability wiring (WithDurable + shipDurable
ship-before-ack in every write handler) plus ORM's postGrant scope comment.
deps: orm v0.6.1 -> v0.6.7 (CreateIfAbsent typed-record ingest the orm store
needs) and commerce v1.49.13 -> v1.49.14 (DatastoreAdapter satisfies orm
v0.6.7's orm.DB CreateIfAbsent, keeping ./apps/ building).
TestRecordShipsSoTakeoverKeepsIt reopens the shipped store through cek. On a CGO
build whose SQLite lacks SQLCipher, PRAGMA key is silently ignored so cek writes a
plaintext file it then cannot migrate on reopen (sqlcipher_export absent) — and the
suite TestMain still injects a dev key because sqlitedrv.EncryptionAvailable() reports
a false-positive yes, so the test hard-failed under the default CGO build.
cekCanReopen probes the actual round-trip: it holds under pure-Go (plaintext) and
real libsqlcipher (encrypted, exercising the .dek sidecar cross-pod restore), and
skips only on the broken-capability build. go test ./... stays green everywhere;
ship-before-return is proven under CGO_ENABLED=0 and by TestRecordOnNonOwnerFailsClosed.
Red L2: durability disabled now routes through disabledDurability(), which logs at
ERROR on a MULTI-REPLICA deployment (>1 CLOUD_PEERS) — per-org stores then survive
only via shard routing + per-pod RWO PVC, so a lost PVC loses data — and at INFO for
single-replica/dev where local-only is expected. The encryption-capable-build-without-
cipher case is treated as a misconfig (fail closed + loud), never a silent plaintext
ship or a silent drop back to the non-HA outage.
Red M3: documented degraded-open recovery — a degraded pod stays read-only for the
cached store's life (in-place re-acquire would CarryForward-restore under the live
handle = stale reads); recovery is a fresh open (pod restart / shard reroute), with
quiesce-close-reopen as the future enhancement.
Red M4/M5: documented the SeaweedFS operational requirements the fence rests on —
object versioning + no-expiry lifecycle on the org-db prefix (lease round is the
system of record; a dropped/rolled-back lease can un-fence a zombie), and RWO
per-writer PVCs for DataDir, never RWX.
The legacy migration was idempotent but re-read the raw-SQL tables on every open. A
completion marker (a kindMeta record under a distinct id from the seq clock, written
in the SAME tx as the migration so it commits iff the migration commits) short-circuits
every later open. Greenfield stores mark done immediately. Test asserts the marker is
set post-migration and that a re-open stays correct (no duplication).
The single-writer fence rests on the gateway evaluating If-None-Match:* (create-only)
and If-Match (version-conditioned) preconditions ATOMICALLY server-side — the one
property the in-process fakes cannot prove. This env-gated integration test races 12
concurrent writers against the REAL S3ConditionalStore and asserts EXACTLY ONE wins
each precondition; two winners = split-brain possible, do not ship durability against
that gateway. Skipped in unit runs; run in staging with CLOUD_DURABLE_IT=1 + S3_ADMIN_*
before durability fences real tenant data (or on a SeaweedFS version bump).
Red M1: forPath held the store-wide c.mu through openDurable→Hydrate (Acquire +
CarryForward round-trips, untimed) — a hung SeaweedFS froze the whole subsystem, a
cache-hit For() included. Now the durable open runs with c.mu RELEASED, deduped by
an in-flight record so concurrent For() for one org opens exactly once (cek cannot
open a file twice); every object-store round-trip (hydrate, Sync, close) is bounded
by durableOpTimeout (30s) so a slow store degrades to bounded latency, never a
deadlock. The local-only path is unchanged (disk I/O under c.mu as before).
Red L3: CloseAll ships each final state with c.mu released and time-bounded (was a
background-ctx ship under the lock) — Durable.Close takes a ctx now.
Red L1: unframe checked m+sl > len(b), which a max-uint64 length wraps past, then
panics the slice. Now checked as sl > len(b)-m (subtraction, no wrap) → fails closed.
Tests: TestDurableForDedupsConcurrentOpens (8 concurrent For → one store), the L1
oversized-length frame fails closed, all durable + research + root OrgStore green.
CRITICAL (red): the orm store read only _entities, so an existing per-org file whose
evidence lived in the old raw-SQL experiment/attempt/artifact tables read as EMPTY —
every logged run silently vanished the instant orm shipped (data intact in-file, but
all reads returned 0). migrateLegacy now runs in openStore: gated on old-table
existence (greenfield skips), idempotent via CreateIfAbsent, fail-secure (a migration
error fails the open rather than serving empty over real data).
- Each version keeps its OLD seq value → canonical/supersession preserved exactly (a
corrected run stays canonical); the append clock is advanced past every migrated seq
so a later ingest still supersedes (proven with a ts=0 correction across the boundary).
- Rows read by COLUMN NAME (SELECT *) → a table left by any older schema (even the
outage-era one missing provenance columns) migrates without a 'no such column'.
- content_hash/revision/status/visibility/consent/provenance + artifact blobs verbatim.
- Regression TestLegacyDataMigration: seed old tables incl a correction pair → open →
assert pre-migration truth (corrected canonical, retained intact, artifact bytes),
a later ingest supersedes, an unrelated ingest doesn't flip it, re-open is a no-op.
Also (red): setArtifactVisibility now does its read-modify-write in a transaction like
setGrant (LOW); a scale note at the loaders documents the in-memory-fold tradeoff +
indexed-Filter escalation (MEDIUM, org-bounded); postGrant's client-supplied project
is confirmed intentional (org is the tenant boundary, project an org-internal target
label) with a clarifying comment (LOW).
Red H1: research.Record (in-process A/B evidence, experiments.go:425) committed
locally but NEVER shipped — an acked write lost on takeover, and unfenced on a
non-owner. Record now Syncs before returning and propagates a not-acked ship as an
error (mirrors the HTTP shipDurable), so a takeover keeps it and a non-owner fails
closed instead of persisting a stale divergent local copy.
Writing the H1 takeover regression surfaced a real restore bug: restore() wrote the
.dek key sidecar BEFORE RestoreFile created the parent dir, so a fresh successor
(orgs/<slug>/ absent) failed the sidecar write → hydrate degraded → EMPTY store =
the lost write. RestoreFile (which MkdirAll's) now runs first, then the sidecar.
The flat-tempdir unit test masked it; added TestDurableRestoreIntoFreshNestedDir.
Red M2: wal_checkpoint(TRUNCATE) result was discarded — busy!=0 ships a partial
snapshot (committed frames still in WAL) as acked = silent lost write. Now
QueryRow'd; fail closed on busy!=0.
Tests: TestRecordShipsSoTakeoverKeepsIt (takeover keeps the evidence, real cek
key-sidecar cross-pod restore), TestRecordOnNonOwnerFailsClosed, and the nested-dir
restore regression — all green.
24 concurrent ingests of distinct content must all land (no lost write, no dup) and
the server-assigned append clock (seq) must be unique + gapless in [1,N] — the
concurrency proof for the per-store seq that replaced SQL AUTOINCREMENT, backed by
OrgDB single-writer + orm writeMu + the ingest tx.
A KYC/verification terminal status is now reached by three orthogonal paths, never a
client-asserted status:
- provider reconcile (pull): company kyc/refresh + compliance verifications/:id/refresh
consult the wired provider for the settled status; Manual stays pending.
- provider webhook (push): compliance verifications/webhook authenticates by HMAC
signature (idv.Webhook, KMS-sealed secret, disabled by default) and reconciles the
referenced check from the provider API, so the request body cannot dictate a status.
- reviewer decision: company kyc/decision (a platform reviewer) and compliance
verifications/:id/decision (an org admin or platform reviewer) are role-gated and
attributed (DecidedBy = the acting user), and produce a DISTINCT reviewer_confirmed,
never a provider_verified.
guardKYCVerified and the compliance decision accept only an attributed provider pass or
reviewer_confirmed. A founder/check records who decided; an unattributed "verified"
fails closed. startKYC clamps a provider's inquiry-time status to pending.
legal: the counsel-review notice is coupled to the category — formation and equity
templates always carry it, forced on override and emitted at render — and an override
body may reference only declared fields.
compliance createAccreditation records only an asserted state; a provider_verified
state routes through the attributed decision endpoint.
Remove the committed cek key sidecars and gitignore the pattern; tests seal their
stores under t.TempDir().
Hanzo Compliance (/v1/compliance): org-scoped KYC/KYB verification through a
provider-agnostic seam, accreditation-state tracking, and a compliance-scoped read
of the shared tamper-evident audit plane (SOC 2 posture). Subject PII is sealed at
rest (cek) and referenced by opaque id everywhere else — never in logs, audit, or
URLs. No path yields a verified status on create; a terminal decision comes only
from the provider (refresh) or an authenticated, audited callback.
Hanzo Legal (/v1/legal): a versioned, org-overridable standardized template library
plus a pure, deterministic merge-field generation engine, a sealed document store,
and e-sign + filing seams. The counsel-review notice is a non-droppable invariant on
formation and securities documents.
clients/idv: the ONE identity/business verification seam — honest Manual default,
config-driven Persona/Onfido/Stripe adapter with a KMS-sealed key and a strict,
fail-closed status classifier. Consumed by BOTH compliance onboarding KYB and company
formation KYC; company now resolves a real provider from config (fail-closed),
replacing the manual-only default.
Boundary invariant, enforced in the data model and on the wire: platform tooling with
licensed providers and professionals in the loop — provider-reported or tracked states
only, never a platform assertion of "compliant" or "legally valid".
The research HTTP tests used the bare app.Fiber().Test(req) whose default client
timeout (1s) is too tight for a cold cek-encrypted per-org SQLite open, so they
flaked as i/o timeouts. Match the rest of the repo (base/exec/world tests):
fiber.TestConfig{Timeout: 30 * time.Second}. Assertions unchanged.
OrgStore gains an opt-in Durability (WithDurable): when set, forPath hydrates each
org's SQLite from the object store BEFORE opening (elected owner CarryForward-seals
to its lease round; non-owner refreshes read-only), binds the handle, and Sync ships
it fenced after a write. A degraded hydrate never blocks the open — the store always
opens (reads local, writes fail closed) so a second replica can never break it. With
no option it is byte-identical to the local-only cache the other 12 subsystems use.
BuildDeps constructs the deployment Durability (buildDurability): the SeaweedFS S3
If-Match ConditionalStore (same s3admin identity as deps.VFS), membership over
CLOUD_PEERS (the SAME set the shard router elects on), and the per-org envelope
Cipher rooted at the KMS master. No object store ⇒ nil ⇒ local-only. An
encryption-capable build with no cipher is refused (never ship plaintext snapshots).
research wires WithDurable(b.Durable) and calls Sync after each write commits
(ship-before-ack): a not-acked ship (deposed/degraded) returns 503 so the client
retries on the org's current owner — no acknowledged write is lost on failover. On a
local deployment the ship is a successful no-op. Scoped to research as the proof; the
seam (WithDurable) is the one other subsystems adopt next.
durable_test.go adds the cek-encrypting-build coverage: the .dek key sidecar ships
with the database bytes and is restored on hydrate, plus the (sidecar,db) frame
round-trip.
v0.6.4's replica/sqlite.go blank-imports modernc.org/sqlite directly, which
registers the "sqlite" database/sql driver a SECOND time alongside cek's
hanzoai/sqlite — a double-register panic at init the moment a binary links both
(which wiring internal/org's durable path into cloud is the first to do). v0.6.6
(469995c + ffd32ca) routes replica through the ONE hanzoai/sqlite driver, so a
single registration stands. Patch bump, FencedStore API unchanged.
Kills the 'no such column' DDL-drift bug class permanently: records are typed Go
values stored as JSON in orm's fixed _entities table, so adding a field is a struct
change with ZERO DDL. orm layers over the per-org *sql.DB cloud.OrgDB already opens
(cek-encrypted, single-writer, WAL) via orm.AdaptSQLite — orm manages the records,
the caller owns the file — so encryption at rest and the HA durability plane are
preserved and the openStore(*sql.DB) seam is unchanged.
Behaviors preserved over the orm model:
- dedup/idempotency: CreateIfAbsent keyed by <project>:<id>:<content_hash>
- versioned/canonical: a per-store monotone seq stamped inside the ingest tx
(never client ts); canonical = latest-appended non-retracted per stable id
- queries (counts, list, totals, projects, artifacts) reimplemented as in-memory
folds over orm Query (per-org stores are small)
- evidence model intact: ingest forces private/non-trainable/non-publishable,
faulted runs retained, artifact sha256 server-derived, provenance first-class
Artifact bytes split into a distinct blob keyspace so the diary feed never loads
blobs, and because orm's _entities id is a global PRIMARY KEY (kind-prefixed ids
keep each kind in its own keyspace). Regression test proves a new provenance field
needs no migration; the 11 evidence-semantics tests pass against the orm store.
Packages the handoff_test.go discipline into one value a per-org SQLite store
wires at open/write/close. Composes the three lanes: ha election (CASFencer
lease = WHO writes), replica.FencedStore (CarryForward-on-takeover + round-fenced
ship = HOW it ships, no acknowledged write lost, deposed writer rejected as
ErrStaleRound), and the org envelope Cipher (durable object at rest).
Snapshot is a raw file copy — checkpoint the WAL on the store's sole connection,
read the actual local file bytes, and ship them framed with the cek .dek key
sidecar. Backend-agnostic: identical whether the local file is cek-encrypted
(production) or plaintext (dev), so an existing on-disk store needs no migration
and cek stays the local at-rest gate.
Hydrate never makes a store unopenable (degrades read-only, writes fail closed) —
the outage was a second replica breaking the store.
durable_test.go proves it over one object store: two replicas contend → one
writes, the other defers, both open, no split-brain; a rolling takeover hydrates
the shipped snapshot with no lost write; a deposed writer is fenced; an
unreachable store degrades read-only, never 'unavailable'.
The object store is the SeaweedFS S3 gateway reached via github.com/hanzoai/s3-go
(package name minio, aliased s3). Rename the import alias minio→s3 and every
minio.X reference to s3.X; MinioConditionalStore→S3ConditionalStore,
NewMinioConditionalStore→NewS3ConditionalStore; fix comments/docs that named the
old MinIO fork. Upstream transitive module names (github.com/minio/*) in
go.mod/go.sum are external deps of hanzoai/s3-go and are left as-is.
No behavior change: pure rename + comments.
Pre-existing failure on clean main, unrelated to the pre-pay work — two causes,
both surfaced now that the metering suite runs:
1) Encryption gate: on an encryption-capable (cgo) build, cek refuses to open
the finance store without CLOUD_KMS_MASTER_KEY_REF. The root package's
TestMain supplies a throwaway dev key; the clients/metering test package had
no TestMain, so the finance-in-process test couldn't open a store. Add the
same dev-key TestMain (mirrors the root; only when the build can encrypt and
no key is provided — CI's real key still wins).
2) Stale ceil-based assertions vs the exact 18-decimal ledger. Since the
Money Int->Atto migration, finance debits the EXACT sub-cent amount: a 1.5c
micros debit lands as 1.5c, leaving 98.5c (/usr/bin/zsh.985) — not a ceiled 98c. The
RecordResult still reports the ceiled 2c (Cents() rounds up for whole-cent
contexts), which is unchanged. Assert the exact ledger balance (money.Cmp
against ParseUSD) instead of bal.Cents(), which ceils 98.5->99. The anti-leak
invariant holds: the sub-cent debit is recorded exactly, never dropped to 0.
Full metering suite green.
The CTO pre-pay acceptance scenario end to end: a freshly provisioned org
starts at a ZERO balance, so a metered request is REFUSED (402, no free floor,
nothing recorded); after a pre-pay deposit lands, the same request is SERVED
and DEBITS the balance; when the balance is exhausted, it is refused again. The
gate (Client.Authorize: funded := available > 0) already enforced this; this is
the explicit acceptance test for the pre-pay model. Stub balance made
thread-safe (setAvailable). Green under -race.
The store returned 500 'research store unavailable' because migrate() ran
CREATE TABLE IF NOT EXISTS (a no-op on an existing table) then CREATE INDEX
ix_exp_git ON experiment(git_sha) — which failed 'no such column: git_sha'
on any org store created before the provenance columns landed. CREATE TABLE
IF NOT EXISTS never adds a column to an existing table.
Fix: after creating the tables, ALTER TABLE ADD COLUMN every current non-key
column (tolerating 'duplicate column name' when already present), THEN the
indexes. migrate() now converges any older schema to the current column set
and is idempotent across re-opens. Adding a column = add it to the CREATE
plus the ensure list. Regression tests reproduce the outage (old table
without git_sha) + idempotent re-open + fresh-db all-duplicates.
The store returned 500 'research store unavailable' because migrate() ran
CREATE TABLE IF NOT EXISTS (a no-op on an existing table) then CREATE INDEX
ix_exp_git ON experiment(git_sha) — which failed 'no such column: git_sha'
on any org store created before the provenance columns landed. CREATE TABLE
IF NOT EXISTS never adds a column to an existing table.
Fix: after creating the tables, ALTER TABLE ADD COLUMN every current non-key
column (tolerating 'duplicate column name' when already present), THEN the
indexes. migrate() now converges any older schema to the current column set
and is idempotent across re-opens. Adding a column = add it to the CREATE
plus the ensure list. Regression tests reproduce the outage (old table
without git_sha) + idempotent re-open + fresh-db all-duplicates.
Red-review fixes on the /v1/help public plane:
- Drop the app per-IP rate limiter. Behind hanzoai/ingress the socket peer IS the
ingress, so a per-IP limiter keys every customer to ONE shared bucket (a global
throttle + trivial DoS) and X-Forwarded-For is client-settable (a fresh value per
request evades the limit and grows the bucket map without bound). The ingress owns
the per-client edge limit; the plane bounds each request instead.
- GET /v1/help/categories now returns only sections that front a Published + public
article, so an internal (agent-only) category name or description never leaks.
- Intake returns an opaque random public_ref in place of the monotonic ticket name,
so an anonymous submitter cannot read the org's ticket volume. The sequential name
stays internal to the agent plane.
- Cap the whole intake body at 64 KiB before parsing.
Tests: clients/help green (adds no-shared-bucket-throttle, category-gating, oversized-
body, opaque-ref assertions).
Complete the Hanzo Support model as DocType fixtures and add the thin
/v1/help public plane — the native-Go replacement for the Frappe Helpdesk
(Vue + Python Frappe on Werkzeug), on Base in the one cloud binary.
Model (clients/help/help.go): add hd-article + hd-article-category (the KB)
and hd-communication (the ticket conversation thread); add a source field to
hd-ticket for inbound-connector provenance. Agents author and triage all of
it on the generic role-gated /v1/framework/hd-* surface — no new agent code.
Public plane (clients/help/subsystem.go): the anonymous face the secure-by-
default engine cannot serve — GET /v1/help/articles + /articles/:slug (only
status=Published AND is_public=1, re-checked on direct fetch), /categories,
and POST /v1/help/tickets (rate-limited customer intake creating the ticket
plus its opening conversation message). The served org is fixed server-side
(CLOUD_HELP_PUBLIC_ORG, else the deployment brand), never client-chosen, and
fails closed when unset. Storage delegates entirely to the framework
in-process API — one engine, no duplicated CRUD.
Wire help as a mount subsystem (apps.go + frozen wire order), alongside
knowledge, the other framework lane with a companion subsystem.
nvidia-smi reports memory.total as [N/A] on a Grace-Blackwell SoC (no
dedicated VRAM counter) so the board showed nothing for spark. A unified
NVIDIA SoC now reports the machine RAM snapped to hardware capacity
(snap bound widened to 8 GiB — GB10 firmware reserves ~6.3 GiB), its sm
arch from compute_cap (12.1 -> sm_121), and the host CUDA toolkit +
driver versions ride the registration, mirroring the AMD rocm/hip pair.
An APU board figure now matches the Apple convention: the unified pool is
the MACHINE memory (128 GiB Strix Halo says 128 GiB), not the GTT tunable.
Kernel-visible MemTotal snaps up to the next 16 GiB DIMM capacity only when
the gap is a plausible firmware reservation (<= 6 GiB); a real carve-out
stays honest. evo: 118 GiB GTT / 124.4 GiB visible -> 131072 MiB.
Free /v1/admin/storage for the operator's S3 object-buckets view (a distinct
storage concern); the DO block-volume + datastore-fill fleet is /block-storage.
No consumers yet, so the rename is clean. Handler storage→blockStorage, file
storage.go→block_storage.go (history preserved). Pairs with the operator SPA
Block Storage page + console repoint.
A python function created with target=fleet invokes as an fn.run job on
the org gpu-jobs queue through the ONE embedded tasks engine (new
View.StartActivity/DescribeActivity seam, tasks v1.51.4): same queue,
same claim loop, same result a direct tasks-API submit gets. invoke
blocks bounded by TimeoutSec (900s ceiling, matching the sandbox);
longer jobs belong on the tasks API. Job outcome maps onto the sandbox
execResult contract so billing gate + metering + invocation recording
are identical across executors; fail-closed when the engine is not
ready. Also: functions suite gains the sibling TestMain encryption
harness (suite now runs on an encryption-capable build without CI env).
Translate the canonical /v1/event stream to each connected platform's
conversion schema and forward it server-side. GA4 (Measurement Protocol)
and Meta (Pixel + Conversions API) work end-to-end; X, LinkedIn, TikTok,
and Reddit are scaffolded against the same Destination interface.
- clients/destinations: the Destination interface + per-platform adapters
+ translator (canonical EVENTS -> normalized conversion) + per-org
registry (Base/SQLite, API secrets KMS-sealed) + the analytics fan-out
consumer. A destination may reuse an integrations OAuth token
(Meta CAPI <- meta_ads) or seal its own; PII match keys are SHA-256
hashed before send.
- clients/analytics: a one-way fan-out sink after the write core, handed
the RAW (pre-scrub) batch so a Conversions-API forwarder can hash the
match keys the warehouse drops; detached + fail-soft, no-op when unset.
- clients/guide: read the analytics funnel (GET /v1/guide/analytics + an
overview fold) and expose destinations_connect as an MCP tool.
Tests: translator mapping, GA4 + Meta send end-to-end (mock + PII
hashing), store tenant isolation, fan-out, and the connect seam.
Restore the fugu-ultra published_claim rows + the test fixture — fugu is just
another model we verify in the arena and can route to, like grok/gpt/opus. The
comments stay neutral (no 'disprove the rival' narration). The rule is only: no
fugu on the Enso MARKETING pages (hanzo.ai/enso — already clean). Reverses the
over-broad purge in 8b3aec0.
Remove the 4 sakana/fugu-ultra published_claim rows + scrub the two comments that
named the competitor (house rule: no competitor names in source). The public
leaderboard (unauth GET /v1/benchmark/leaderboard) no longer exposes it. Legit
arena models (grok/gpt/opus/fable etc., provider-reported) are unchanged. Removed
the fugu-ultra test fixture (594 attempts remain, test still >=500 green).
The data source for admin.hanzo.ai's realtime Block Storage board, so we can
watch the analytics datastore fill and scale DO storage before it runs out.
SuperAdmin only (the s.guard wrap). Two REAL sources, each degrading independently:
- Fleet inventory (count · total capacity · monthly cost · per-volume region +
attachment) from the DigitalOcean API — a new Volumes() on the existing
DO_API_TOKEN client (paginated). DO gives capacity + attachment but NOT fill %,
so a volume's used/pct stay ABSENT (the console renders "—", never a fabricated
number).
- The analytics datastore's own fill from ClickHouse system.disks (the 200Gi PVC)
over the SAME shared aiobject.DatastoreQuery the analytics + compute lenses use —
no second connection. This is THE number the operator scales on. Near-full
volumes raise an alert (warn ≥ 80%, critical ≥ 90%).
Honest by construction: DO unconfigured → empty fleet; datastore not connected →
no datastore card. admin creates no table and holds no storage state — it only reads.
Pure buildStorageSnapshot / alertLevel / datastoreFillFromRow are unit-tested
(fleet totals, cost, absent per-volume fill, the datastore card + alert, the
system.disks bytes→GiB→pct math, zero-capacity → nil). Pairs with console 8.4.151.
A linked node with uv claims fn.run jobs from the org gpu-jobs queue: an
inline Python script executed in an ephemeral uv run env on the node own
GPU stack, result = output tail + exit code + duration; nonzero exit
fails the activity with the traceback. Poison-loop guard becomes
per-lane, so a render-less node (evo) still claims its own lanes and
declines renders back to the queue. E2E-proven: submit via tasks API ->
evo claim -> uv run (rocm-smi visible) -> result GET.
Rebased onto origin/main (experiments primitive landed @541f1b4). Campaign A/B now
composes clients/experiments instead of a nil-seam placeholder:
- experiment.go: AssignFunc composes experiments.Assign (bucketing); AnalyzeFunc
composes experiments.Analyze (pull-model — it reads the metric from analytics
itself, no duplicate measurement). Both nil-safe: single-creative honest default.
- metrics.go: GET /v1/campaign/:id/metrics embeds the A/B analysis (abTest) when a
campaign runs >1 creative and an experiment is wired.
- apps/wire_seams.go: SetExperiment wires Assign+Analyze at the composition root
(campaign-linked experiments live in the org's default project).
Tests: assigned variant flows to the executor plan (utm_content); single-creative
never assigns; assign-error fails soft to the default creative.
Top-level go-to-market plane: a Campaign is a VALUE that spans channels; a
Channel is an orthogonal EXECUTOR it fans out to. Each channel consumes the
connector plane via integrations.TokenFor — the campaign object never touches a
credential.
- clients/campaign: Campaign model + store (per-org SQLite, cek at rest), the
Channel interface + injected-func registry, /v1/campaign surface (CRUD +
launch/pause fan-out + channels add/remove + metrics), experiment seam for
creative A/B (nil-safe).
- clients/ads/provider.go: the ad-network execution edge (LaunchPaid/PaidSpend/
PausePaid) — resolves the org's ad token via TokenFor, Meta executed for real,
fail-closed on a disabled connector. Standalone POST /v1/ads/campaigns/:id/launch.
- clients/analytics/campaign.go: CampaignMetrics — the ONE campaign-metrics query
over hanzo.events, org + utm_campaign bound positionally (tenancy invariant).
- apps/wire_seams.go: paid channel wired at the composition root (campaign never
imports ads, ads never imports campaign).
Tests: fan-out, connector-consumes-token (httptest Meta), tenant isolation,
connector-disabled-blocks-send, spend fan-in, analytics tenancy predicate.
A unified-memory APU now reports the cpuinfo marketing name, normalized
from BIOS shouting: AMD RYZEN AI MAX+ 395 w/ Radeon 8060S ->
"AMD Ryzen AI Max+ 395 w/ Radeon 8060S (gfx1151)". Model numbers keep
their casing; discrete cards and non-Ryzen hosts keep their names.
Business model: no trial credits. Onboarding no longer grants any starter
credit — onboardFirstRun drops the commerce GrantCredit call, the account
state drops the commerce client, and provision's result drops trialGranted.
The first-run path still drives the ONE atomic IAM provision (org + admin
move + hashed org-scoped credential); the org just starts at a zero balance.
Removed payout.Client.GrantCredit (its only caller was this funding path —
no dead code); the additive Deposit + the referral/affiliate/author programs
are untouched.
Test: first-run drives provision ONCE (owner/name from get-user, resolved
slug), no funding call; a retry converges.
/v1/experiments is the VALUE that composes three existing planes, never a fourth
engine. ASSIGNMENT = flags (deterministic subject->variant via engineEvaluate; no
second bucketing, no assignment store). MEASUREMENT = analytics (one org-scoped
hanzo.events query; the eventsWhere isolation invariant). EVIDENCE = research
(per-variant samples as kind=ab rows; two-proportion z significance is a pure
function over them). Lifecycle: create writes the multivariate flag def; assign is
a flags evaluation; analyze folds analytics outcomes per variant by joining each
subject to its flags assignment; decide promotes a winner by rewriting the flag
rollout to 100%.
Seams added (additive, one-way): flags.Assign/PutDef/GetDef, analytics.Outcomes,
research.Record/List. clients/campaign composes experiments.Assign/Analyze to run a
creative A/B without reinventing assignment or evidence.
Variant KIND is orthogonal: variant.payload can be a feature config, ad-creative
id, email subject, or model id.
- An APU whose VRAM is a token carve-out now reports its GTT unified pool
(Strix Halo: 1 GiB VRAM carve-out vs ~118 GiB usable) with unified=true;
discrete cards keep dedicated VRAM. Fixes evo showing 1 GB.
- gfx family naming: gfx1151 -> AMD Strix Halo (also Strix Point, Phoenix),
so the board says what the silicon is, not just the iGPU series.
- Inventory records the native gfx arch per GPU and the host ROCm + HIP
versions (/opt/rocm/.info/version, hipconfig), surfaced on /v1/fleet/workers.
- docs: bring-your-gpu.md converged on hanzo link (gpu connect is gone).
The production signup path (POST /v1/iam/onboard) did create-organization THEN
move-user as two non-atomic S2S calls — a fault between them orphaned the org
(the exact bug IAM's provision() fixed), and it minted no credential and
granted no trial. First-run onboarding now calls the ONE atomic
POST /v1/iam/admin/provision (service-token) instead: org + admin move +
hashed credential + trial claim converge, and a mid-flight retry resumes on
the founder's own org rather than orphaning it.
On the one-time trial grant (the once-per-verified-identity claim IAM
returns), it funds the identity's CANONICAL org via commerce
POST /v1/billing/credit with a per-identity idempotency key
(starter:<verified-email>) — so the per-org dedup is a per-identity dedup, the
anti-farm gate (one trial per identity, not per org). Funding is best-effort +
idempotent: a hiccup never fails an onboarding that already provisioned.
The additional-org flow (caller already has a home org) is unchanged
(create-without-move). A legacy fallback keeps the old pair when the service
token is unset, so a partial deploy still onboards.
Test: first-run drives provision ONCE (owner/name from get-user, resolved
slug), funds with the per-identity key + bounded 500c to the canonical org,
and a retry grants no second trial → never double-funds.
Deposit (POST /v1/billing/deposit) is additive — a retried onboard would
double-fund. GrantCredit posts the idempotent /v1/billing/credit: commerce
dedups on idempotencyKey, so the same key credits at most once. The starter
grant uses a per-identity key against the identity's canonical org, so the
per-org dedup is a per-identity dedup — the trial-abuse gate (one trial per
verified identity, not per org).
A *Client method only (not the shared Commerce interface), so the authors/
affiliates seams are unaffected.
Tests: posts /v1/billing/credit with the per-identity idempotencyKey +
bounded 500c + starter-credit tag + org namespace; unconfigured is an honest
error.
Canonical ordering used client ts first. But the backfill importer stamps a wall-clock ts
(~1.7e9) while the SDK live path sends none (ts=0), so an SDK correction/retraction (ts=0)
of a backfilled experiment (ts=big) sorted BEFORE it and NEVER superseded — the stale
value stayed canonical while the producer saw added=1 (the exact 91.4→69.7 stale-correction
case in the intended backfill+live-SDK workflow). canonicalExpWhere/AttWhere now order by
the server's monotonic append clock seq (latest-APPEND-wins, un-forgeable by a client); ts
is display-only (measured-at). datastore.go notes the deferred canonical OLAP read must be
seq-authoritative too (never argMax(ts)).
Cross-tool PoC (TestSupersessionIsSeqNotClientTS): ts=big backfill → ts=0 SDK correction
wins (69.7) AND ts=0 SDK retraction withdraws (canonical=0); attempt correction likewise;
all 17 tests green.
Red MEDIUM: meta_ads is documented read-only, but metaScopes carried
business_management — read+WRITE to Business Manager assets (users, ad
accounts, system users). Ad-account enumeration uses /me/adaccounts under
ads_read, so no read path needs it; the custodied ~60-day token no longer
carries business-asset write authority.
Harden the forbidden-scope guard to reject business_management and any
*_management / rw_* grant (not just ads_management), so a write scope cannot
regress. Add two tests: the short-token fallback (fb_exchange_token leg fails →
seal the short token with its real ~now+3600 expiry, no refresh) and a
cross-provider state-confusion negative (a meta_ads state replayed at
google_ads/callback is rejected as invalid state).
M1: the artifact sha256 was client-asserted + never verified (poisonable, hash-addressed
unearned). Now the caller submits the BYTES (base64 content); the SERVER hashes them and
that sha256 is the identity + ref (sha256:<hash>). A client-asserted sha256 must MATCH
(else 422); content is required; poisoning needs a preimage. Bytes are stored + retrievable
by hash (GET /v1/research/artifacts/:sha256, org-scoped). First-writer-wins resolved.
M2: Artifact.ref was an ungated file:// twin of the SSRF-gated endpoint. The server now
DERIVES ref = sha256:<hash> and ignores any client ref — no file:///etc/passwd can be
stored or served back.
M3: ssrfSafe comment corrected to best-effort INGEST HYGIENE (not the dial-time control);
added the ranges the stdlib predicates miss — CGNAT 100.64/10 (Alibaba IMDS
100.100.100.200), IANA protocol 192.0.0/24 (Oracle IMDS 192.0.0.192), 0/8, TEST-NETs,
198.18/15, 240/4, NAT64, v6 doc; named dialTimeGuardTODO as the greppable pin required
before any endpoint dialer ships.
17 tests pass incl content-address poisoning-rejected + blob round-trip + new SSRF ranges.
revision is now part of content identity (hashExp/hashAtt) so a retraction of an
otherwise-identical run is a DISTINCT retained version, not an INSERT-OR-IGNORE no-op.
Canonical redefined: THE single latest version by (ts,seq); if that latest is retracted
the id has NO canonical (withdrawn) — the NOT EXISTS no longer filters revision, so a
newest-retraction wins and no earlier version resurfaces. Also canonicalize hashed JSON
(sorted-key, whitespace-free) so a re-serialization never mints a spurious version (L3).
Red PoC: retract → withdrawn from canonical (0), retained keeps both (2); a later
correction restores canonical. 16 tests pass.
Add nine org-scoped connectors for the Business AI Guide's Account-Linking
step, on the same registry + KMS-sealed custody as google/github/cloudflare:
meta_ads OAuth Advertising Facebook/Instagram Ads + Pages (long-lived token)
google_ads OAuth Advertising reuses google.go plumbing, adwords scope
google_analytics OAuth Analytics reuses google.go plumbing, analytics.readonly
microsoft_ads OAuth Advertising Microsoft Advertising (Bing), refresh token
tiktok_ads OAuth Advertising TikTok for Business, durable token
reddit_ads OAuth Advertising Basic-auth client, refresh token
linkedin_ads OAuth Advertising read + reporting, optional refresh
warpcast apikey Social Farcaster via Neynar key (keyVerify)
whatsapp apikey Messaging WhatsApp Cloud API token + phone id (keyVerify)
Each is one declarative provider file over the existing Provider contract;
oauth_http.go is the shared bounded, token-free transport the form/JSON
exchanges reuse. Scopes are least-privilege (read/reporting; write scopes
added when a write path is wired). All AdminOnly — linking a client's
marketing account is an org-admin action. Tokens are read by the ads /
analytics / marketing readers via integrations.TokenFor(org, provider, name).
Tests: authorize/exchange against httptest stand-ins, KMS seal-before-row,
verify-before-store fail-closed, tenant isolation, org-admin gate, and
token-free errors/responses per provider.
The evidence plane (HIP-0512) was built + committed but orphaned — absent from
apps.Wire(), so it 404'd in prod and the board showed only its seed snapshot.
Wire it as a non-staged subsystem (mounts under the production mount-all default),
the arena's sibling right after benchmark, and mount its embedded R&D Ops Board at
/research (same-origin with the plane it reads).
- apps.go: import + {Name:"research", Mount, Shutdown} after benchmark
- wire_test.go: frozen sequence updated (research, non-staged, has Shutdown)
- research.go: mount the embedded board UI at /research (like tasks/ui)
- clients/research/ui: the board embed + serving test
go build ./cmd/cloud links (617MB); apps wire/staged guards + full research
suite green. Goes live at api.hanzo.ai/v1/research + /research on image deploy.
HIGH-2 (exec-kubeconfig RCE) + HIGH-3 (SSRF guarded the wrong field): one hardened
fold entrypoint. fleet.SafeRESTConfig — which dynFromKubeconfig/Register now funnel
through (so it also closes the shared visor.attachCluster BYO path) — REJECTS a
kubeconfig that carries an exec or auth-provider credential plugin (client-go would
run a local binary with the pod's full env) and SSRF-guards restCfg.Host, the ACTUAL
dial target the node-list hits (https-only; no loopback/private/link-local/IMDS/
unspecified/multicast). The guard now covers the kubeconfig SERVER, not the
discovery-reported endpoint (which diverges for DO/Azure). Bypass env
FLEET_ALLOW_PRIVATE_HOSTS for loopback test apiservers only.
HIGH-1 (GCP credentialJson SSRF/LFI/exfil): validateGoogleCredential fail-closes the
customer credentials JSON BEFORE google.CredentialsFromJSON — only a service_account
key is accepted (token_uri pinned to a google host); an external_account (WIF) config,
whose credential_source.url/file and token_url are attacker-chosen (pod SSRF /
arbitrary file read / token exfiltration, firing during verify), is rejected. Keyless
WIF becomes a Hanzo-owned config (follow-on), never customer-authored.
MED-4 (AWS region SSRF): a region is validated against ^[a-z]{2}-[a-z]+-\d+$ before
it is interpolated into the STS/EKS host, so "evil.com/" can never redirect the
assumed-role token off-account.
Venue's wrong-field endpoint pre-guard is removed (fleet is the one gate). Tests:
fleet SafeRESTConfig (exec/auth-provider/non-routable/non-https reject, public allow,
bypass); venue exec-kubeconfig-not-folded, server-diverges-from-endpoint-guarded,
hostile-external_account-rejected, validateGoogleCredential, validRegion. Fold-consumer
suites (visor/ml/fleet) green; cmd/cloud links. (Harness: fiber v3 Test timeout raised
off its 1s default so CI-box load can't flake the fold's TLS dial.)
Same discovery interface as DO/AWS/GCP, folding into the ONE fleet. An org
registers an Azure AD app (tenant + client): a client_secret drives the
service-principal flow; its absence selects KEYLESS Workload Identity Federation
(Hanzo presents its own federated OIDC token as a client assertion — no customer
secret stored). Discovery is plain ARM REST (net/http, no azure-sdk-for-go): GET
managedClusters per subscription, POST listClusterUserCredential for each
kubeconfig, fold. Tests cover both the SP and keyless-WIF paths (httptest AAD +
ARM stubs) plus token-free errors and keyless-seal.
The go:embed console (served BY the cloud binary) drives the Base product's
data plane at /v1/collections/* but has no Next BFF (pruned by build:embed), so
the calls hit cloud's native /v1 router — which had no such route → 404, and the
whole Base product (Bases manager + Records) went dark on console.hanzo.ai.
Add a principal-gated reverse proxy (clients/base/collections.go, the clients/o11y
pattern) that forwards /v1/collections[/*] 1:1 to the managed Base (base.hanzo.ai,
in-cluster Service; overridable via CLOUD_BASE_ORCHESTRATOR_URL/HOST) — the
in-binary replacement for the retired console BFF. cloud can't mint a hanzo.id JWT
(JWKS-only), so it forwards the caller's OWN Bearer; the managed Base validates it
and scopes tenants by the token subject. allowCollections reproduces the BFF's
least-privilege allowBaseSurface (collections data plane only; no Base admin tunnel).
Always on, before the embed gate. Tests: allow-list, director (Host+path+bearer),
principal gate (403 fail-closed).
An org links labeled DigitalOcean / AWS / GCP accounts (verified live, KMS-sealed,
keyless where possible); venue discovers each account's native Kubernetes clusters
and folds them into clients/fleet.Register — the same registry visor surfaces at
/v1/clusters and ml federates onto. No second cluster registry.
- DigitalOcean: PAT -> GET /v2/account, list DOKS clusters, pull each kubeconfig (godo).
- AWS: cross-account STS AssumeRole pinned by external id (confused-deputy), eks
List/DescribeCluster, k8s-aws-v1 presigned token. Hand-rolled REST + owned SigV4
(pinned by the canonical get-vanilla vector); no aws-sdk-go-v2 service trees.
- GCP: Workload Identity Federation or SA key via oauth2/google, container REST
clusters.list; no google container SDK / gRPC.
Multi-credential per org (labeled, /orgs/{org}/cloud/{provider}/{label}), org-scoped
and tenant-isolated, admin-gated mutations, SSRF guard on discovered endpoints,
per-new-cluster billing meter through the shared compute fee. One discovery
interface, thin providers behind it. Zero new module deps (godo + oauth2 present).
Extend the ONE breakdown endpoint with three org-scoped behavior lenses over
hanzo.events — answering WHERE people go, WHAT they look at, and WHERE they come
from — as ranked {key, pageviews, visitors, pct} lists alongside models/products:
- topPages GROUP BY path (WHERE event='$pageview')
- topReferrers GROUP BY referrer_domain, self/empty referrer bucketed "(direct)"
- topSources GROUP BY utm_source, empty utm bucketed "(none)"
Read-side only: no ingest/write-core/schema change. Each lens rides eventsWhere
so the org is bound POSITIONALLY (never interpolated) — same tenant boundary as
every other query. pct is share of the TRUE in-window pageview total via
`sum(pageviews) OVER ()`, so a top-N list honestly shows the long tail. Every
lens degrades to honest-empty (available:false, empty items, Debug log) when the
events table is absent/errored — never a 500 (mirrors the overview web lens).
visitors=uniqExact(distinct_id), pageviews=count of $pageview — identical to the
web overview lens.
Tests: breakdownSQL binds org positionally + filters $pageview + buckets
direct/none + window-fn total; buildBreakdown pct is share of in-window total
(not returned-rows sum); honest-empty on datastore error. gofmt+vet clean, no
go.sum drift.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the research-diary artifact record (raw-artifact retention class): a
dashboard-snapshot (PNG of the canonical board) or ai-report (generated page),
addressed by sha256 content hash — a re-POST of the same bytes is a no-op (the
hash-addressed idempotency). The bytes live at ref (blob store); this stores the
verifiable manifest with the same provenance (project, git_sha/branch/dirty,
lib_versions) as a run.
POST /v1/research/artifacts record one artifact (idempotent by sha256)
GET /v1/research/artifacts?run=&project=&since= diary feed, newest-first
Private by default; public only via the separate visibility grant (POST
/v1/research/grants with sha256), the same rule as runs; no implicit training/commons
rights. Rolls up to hanzo.research_artifact (append-only, keyed by sha256). 14 tests
pass.
Reshapes the AttemptStore per the updated ToS/DPA/Research-Supplemental terms:
- VERSIONED, not write-once. A correction APPENDS a new version under the same
stable id; the prior is RETAINED (superseded), never mutated. Version identity is
content_hash over the measurement AND its provenance, so an idempotent re-ingest is
a no-op while a corrected number or a run on a new commit/lib version is a new
retained version. revision ∈ {original,corrected,retracted}; canonical/superseded
are derived (latest non-retracted per stable id).
- RETAINED vs CANONICAL exposed as distinct counts everywhere (ingest response,
/totals, /projects). retained is the truth; canonical is the deduped view, so dedup
never reads as loss. faulted/failed runs are retained (negative results are evidence).
- PROVENANCE first-class + queryable: project (column), git_sha/git_branch (indexed),
git_dirty, lib_versions (structured JSON) — the longitudinal 'which lib version
regressed X' record.
- PRIVATE by default; visibility (private/org/public) + trainable + publishable are
each a SEPARATE authorized grant (POST /v1/research/grants), never implied by upload.
- Retention classes separable (nonpersonal metadata/provenance permanent; raw
artifacts own retention; secrets never stored). Honest durability label: versioned ·
append-only; cloud mirroring rolling out — NOT yet immutable/replicated/recovery-tested.
12 tests pass (pure-Go): idempotent-by-content, correction-appends-version,
provenance-distinct-versions, faulted-retained, private-by-default+grant,
projects/totals canonical+retained, SSRF, HTTP round-trip+provenance+tenant-isolation,
grant-separate-from-upload.
Adds the Hanzo Research surface (HIP-0512 §"Hanzo Research"): every experiment
across every product accrues as immutable, queryable evidence under one
discriminator (kind ∈ benchmark|kernel-perf|training|ablation|policy-eval).
Two planes: each org's transactional SQLite (per HIP-0302, physical file
isolation) is the durable source of truth; it rolls up best-effort into
hanzoai/datastore (ClickHouse) via the account-usage warehouse pattern for the
cross-project OLAP surface. A datastore outage degrades to rolled_up:false, never
a failed ingest.
Ingest is idempotent by stable id — attempt by (project,benchmark,item,model),
experiment by (project,<kind>:<subject>:<task>) with a ts-guarded upsert for
latest-run-canonical — so a corpus imports exactly once however many times it is
replayed (the no-data-loss guarantee) and an out-of-order replay of an older run
cannot regress a newer number.
Surface:
POST /v1/research/experiments ingest a batch → SQLite → roll up
GET /v1/research/experiments list, latest-run-canonical (?project= ?kind=)
GET /v1/research/projects every project + real totals (ops board)
GET /v1/research/totals headline aggregate + per-kind (?project=)
Org is the physical tenant boundary (validated principal, never a client field);
project is a first-class column so the org's ops board aggregates across its
projects in one query. BYO endpoint is SSRF-gated at ingest (loopback/private/
link-local/metadata denylist, https required). status is a stored run-state field
(default complete); the live leasing state machine is the durable-execution
increment.
Tests (9, pure-Go): idempotent double-ingest, latest-run-canonical + no-regress,
attempt immutability, project/totals aggregates, SSRF denylist, HTTP round-trip +
tenant isolation + fail-soft roll-up.
A non-text Workers AI modality (whisper audio, vision/classification image
bytes) yields no prompt text, so aiPromptText → "" → EstTokens → 0 → the
token-denominated BYO fee → 0. ResourceMeter.Gate short-circuits on a
0-cent cost BEFORE calling commerce Authorize, so a frozen / broke /
over-cap org got ungated, unbilled Hanzo-proxied inference — no balance
check, no debit row.
Root cause: a token-denominated fee cannot price a non-text modality. Fix:
floor the per-CALL fee. BYOInferenceFeeMicros now returns max(token fee,
BYOFloorMicros) with a non-zero default floor ($0.0001), so:
- the gate reservation is ALWAYS >= 1 cent → the gate ALWAYS runs → a
frozen/broke/over-cap org is REFUSED, never proxied;
- every /ai/run leaves a debit row >= the floor → no silent proxied call.
This closes F-1 and the F-2 unbilled-call gap (same root cause) together.
Also reorder aiRun: the balance/freeze gate now runs BEFORE account
resolution, so a refused org makes ZERO Cloudflare contact (no discovery,
no run). The floor is one shared, generic knob in the inference-billing
spine (CLOUD_AI_BYO_FLOOR_UUSD) — not a Cloudflare-specific path.
Tests: a broke org + a non-text model is refused 402 with zero CF contact
and zero debits; a funded text run whose model reports no usage still bills
the floor. BYO fee/floor math unit-tested with the floor isolated.
Deliver was the only run-start path with no bound. Add three orthogonal per-org
bounds in the ONE shared startRun, plus the loop/redelivery hardening:
HIGH-1 amplification (self-trigger loop / fan-out / hook-hammer):
- concurrency: orgRunLimiter now guards EVERY run-start (moved into startRun; runFlow
parity), full → 429.
- durable rate budget: CountRunsSince (persisted rows, survives restart) caps run-starts
per rolling minute (CLOUD_AUTOMATIONS_RUNS_PER_MIN, default 300), enforced BEFORE the
insert → a fan-out/loop hits the ceiling and stops.
- causation depth: TriggerEvent.Depth → FlowRunInput.Depth; Deliver refuses at
maxCausationDepth so an in-platform cycle terminates. /hooks reads X-Causation-Depth;
the seam carries depth.
MED-1: the github push webhook skips a bot/App-authored push (isBotActor "[bot]") — our
own outbound mirror pushes AS the App bot, so "on push → our push → push" cannot loop.
MED-2: engine readiness is checked BEFORE the run-row insert, and a transient start
failure DELETEs the row (not FAILED) — a not-ready/crash no longer burns the DedupeKey,
so redelivery retries instead of dropping the event.
LOW-1: /hooks with no X-Idempotency-Key content-hashes the body, so a hammer of identical
POSTs collapses to one run.
Tests (cgo + pure-Go, race-clean): TestDeliverLoopBounded (in-platform cycle terminates),
TestDeliverRateCapped (durable budget), TestDeliverEngineNotReadyRetryable (no key burn),
TestInboundHookContentHashDedupe, TestBotActorGuard.
Polish toward one-way + compose, no behavior change:
- armTrigger: the ONE trigger-SOURCE arrival plane (cron schedule for POLLING,
routing-index subscription for WEBHOOK/APP_WEBHOOK, nothing for MANUAL),
extracted out of setEnabled so status-flip and trigger-arrival are separate
concerns. The action chain is untouched here — any trigger source pairs with any
actions (a product, not a switch).
- startRun: the ONE way a firing turns (rule, run id, event) into a durable run —
persist-gate then dispatch, threading the event payload. Shared by the manual
/run and event Deliver paths; a cron tick still starts through the engine
schedule. Deliver collapses to pure match + dispatch; runFlow to a lean handler.
Tests green (cgo + pure-Go), race-clean.
Wires the existing verified provider webhooks into the automations Deliver plane
via a composition-root seam, so a real external event fires subscribed flows —
without integrations importing automations (which imports it for credential
custody).
- integrations.SetAutomationTrigger: a primitive-typed seam (nil-safe no-op when
automations is disabled), set once at the apps composition root to
automations.Deliver.
- github push webhook now also fires github/push to subscribed flows for the
signature-verified installation org (best-effort; ev.After is the dedupe key).
- stripe/channels reach the SAME Deliver with a one-line fireTrigger call.
Test: the seam passes the verified event verbatim, fail-closes on empty org, and
no-ops unwired.
Adds the IFTTT inbound plane to the ONE automations engine: an external event
(provider webhook, inbound message, scheduler tick) starts every enabled flow
whose WEBHOOK/APP_WEBHOOK trigger matches (source, event), threading the event
payload into the run as {{trigger.*}}.
- trigger.go: Deliver(ctx, org, TriggerEvent) — org-scoped match over the routing
index, per-org durable start via the shared engine (runStarter -> executeFlow),
at-most-once by DedupeKey (CreateRunIfAbsent gate), fail-closed on a missing org.
- store: automations_triggers routing index (org, provider, event -> flow+version),
maintained by setEnabled — the inbound analog of the POLLING cron schedule.
- engine: seed outputs["trigger"] so actions read the firing event.
- POST /v1/automations/hooks/:source/:event — authenticated inbound sink.
- each run emits ONE o11y event beside the exactly-once meter + audit.
- main_test.go: cek dev-key TestMain (mirrors clients/sync) so the suite runs under
cgo and pure-Go alike.
Tests: trigger->action fire + payload threading (real embedded engine), tenant
isolation, idempotency, fail-closed, HTTP org gate, enable/disable lifecycle.
LOW-1: cache the installation grant set per-installation-id (45s TTL) so a
status-polling console no longer re-enumerates up to 100 upstream repo pages
on every Pages request. Keyed strictly by installation id (never org name) and
TTL-bounded, so a reconnect gets a fresh key and a revoked grant cannot outlive
the TTL — the cache can never widen cross-tenant scope.
LOW-2: pagesErr now distinguishes a rate-limit 403/429 (x-ratelimit-remaining:0,
Retry-After, or a rate-limit body) — surfaced as 429 with the retry hint — from a
genuine permission 403 that keeps the actionable re-authorize message.
INFO-1: truncateBody redacts credential-shaped substrings (gh*_ / github_pat_ /
Bearer <value>) from any surfaced provider error body — defense in depth.
Adopts Red's github_pages_adversarial_test.go (two-installation cross-tenant +
injection/cname/token/confused-deputy matrices) and adds fast-follow tests for
the grant-cache keying/collapse, the 403/429 split, and the secret scrub. Full
integrations suite green under -race.
Five org-authed routes, siblings of the repo list/import, addressing one
repo as a resource on the same short-lived installation token:
GET /v1/integrations/github/repos/:repo/pages status + live URL + custom domain
POST /v1/integrations/github/repos/:repo/pages enable/configure (branch source or Actions)
PUT /v1/integrations/github/repos/:repo/pages set/clear custom domain, HTTPS, source
DELETE /v1/integrations/github/repos/:repo/pages disable
POST /v1/integrations/github/repos/:repo/pages/builds request a build
Org comes from the validated principal; the repo name is intersected with
the installation's granted set and the owner is taken from GitHub's own
full_name, so a caller can neither address an ungranted repo nor inject an
owner into the GitHub API path. The token rides only the Authorization
header. Fail-closed: unconfigured 503, unconnected 409, ungranted 404.
The benchmark subsystem was added to Wire() (native /v1/benchmark arena)
without updating the frozen mount-order golden, so TestWireOrderMatchesFrozen
fails on main (93 Wire specs vs 92 frozen). Add the missing entry in its
Wire() position (after evals). Mechanical refreeze, independent of the
Cloudflare asset plane in the parent commit.
Consolidate the per-org Cloudflare asset plane under the first-class
/v1/cloudflare/* prefix (sibling of /v1/dns, /v1/domain): connecting the
provider stays on the integrations plane, managing resources moves here.
Surface + core-manage each resource against the CF v4 API with the org's
own KMS-sealed token (fail-closed, tenant-isolated on the validated
principal):
- Zones + Analytics (read): list/get zones, zone traffic dashboard
- Pages, Workers (handlers unchanged, prefix relocated)
- Workers AI: POST /ai/run/{model} metered inference
- R2, KV (+ key value get/put/delete), D1 (+ query): wired for real
(were Phase-2 501 stubs)
Thin hand-rolled REST over the CF v4 API on net/http, decomplected to one
seam: client.send is the ONE authed request/read primitive (Bearer-only,
bounded, token-free); envelope unwrap (exec/cfDo/cfUpload), verbatim relay
(pass), raw value (getRaw) and the AI run (runAI) all compose over it, so
the request/auth/error dance is written once. acctClient/acctWrite are the
ONE auth+account preamble every account-scoped handler shares — each
resource handler is a thin, orthogonal body over those two seams.
Workers AI meters through the ONE unified usage/billing spine
(cloud.AIMeterProvider "ai") and emits the ONE gen_ai o11y span
(clients.StartGenAISpan), at the thin BYO fee since the org's own token
already paid Cloudflare for the compute. New shared, generic (not
CF-specific) inference-billing surface in metered_ai.go: BYOFeeBps /
BYOInferenceFeeMicros / MicrosToGateCents + exported AIMeterProvider and
EstTokens. The gen_ai span constructor is extracted from aihttp.go so
chat, embed and workers-ai share one.
Least-privilege connector scopes grown to match the now-wired surfaces
(R2/KV/D1/Workers-AI/Analytics). Model, key and id path segments are
validated (anchored charset + no-traversal) so no caller value can
smuggle path structure or a host into an upstream URL.
Tests (httptest CF stub + fake commerce): tenant isolation, the model
SSRF guard, the org-admin mutation gate, raw KV value relay, D1 query,
and the Workers AI debit landing on the unified "ai" spine.
Registers the research subsystem in apps.Wire() (after benchmark, before
treasury) with its Shutdown, and freezes its position — and benchmark's, which
closes a pre-existing Wire/frozen gap on main — in TestWireOrderMatchesFrozen.
Make POST /v1/event the single canonical ingest door: one handler
implementation, one write core (ingestEvents), serving every wire shape
and every auth context. The other five routes become thin aliases/shims
delegating to it.
- decodeIngest: ONE wire-tolerant decoder accepting a bare Event object,
a bare [Event] array, AND the {batch:[…]} | {events:[…]} envelope the
Segment/beacon/publishable paths speak — all yielding the SAME
[]CaptureEvent the write core consumes.
- eventTenant: auth is the orthogonal pluggable concern on the one door —
(1) IAM bearer, (2) write-only publishable key (pk_, folded in from the
old /v1/ingest), (3) out-of-band IAM access key; fail-closed 403 with NO
brand-host fallback on the canonical door.
- ingestBody: the ONE ingest core (tolerant decode → foldException → the
ONE write core). eventIngest, the site-host carve (eventWithOrg), and the
/v1/ingest alias all funnel through it; auth is the only per-door
difference.
- Site-host carve repointed to /v1/event (eventWithOrg forces org from the
resolved Site); isAnalyticsPath adds /v1/event so a published-site beacon
POSTing <host>/v1/event lands host-forced. The org-forced-from-host
adversarial invariant holds for /v1/event too.
- /v1/ingest is now a thin DEPRECATED alias of eventHandle (one-shot
deprecation log, $source=ingest tag); /v1/ingest/keys minting unchanged.
- /v1/analytics{,/batch}, /v1/tracker, /v1/insights/e kept as thin
foreign-protocol shims (external Segment/PostHog compat only) funnelling
through the SAME write core; captureWithOrg now delegates to ingestBody.
Write core and hanzo.events schema unchanged. Tests prove the three wire
shapes land identically, pk_/bearer/host auth contexts resolve the same
tenant into a byte-identical warehouse row, and the exact app beacon
{batch:[ev]} body is accepted on /v1/event via the site-host carve.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the Hanzo Research surface (HIP-0512 §"Hanzo Research"): every experiment
across every product accrues as immutable, queryable evidence under one
discriminator (kind ∈ benchmark|kernel-perf|training|ablation|policy-eval).
Two planes: each org's transactional SQLite (per HIP-0302, physical file
isolation) is the durable source of truth; it rolls up best-effort into
hanzoai/datastore (ClickHouse) via the account-usage warehouse pattern for the
cross-project OLAP surface. A datastore outage degrades to rolled_up:false, never
a failed ingest.
Ingest is idempotent by stable id — attempt by (project,benchmark,item,model),
experiment by (project,<kind>:<subject>:<task>) with a ts-guarded upsert for
latest-run-canonical — so a corpus imports exactly once however many times it is
replayed (the no-data-loss guarantee) and an out-of-order replay of an older run
cannot regress a newer number.
Surface:
POST /v1/research/experiments ingest a batch → SQLite → roll up
GET /v1/research/experiments list, latest-run-canonical (?project= ?kind=)
GET /v1/research/projects every project + real totals (ops board)
GET /v1/research/totals headline aggregate + per-kind (?project=)
Org is the physical tenant boundary (validated principal, never a client field);
project is a first-class column so the org's ops board aggregates across its
projects in one query. BYO endpoint is SSRF-gated at ingest (loopback/private/
link-local/metadata denylist, https required). status is a stored run-state field
(default complete); the live leasing state machine is the durable-execution
increment.
Tests (9, pure-Go): idempotent double-ingest, latest-run-canonical + no-regress,
attempt immutability, project/totals aggregates, SSRF denylist, HTTP round-trip +
tenant isolation + fail-soft roll-up.
Re-ship increment 3a (the subscription/DNS plan CHARGE authority). Restores the
mount reverted in 43f5656d7 after the v1.49.12 seed bug, now on the RED-approved
fix commerce v1.49.13 (fix/plans-seed-roundtrip):
- models/plan Metadata_ datastore:",noindex" (envelope round-trips; team.minSeats
no longer served null)
- bundle expansion builds a LOCAL zero-price childSnapshotPlan — never writes a
$0 partial row into the plan authority (fixes world-pro/world-team $0 under-charge)
- corrective boot seed force-corrects unmanaged partial rows to the embed while
leaving admin edits (Managed) authoritative
Mount mirrors the 2b catalog mount: planapi.AdminRoute(storeV1, commercebilling.
SeedRows) on the /v1 bundle + own /v1/plans in commercePrefixes so it reaches
commerce (requireSuperAdmin-gated), not the AI /v1/* 402 catch-all. PUBLIC read
stays GET /v1/billing/plans. Money-path suites green (TestProdPath_SeedReadbackAllFields,
TestProdPath_CorrectsPreexistingBadRows); cloud prefix tests green.
Published sites POST analytics beacons to <site-host>/v1/analytics (and
/v1/insights/e), but that hit the static site server and 405'd — while
/v1/base already routed via the sites base-carve. Mirror the base carve
exactly for analytics ingest, so a page on yadota.hanzo.app (or a bound
custom domain) can send its own beacons into hanzo.events.
sites: add analyticsHostHandler + SetAnalyticsHostHandler + isAnalyticsPath
(prefix /v1/analytics OR ==/v1/insights/e). In Middleware, for both the slug
host and the custom-domain branch, carve the ingest POST to the handler with
org = resolved Site.Org (server-supplied, host-derived — never the caller/
body), gated on method POST so the authenticated GET read lenses
(/v1/analytics/overview|timeseries|top on api.hanzo.ai) are never hijacked.
The base carve stays first and independent.
analytics: extract captureWithOrg / insightsWithOrg — the Segment/beacon and
PostHog decode+ingest cores with an EXPLICIT org — so the deprecated aliases
(org via captureTenant) and the site-host carve (org forced from the Site)
share the ONE write core (ingestEvents), no copy. build() installs the carve
via sites.SetAnalyticsHostHandler, gated by the same already-existing flag the
anonymous ingest path uses (CLOUD_ANALYTICS_PUBLIC_CAPTURE, default ON); off
⇒ a site host 405s a beacon POST, unchanged.
Tests: sites carve routing (slug + custom host force Site.Org, not body/header
claim; GET serves static; /v1/base still routes to base; self host Continues;
non-live 405; no-handler 405; isAnalyticsPath). analytics end-to-end through
Mount+sites.Middleware (forced site org, 503-not-403 discriminator; empty
batch 200; custom domain; GET not hijacked; non-site host uses the normal
gate; disabled when public capture off).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The render mirror sweeps every local studio output into the org library
with the WORKER token, so on a box whose login org differs from the
content org it misroutes files; until now the only disable was pointing
--studio-url at a dead port. --mirror=false is the honest switch; default
true keeps current behavior. Ported from feat/per-gpu-queue 9468354
(superseded branch — the gpu command family there became link on main).
Belt-and-suspenders on the brand apex: New() now filters IsReserved labels out of the
first-party allowlist (logging the drop), so an operator setting CLOUD_SITES_FIRSTPARTY=
login (or api/wallet/console/…) can never turn an auth-sensitive host into a publishable
site. Default cd,flow,gallery are clean and unaffected. TestFirstPartyDropsReserved pins it
(cd/flow/gallery serve; api/login/wallet dropped). Closes RED's last actionable finding —
F-1 (org-pin) already fixed in ba9330c; host-parsing + Base-path RED-confirmed sound.
RED review of the first-party allowlist found the real risk: siteSlug returns a bare
slug and serve() resolved it via ResolveUniqueLiveSlug (the unique-LIVE-slug-across-ALL-
orgs fallback). So cd.hanzo.ai → Resolve('cd') → a CUSTOMER's project named 'cd' could be
served on our internal brand host — and via the OAuth-redirect path in reserved.go that is
account takeover. The allowlist gates WHICH labels serve; it did NOT pin WHOSE project.
Fix: first-party resolution is PINNED to the owning org (hanzo). siteSlug now returns a
firstParty flag; serve()/resolveLivePinned route a first-party host through the new
Resolver.ResolveOrg → store.ResolveOrgLiveSlug (WHERE org=? AND slug=? AND status='live'),
which can NEVER return another org's project. Multi-tenant hanzo.app is unchanged (still
unique-across-orgs). Fail-closed: no owning org (CLOUD_SITES_FIRSTPARTY_ORG, default hanzo)
⇒ first-party disabled entirely.
TestFirstPartyOrgPinned proves it: a first-party host calls ResolveOrg(hanzo,slug) and NEVER
the unique-slug Resolve; a multi-tenant host does the opposite. tsc/build green.
The shipped hanzo is the Rust CLI (~/work/hanzo/cli, curl hanzo.sh); the Go
cmd/hanzo was only ever a local 'make hanzo' control-plane build (CI/release
already build only cloud). Remove the make target so cloud builds exactly the
one stateless unified-API binary. cmd/hanzo + cli/ stay as reference for the
still-to-port client-side tools (GPU fleet worker link, runner, engine,
security); the code wrapper + zen-tier 1M mechanism are now in the Rust CLI.
Claude-Session: https://claude.ai/code/session_01QSN1woYbvENByMbGUqQ9Me
Converge cd/flow/gallery.hanzo.ai off the legacy s3://cdn staticFiles plane onto the
Projects PaaS plane (clients/sites → hanzo-sites/hanzo/<slug>). The site router served
only <slug>.hanzo.app (user sites); hanzo.ai is a SelfDomain it never resolved.
hanzo.ai is INTERNAL-ONLY (users get <slug>.hanzo.app), so it uses the OPPOSITE security
model to the multi-tenant apex: sites are OPT-IN via an explicit allowlist
(CLOUD_SITES_FIRSTPARTY=cd,flow,gallery on CLOUD_SITES_FIRSTPARTY_APEX=hanzo.ai). ONLY an
allow-listed label serves; EVERY other hanzo.ai host (api/console/iam/kms/world/chat/…,
listed in reserved.go or not) falls through PROTECTED by default — so a first-come project
can never shadow a real internal host (the reserved.go OAuth account-takeover). No
denylist-completeness burden on the brand's own domain. hanzo.app (user sites) unchanged.
Tests: TestSiteSlugFirstParty pins the boundary (allow-listed serve; api/iam/kms/world/
unlisted all protected; dotted key can't match; hanzo.app unaffected). tsc/build green.
Architecture per CTO directive: read/leaderboard/compare/worker go through the
AttemptStore interface — fileStore is LOCAL-DEV (DataDir JSONL); the cloud backend
(relational + object store, stateless API) swaps in behind the same interface, no
handler change. NEVER pod-local writes in prod.
- store.go: AttemptStore interface + fileStore (append-only, idempotent by item×model)
- runner.go: execution worker (OpenAI-compatible call → extract → score → Append,
cache-before-spend via Has); items from the evaluator plane
- presets.go: POST/GET /v1/benchmark/presets — design-your-own router blend (enso-<name>)
- benchmark_test.go: PROVES on real attempts — fable-5 measured 81.3 vs published 94.6
(gap +13.3), grok vs gpt-5.6-sol paired n=198 net+5 McNemar p=0.30, exact-McNemar unit
Fugu Ultra GPQA claim 95.5 vs our harness 94.4% (answered) is the flagship
replicate-or-disprove. Builds clean; unified binary links.
Sibling to /v1/eval, mounted into the unified cloud binary (apps.go). The benchmark
ARENA: run the top-14 canonical public benchmarks against any model/endpoint under ONE
standardized harness, and replicate-or-disprove a published claim. Provenance-first,
never blended: published_claim (vendor-reported) vs hanzo-measured (our harness) are
separate planes — the gap is the signal (Fugu claims GPQA 95.5; our harness measures
~92-95 by fault accounting; fable-5 claims 94.6, we measure 81.3).
Routes: GET /catalog (top-14, native flags), GET /leaderboard?benchmark= (per-model
measured ∥ published, coverage-aware), GET /compare?a=&b= (paired common-set +
rescue/damage + exact McNemar — the only valid arm-vs-arm test), POST /runs (BYO model
or endpoint; cache-before-spend; execution worker follow-on). Builds clean; unified
binary links.
Every new project now gets analytics collection and a Base data space
(form/forum/data submissions) wired by default — no opt-in — through the ONE
create path.
- createProject applies setProjectDefaults(analytics ON unless the caller passes
analytics:false; space_id = "<org>/<slug>") then best-effort provisions the
Base space. /v1/sites (ensureProject) routes through the same helpers, so
defaults live in exactly one place.
- Fail-soft: a Base provisioning error (incl. embed disabled) is logged and
swallowed — it never fails project creation, mirroring the CF edge-purge policy.
- clients/base.EnsureSpace: idempotent, in-process (no HTTP) provisioner that
opens the org's per-org Base app and ensures the public-create "submissions"
collection exists.
- store: additive analytics/space_id columns (migration matches the existing
cache_control/last_purge_at ALTER pattern); analytics defaults ON for existing
rows too. Exposed on the project view as analytics/space.
Tests: analytics default-ON, analytics:false opt-out, space provisioned on
create, and provisioning failure is fail-soft (create still 201, persists).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Increment 3a — the subscription/DNS plan CHARGE authority. Mirrors the 2b catalog
mount: register commerce's plan SuperAdmin CRUD (api/plan -> planApi.AdminRoute)
on cloud's /v1 bundle group, injecting the embed seed source
(commercebilling.SeedRows — the SAME @hanzo/plans embed the boot seed +
resolveSubscriptionPlan read), and own /v1/plans in commercePrefixes so it reaches
commerce, not the AI /v1/* 402 catch-all.
commerce v1.49.12 (feat/plans-sot cherry-picked onto v1.49.11): models/plan
authority + runPlansSeed (env COMMERCE_PLANS_SEED, idempotent/count-gated) +
resolveSubscriptionPlan reads the editable authority with embed fallback. The mint
gates score the IMMUTABLE embed (paidTier/IncludedMonthlyCents), NOT the DB price —
an admin edit never moves a charge gate. Seed == embed (TestSeededPricesEqualEmbed),
so first boot changes NO charge. Each handler requireSuperAdmin-gated (anon -> 403).
- Add POST /v1/projects/:slug/purge (mirrored at /v1/platform/sites/:slug/purge):
resolve (org,slug), flush the edge cache-tag site-<org>-<slug>, stamp
LastPurgeAt, persist, return 200 Project. Never touches the S3 origin. Org-scoped
exactly like deploy (403 no org, 404 unknown slug). CF-unconfigured/failing purge
is non-fatal (stamps + 200), matching deploy's behavior.
- DRY: extract purgeTag (the ONE cf.PurgeTags + cache-tag derivation + failure
policy) and purgeEdge (purge + stamp). deploy(onPublish), setDomains, del, and the
new purge handler all route through purgeTag — no copy-pasted CF call.
- Cache-Tag on served responses already emitted in sites.streamSite (unchanged);
tightened its comment + CacheTag doc to state the emit/purge pairing invariant.
- Scrub stale svc suffix from prose (6→0): projectsvc→projects, tasksvc→tasks,
cloud-mlsvc comment reworded; zapsvc test fixture → zap. msvc (third-party) left.
- Enforce one vocabulary in touched comments (project / deployment / S3 origin /
edge cache-tag / purge); no behavior change.
- Tests: purge_test.go — stamps LastPurgeAt, CF-unconfigured→200, 403/404 scope,
S3 origin untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One key-verify mechanism (keyVerify) plus the payment, model, messaging, and
SaaS connectors an online store or AI startup needs, on the user-scoped
/v1/connectors plane. Each provider is declarative data — origin, path,
credential placement, offline checks — over a single fail-closed, token-free
verify; a new connector is a registration, not another hand-rolled HTTP dance.
Providers: stripe, paypal, square, shopify; gemini, groq, mistral, cohere,
together, replicate, huggingface, openrouter, xai, fireworks, deepseek,
pinecone; sendgrid, resend, postmark, mailchimp, klaviyo, twilio; hubspot,
notion, linear, airtable.
A customer key seals to KMS only on a 2xx from the provider's cheapest
authenticated read; non-2xx and transport errors store nothing and never echo
the credential. authClient no longer follows redirects, so a 3xx fails closed
and custom credential headers are not forwarded across a hop.
Pulls the free-tier auto-routing flash cap (ai#111): a CONFIDENT free-tier org's
`auto` is confined to the flash pool (blended-price ceiling), so a non-paying
caller is never routed to a premium model. Enso stays the default; paid/trial,
explicit model, and X-Max-Cost still win. Single-commit jump; cloud builds clean
against v1.830.0 (CGO_ENABLED=0).
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Periodic reconcile scheduler (clients/sync) that keeps every poll sync current (env-gated CLOUD_SYNC_RECONCILE_INTERVAL, leader-safe on the Recreate singleton writer, bounded by the shared reconcileSem), plus OrgStore.Each cross-org sweep primitive and store.ListAll. gitToken now falls back App-token -> GIT_MIRROR_TOKEN -> anonymous instead of hard-failing, so github reconciles (public immediately, private once GIT_MIRROR_TOKEN is set) stop freezing. Tests: scheduler_test, orgdb_test TestOrgStoreEach, git_provider_test TestGitTokenFallback; go vet ./... clean.
admin.hanzo.ai's editor (console v8.4.154) CRUDs /v1/catalog/entries, but
commerce's setupRoutes wires only /v1/commerce/* — the /v1/catalog CMS
(api.Route -> catalogApi.AdminRoute, standalone-only) 404'd in the co-resident
binary. mountCommerce is the adapter for exactly these setupRoutes-skipped
families: mount catalogApi.AdminRoute on the existing /v1 bundle group (storeV1:
AddHost+RequestContext+errorScope+IAMTokenRequired — the same chain the
standalone /v1 bundle uses), and own /v1/catalog in commercePrefixes so it
reaches commerce, not the AI /v1/* balance catch-all.
No commerce module change: catalogApi.AdminRoute already ships in v1.49.9. Each
handler is requireSuperAdmin-gated (anon -> 401/403, cross-tenant data — never
org-scoped). Editor unchanged; closes the admin-edit half of the loop.
Picks up the #146 sibling fix: the co-resident billing READS (ListInvoices,
ListBillingSubscriptions, ListPayouts, GetPaymentConfig, DownloadInvoicePDF) no longer
nil-deref-panic (→502) on the embed path when no org is in Locals — they resolve the org
via nil-safe GetOrganizationOK. Compile-verified clean:
go build ./apps/... ./clients/account/... ./clients/commerceclient/...
Picks up the #146 fix: GET /v1/billing/spend-alerts (the console Budgets page) no
longer panics (→ 502) on the co-resident embed path when no org is in Locals.
commerce ListSpendAlerts + its CRUD siblings + billingSubject now resolve the org
via the nil-safe GetOrganizationOK. Compile-verified clean:
go build ./apps/... ./clients/account/... ./clients/commerceclient/...
The team.guests cap ran only at selectWorkspace (the guest's later login),
never at sendInvite — the actual guest add. A workspace admin could invite
guests past the plan cap and the add itself was ungated. Call the SAME
entitle seam at the add point, OBSERVE-MODE consistent with the login gate:
it logs the over-cap denial and admits today, and the 402 arms the day
enforcement returns. Commerce/plans errors admit inside entitle, so a
licensing outage never bricks an invite. Tests prove the gate runs at the
add point (commerce + plan cap consulted, genuine over-cap rank) yet admits,
and that an infra error still records the invite.
createContent seeded a brand-new doc's live update log with a bare
Get(miss)-then-Put. In the gap between the miss and the Put, a concurrent
first edit on the live WS lane could land and then be overwritten by the
seed (lost update), or the seed could overwrite a log that just went live.
Seed THROUGH the collab hub (seedIfAbsent): it reuses the room's flushMu/mu,
so the empty-check and the set are atomic under the room lock and the seed
and the field's live editor serialize on ONE room. The seed applies only
when the log is still empty — never clobbers a live edit. Deterministic +
concurrent (-race) tests prove the live edit always survives.
EnsureWorkspace was check-then-insert (WorkspacesOf -> INSERT) with no
uniqueness on the personal-workspace identity, so two concurrent logins for
the same (org, account) could each see 'none' and mint a duplicate personal
workspace (reproduced: 32 concurrent logins -> 6 rows). Add a partial-unique
index on (owner_org, owner) — EnsureWorkspace is the sole creator and always
writes owner=account — with a dedup that converges any pre-existing duplicates
to the earliest row, and make the create an idempotent upsert (ON CONFLICT DO
NOTHING then adopt the winner). Concurrent EnsureWorkspace now yields exactly
one row. Heal logic extracted to adoptExisting, reused on both paths.
Seats swallowed the row Scan error, so a genuine DB failure reported (0, 0)
— indistinguishable from an org with no members, under-counting billed
seats. Propagate the error (the aggregate COUNT always returns one row, so
any Scan error is a real failure); the wallet's /billing/plan read now
502s honestly and retries instead of rendering a truthful-looking 0.
DECOMPLECTION. The operator SPA (admin.hanzo.ai) authenticates via /v1/signin (a
cloud PKCE session → the X-User-* principal the gateway/middleware mints) but read
its IDENTITY from /v1/get-account, which the embedded IAM (casibase) answers from ITS
OWN session cookie. A PKCE session is not a casibase session, so get-account returned
owner:"hanzo" (anonymous) or "Unauthorized operation" — and the SPA SuperAdmin gate
(owner==admin && isAdmin), reading that, bounced the operator UI to login even though
the SAME session got 200 from every /v1/admin/* route. Identity source and auth source
disagreed: two session models for one surface.
Now identity IS the principal. AccountFromPrincipal (registered after IdentityMiddleware,
before MountAll's casibase mount) answers /v1/get-account from X-User-Owner/Name/Email/
IsAdmin when a VALIDATED principal is present (owner = HOME org, so a SuperAdmin
org-switched into a tenant stays one); with no principal it falls through to casibase
unchanged (anonymous sign-in + legacy casibase-session callers untouched). One truth,
additive, fail-open. Fixes the operator console UI being unusable via browser login.
The commerce S2S fallback set available = balance whenever the reported
available was 0, so a fully-held wallet (balance == holds, available 0)
rendered its entire balance as spendable — money the prepaid gate would
refuse. Derive available as balance NET OF holds when commerce reports only
a balance, clamped at 0 (holds beyond balance never go negative). Extracted
as spendableCents with a table test; a fully-held org proves the handler.
A workspace invite passed the body role straight into the org-level IAM
grant, so a single-workspace admin could make an invitee an org IAM admin/
owner (trusted elsewhere via IsAdmin). Cap the org grant at member; the rich
role stays workspace-scoped on the local roster row. Test proves the cap.
Two defects in the single-replica collab hub (collabws.go):
1) BLOCKING race → 'panic: close of closed channel' + registry corruption. join
registered its peer in rm.peers AFTER releasing h.mu, so in that gap a concurrent
leave of the last old peer would GC the room (delete h.rooms[key] + close(rm.stop),
ending the flusher). The resurrected room then panicked on its next leave
(close of an already-closed channel) and could evict a live room (split-brain).
This is the ordinary 'reconnect by the sole editor' shape. Fix: register the peer
while STILL holding h.mu, before the slow VFS load — a concurrent leave can no
longer see the room as empty. Lock order stays h->rm. A failed first-open now
undoes the join via leave() so it leaks neither the peer nor the room+flusher.
2) Silent lost updates on flush. flush cleared rm.dirty BEFORE the Put and discarded
the Put error, so a transient VFS failure dropped the buffered window (dirty already
false => neither the ticker nor last-leave retried). Fix: clear dirty only on a
SUCCESSFUL Put, re-arm it on failure, and serialize the persist (flushMu) so two
overlapping flushes cannot reorder their Puts.
TDD: TestCollabHubConcurrentJoinLeaveNoPanicNoLeak hammers join+leave on one doc from
500 goroutines under -race (reproduces the panic + a room leak without the fix, clean
with it). TestCollabFlushRetriesAfterPutError proves a failed Put keeps the room dirty
and nothing is persisted, then a later Put persists and clears dirty.
ledgerFinance.Balance discarded store.Balance's error (bal, _ := ...), so a real
read failure — a DB Scan error or a corrupt/garbled stored-balance ParseInt failure —
returned (zero, nil), indistinguishable from a genuine zero. That defeated the
documented 'a balance that cannot be read is unknown, and unknown is not broke'
invariant every caller relies on: /v1/billing/balance and /v1/finance/balance showed
a FUNDED customer $0.00, and the AI prepaid gate (metering.fetchAvailable) refused a
funded org's request. Propagate the error; the callers' guards already render 503 on
a non-nil read error. Direction was already safe (zero only under-reports), so this is
correctness + observability, not an over-charge.
TDD: TestBalanceReadErrorSurfaces closes the store's DB out from under the cached
handle and asserts Balance now returns an error, not (0,nil). Fails before, passes after.
The codexLike provider hardcoded '-c model_context_window=262144' (+auto_compact
235929) for EVERY model, so enso / zen5 — which the gateway serves at 1M
(ai flagshipWindow pin, live) — were capped at 256K in the 'hanzo code' Codex
wrapper. That surfaced to the user as 'maximum context exceeded' at 262144 even
after switching to zen5-pro.
Make the window model-aware: codeContextWindow(model) → 1M for the enso/zen5
flagship tiers, 131072 for flash; auto-compact at 90%. Threads the served model
into provider(base, model). Only codexLike sets provider (codex has no carrier,
so its model IS the served zen id). Tests updated + a sizing test added.
Claude-Session: https://claude.ai/code/session_01QSN1woYbvENByMbGUqQ9Me
The deprecated-alias 'log once per path' used a package sync.Map (Go's HashTrieMap),
which entered a 'ran out of hash bits while inserting' panic state under the hot
ingest path — so the Recover middleware turned EVERY POST /v1/analytics (and /batch,
/tracker) into a 500. Events stopped landing in hanzo.events once the map degraded.
The alias set is tiny (3 paths); swap the sync.Map for a mutex-guarded map — no trie
state to corrupt, cannot panic. Behavior unchanged (still logs each alias once).
go build ./... green.
The agent-run path made ONE completion call; when the default agent model
(deepseek-v4-flash) returned a transient upstream 429 'Platform overloaded'
(~1 in 3 under load), the run recorded an error and the bot reply was dropped.
- types: add ErrUpstreamBusy sentinel — the shared vocabulary for a transient,
safely-retryable upstream failure (429/5xx/empty-choices/'overloaded').
- aihttp: classify + tag transient chat-completion failures with ErrUpstreamBusy
(errors.Is-detectable); permanent errors (400/auth/unserved model) stay
untagged and fail fast. Message preserved; no control-flow change for
non-agent callers (interactive chat untouched).
- agents/executeRun: bounded retry (3 attempts, equal-jittered backoff, ctx-aware)
on ErrUpstreamBusy, then ONE failover to the reliable model (CLOUD_AI_FALLBACK_MODEL,
default 'best') if the agent's own model stays throttled. Retrying a completion
is side-effect-free, so metering still debits EXACTLY once on the eventual
success (runAgent meters only r.Status==ok) and never on failed attempts.
- Bill the model ACTUALLY used (r.Model) — a failover run bills 'best', not the
throttled model it started on.
- Scoped to the autonomous agent/bot run path ONLY; interactive user-facing
chat/completions behavior is unchanged.
TDD: 429-twice-then-200 -> one ok run + one debit; persistent 429 + failover
exhausted -> clean error run + no debit; failover-success bills the used model;
transient-classifier unit test (429/503/500/empty-choices busy, 400 not).
Default stays --normalvram (safe for small BYO GPUs); big-memory boxes (GB10
128G unified) can set HANZO_STUDIO_VRAM=--highvram so the render backend keeps
models GPU-resident. Additive + opt-in; other workers unchanged.
cloud's last tie to the retired Casdoor/Beego fork (hanzoai/iam-v1) is gone:
- clients/iam: embeds the clean hanzoai/iam (zip-native + hanzoai/orm) via
iamserver.Mount over its own SQLite store; fail-closed 503 on boot failure.
Removes the dual-impl identitySpec (CLOUD_IAM_IMPL=iam2 branch) + clients/iam2.
- clients/platform + clients/deploy: read the IAM-owned Project resource via
iam/pkg/store over the embedded DB() instead of iam-v1's object-store ormer.
- cmd/hanzo: retires the `hanzo iam` Casdoor daemon subcommand.
- go.mod: the last transitive iam-v1 edge was hanzoai/ai/object; bump ai
v1.829.3 -> v1.829.4 (Casdoor-free cut). go mod tidy drops iam-v1 entirely.
Remaining iam-v1 references are accurate "retired/gone" doc comments only.
Tests green: clients/iam, clients/platform, clients/deploy. Wire frozen-order
golden consistent (iam2->iam collapse).
Co-authored-by: zeekay <ai@hanzo.ai>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The embedded console served index.html for EVERY non-asset path, so a deep
load of /auth/callback hydrated '/' instead; AuthGate discarded the ?code and
bounced to /signin — sign-in could never complete on the embedded console.
Now an extensionless path tries the static-export route shell (<route>.html)
first, with the index treatment (no-cache + white-label title rewrite, one
shared brandTitle); direct .html requests are also no-cache.
Router-config routes (/v1/router/{policy,defaults,ledger,rewards,artifact-meta}
and /v1/org/settings) now serve over api.hanzo.ai. The v1.829.5 refactor moved them
to ZAP-native handlers and deleted their beego routes, but :8000 is the beego
web.Router — the ZAP registry backs a separate transport — so they 404'd in prod
(console Router->Policy, chat/app routing defaults). v1.829.6 adds RouterConfigBridge
(one beego adapter dispatching in-process through the SAME gateway registry, so the
ZAP handler stays the sole impl) and restores their isBalanceExempt entries.
detectGPUs now reports AMD accelerators as first-class resources alongside NVIDIA
and Apple Metal — discrete Radeon cards and gfx APUs alike (evo's gfx1151 Radeon
8060S on the RYZEN AI MAX+ 395). Resolution order: rocm-smi --showproductname (name
+ gfx target), then the kfd topology under /sys (GPU nodes by simd_count, gfx from
gfx_target_version), then a vulkaninfo summary; VRAM filled from amdgpu sysfs
mem_info_vram_total. Pure parsers are unit-tested against evo's real rocm-smi CSV
and kfd properties; the gfx_target_version decode (110501 → gfx1151) is covered.
The unified Go binary is hanzo dev — the unified Hanzo Go binary
Usage:
hanzo <command> [flags]
Control commands (gcloud/doctl-style):
agent invoke a managed Hanzo agent to run a task (headless)
apps list/get the platform apps board (declared/running/drift)
auth manage authentication + stored identities (login, logout, whoami, list, switch, token)
bot launch a computer-using agent (booted desktop or terminal)
build enqueue a platform-native build (runner fabric)
clusters provision/list/select dedicated DOKS clusters
code launch a coding agent (claude, codex, dev) on a Hanzo cloud model
config view/edit ~/.hanzo/config preferences
deploy drive a platform redeploy (rolling restart, zero-downtime)
engine run a local hanzo-engine (OpenAI + Anthropic model server)
k8s deploy-target helpers (current target)
link bring this machine into the Hanzo cloud fleet as a node (fabric + compute worker)
login authenticate against Hanzo IAM (hanzo.id) and store a token
logout remove stored credentials
run launch a workload on Hanzo compute (container or function)
runner run this machine as a JIT CI runner for your org (GitHub Actions)
security scan files for hardcoded secrets (local guardrail; no server/auth)
status show the org's fleet — every node with each of its GPUs
unlink take this machine out of the fleet (deregister + stop hanzod)
whoami show the current identity from the stored token
Service subcommands (server mode):
account serve the account subsystem standalone
account-bridge serve the account-bridge subsystem standalone
admin serve the admin subsystem standalone
admission serve the admission subsystem standalone
ads serve the ads subsystem standalone
affiliates serve the affiliates subsystem standalone
agent serve the agent subsystem standalone
agents serve the agents subsystem standalone
agentskills serve the agentskills subsystem standalone
ai serve the ai subsystem standalone
analytics serve the analytics subsystem standalone
audit serve the audit subsystem standalone
authors serve the authors subsystem standalone
authz serve the authz subsystem standalone
automations serve the automations subsystem standalone
base serve the base subsystem standalone
billing serve the billing subsystem standalone
bots serve the bots subsystem standalone
captable serve the captable subsystem standalone
catalogsync serve the catalogsync subsystem standalone
channels serve the channels subsystem standalone
cloud serve the full unified surface (all enabled subsystems, one listener)
cloudflare serve the cloudflare subsystem standalone
code serve the code subsystem standalone
commerce serve the commerce subsystem standalone
company serve the company subsystem standalone
content serve the content subsystem standalone
crm serve the crm subsystem standalone
dataroom serve the dataroom subsystem standalone
datastore datastore-fork analytics DB — not a Go serve target (see help text)
deploy serve the deploy subsystem standalone
dns serve the dns subsystem standalone
do serve the do subsystem standalone
domain serve the domain subsystem standalone
entitlements serve the entitlements subsystem standalone
evals serve the evals subsystem standalone
exec serve the exec subsystem standalone
flags serve the flags subsystem standalone
framework serve the framework subsystem standalone
functions serve the functions subsystem standalone
gateway serve the gateway subsystem standalone
git serve the git subsystem standalone
graph serve the graph subsystem standalone
guide serve the guide subsystem standalone
iam serve standalone Hanzo IAM (full Beego server: login UI, OAuth2/OIDC, LDAP/RADIUS)
ingress serve the ingress subsystem standalone
integrations serve the integrations subsystem standalone
kafka serve the kafka subsystem standalone
kms serve the kms subsystem standalone
knowledge serve the knowledge subsystem standalone
licensing serve the licensing subsystem standalone
link serve the link subsystem standalone
marketing serve the marketing subsystem standalone
marketplace serve the marketplace subsystem standalone
metrics serve the metrics subsystem standalone
ml serve the ml subsystem standalone
notify serve the notify subsystem standalone
o11y serve the o11y subsystem standalone
paas serve the paas subsystem standalone
plan serve the plan subsystem standalone
platform serve the platform subsystem standalone
plugins serve the plugins subsystem standalone
pricing serve the pricing subsystem standalone
product serve the product subsystem standalone
projects serve the projects subsystem standalone
prompts serve the prompts subsystem standalone
provisioning serve the provisioning subsystem standalone
pubsub serve the pubsub subsystem standalone
referrals serve the referrals subsystem standalone
rollingcap serve the rollingcap subsystem standalone
runtime serve the runtime subsystem standalone
sbom serve the sbom subsystem standalone
security serve the security subsystem standalone
settings serve the settings subsystem standalone
sign serve the sign subsystem standalone
social serve the social subsystem standalone
storage serve the storage subsystem standalone
sync serve the sync subsystem standalone
tasks serve the tasks subsystem standalone
team serve the team subsystem standalone
templates serve the templates subsystem standalone
tools serve the tools subsystem standalone
tracker serve the tracker subsystem standalone
treasury serve the treasury subsystem standalone
usage serve the usage subsystem standalone
validators serve the validators subsystem standalone
visor serve the visor subsystem standalone
wallets serve the wallets subsystem standalone
websearch serve the websearch subsystem standalone
world serve the world subsystem standalone
x402 serve the x402 subsystem standalone
zen serve the zen subsystem standalone
zero-trust serve the zero-trust subsystem standalone
Meta:
help show this message
version print version and exit
Flags are per-subcommand (e.g. `hanzo cloud --enable=iam,kms --brand=hanzo`,
`hanzo kms --listen=:8443`). Run a subcommand to see its config via env/flags. (HIP-0106): link composes fabric join (hanzo dev — the unified Hanzo Go binary
Usage:
hanzo <command> [flags]
Control commands (gcloud/doctl-style):
agent invoke a managed Hanzo agent to run a task (headless)
apps list/get the platform apps board (declared/running/drift)
auth manage authentication + stored identities (login, logout, whoami, list, switch, token)
bot launch a computer-using agent (booted desktop or terminal)
build enqueue a platform-native build (runner fabric)
clusters provision/list/select dedicated DOKS clusters
code launch a coding agent (claude, codex, dev) on a Hanzo cloud model
config view/edit ~/.hanzo/config preferences
deploy drive a platform redeploy (rolling restart, zero-downtime)
engine run a local hanzo-engine (OpenAI + Anthropic model server)
k8s deploy-target helpers (current target)
link bring this machine into the Hanzo cloud fleet as a node (fabric + compute worker)
login authenticate against Hanzo IAM (hanzo.id) and store a token
logout remove stored credentials
run launch a workload on Hanzo compute (container or function)
runner run this machine as a JIT CI runner for your org (GitHub Actions)
security scan files for hardcoded secrets (local guardrail; no server/auth)
status show the org's fleet — every node with each of its GPUs
unlink take this machine out of the fleet (deregister + stop hanzod)
whoami show the current identity from the stored token
Service subcommands (server mode):
account serve the account subsystem standalone
account-bridge serve the account-bridge subsystem standalone
admin serve the admin subsystem standalone
admission serve the admission subsystem standalone
ads serve the ads subsystem standalone
affiliates serve the affiliates subsystem standalone
agent serve the agent subsystem standalone
agents serve the agents subsystem standalone
agentskills serve the agentskills subsystem standalone
ai serve the ai subsystem standalone
analytics serve the analytics subsystem standalone
audit serve the audit subsystem standalone
authors serve the authors subsystem standalone
authz serve the authz subsystem standalone
automations serve the automations subsystem standalone
base serve the base subsystem standalone
billing serve the billing subsystem standalone
bots serve the bots subsystem standalone
captable serve the captable subsystem standalone
catalogsync serve the catalogsync subsystem standalone
channels serve the channels subsystem standalone
cloud serve the full unified surface (all enabled subsystems, one listener)
cloudflare serve the cloudflare subsystem standalone
code serve the code subsystem standalone
commerce serve the commerce subsystem standalone
company serve the company subsystem standalone
content serve the content subsystem standalone
crm serve the crm subsystem standalone
dataroom serve the dataroom subsystem standalone
datastore datastore-fork analytics DB — not a Go serve target (see help text)
deploy serve the deploy subsystem standalone
dns serve the dns subsystem standalone
do serve the do subsystem standalone
domain serve the domain subsystem standalone
entitlements serve the entitlements subsystem standalone
evals serve the evals subsystem standalone
exec serve the exec subsystem standalone
flags serve the flags subsystem standalone
framework serve the framework subsystem standalone
functions serve the functions subsystem standalone
gateway serve the gateway subsystem standalone
git serve the git subsystem standalone
graph serve the graph subsystem standalone
guide serve the guide subsystem standalone
iam serve standalone Hanzo IAM (full Beego server: login UI, OAuth2/OIDC, LDAP/RADIUS)
ingress serve the ingress subsystem standalone
integrations serve the integrations subsystem standalone
kafka serve the kafka subsystem standalone
kms serve the kms subsystem standalone
knowledge serve the knowledge subsystem standalone
licensing serve the licensing subsystem standalone
link serve the link subsystem standalone
marketing serve the marketing subsystem standalone
marketplace serve the marketplace subsystem standalone
metrics serve the metrics subsystem standalone
ml serve the ml subsystem standalone
notify serve the notify subsystem standalone
o11y serve the o11y subsystem standalone
paas serve the paas subsystem standalone
plan serve the plan subsystem standalone
platform serve the platform subsystem standalone
plugins serve the plugins subsystem standalone
pricing serve the pricing subsystem standalone
product serve the product subsystem standalone
projects serve the projects subsystem standalone
prompts serve the prompts subsystem standalone
provisioning serve the provisioning subsystem standalone
pubsub serve the pubsub subsystem standalone
referrals serve the referrals subsystem standalone
rollingcap serve the rollingcap subsystem standalone
runtime serve the runtime subsystem standalone
sbom serve the sbom subsystem standalone
security serve the security subsystem standalone
settings serve the settings subsystem standalone
sign serve the sign subsystem standalone
social serve the social subsystem standalone
storage serve the storage subsystem standalone
sync serve the sync subsystem standalone
tasks serve the tasks subsystem standalone
team serve the team subsystem standalone
templates serve the templates subsystem standalone
tools serve the tools subsystem standalone
tracker serve the tracker subsystem standalone
treasury serve the treasury subsystem standalone
usage serve the usage subsystem standalone
validators serve the validators subsystem standalone
visor serve the visor subsystem standalone
wallets serve the wallets subsystem standalone
websearch serve the websearch subsystem standalone
world serve the world subsystem standalone
x402 serve the x402 subsystem standalone
zen serve the zen subsystem standalone
zero-trust serve the zero-trust subsystem standalone
Meta:
help show this message
version print version and exit
Flags are per-subcommand (e.g. `hanzo cloud --enable=iam,kms --brand=hanzo`,
`hanzo kms --listen=:8443`). Run a subcommand to see its config via env/flags., delegated to the Rust CLI installed as hanzo code claude · /home/z/work/hanzo/cloud · start
model routing: on → api.hanzo.ai (prompts + code go here; usage metered to your org)
session stream: on → https://hanzo.bot/sessions/sess_4729b3a0a623c55d0d3b7b08fe8ebe35
resume: hanzo --resume 4729b3a0a623c55d0d3b7b08fe8ebe35) with compute-worker
registration (CPU cores+model, memory, each GPU), heartbeats, claims gpu-jobs.
unlink is idempotent; status renders the fleet with each GPU distinct. hanzo dev — the unified Hanzo Go binary
Usage:
hanzo <command> [flags]
Control commands (gcloud/doctl-style):
agent invoke a managed Hanzo agent to run a task (headless)
apps list/get the platform apps board (declared/running/drift)
auth manage authentication + stored identities (login, logout, whoami, list, switch, token)
bot launch a computer-using agent (booted desktop or terminal)
build enqueue a platform-native build (runner fabric)
clusters provision/list/select dedicated DOKS clusters
code launch a coding agent (claude, codex, dev) on a Hanzo cloud model
config view/edit ~/.hanzo/config preferences
deploy drive a platform redeploy (rolling restart, zero-downtime)
engine run a local hanzo-engine (OpenAI + Anthropic model server)
k8s deploy-target helpers (current target)
link bring this machine into the Hanzo cloud fleet as a node (fabric + compute worker)
login authenticate against Hanzo IAM (hanzo.id) and store a token
logout remove stored credentials
run launch a workload on Hanzo compute (container or function)
runner run this machine as a JIT CI runner for your org (GitHub Actions)
security scan files for hardcoded secrets (local guardrail; no server/auth)
status show the org's fleet — every node with each of its GPUs
unlink take this machine out of the fleet (deregister + stop hanzod)
whoami show the current identity from the stored token
Service subcommands (server mode):
account serve the account subsystem standalone
account-bridge serve the account-bridge subsystem standalone
admin serve the admin subsystem standalone
admission serve the admission subsystem standalone
ads serve the ads subsystem standalone
affiliates serve the affiliates subsystem standalone
agent serve the agent subsystem standalone
agents serve the agents subsystem standalone
agentskills serve the agentskills subsystem standalone
ai serve the ai subsystem standalone
analytics serve the analytics subsystem standalone
audit serve the audit subsystem standalone
authors serve the authors subsystem standalone
authz serve the authz subsystem standalone
automations serve the automations subsystem standalone
base serve the base subsystem standalone
billing serve the billing subsystem standalone
bots serve the bots subsystem standalone
captable serve the captable subsystem standalone
catalogsync serve the catalogsync subsystem standalone
channels serve the channels subsystem standalone
cloud serve the full unified surface (all enabled subsystems, one listener)
cloudflare serve the cloudflare subsystem standalone
code serve the code subsystem standalone
commerce serve the commerce subsystem standalone
company serve the company subsystem standalone
content serve the content subsystem standalone
crm serve the crm subsystem standalone
dataroom serve the dataroom subsystem standalone
datastore datastore-fork analytics DB — not a Go serve target (see help text)
deploy serve the deploy subsystem standalone
dns serve the dns subsystem standalone
do serve the do subsystem standalone
domain serve the domain subsystem standalone
entitlements serve the entitlements subsystem standalone
evals serve the evals subsystem standalone
exec serve the exec subsystem standalone
flags serve the flags subsystem standalone
framework serve the framework subsystem standalone
functions serve the functions subsystem standalone
gateway serve the gateway subsystem standalone
git serve the git subsystem standalone
graph serve the graph subsystem standalone
guide serve the guide subsystem standalone
iam serve standalone Hanzo IAM (full Beego server: login UI, OAuth2/OIDC, LDAP/RADIUS)
ingress serve the ingress subsystem standalone
integrations serve the integrations subsystem standalone
kafka serve the kafka subsystem standalone
kms serve the kms subsystem standalone
knowledge serve the knowledge subsystem standalone
licensing serve the licensing subsystem standalone
link serve the link subsystem standalone
marketing serve the marketing subsystem standalone
marketplace serve the marketplace subsystem standalone
metrics serve the metrics subsystem standalone
ml serve the ml subsystem standalone
notify serve the notify subsystem standalone
o11y serve the o11y subsystem standalone
paas serve the paas subsystem standalone
plan serve the plan subsystem standalone
platform serve the platform subsystem standalone
plugins serve the plugins subsystem standalone
pricing serve the pricing subsystem standalone
product serve the product subsystem standalone
projects serve the projects subsystem standalone
prompts serve the prompts subsystem standalone
provisioning serve the provisioning subsystem standalone
pubsub serve the pubsub subsystem standalone
referrals serve the referrals subsystem standalone
rollingcap serve the rollingcap subsystem standalone
runtime serve the runtime subsystem standalone
sbom serve the sbom subsystem standalone
security serve the security subsystem standalone
settings serve the settings subsystem standalone
sign serve the sign subsystem standalone
social serve the social subsystem standalone
storage serve the storage subsystem standalone
sync serve the sync subsystem standalone
tasks serve the tasks subsystem standalone
team serve the team subsystem standalone
templates serve the templates subsystem standalone
tools serve the tools subsystem standalone
tracker serve the tracker subsystem standalone
treasury serve the treasury subsystem standalone
usage serve the usage subsystem standalone
validators serve the validators subsystem standalone
visor serve the visor subsystem standalone
wallets serve the wallets subsystem standalone
websearch serve the websearch subsystem standalone
world serve the world subsystem standalone
x402 serve the x402 subsystem standalone
zen serve the zen subsystem standalone
zero-trust serve the zero-trust subsystem standalone
Meta:
help show this message
version print version and exit
Flags are per-subcommand (e.g. `hanzo cloud --enable=iam,kms --brand=hanzo`,
`hanzo kms --listen=:8443`). Run a subcommand to see its config via env/flags. is a
superset — non-Go verbs pass through to hanzo-node so nothing breaks.
The Go unified binary takes the `hanzo` name (HIP-0106). fabricCLI now resolves the
Rust fabric/dev CLI as `hanzo-node` (then a self-guarded `hanzo`), and cmd/hanzo
delegates any verb that is neither a Go control verb nor a served subsystem —
node, dev, wallet, network, … — to it via cli.Passthrough. So one `hanzo` name
serves both: link/unlink/status + the whole Go surface native, and `hanzo node up`
(the fabric that link composes) plus the Rust dev verbs handed through unchanged.
unlink is now idempotent: runDisconnect treats an already-terminal (409) or absent
(404) fleet row as the desired end state (no-op with a clear notice), and unlink
runs stopFabric unconditionally so a deregister error never leaves hanzod running —
the deregister error is reported, not short-circuited. Tests cover the 409/404
idempotent path and the 500 error-surfacing path.
Sweeps the remaining `gpu connect` prose in comments to `link` (engine, runner,
visor fleet/board/visor, and the gpu/fleet spec + engine test headers).
The live wallet Seats:0 for maxpower was NOT the orgs claim (Dave's claim already
lists maxpower/admin). Dave OWNS a maxpower workspace, but a team-go migration left
his own member row is_bot=1/active=0, so Seats (which filters active=1 AND is_bot=0)
excluded him while getUserWorkspaces still listed the workspace (no flag filter).
EnsureWorkspace early-returned on the existing workspace without ever correcting
the row, so every re-login kept 0.
A user who just authenticated through IAM is by definition an active, non-bot member
of their own workspace: EnsureWorkspace now forces the caller's OWN row (never
anyone else's) to active=1/is_bot=0 on the existing-workspace path. Idempotent.
TestEnsureWorkspaceHealsMigratedMember reproduces Dave's exact shape red→green.
Two live hanzo.team defects:
1) Wallet Seats:0 for the caller's own org. orgsClaim dropped the HOME org (the
org the wallet, Seats, and every account-store surface scope to via extra.org)
whenever the IAM orgs claim was non-empty but did not itself list home — the
fallback only fired for an EMPTY claim. So establishSession never ensured a
home-org workspace, and Seats(home) returned 0 for an org that has the caller
as a member. Home is now unconditionally in the set, so its workspace (hence a
seat) is ensured at every login. Reproduced in TestOrgsClaimAlwaysIncludesHome.
2) New-Issue dialog description dropped on create. createContent stored only the
markup SNAPSHOT blob; the collaborative editor replays the Y.js update log the
WS lane serves (ydoc-<id>-<field>), a different blob, so the description showed
empty. createContent now also seeds that log from the front-supplied Y.js
update (never clobbering an existing/live log; scoped to createContent).
TestCollabCreateContentSeedsYLog covers it.
link composes the two node memberships under one verb: it starts hanzod via the
canonical `hanzo node up` (best-effort; --no-fabric skips it) and runs the
compute-worker loop that registers this host's CPU (cores + model), memory, and
each GPU as its own resource, then heartbeats and claims jobs from the org queue.
unlink deregisters the node and stops hanzod; status renders the fleet with each
GPU shown distinctly and this box highlighted. Works on a CPU-only node.
Replaces the gpu connect|status|disconnect surface (one way, no alias). Adds a
CPU model field to the advertised inventory — CLI registration and the visor
fleet record in lockstep. The worker machinery (register/heartbeat/claim, studio,
engine advertise) is unchanged; only the command layer and inventory grow.
The admin cockpit's commerce reader (/v1/admin/finance COGS, /v1/admin/usage,
costs) built a PLAIN http.Client but its base is commerceinproc.BaseURL() — which
returns the 'http://commerce.inproc' placeholder when commerce is co-resident. A
plain client DNS-resolves that host → 'lookup commerce.inproc: no such host', so
the admin finance/cost god-view silently errored ('commerce unreachable'). Swap to
commerceinproc.Client() — the self-routing transport metering already uses:
in-process dispatch for the placeholder host, plain HTTP for a split-deploy URL.
Only this admin client hit it (the others use real env URLs).
The per-tier rolling cap only governed SEEDED subscription tiers (free/pro/…).
Pay-as-you-go / empty / unknown tiers fell through to uncapped — a burst-spend
hole for the entire non-subscription user base (every current org is
pay-as-you-go). Add ai_rolling_cap_cents_default: when a caller's tier has no
specific cap, the reader falls back to it, so EVERY caller gets a rolling ceiling
once set. Default 0 = opt-in (no behavior change until an admin sets it in the
cockpit); tier-specific caps still win. Fail-open on tier/sum error unchanged.
Test covers the fallback (pay-as-you-go over default → deny; tier-specific beats
default; empty+no-default still admits).
A parallel merge (feat(validators): NFT-gated node provisioning) added the
validators subsystem to Wire() at position 45 (after ads) without updating the
frozen golden, so TestWireOrderMatchesFrozen fails 91 vs 90 — the failing test
behind main's red CI once the go.sum compile error is fixed. Refroze to match
Wire() order + flags (ownsHealth=false, hasShutdown=true).
Main CI/CD has been red for 8+ commits: go.sum was missing transitive
entries (mongo-driver/bson via golang-set, btcd/chainhash/v2 via btcec,
hanzos3/go-sdk via zapdb, go-json-experiment/json + luxfi/filesystem via
luxfi/node) so go vet/test/build all fail before any test runs. A parallel
dep bump landed without tidy. Pure require-list + go.sum reconcile; versions
unchanged (luxfi/node stays v1.36.15).
Group("/v1/validators").Post("") registers "/v1/validators/", which the
portal's bare POST /v1/validators would miss. Register the list+provision
collection root via app.Get/app.Post like clients/wallets et al.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(apps): refreeze wire golden — add rollingcap subsystem
Wire() mounts 90 subsystems but the frozen sequence listed 89: the rollingcap
gate (mounted after billing) was added to Wire() without refreezing the golden,
so TestWireOrderMatchesFrozen has been red on main. Add the missing entry at its
Wire() position — the same maintenance the earlier dns+cloudflare refreeze did.
Pre-existing drift, unrelated to the per-GPU queue work; folded in so cloud CI's
go-unit gate (which runs ./apps/) is green again.
* feat(fleet): per-GPU render queue — targeted lanes, visibility, cancel, usage
Make all BYO-GPU render work flow through the org's gpu-jobs queue, visible and
manageable per GPU, and close the hidden direct-submit hole.
cli/gpu.go
- Two-lane claim: the worker claims its OWN lane ("gpu:<identity>") FIRST, then
the shared "gpu-jobs" lane — targeting is the taskQueue VALUE within the one
gpu-jobs namespace, so a job pinned to spark is never starved and no worker
steals another GPU's targeted job.
- Render submit moves from the open POST /prompt to the gated
POST /v1/worker/execute with X-Worker-Token (KMS STUDIO_WORKER_TOKEN).
- Worker reports live GPU utilization (nvidia-smi) each heartbeat via
POST /v1/fleet/samples.
cli/studio.go
- Launch the local ComfyUI with --listen 127.0.0.1 --worker-mode (was 0.0.0.0,
unauthenticated) — the worker dials loopback, so binding wider only exposed an
open /prompt; worker-mode gates the submit seam.
clients/visor
- GET /v1/fleet/jobs?gpu=&status= — the org's queue, each row tagged with the
GPU it targets (''=shared lane) + the claiming worker; ?gpu=X matches target OR
claimant; status normalized to queued|running|completed|failed|canceled; the
full ComfyUI graph is omitted (cheap SaveImage label instead).
- POST /v1/fleet/jobs/:id/cancel {run,reason} — org-scoped cancel via the tasks
CancelActivityForOrg wrapper (tasks v1.51.2).
- POST /v1/fleet/samples — BYO util ingest into the existing samples warehouse
the board already overlays; fleetUnit gains Queued/Running per-GPU depth.
Reuses ActivitiesForOrg (one engine, one tenant key), fail-soft by source.
TDD: claim precedence, gpuTarget, status normalize, filter, per-node counts,
sample build, route tenancy — ./cli/ + ./clients/visor/ added to the CI go-unit
gate so they run every push.
* fix(gpu): detach + bound the util sampler so a hung nvidia-smi can't wedge the worker
sampleGPUs shelled nvidia-smi with no timeout and reportSample ran SYNCHRONOUSLY in
the worker's select loop (heartbeat tick + render-progress tick). Under GPU/driver
pressure nvidia-smi can hang, blocking the whole loop → no heartbeats, no claims →
the machine flaps offline and stops rendering mid-run.
- sampleGPUs takes a ctx and runs the probe via exec.CommandContext under a 5s cap
(self-cancels instead of hanging); the probe is an injectable package var.
- reportSample detaches probe+POST onto its own goroutine under a 20s budget and
returns immediately, so the select loop is never blocked. At most one report in
flight per ticker site (interval >> budget); self-cancels on worker shutdown.
Test: a hung sampler (blocks until its bounded ctx fires) — reportSample still
returns to the caller at once, proving claim/heartbeat can't wedge (-race clean).
* fix(fleet): adversarial batch — pagination, render preflight, filter/stall/token hardening
F1 (MAJOR): the queue + fleet reads no longer truncate at 100 rows. gpuJobs and
byoWorkers cursor-walk the org's namespace to completion via the new paginated
tasks read (ActivitiesPageForOrg, tasks v1.51.3); gpuJobs then recency-sorts and
bounds terminal history (all live jobs kept + last 50 terminal). Past ~100 lifetime
renders a busy org no longer hides live jobs or drops online workers.
F2 (MAJOR): worker render preflight. A node advertises studioCap + claims render
lanes ONLY when it can serve — STUDIO_WORKER_TOKEN present AND a studio reachable
(or launched via --studio-dir). Otherwise it heartbeats as present but claims
nothing (no poison loop of 403→FAILED→reclaim), with a loud one-time operator
warning. Re-evaluated each heartbeat so a studio dying/recovering flips claiming.
F3: ?gpu= filter is case-insensitive (node ids are lower-case).
F4: a running job past its lease surfaces as 'stalled' (worker died, not yet reaped)
instead of 'running' forever.
F5: every loopback studio call (execute/history/view/upload/queue) sends
X-Worker-Token, robust to the worker-mode gate widening scope.
F7: corrected the '/prompt' → '/v1/worker/execute' error string.
Tests: >100-row ordering/bound + stall + case-insensitive filter + cancel err→HTTP
(404/409) mapping; terminal complete/fail hit the exact ns+wf+run path (catches a
routing regression a 200-everything stub would miss); SharePolicy.reject fallback;
not-ready node claims nothing; studioCap gating. tasks v1.51.3 adds
ActivitiesPageForOrg with a >100 pagination test.
---------
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The shared hanzo-app OAuth client trusts https://stg.hanzo.app/callback, but `stg` was not a reserved subdomain — an attacker could first-come-claim stg.hanzo.app, run authorize(client_id=hanzo-app, redirect=stg.hanzo.app/callback) (IAM exact-matches), and harvest a logged-in user code minted with aud=hanzo-app that api.hanzo.ai trusts → account takeover. stg.hanzo.app is 404/unbound today, so reserving closes it safely.
Adds `stg` to sites.baseReserved + reserved-superset test (reserved ⊇ {www,stg}) + storage-layer BindHost(stg)-rejected test. Cherry-picked ONLY the isolated clients/sites files from 5ac8f89; the larger per-app IAM client changes (appauth.go/projects.go/iam) stay OUT of main pending red re-review.
Verified: go build ./... ok; CGO=0 go test ./clients/sites/... ok; BindHost(stg)→errReservedHost, stg does not resolve.
Phase-1 of GDA/SDM validator onboarding on lux.cloud. New /v1/validators/*
subsystem: a caller proves wallet control (EIP-191 personal_sign challenge,
address recovered server-side) AND on-chain ownership of a Validator-tier
GenesisNFT on Ethereum mainnet (ownerOf against 0x31e0F919C67ceDd2Bc3E294340Dc900735810311,
reusing the luxfi/geth read path), then the endpoint:
- generates a luxd staking identity (TLS+BLS+ML-DSA-65 -> strict-PQ NodeID)
via luxfi/node/staking — byte-identical to genesis/cmd/venuekeygen — and
SEALS it into KMS (never plaintext; fail-closed);
- persists the org -> tokenId -> slot entitlement (per-org SQLite);
- writes a NEW-node LuxNetwork CR (group node.lux.cloud, ns lux-validators)
+ KMSSecret sync — three hard guards make it structurally incapable of
touching the live hand-managed luxd StatefulSets;
- ENQUEUES an owner-gated registration (pending_owner_approval, NEVER
auto-submitted to any P-Chain).
Adds github.com/luxfi/node v1.36.15 (requires cloud's exact pinned
crypto/ids/geth — zero version skew). 13 tests pass incl. a live ETH-mainnet
ownerOf read; go vet clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- F2-1: gate the co-resident spend-alert CRUD WRITES (POST/PATCH/DELETE) to an ORG ADMIN
/ SuperAdmin / trusted S2S token (requireSpendCapAdmin) — commerce's user group admitted
any authenticated member, so a compromised member key could DELETE the org's cap
(unbounded spend) or POST a 1c enforce cap (org-wide 402 DoS). Reads (list/authorize)
stay member/S2S-open. Exports accountclient.IsServiceToken for the gate.
- bump github.com/hanzoai/ai v1.829.0 → v1.829.3 (router allowlist HARD floor + no
unowned-OrgSettings clobber across all writers: beego/ZAP/trainer/generic-setter).
- bump github.com/hanzoai/commerce v1.49.7 → v1.49.8 (spend-cap fails OPEN on unknown
spend, not closed — no 402-storm on a finance-read blip).
The universal sync engine's git provider drove an EXTERNAL Gitea store
(giteaFromEnv → gitea.mirrorIn / ensurePushMirror). Repoint it to the native git
object-plane seams already registered by clients/git at Mount, so the ONE git
store IS the in-binary /v1/git plane and no byte transits an external git host:
- inbound (source push) → cloud.InboundGitSync (fast-forward-only advance;
a diverged native ref is a Conflict, native preserved)
- reconcile pull/both → cloud.ImportGitRepo (ff mirror every branch in;
MirrorURL=source registers the native→source push-back)
- reconcile push-only → cloud.EnsureGitMirror (declare the outbound target;
the native mirror_out lifecycle does the pushing)
sync_api.reconcileOutboundMirror likewise moves onto cloud.EnsureGitMirror — the
ONE outbound-target registrar — so a sync's mirror target is never split across
two stores. gitea.go + gitea_test.go (the entire external-Gitea client) are now
dead and removed: forwards-only, no dead code, DRY.
This also FIXES the provider being inert in prod: giteaFromEnv fails closed
without GITEA_TOKEN/URL (unset on cloud), so no git sync could reconcile; the
native seams are in-process and need no external config. resolve() (the pure
decision core) is unchanged — TestGitResolve green; build + vet clean. The
KMS-gated store tests (TestSyncValidation) fail identically on origin/main
(CLOUD_KMS_MASTER_KEY_REF test-env requirement), orthogonal to this change.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
The native /v1/git UI (ui.go) was reachable only under /git/* on every host, so
git.hanzo.ai/ and git.hanzo.ai/<org>/<repo> fell through to the console SPA
catch-all (webui.go app.All("/*")) — the console shadowed the git host root.
Extend the existing onGitHost host-routing (already guarding the root smart-HTTP
/:org/:repo/* clone paths) to the UI: register the SAME handlers at the root
("/", "/:org/:repo", tree/blob/commits) gated to the git host. On api/console
they fall through (c.Next()) to the console catch-all, so a bare /:org/:repo
never shadows it there; on git.hanzo.ai they serve the native browser.
URLs are now canonical per host — one and only one way: base "" on the git host
(git.hanzo.ai/<org>/<repo>, matching the clone URL) and "/git" where the console
embeds the browser. Thread that base through render/templates/href-builders; the
UI clone box shows the clean git-host form (git.hanzo.ai/<org>/<repo>.git).
Non-destructive: additive routes behind a host guard; smart-HTTP clone routing
(distinct /info/refs|/git-*-pack tail) and console/api hosts are unchanged.
TestRootUI_HostGuard covers git-host serve + api-host fall-through.
Co-authored-by: hanzo-dev <dev@hanzo.ai>
Picks up the per-org enabled-models allowlist + savings-vs-quality dial: OrgSettings
RouterEnabledModels/RouterQualityBias, the /v1/get-router-policy + /v1/update-router-policy
carry them (GET also returns the servable-model catalog), and resolveAutoModel enforces
the allowlist (both heuristic + engine paths) and the dial (cost-budget narrow + SLO
tighten). Opt-in: an org that sets neither routes exactly as before.
Decision + per-site migration runbook: every first-party static site (cd/flow/
gallery/yadota) becomes a Project served by the ONE host-router (clients/sites),
bundle moved to the canonical <bucket>/<org>/<slug> layout (no external-prefix
special case — resolver keeps one code path). <slug>.hanzo.ai becomes a bound host
routed through cloud, mirroring the *.hanzo.app wildcard edge. Steps 1-2 additive
(no live-routing change); the cutover (route *.hanzo.ai via cloud, delete the
staticFiles Middleware+IngressRoute) is a reviewed per-host flip, cd.hanzo.ai LAST.
End state: static-sites.yaml holds zero first-party sites — one router, one S3
layout, one store; sites sourced from hanzo-apps.
'iam-v1 is dead; use the new clean iam for all things.' cloud imported the dead
Casdoor fork github.com/hanzoai/iam-v1 in 6 files for exactly ONE type: OrgRef
{Org,Role} (a JWT-claim membership ref). The v2 clean-room iam (github.com/
hanzoai/iam) now EXPORTS it at pkg/model.OrgRef (= schema.OrgRef, byte-identical
JSON) as of v1.32.1. Repoint all 6 (auth_identity, token_validator,
clients/team/{invite,account}, + 2 tests) to model.OrgRef and bump iam→v1.32.1.
cloud's own code now has ZERO direct iam-v1 imports. iam-v1 remains ONLY as a
TRANSITIVE dep via hanzoai/ai/object, which still couples to the full Casdoor
IAM API (Claims/GetUser/GetOrganization/MFA…) — a major separate migration
(ai's domain), not an OrgRef swap. Builds clean; model.OrgRef resolves.
Co-authored-by: zeekay <ai@hanzo.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The GET list + GET /authorize are served co-resident, but POST/PATCH/DELETE
/v1/billing/spend-alerts were NOT — so a customer creating/editing/removing a usage
cap fell through the account bridge's /v1/billing/* wildcard (billingForwardable
includes POST spend-alerts), forwarded to COMMERCE_URL (= this binary), and
self-dispatched into the SAME 502 loop authorize hit. Net: self-service cap
management was impossible in the unified binary (every write 502'd).
Register CreateSpendAlert / UpdateSpendAlert / DeleteSpendAlert co-resident with the
exact chain commerce's own route table gates them (api/billing/handlers.go:322-325,
user group userRequired = TokenRequired) + the global RequestContext: an IAM JWT OR
the COMMERCE_SERVICE_TOKEN, org from the gateway-pinned X-Org-Id. Org-scoped by
namespace (a caller writes only their OWN org's caps; a foreign :id misses in their
namespace), so no PinBillingSubject — spend-alerts are org-level. Specific routes
shadow the bridge wildcard (order 100 < 122). Completes the self-service cap CRUD:
list + authorize (already co-resident) + create/edit/delete (this).
Re-land of the crew seed (a parallel force-push to cloud main dropped it).
personalities.go: dev/des/vi personas + idempotent SeedPersonalities(ctx,org)
(one registry, UNIQUE(org,name); no-op without a model). account.go OAuth
callback seeds per-org after EnsureWorkspace (best-effort, never blocks login).
Native TestSeedPersonalities green. TEAM_AGENTS_ENABLED=1 already live +
AIDefaultModel=deepseek-v4-flash → the crew materializes and answers @-mentions.
Co-authored-by: zeekay <ai@hanzo.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The named personas from the old hanzo.ai site, brought into Hanzo Team as
ordinary rows in the ONE agents registry: nothing special-cased downstream —
they list, project into the Team roster as bot members (bots.go), and answer
@-mentions through the SAME agents.RunOnBehalf path every agent uses.
- clients/agents/personalities.go: the canonical crew (dev=builder, des=designer,
vi=visionary) + SeedPersonalities(ctx, org) — idempotent via the registry's
UNIQUE(org,name); no-ops without a default model (never a half-seeded org) or
an unmounted subsystem (safe to call best-effort on the login path).
- clients/team/account.go: the OAuth callback seeds the crew per-org right after
EnsureWorkspace — a new org gets its default office AND its default crew
together. Best-effort: a seed hiccup NEVER blocks login.
Native test green: TestSeedPersonalities — creates the crew, ListForOrg returns
the @dev/@des/@vi handles with model+prompt, re-seed is a 0-create no-op (no
dup), no-model is a clean no-op. To make them TALK, the Chunter responder flips
on with TEAM_AGENTS_ENABLED=1 (the deploy env) — the pipeline is already built.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The request-edge metering gate (clients/metering scopeAuthorize) reads the per-scope
spend-cap verdict at GET /v1/billing/spend-alerts/authorize over commerceinproc. With no
co-resident handler it fell through to the account bridge's /v1/billing/* wildcard, which
re-forwarded it to COMMERCE_URL (the public api.hanzo.ai edge = this binary) BY PATH through
the same transport, re-entering the wildcard until the depth-8 guard refused -> 502. The live
pod logged ~135x/30m of this (org "maxpower"). The cap is a policy overlay so the gate FAILS
OPEN (outer status 200, no traffic blocked), but every authorize call burned 8 full-app
dispatches and the spend cap never actually evaluated (always fail-open) — a silent policy hole.
Register commerce's own AuthorizeSpendCap co-resident in mountCommerce (order 100, ahead of the
bridge at 122) so the specific route shadows the wildcard and the gate hits the real handler at
depth 1 — the same co-resident move already made for plans/spend-alerts/invoices/etc. Service-
token chain (RequestContext + TokenRequired), mirroring commerce's OWN gate on this route, not
the IAM/PinBillingSubject console chain (a service token is not an IAM JWT, so IAMTokenRequired
would leave GetOrganization unset and AuthorizeSpendCap would 500).
Regression test in clients/commerceinproc/selfdispatch_test.go pins both arrangements: without
the specific route the wildcard self-loops to the depth-8 refusal (the prod 502 signature); with
it the handler serves once at depth 1 and the wildcard never fires.
GET /v1/deploy/applications listed only App CRs (the ~72 pod-backed services), so
every static-plane SITE — cd.hanzo.ai itself, flow, gallery, yadota, … — was
invisible on the CD dashboard. A site has no App CR / Deployment: it is a
`staticFiles` Middleware (S3 origin `s3://cdn/<slug>`) + an IngressRoute (its host),
served straight from S3 with zero pods.
clients/deploy/sites.go: listSiteApplications enumerates the staticFiles Middlewares
per namespace and joins each to its IngressRoute host, projecting one Application row
per site with Role:"site", Repository=the S3 origin, Endpoints=[https://<host>], and
— since a static site is served from exactly its declared prefix — always Synced
(Version==RunningVersion=="static"). Health = routed (Healthy) vs defined-but-unrouted
(Missing). Best-effort (mirrors runningVersions): a missing-CRD/RBAC list error logs
and yields nothing, so the services half of the board always renders.
deploy.go: middlewaresGVR + ingressRoutesGVR (hanzo.ai/v1alpha1). applications.go:
fold site rows into the per-namespace scan + the summary. deploy_test.go:
TestListSiteApplications (a staticFiles Middleware + IngressRoute → one role:"site"
row; an unrouted Middleware → Missing, no endpoints).
Aligns with "delivery is the cloud deploy engine": the CD dashboard now renders the
WHOLE delivery surface — every service AND every site.
Claude-Session: https://claude.ai/code/session_81b00bd9
deps.AI (chat completions, a WRITE endpoint: agents/guide/crm/content/
sitegen/code-ask) and deps.Embed (embeddings, READ-ONLY: code-index + KB)
were ONE shared client authenticated by CLOUD_AI_API_KEY — a read-only
publishable (pk-) key from secret cloud-ai-embed-key. The gateway 403s a
pk- key on any write endpoint ('Publishable keys can only access read-only
endpoints ... use a secret key'), so bot replies intermittently failed.
Split the credential by concern:
- pickCompletionsClient (deps.AI) REFUSES any pk- key (publishableKey guard)
and authenticates with the binary's IAM M2M identity (IAM_CLIENT_ID/SECRET,
already deployed) — the durable no-static-key path. A secret sk-/hk- static
key is still honored as an operator override.
- pickEmbedClient (deps.Embed) keeps the pk- key UNCHANGED (correct
least-privilege for a read-only call); falls back to the M2M resolution when
no static embed key is set.
Metering unchanged: both clients wrap the ONE meteredAIClient path. No CR/KMS
change needed — the M2M identity is already in the cloud CR.
Tests (pick_test.go): completions refuse pk- and present the M2M bearer;
completions honor an sk- key; pk--only + no M2M fails closed (never sends the
pk- key to the write endpoint); embeddings present the pk- key; embed falls
back to M2M with no key; BuildDeps splits both credentials end to end.
- establishSession EnsureWorkspace's the FULL verified membership set, not just
the home org, so a non-home org's wallet counts the caller as a seat instead
of "0 Members in your org". The multi-org lane unioned getUserWorkspaces across
every org but left the seat/member projection seeded only for the home org; one
membership set now drives the token, the workspace union, AND the seat count.
- wallet page: an empty commerce plan renders "Free" + an Upgrade CTA (the login
gate admits a no-subscription org on the effective Free tier) instead of a bare
"—" that read as a data failure. Never fabricates a tier.
- wallet header reflows at narrow widths: the Top up button holds its intrinsic
size (flexShrink 0) under a wrapping row, so it stays whole at 390px instead of
clipping to "Top".
- test: Seats counts distinct active non-bot members (owner + guest; bot and
inactive excluded; tenant-scoped).
Adds a write-only publishable key and a direct-to-ClickHouse ingest ALONGSIDE
the existing /v1/event + Kafka-tier pipeline (nothing removed).
- pk_<b64url(org)>.<b64url(hmac)> — org sealed under HMAC-SHA256(CLOUD_INGEST_KEY_SECRET,
org). Ingest-only BY CONSTRUCTION: the pk_ underscore prefix is outside
isAPIKey's set, so SanitizeIdentity/OrgForKey refuse it — it can never become a
bearer principal, so it can never read. Verify is one HMAC compute (no IAM/DB
hop) — the lowest-latency path. Fails closed when the secret is unset.
- POST /v1/ingest — {batch:[WireEvent]} authed by pk_, org stamped from the SIGNED
key (never the body), funneled through the ONE write core (ingestEvents) into
hanzo.events, tagged source=ingest.
- POST /v1/ingest/keys — an org owner (validated principal) mints a pk_ for its
OWN org.
- GET /v1/errors — type:'error' read lens (validated principal; reads never accept
the write-only key). error is now first-class in canonicalType/resolveEventName
('error'/$error); the WireEvent error object folds into properties.$exception.
Tests: mint↔verify round trip, fail-closed matrix (forged org, wrong secret,
malformed), exception folding, canonicalType.
Picks up the sites.*/base.* namespace grouping in the embedded plans bundle so
/v1/plans/vocab matches the canonical entitlements vocabulary. Subscription caps
(ai.rolling_cap_usd/_window_hours, sites/base included) already flowed via the raw
__PLANS_DATA__ injection; this closes the last display-path drift.
commerce's api.Route() billing bundle is never compiled into the cloud binary, so
GET /v1/billing/{invoices,subscriptions,spend-alerts,payouts,payment-config} had no
handler here and fell through to the account bridge's /v1/billing/* wildcard. The
bridge forwards to COMMERCE_URL, which defaults to the public api.hanzo.ai edge —
i.e. THIS binary — re-entering the same bridge in an unbounded self-dispatch loop
that surfaces as a 502. In prod there is no separate commerce backend to point
COMMERCE_URL at (the in-cluster commerce Service selects the cloud pods), so
co-residence is the only way to break the loop.
Register commerce's own read handlers on the shared app (order 100, shadowing the
bridge wildcard at 122), behind RequestContext + IAMTokenRequired (org namespace from
the gateway-validated X-Org-Id) + a new PinBillingSubject middleware that carries the
SAME subject-pinning the bridge applies (reusing resolveCaller/scopedBillingSearch/
account.Payer) so a co-resident read can never widen past the caller and an
unvalidated caller is refused before the handler runs. This is the same co-resident
move already made for plans/usage/balance.
Bump the embedded hanzoai/commerce dep to v1.49.7 (catalog SOT + public/admin catalog API).
commerce's legacy api.Route() billing bundle (ListPlans, invoices, subscriptions)
is NOT registered by the co-resident embed — setupRoutes wires only /v1/commerce/*
— so /v1/billing/plans had no handler in the cloud binary. The account bridge's
/v1/billing/* wildcard (order 122) then forwarded the read back to commerce at
COMMERCE_URL, which defaults to the public api.hanzo.ai edge, re-entering the same
bridge in an unbounded self-dispatch loop that surfaced as
'commerce unreachable: Get https://api.hanzo.ai/v1/billing/plans' -> 502.
Register commerce's static ListPlans on the shared app in mountCommerce (order 100,
ahead of the bridge) so the specific route shadows the wildcard and plans serve
in-process — the same co-resident move billing.go already makes for usage/balance.
append() only flushed when the NEXT append found the debounce due, so a
typing burst followed by idle sat dirty in memory until the last peer
left. The per-room flusher ticks the same debounce; GC closes it.
Backgrounded tabs throttle the provider's awareness renewals; without
server pings the idle read deadline fires an abrupt close the provider
surfaces as 1006 — the 'cannot connect to collaboration service' banner.
The browser's network stack auto-pongs even when throttled; each pong
extends the read deadline.
The front's @hocuspocus/provider (2.15) speaks its protocol at the bare
/collaborator path with the doc id IN-BAND, which no ingress rewrite can
bridge to the collab relay's /v1/collab/<id> mux — so the live lane joins
the snapshot RPC lane in clients/team: collabws.go serves the hocuspocus
wire (Auth in-band with the SAME HS256 session token + workspace pin +
membership gate as collab.go, SyncStep1 -> replay + empty-diff Step2 +
server Step1, SyncStatus acks, awareness echo keepalive) over zip/wsx,
persisting the Y.js update log per doc on deps.VFS under the tenant-scoped
blob key. The server never parses update payloads: Y.js updates are
commutative + idempotent, so log replay converges; a lone peer's
full-state SyncStep2 (the reply to the server's empty-SV Step1) replaces
the log — compaction without a server-side CRDT. Rooms are in-process
(cloud pins replicas=1, single writer).
Bumps iam to v1.31.34 and carries the verified `orgs` membership-set claim end
to end, so a user's team workspaces union across every org they belong to.
- cloud.VerifiedIdentity gains Orgs []iam.OrgRef, copied from the verified
claims (idClaims parses the signed `orgs`); empty on legacy tokens.
- establishSession folds the full membership set into the session token
(extra.orgs), fallback [{owner, admin}] for a legacy token; extra.user carries
the IAM id for a mid-session refresh. Still fails closed on empty owner. The
short workspace token (rides the transactor URL) stays minimal (extra.org only).
- getUserWorkspaces unions WorkspacesOf across every session org, each
WorkspaceInfo tagged with its owning org for the client switcher.
- selectWorkspace resolves an EXPLICIT (org∈session, slug) — clean BadRequest on
absent, WorkspaceAmbiguous on a slug in two orgs, never a silent default.
getWorkspaceInfo resolves the token's workspace claim, killing the wss[0]
default the same way. Single-workspace fast path unchanged (front selects by
URL). Cross-tenant isolation preserved (every lookup owner_org-scoped).
- Invite plane (clients/team/invite.go): sendInvite resolves the invitee in IAM
(get-user), POSTs add-membership as the confidential hanzo-team app
(client_secret_basic, CapMembershipAdmin), and writes the local member row;
owner/admin only. getMemberships is the mid-session refresh (live get-memberships,
session-set fallback). AddMember upserts the roster row idempotently.
Tests: table tests for the orgs round-trip (+legacy fallback), the workspace
union, cross-org select, no-default/ambiguous refusals, getWorkspaceInfo
no-default, guest-cap unaffected, invite writes membership+row (mock IAM) +
admin gate, refresh live/fallback, and VerifiedIdentity.Orgs claim flow.
The Team front was losing two core lanes against the native backend:
- Issue/doc rich-text creation dead-ended: the front's collaborator-client
POSTs createContent/updateContent/getContent to /collaborator/rpc/:documentId,
which only had the Y.js WS relay behind it — every RPC 404'd, so
createMarkup threw and tracker issues/documents could not be created.
collab.go now serves that contract on deps.VFS (same tenant-scoped blob
keys as files.go): snapshots at makeCollabJsonId ids, membership-gated,
no-oracle 404s. Ingress path-splits /collaborator/rpc → cloud (universe).
- Channels/DMs vanished from the chat navigator on reload: the nav lists
notification:class:DocNotifyContext per {user} (upstream server triggers
materialize them; we never did). seed.go's trigger now projects contexts
from chunter Channel/DirectMessage membership — create on member add,
remove on leave, lastUpdateTimestamp bump per message (heals pre-existing
channels on first message) — and mirrors every write as a derived tx that
tx() broadcasts so live sessions refresh.
Tests: collab RPC round-trip + tenancy red bars; channel→context projection
(create/touch/leave). chat_test's local clChannel const moved to seed.go.
The hanzo.team login page goes native: the SPA form now authenticates
straight against Hanzo IAM (there are no local accounts) and the social
buttons land directly in the provider OAuth flow.
- account RPC "login": server-side IAM password grant — the SAME two-step
the platform e2e auth helper locks (POST /v1/iam/login responseType=code,
then the confidential code exchange) — followed by the EXACT session
establishment the OAuth callback runs (now ONE shared establishSession:
userinfo → verified owner claim → workspace ensure → HS256 token). The
password rides only in the body of the one IAM login call, is never
logged or persisted, and bad credentials answer a clean 401 with the
platform status the form already translates.
- /auth/google and /auth/github: same authorize hop as /auth/openid (the
one registered callback) carrying provider_hint=provider-google/github,
so hanzo.id auto-federates straight into the provider (console-proven,
id >= 0.2.6); explicit ?provider_hint= passes through verbatim.
- /providers now surfaces Google + GitHub + Hanzo so the SPA renders the
three buttons with zero client wire changes.
- Status.Severity is the platform's STRING enum ("ERROR"), not an int the
SPA compares against nothing — error styling and retry now behave.
Tests: password login mints a verifying session (mock IAM), bad creds 401
with no password in logs or response, provider_hint mapping, providers
surface. TestPersistenceCRUD remains the known pre-existing host red.
Installs the ai gate's per-tier rolling AI-spend cap — the Anthropic-style burst
limit that resets continuously (usage older than the window drops out of the
trailing sum; no reset job). Composes three co-resident globals and owns no state:
aiobject.TierReader() — the caller's commerce plan tier
finance.Current() — the ledger's windowed usage sum (SumUsageSince)
flags.Int(key) — the admin-editable per-tier caps
The two knobs (window hours + per-tier cap cents) are platform switches, so
admin.hanzo.ai renders and edits them LIVE via the existing /v1/admin/flags
cockpit — zero bespoke admin UI. Seed defaults mirror @hanzo/plans subscription.json
(developer $0.75 / pro $2.50 / plus $12 / max $25 per 3h window).
- clients/rollingcap: the new subsystem (own package — it imports clients/flags,
which imports root cloud, so the wiring lives above that edge). Registers the
switches (init) + installs the reader (Mount, no routes). FAILS OPEN on any
tier/finance error — a commerce blip must never 429 a paying caller.
- types.FinanceClient: expose SumUsageSince (the trailing-window source) on the
interface (was only on the concrete ledger); billing/marketing test mocks updated.
- apps.go: mount rollingcap after commerce/plan (its globals are wired by then).
- go.mod: hanzoai/ai v1.827.1 → v1.828.1 (the SetRollingCapReader hook).
Inert until deployed on the unified binary; the hook is nil elsewhere. Tests cover
the decision table (over/under-cap, window-off, unknown/uncapped tier, fail-open on
tier+sum errors) + Mount no-op when globals unwired + seed coverage.
No org has subscription rows yet and no self-serve checkout exists, so the
definitive-no 402 on selectWorkspace locked every user out (live 2026-07-20,
front rendered it as NoLoaderForStrings). Log the denial, admit, and bring
enforcement back with the card-on-file subscribe path.
Root cause of the "broken module graph" (make test / go build ./... failing with
oxy invalid-version, ugorji/koanf ambiguous imports, k8s.io/kubernetes staging
referencing removed API groups): `~/work/hanzo/go.work` auto-shadows this tree but
does NOT list ./cloud. In that workspace mode Go drops cloud's own go.mod
directives — the oxy replace, the ugorji monolith exclude, and the k8s.io/*
staging pins — so the graph that those directives keep consistent falls apart.
cloud is a standalone deploy unit (own go.mod/Dockerfile/binary) and must not join
that workspace (merging its k8s/otel tree with o11y's reintroduces the koanf split
ambiguity; the parent workspace is independently red on koanf).
Fix: the Makefile forces GOWORK=off for all go targets — exactly how CI and the
Dockerfile build (fresh checkout, no parent go.work). No go.mod change was needed;
the existing directives are correct for module mode. A committed go.work was
rejected: it would flip the Dockerfile into workspace mode after its -mod=readonly
`go mod download` step.
Proof (GOWORK=off, == what make now runs): `go build ./...` exit 0, `go vet ./...`
exit 0, `go mod tidy` stable (no go.mod change). `make build` exit 0. `go test
./...` runs (was fully blocked before): 128 ok / 103 no-test / 10 fail, every
failure runtime not module-graph — encrypted-OrgDB tests that need CGO+libsqlcipher
(the Dockerfile's -tags libsqlite3 stage), a bundle-embed test needing make
deploy-ui, and pre-existing behavior tests (metering/zt/o11y). See LLM.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- clients/team/wallet: small Vite/React page on @hanzo/ui@8 (balance
three-bucket split, current-period usage, plan + seats, top-up link to
billing.hanzo.ai), mobile-first monochrome; committed dist go:embed'd
(the console/tasks one-binary precedent).
- clients/team/billing.go: session-gated serve of the embed + GET
/v1/team/billing/plan (seats/guests from the org's member rows, plan +
team.guests cap through the same commerce/plans seams entitle uses);
orgPrincipal is the ONE token→tenant resolution (files plane rebased on it).
- money reads stay on cloud's own /v1/billing/balance + /v1/usage/summary:
the hanzo_iam_token cookie the team callback sets is now a validated
principal (aud hanzo-team appended to defaultJWTAudiences, forwards-only),
so the org is pinned server-side from the verified claim — no second
auth mechanism.
- tests: billing 401 unauth (through real Mount), embedded shell + bundle
served authed, plan org-scoped across two tenants, audience pin.
Finishes the one convertible subsystem the group sweep skipped: team's routes
were spread across 5 register funcs whose receiver was named `g` (colliding
with the group var). Resolved by passing the group as a zip.Router param
(register(r zip.Router, ...)) instead of *zip.App — one `tg := app.Group(
"/v1/team")` in Mount, threaded to acct/bridge/files.register + the two inline
transactor routes, all rewritten to relative paths. (Note *zip.App does NOT
satisfy zip.Router — App.Fiber() returns *fiber.App vs the interface's
fiber.Router — so the two test harnesses now pass app.Group("/v1/team") too.)
Route table preserved (13 team routes byte-identical); team test suite green
(exercises the real /v1/team/* paths end-to-end); combined build +
TestWireOrderMatchesFrozen pass. deploy stays flat by design — it already DRYs
via const dashPrefix and its loginPath/callbackPath vars are reused for
redirects (scope.go), so grouping would risk redirect paths for no real gain.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up to 256b848e: the doc comments still named packages by their old
svc-suffix (iamsvc.Mount, pricingsvc, provisioningsvc, mlsvc, evalsvc, plansvc,
productsvc, syncsvc, gatewaysvc) — stale references to symbols that are now bare.
Corrected to the real names across config/middleware_identity/eval/gateway/git/
ml/pricing/projects. Comment-only; no code change.
Deliberately kept: cloud-mlsvc at clients/ml/ml.go (a real ClusterRoleBinding
name in ml-rbac.yaml, correctly referenced) and the "zapsvc" test-fixture repo
strings (test data, not an identifier).
Also gofmt'd clients/pricing/admin_http_test.go (pre-existing import-order drift).
- token: every session token carries exp (30d; workspace 12h); Decode enforces
exp/nbf with 60s skew; pre-rollout no-exp tokens honored until a fixed
legacy cutoff (constant, no env). The "secret" fallback literal is GONE —
empty secret is a hard ErrNoSecret and the TEAM_DEV_INSECURE hatch is dead.
- account: OAuth state is a random nonce bound to a short-lived cookie
(navigateUrl rides in the cookie), verified one-shot on callback; the tenant
comes ONLY from the RS256/JWKS-verified IAM token owner (cloud.NewTokenValidator)
— fail closed, no default org.
- transactor: WS upgrade enforces an Origin allow-list (same host, team
surfaces, *.hanzo.ai, absent Origin for non-browser); serves the front's
/api/v1/statistics poll target (own-workspace sessions only).
- entitle: selectWorkspace requires the org's 'team' license — definitive no
→ 402 + upgradeUrl billing.hanzo.ai; guest role capped by the plan's
team.guests entitlement (join order); infra errors ALWAYS admit so the gate
can never brick login mid-rollout.
One name per thing, no compound-word cruft. The `svc` suffix was never a real
package (zero `package *svc`) — only import aliases and abbreviated test helpers.
- Import aliases → bare package names: plansvc→plan (commerceclient),
captablesvc→captable + dataroomsvc→dataroom (company/adapters). No stutter, no
alias where the bare name is unambiguous.
- Test helpers spelled out: fakeSvc→fakeService, testSvc→testService,
newSvc→newService — across admin/agents/deploy/domain/functions/ingress/
integrations/ml/platform/provisioning/storage/wallets tests, callers updated
in-package.
- Stale `// Package …svc` doc-comment prose corrected to the real package name
(exec/iam/plugin/pricing/product/provisioning/sync/tasks).
Naming only — no logic change. go build + test-compile green on all 21 packages.
Note: the clients/team package (filesSvc/fsvc rename) is excluded here — it has
concurrent in-progress work; its svc cleanup lands with that change.
Bump github.com/hanzoai/plans v1.4.0 -> v1.4.1 (catalog: pro repriced $20
on hanzo_pro_20, new plus $100, max $200, team $25/user per-seat minSeats 2,
team.guests entitlement, team namespace).
Pin the contract in clients/plan tests: TestPlans_Ladder freezes the
subscription ladder prices + stripe lookup keys + team per-seat/minSeats;
TestLicenseEntitlement_TeamProduct freezes the hanzo.team entitlement gate —
licensing.product:team emitted for pro, plus, max AND team, engine on max,
never on developer. Vocab namespaces 9 -> 10 (team).
Smoke-booted: /v1/plans/subscriptions serves the new ladder,
/v1/plans/entitlements/team carries licensing.product:team.
A published site host now serves its org own Base data plane (HIP-0014). The
sites middleware, on a /v1/base|/v1/realtime|/_/ path, calls an injected per-org
Base handler with the org the SUBDOMAIN resolves to (Site.Org) — never the
caller — so an anon page reaches its own Base, authz by Base collection rules.
One seam (sites.SetBaseHostHandler, mirroring SetResolver; no import cycle),
gated by CLOUD_BASE_PUBLIC_HOST (default OFF): absent the flag a site host serves
only static files, unchanged. This is what makes maxpower.hanzo.app/_/ (admin) +
the public contact form + anon realtime chat work — the token supersedes the
key/host for signed-in users (org from IAM), keys/host are the tokenless path.
Partial fix for red MEDIUM-1 (unbounded shared purge = cross-tenant blast
radius). Per-tag coalescing + process-wide per-minute ceiling in the Purger.
NOT finished: MEDIUM-2 (release retention GC), LOW-1 (reject rel=="."),
LOW-2 (empty dest ETag = fail). Do not merge until complete + red re-review.
The org-scoped two-label design (<slug>.<org>.hanzo.app) was never servable: a
k8s wildcard Ingress host and a Let's Encrypt wildcard cert each match exactly
ONE label, so the two-label host neither routes nor gets TLS. Publish still
stamped it as liveUrl and bound it, so every 'Visit' link and the console/app
cards pointed at a dead host, and the sites edge served nothing.
One host, one way:
- siteURL → https://<slug>.<apex> (bare); siteHost → bare <slug> (the global
first-come binding key — matches TestSiteHostBindingIsFirstComeAndTenantSafe,
which already asserted bare-host first-come). A second org publishing the same
slug is refused the subdomain and serves at its S3 URL only.
- siteSlug parses ONLY the bare host; a dotted key falls through to the API
pipeline. unique-live-slug resolve (added earlier) keeps pre-binding publishes
servable with no backfill.
- tests updated to the bare-host contract throughout.
Static sites had no way to put content at a site's prefix through the API.
Add one: a release plane on the existing site engine.
A release is an immutable prefix whose id is a digest of the object manifest
it was promoted from; the site record holds a pointer to the release it
serves, and siteResolver resolves through it. Publishing copies server-side
within the object store, so no bytes traverse the API and no client holds an
S3 credential. Rollback is the same pointer flip aimed at an older release.
Isolation: the source is a path relative to the caller's own org space. The
org segment comes from the validated principal (the one org rule this package
already uses for site prefixes) and the bucket never comes from the request,
so a caller has no syntax for naming another tenant's data. safeRel roots and
cleans both the source and every object key.
Atomicity: the release row is written only after every object lands, and
activation is one statement whose WHERE requires that row in the same tenant,
so a partially-copied release cannot be pointed at. ActivateRelease is the
sole writer of the pointer on the activate path.
Releases live in <org>/.releases/<slug>/<id>/, a sibling of the mutable
serving prefix, so a full-artifact deploy reclaims the pointer without
destroying retained releases. Caps reuse the artifact budget.
POST /v1/sites/:slug/publish
POST /v1/sites/:slug/releases
GET /v1/sites/:slug/releases
POST /v1/sites/:slug/releases/:release/activate
Mirrored under /v1/platform/sites. Inert for existing sites: an empty pointer
serves the legacy prefix.
45 subsystems converted from flat full-path registration to the idiomatic zip
app.Group("/v1/<prefix>") + relative-path pattern — DRY the prefix, one and
only one way. Route-preserving: proved Group(p).<M>("/rel") == flat
app.<M>("p/rel") byte-for-byte; bare-prefix root routes kept FLAT (Group(p).
Get("") would add a trailing slash). Multi-prefix / dynamic-path / cross-
function-collision subsystems deliberately left flat.
Every converted subsystem gated on route-table preservation + go build + go vet;
combined ./clients/... + ./apps/... compiles clean; TestWireOrderMatchesFrozen
passes (Wire()/composition root untouched — grouping is inside each Mount).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
feat/fleet-byo-cpu-spec: gpu-connect nodes report CPU arch + cores + RAM on
/v1/fleet (cli/gpu.go, clients/visor board+fleet). Redundant #331 flags-test
tweak dropped in favor of main's. Tests: visor ok, cli ok, apps ok (CGO-off);
flags green under CGO-on (its one CGO-off failure is pre-existing, matches main).
BYO nodes that dial in via `hanzo gpu connect` reported only their GPUs, so the
/v1/fleet board (and the world Fleet panel) showed NO CPU arch or system memory
for them — unlike code-linked run-targets, whose Spec already carries
arch/cpus/memory. evo-2 (Strix Halo, x86_64) and spark (GB10, aarch64), both
128 GB, appear on the board as BOTH a run-target AND a gpu-connect worker; the
BYO rows surfaced blank arch/memory.
Add the host's static CPU spec to the fleet presence record, read from the real
machine (never hardcoded), in the SAME convention the fleet already uses for
code-linked nodes so a machine shows ONE arch string across both rows:
- reporter (cli/gpu.go): registration gains arch, cpus, memory. detectArch is
`uname -m` (aarch64 | x86_64 | arm64) to match the existing fleet convention
(NOT runtime.GOARCH's arm64/amd64). detectMemTotal reads /proc/meminfo
MemTotal on Linux (evo-2, spark) / sysctl hw.memsize on Darwin; cpus =
runtime.NumCPU. 0/"" when unknown, never faked.
- decoder (clients/visor/fleet.go): fleetRegistration + byoWorker mirror the
three fields (lockstep with the CLI) and byoWorkers populates them.
- board (clients/visor/board.go): workerUnits -> byoUnit fills
fleetSpec.Arch/CPUs/Memory (the fields agentUnits already sets), so a
gpu-connect node and a code-linked node describe themselves identically.
Verified on a real GB10 (spark-class): detectArch=aarch64 (uname -m), nproc 20,
/proc/meminfo 127600528 kB -> 130662940672 bytes — byte-identical to how the SAME
box already reports as a code-linked run-target (arch=aarch64 cpus=20
memory=130662940672).
Tests: parseMemTotalKB, detectMemTotal (real host >0), detectArch (uname -m
convention, not GOARCH), buildRegistration host spec; fleetRegistration decode +
byoUnit projection + unknown-spec omitted.
The one admin mint surface. SuperAdmin-only (core.Guard); forwards verbatim to
commerce's already-mint-gated POST /v1/billing/credit-grants (middleware.Mint →
PlatformOnly) via COMMERCE_SERVICE_TOKEN, scoped to the target org, and writes one
tamper-evident audit record. Commerce stays the sole credit-grant ledger — no
in-process mint. NOT deployed; for red review (mint surface).
Assisted-by: neo:claude-opus-4-8
The org-scoped host redesign (<slug>.<org>.hanzo.app) left the edge unservable:
a k8s Ingress host and a Let's Encrypt wildcard each match exactly ONE label, so
the two-label shape never routes nor gets TLS, while the one-label product URL
every surface advertises (palette, share copy, publish toast) was rejected by
siteSlug and fell through to the console pipeline. Net: no published site
resolved at all.
Fix, preserving the org-scoped design:
- siteSlug accepts a bare non-reserved <slug>.<apex> label again (org-scoped
two-label parsing unchanged, ready for per-org certs later)
- siteResolver falls back for bare keys: explicit site_hosts binding first,
else ResolveUniqueLiveSlug — serve iff EXACTLY ONE live project owns the
slug across orgs; ambiguous or draft ⇒ honest 404. Deterministic,
hijack-safe (reserved labels rejected at the host boundary), and
migration-free for publishes that predate host binding.
- tests: bare-host parse cases + unique/ambiguous/draft resolve proofs
The .worktrees/ directory holds local git worktrees (dev infra), never part of
the tree — mirrors the existing .claude/ rule so a working checkout stays clean.
feat/kms-reseal-migration: split reseal tokenFunc into src/dst faces (CR app-name
credential vs per-org <org>-platform-kms), assert minted-token owner==target org
(refuse admin), flag empty source folders as seeding-wedge risk. Tests: apps ok;
cmd/kmsreseal ok (full suite green under CGO; new auth/owner tests green under CGO-off).
The reseal migration reads from the standalone KMS and writes into cloud KMS —
two faces with DIFFERENT identities. Split the single tokenFunc into srcAuth/dstAuth:
src uses the CR app-name credentialsRef (the standalone accepts it); dst uses the
per-org <org>-platform-kms credential (cloud accepts it dynamically, admin-denied,
no static audience widening).
Defense-in-depth (LOW-1): decodeJWTOwner reads the minted token owner/isAdmin claims
locally and asserts owner == target org (refusing admin tokens) before any read/write,
so a misscoped credential fails its target instead of acting on the wrong org. An
empty source folder is flagged as a seeding-wedge risk instead of silently skipped.
Tests (auth_test.go): decodeJWTOwner, owner-mismatch + admin-refusal gates,
dual-face token brokering.
feat/cd-detail-endpoints: serve the three per-app endpoints the ArgoCD SPA
detail view calls, scoped by the same resolveScope/findNamespace path as
dashApp. Tests green: clients/deploy, apps.
The ArgoCD SPA's application-detail view calls three per-app endpoints the
projection did not serve, spamming "404 page not found" toasts. Add them,
scoped by the same resolveScope/findNamespace path as dashApp — a SuperAdmin
sees the whole fleet, a validated org member sees only its own apps, a
cross-tenant name is a clean 404 (no oracle), an unvalidated caller fails
closed:
- GET /applications/:name/syncwindows -> the permissive-empty
ApplicationSyncWindowState (no sync windows run; canSync true).
- GET /applications/:name/revisions/:revision/metadata -> honest minimal
RevisionMetadata (message = the revision, HEAD resolves to the declared
image tag; date = the CR creation time; author empty). Image-based deploys
carry no git commit and the manifest repo is not the app's source, so no
author is fabricated and it never 404s.
- GET /stream/applications/:name/resource-tree -> the live ApplicationTree
as SSE (data: {"result": tree}), the scope gate before any emission,
emitted once then refreshed on the keep-alive interval, honoring ctx cancel.
A routed run's machine pushes with its own credential and streams into the
session, but cloud still owns the completion — the integrity gate, the PR row, and
the session's terminal state (the machine never closes the session, so it was
staying "running" forever). Give a routed run the SAME cloud-side completion the
local keystone path runs after a sandbox push.
- completeChanged: the shared terminal for a run that reported changes — VerifyRef
the pushed branch LANDED (fail-closed to a session error + no PR if absent), file
the native PR, mirror done, close the session done. The local path (Run) now calls
it too, so the two paths cannot drift.
- finalizeRouted: maps a machine's terminal report onto that completion — reported
failure closes the session error (no PR), no-changes closes done (no PR), a changed
push runs completeChanged. No secret crosses; cloud only reads the ref it can see.
- DeliverRoutedRunActivity runs the completion once, after a real report, on a
cancel-immune bounded context, so a completed run is never re-executed by a retry.
The completion seam is injected at the composition root (NewDispatcher), the same
injected-seam shape index_on_push uses, so the free-function activity reaches the
dispatcher's git/tracker/session seams without a global Dispatcher.
- RoutedRun carries Actor + AgentRef (cloud-side only, never sent to the machine) so
the completion attributes the session close and files the PR with the right
assignee.
Also document the mailbox's single-replica dependency at its definition (accepted,
inherited from cloud's KMS-lock replicas:1) with a future replica-aware note.
Tests: routed changed+verify -> PR filed + session done; verify fails -> no PR +
session error; no changes -> done no PR; reported error -> error no PR (verify never
runs); NewDispatcher wires the seam; the durable type bridge preserves attribution.
The CD projection synthesized an argocd.argoproj.io/instance label on every app
(visible on every card). It is Hanzo-native CD, not ArgoCD — the App CRs carry
hanzo.ai/* labels. Emit hanzo.ai/instance instead; env + org labels unchanged.
(The argoproj.io/v1alpha1 response SHAPE stays until the @hanzo/gui FE that reads
@hanzo/ui/cd native types replaces the ArgoCD SPA.)
Resolve each /v1/deploy read request's scope from the validated identity —
the same boundary clients/platform.tenant uses (validated principal +
injective provisioning.SanitizeOrg + the c.IsAdmin SuperAdmin predicate).
A SuperAdmin sees the whole fleet; a validated org member sees only its own
org's apps (hanzo.ai/org label, tenant-<org> namespace); anyone else is
refused. Scoped reads: applications list/detail/resource-tree, clusters,
projects, and the SSE stream. sync/rollback + the argocd bootstrap stay
SuperAdmin-only.
projectApp reads app.kubernetes.io/part-of into spec.project (default when
absent) and surfaces hanzo.ai/org. The projects endpoint reflects the
IAM-owned (org,name) Project resource in-process — org-scoped for a normal
org, all orgs for a SuperAdmin — with a synthesized default so every app's
spec.project resolves. IAM stays the single source; no CD-side project row.
Integrates the last two router branches now on ai main:
- per-org RoutingPolicy on the hot path (decomplected per-org routing)
- context_window surfaced in /v1/models
(do-ai premium routes + judge/MFJP + mean-field + RouterCostCeiling slider
+ the judge-panel deadlock fix already shipped via v1.826.7, already live.)
clients/... compiles clean against v1.827.0.
A malformed watch event carrying a typed-nil *unstructured.Unstructured would
nil-deref on GetName() in forwardWatch. The read plane installs no panic
recovery around detached goroutines, so that crash would take down the whole
process. Guard the typed-nil, and recover at the spawn site so no future
malformed event can crash the plane. Adds a regression test.
Red review: SHIP (this closes the sole LOW finding).
The ArgoCD-UI-compatible surface returned nothing at three endpoints the
applications view calls, so the SPA error-toasted on load:
GET /v1/deploy/clusters -> 404
GET /v1/deploy/projects -> 404
GET /v1/deploy/stream/applications -> 404
Add all three as read-only projections over the SAME App-CR source dashAppList
reads (listAppCRs + runningVersions + projectApp: one source, one projection),
SuperAdmin-gated by guard(), safe on cloud-reader (no writer/commerce imports):
- /clusters -> ClusterList of the destinations the fleet reconciles into,
deduped, always including the in-cluster destination, with a per-cluster
application count. argoCluster has no config field, so a cluster credential
cannot be surfaced by construction.
- /projects -> AppProjectList: prefers real argoproj.io/v1alpha1 AppProject CRs
when that CRD is served (reshaped to only the intended spec fields), otherwise
synthesizes one permissive project per distinct App-CR project name (default
always present).
- /stream/applications -> the applications watch as SSE: one ADDED event per
current App CR, then live ADDED/MODIFIED/DELETED from a per-namespace watch,
held open with keep-alives. Every watch + goroutine is bound to the request and
torn down on disconnect; degrades to keep-alive only if the watch verb is not
granted; fails closed (503) with no cluster client.
Tests (go test -race green): cluster dedupe + always-in-cluster + never-emits-
credentials; project distinct/default + synth-permissive + real-CR-only-intended-
fields; stream ADDED-per-app + zero-app-no-panic + honors-ctx-cancel + SSE-headers;
all three routes 403 without SuperAdmin.
scopeRateLimiter reads its own rules via a co-resident commerce self-dispatch (GET /v1/billing/spend-alerts) on every authed request; the rule cache fills only after the fetch returns, so the self-dispatch re-enters scopeRateLimiter with a cold cache → unbounded in-process recursion → writer stack-overflow (single request) / OOM (concurrent). Dump-attributed (goroutine 4423, 21,823 setRequestCancel) and real-binary A/B verified on current main+fix (GET returns, POST 402s, 40-concurrent peaks 228 goroutines, 0 pileup). Exempt the commerce config surface from its own gate + an on-path depth backstop.
Post-mortem containment for the v1.801.104 writer crash. To be unambiguous on
root cause: the fatal was the commerce co-resident dispatch reentrancy
(stack: apps.mountCommerce.IAMTokenRequired.func5 → commerce@v1.49.3
iammiddleware.go:157 → unbounded net/http.setRequestCancel goroutines), the bug
PR #341 fixes, introduced by 169beab (enso per-tier gate) in the .99→.104 range —
NOT this responder. The responder makes ZERO outbound calls at boot (it fires only
from session.tx, the live client-WS write path; never from reconcile/replay). Any
build of current main still crashes until #341 lands, independent of this change.
That said, an unbounded per-message responder IS a foot-gun, so this makes it
safe-by-default and bounded regardless:
- OFF by default: Mount wires the LLM seam ONLY when TEAM_AGENTS_ENABLED=1. A nil
runAgent makes maybeAgentReply return at the top → NO outbound model call can
fire. An un/mis-configured binary is provably inert.
- Fresh-only: a message created before this process booted (>60s grace) is a
replay/backfill and is NEVER answered — kills the "replayed backlog fans out into
thousands of HTTP calls" failure mode.
- Single-flight per (workspace, space, bot): a burst to one conversation collapses
to one turn; duplicates dropped, not queued.
- Hard concurrency cap: a global semaphore (TEAM_AGENTS_MAX_CONCURRENCY, default 4,
clamped 1..64) bounds in-flight turns; over the cap, DROP.
- Circuit breaker per agent: after 3 consecutive failures skip the agent for 60s —
the backoff that turns a publishable-key 403 storm into a quiet trickle. No
retries, ever.
TDD (all -race green): TestNoReplyToBacklogAtBoot boots against a 500-message
backlog and asserts ZERO runner calls (then one fresh post IS answered);
TestConcurrencyCapBounded (cap=2, 8 msgs → exactly 2 in-flight, rest dropped);
TestSingleFlightPerConversation (5 msgs, 1 conversation → 1 turn);
TestCircuitBreakerBacksOff (persistent failure → runner called exactly threshold
times, circuit opens). Existing responder + roster tests unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ArgoCD-UI-compatible surface returned nothing at three endpoints the
applications view calls, so the SPA error-toasted on load:
GET /v1/deploy/clusters -> 404
GET /v1/deploy/projects -> 404
GET /v1/deploy/stream/applications -> 404
Add all three as read-only projections over the SAME App-CR source dashAppList
reads (listAppCRs + runningVersions + projectApp: one source, one projection),
SuperAdmin-gated by guard(), safe on cloud-reader (no writer/commerce imports):
- /clusters -> ClusterList of the destinations the fleet reconciles into,
deduped, always including the in-cluster destination, with a per-cluster
application count. argoCluster has no config field, so a cluster credential
cannot be surfaced by construction.
- /projects -> AppProjectList: prefers real argoproj.io/v1alpha1 AppProject CRs
when that CRD is served (reshaped to only the intended spec fields), otherwise
synthesizes one permissive project per distinct App-CR project name (default
always present).
- /stream/applications -> the applications watch as SSE: one ADDED event per
current App CR, then live ADDED/MODIFIED/DELETED from a per-namespace watch,
held open with keep-alives. Every watch + goroutine is bound to the request and
torn down on disconnect; degrades to keep-alive only if the watch verb is not
granted; fails closed (503) with no cluster client.
Tests (go test -race green): cluster dedupe + always-in-cluster + never-emits-
credentials; project distinct/default + synth-permissive + real-CR-only-intended-
fields; stream ADDED-per-app + zero-app-no-panic + honors-ctx-cancel + SSE-headers;
all three routes 403 without SuperAdmin.
Bots-as-members (bots.go/roster reconcile) already projects each org agent as a
workspace Employee, but a message to a bot did nothing — the AI was present and
mute. This adds the WRITE/response half: when a human posts a Chunter ChatMessage
addressed to an active bot member — a DirectMessage whose participants include the
bot, or a channel message that @-mentions it — the transactor runs that agent
through agents.RunOnBehalf (the ONE billed/metered/recorded in-process run path)
and posts the model's answer back into the SAME conversation as that bot, via the
SAME applyTx + hub.broadcast write the SPA and roster projection use.
- chat.go: parseChatMessage, replyTargets (DM-member OR @mention addressing),
maybeAgentReply (cheap gate → agents list → per-bot async turn), replyAsBot
(recovered + 90s-bounded goroutine; never blocks the WS loop). plainText/
htmlMarkup bridge stored markup ↔ LLM text. Loop guard: a bot-authored message
never triggers a reply.
- transactor.go: transServer gains runAgent (the LLM seam) + log; session.tx fires
maybeAgentReply on the client write path only (roster/sync call applyTx directly,
so a projection can never trigger a reply).
- bots.go: agentReplyRunner adapts agents.RunOnBehalf (error-status run → post
nothing, never an empty bubble).
- team.go Mount wires runAgent=agentReplyRunner. Responder is off (nil runner) when
unwired, so the path is fully additive.
TDD: 16 tests — parse/ignore, plainText/htmlMarkup, DM + mention addressing, the
full DM reply loop with a fake runner (asserts on-behalf-of user, agent id, plain
prompt, reply authored by the bot in the same conversation), loop-guard, and
disabled-when-no-runner. All clients/team tests green (CGO_ENABLED=0), go vet clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cd.hanzo.ai rendered but every API call 403'd with no way to sign in: the IAM
session cookie is host-scoped to hanzo.id, so no other host's session can ever
authorize cd, and clients/deploy had no login route.
Adds GET /v1/deploy/login + /callback + POST /logout against the admin-console
app (org=admin; hanzo-cloud's org is hanzo, which is why admin users were never
found). Public PKCE client, no secret. The callback verifies the exchanged token
through cloud's own JWKS validator and decides SuperAdmin on VERIFIED claims
before minting; the unverified decode is gone. session/userinfo becomes a public
bootstrap so the SPA can discover it is signed out. guard() is unchanged.
DEPLOY_PUBLIC_URL is now REQUIRED (login/callback 503 without it).
blue built, red cleared: gate stronger than before, zero regressions.
Red review of the sign-in round trip found no security defect, and four ways it
would fail in practice. All four are the same shape: correct in the happy path,
unhelpful or unreachable in the real one.
MINT ONLY WHAT THIS DEPLOYMENT WILL ACCEPT. The callback now runs the exchanged
token through cloud's OWN validator before writing the cookie, and decides the
admin-org question on VERIFIED claims. The audience allowlist is env-overridable
and jwtAudiencesFromEnv REPLACES the baked default, so a deployment whose
CLOUD_JWT_AUDIENCES / GATEWAY_ALLOWED_AUDIENCES omits this console's client_id
minted a cookie the boundary refused on the next request — 403, document-bounce
to sign-in, IAM session still live, instant code, mint, 403, forever. It now
fails once, with the reason and the knob to turn. The unverified claim decode is
gone with it.
That validator is exported from cloud (NewTokenValidator) rather than rebuilt
here: SanitizeIdentity validates on the way in, a subsystem minting a session
needs the same verdict a moment earlier, and two copies of it could drift into
exactly the mint-then-refuse loop above. jwksURLFor is now the one derivation
both share.
SIGN-IN HAS TO BE REACHABLE FROM WHERE THE USER IS. The dashboard is an XHR
client, so guard()'s document bounce never fires for it — it got a 403 and dead
-ended with no route to sign-in. /v1/deploy/session/userinfo is now the one
public bootstrap route: {loggedIn:false} plus the sign-in URL for an anonymous
caller, the real identity for a SuperAdmin. It discloses no identity, no cluster
state, no configuration, and gates nothing; every route that returns fleet data
or mutates a CR stays guarded.
The OAuth origin is now configuration, not the Host header. Deriving a redirect
from caller-controlled input is only ever saved by the registry's exact-match
check — a second lock covering for a broken first one. With no DEPLOY_PUBLIC_URL
sign-in fails closed naming the knob.
Logout is POST: as a GET any site could sign a SuperAdmin out by navigation,
which a SameSite=Lax cookie still rides. The session lifetime comes from the
verified expiry, clamped — an already-expired token no longer becomes an
eight-hour cookie, and a bogus far-future exp no longer becomes a decade-long
one. Both cookies take the __Host- prefix, which the browser only honours for
Secure, Path=/, Domain-less cookies, so a sibling *.hanzo.ai host cannot shadow
them.
Every /v1/deploy route gates on c.IsAdmin(), which SanitizeIdentity mints only
from a validated IAM principal whose org is the reserved admin org. The console
had no way to establish one: the IAM session cookie is host-only on hanzo.id, so
a session from hanzo.id or admin.hanzo.ai is never presented to cd.hanzo.ai, and
the whole surface 403'd with no sign-in anywhere. Add the round trip.
GET /v1/deploy/login redirect into IAM authorize (PKCE S256, CSRF state)
GET /v1/deploy/callback exchange the code, mint the session, land on returnTo
GET /v1/deploy/logout clear the session for this host
The session is cloud's EXISTING one: the callback writes the IAM access token to
hanzo_iam_token, the first name in cookieTokenNames, which SanitizeIdentity
already reads and independently verifies (signature, issuer, audience, expiry)
into the same principal a Bearer yields. No second session mechanism, and the
gate, the validation and the SuperAdmin predicate are unchanged.
Fail closed at every step: no code is redeemed unless the returned state equals
the nonce in the HttpOnly flow cookie this browser started with (login CSRF), a
principal outside the admin org is refused a cookie outright, and a return path
that is not a path on this host collapses to /. The cookie is HttpOnly, Secure,
SameSite=Lax and host-only, so page JS cannot read the token and no cross-site
POST carries it.
guard() keeps c.IsAdmin() as the only gate; only the shape of the refusal is
negotiated. A browser navigation is sent to the sign-in page instead of a dead-end
403; every API call keeps its 403, decided by Sec-Fetch-Dest/Mode when present and
never inferred from Accept alone, and a non-GET is never redirected.
admin-console is the client because its IAM organization is the admin org;
hanzo-cloud is owned by admin but organized under hanzo, so it resolves
admin-org users in the wrong org and never finds them.
Auth-path org resolution hit the datastore on every request, allocating the
Organization before the blocking store call, so requests stalled on the
connection pool each pinned one and the heap tracked the backlog. Confirmed
from a live goroutine profile: 46 waiters in sql.(*DB).conn under
org.Resolve <- IAMTokenRequired, organization.New at 26.4% of a 1301MB heap.
Carries three fixes uncovered while landing it:
- follow commerce's resolver consolidation (middleware/svcorg -> pkg/org)
- point the go-unit test list at clients/flags; the stale clients/featureflags
path failed setup on a missing directory and had CI/CD red on main
- assert the post-#331 flags contract: runtime flags ignore env, boot-time
ReadOnly rows still read it. That test asserted the override #331 removed
and never ran because of the stale path above.
#331 stripped Env from waitlist_*/public_signup/gateway_* so /v1/flags is
the single source of truth, but left this test asserting the env override
it had just removed. The stale clients/featureflags path meant the package
never ran in CI, so it stayed green.
Now pins both halves: a runtime flag holds its default against an env var,
and a boot-time ReadOnly row still reads env.
clients/featureflags was renamed to clients/flags; the stale path made
'go test' fail setup on a directory that does not exist, which has been
failing CI/CD on main.
Auth-path org resolution hit the datastore on every request and allocated
the Organization before the blocking store call, so requests stalled on the
connection pool pinned one each and the heap tracked the backlog. v1.49.3
serves request-owned copies from a bounded LRU and refuses credential-named
orgs at the model layer.
Brings the full run into the deployed service: 55-model DO GenAI catalog (Claude
opus-4.8/sonnet-5/fable-5/haiku, GPT-5.6/5.5/4o/o3, deepseek-v4-pro, llama-4, qwen,
glm, kimi — capabilities declared per live probe), the mean-field congestion router
(gated), the live /v1/router/judge-panel endpoint, the Mean-Field Judge Panel, and
geo-aware consent. Prod model ConfigMap (universe) syncs the catalog data separately.
Claude-Session: https://claude.ai/code/session_01SpMZ69ur3tjAXCiwaa7Wv2
Replaces the pseudo-version pin (v1.49.2-0.20260719024505-24ff20a68f52, the
Bug-A iterator-leak fix only) with the released v1.49.2, which also carries the
Bug-B guard: org.Resolve skips the doomed GetById for a legacy all-digit cached
id (IAM Valkey's stale 1772587477 for 'hanzo') and resolves by name, so the
(*Query).ById legacy-numeric path that hot-looped in v1.801.95 is never taken.
Keeps ai v1.826.4 (in-proc TierReader). go.mod+go.sum only.
The auth fix let the metering cap check actually reach commerce AuthorizeSpendCap; a
legacy-org GetById hot-loop there then HUNG every completion (no timeout on the
in-proc authorize) — a cap that can block/hang the completion path is worse than one
that does not enforce. scopeAuthorize now runs the authorize under a strict 1.5s
deadline AND a select-based hard timeout that returns even if the in-proc handler
goroutine is STUCK (an unresponsive hot-loop cannot be interrupted, so ctx alone would
not unblock). On timeout OR any error -> AuthorizeVerdict fails OPEN (allow) — a slow,
broken, or hot-looping commerce ALWAYS allows, never waits. OnCapError logs each
fail-open so a degraded cap is observable. Regression test: a 10s-hanging authorize
returns an ALLOW in ~1.5s (completion never hangs).
The commerce hot-loop itself (the root cause) is fixed separately; this timeout is the
non-negotiable safety net that makes the cap path unable to hang regardless.
v1.826.4 ships the LLM-as-judge dense-reward loop fully activated: the Mean-Field
Judge Panel (diverse calibrated judges, reputation-weighted consensus), geo-aware
consent (EU/UK/EEA explicit opt-in via CF-IPCountry, non-EU opt-out default), judge
config dynamic at admin.hanzo.ai (OrgSettings "*" row, no env), MFJP enabled by
default on a diverse cheap panel, and internal dev orgs seeded on. Judge scoring uses
the existing probe service bearer (no new secret). Also carries the MFJP + scientific
proof from v1.826.3.
Claude-Session: https://claude.ai/code/session_01SpMZ69ur3tjAXCiwaa7Wv2
commerce 24ff20a6 closes single-row query iterators (Query.First). This
stops the Postgres pool leak that starved org.Resolve on the co-resident
balance + per-tier gate path — the 'context deadline exceeded' that made
the Enso per-tier SKU gate fail open and spiked chat latency to 10-40s.
Wire() gained the /v1/dns zone plane (after projects) and /v1/cloudflare edge
plane (after integrations) but the frozen golden in wire_test.go was not
updated, so TestWireOrderMatchesFrozen failed (87 specs vs 85 frozen) — which
red-lit cloud's CI/CD and blocked the auto-release image build. Refreeze the
golden to the exact runtime sequence (verified position-by-position, 87==87).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ONE Kubernetes noun, proxied to Visor (clients/visor/k8s.go): list the
org's DOKS clusters, one cluster's detail (node pools + worker nodes), DEPLOY
(create) / delete clusters, and the fleet-wide worker nodes. Reads are org-scoped
by the validated IAM owner; mutations (create/delete) are admin-gated
(principal.IsSuperAdmin || IsOrgAdmin) — real house-account infra spend.
Consolidates the worker-node consumption: managedMachines now reads
/v1/k8s/nodes (was /v1/kubernetes-nodes), matching Visor's consolidated path —
no parallel kubernetes-* surface remains.
- k8s.go: listK8sClusters / getK8sCluster / createK8sCluster (admin) /
deleteK8sCluster (admin) / listK8sNodes; wire structs + view mappers.
- visor.go: mount the /v1/k8s/* group; managedMachines -> /v1/k8s/nodes.
- tests: proxy + tenant-scoping, detail shape, nodes, and the admin gate
(a non-admin create/delete is refused BEFORE reaching Visor); the fleet
DOKS-node fake tracks the new /v1/k8s/nodes path.
Claude-Session: https://claude.ai/code/session_01SpMZ69ur3tjAXCiwaa7Wv2
- buildJobSpec doc claimed the REVERTED over-hardening (allowPrivilegeEscalation=
false, all caps dropped); correct it to the actual documented rootless posture
(defaults left for rootlesskit newuidmap) + point at the securityContext.
- tenantPullSecretName comment overstated "cloud-api holds no secrets grant";
clarify cloud's only Secrets write is the per-tenant KMS-auth creds in a TENANT
ns, and that the isolated build ns must stay OFF the tenant-RBAC selector so no
secrets grant is projected there (R6).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GET /v1/platform/projects 500'd on console dashboard init. The iamStore guard
converts a nil co-resident IAM object store into a typed 503, but listProjects
re-stamped ANY store error as a 500 (zip.Errorf(500, "list: %v", err)),
discarding the status — so a signed-in session's first read broke dashboard
init with {"status":500,"error":"list: platform requires the co-resident IAM
store, which is not initialized"}.
The dashboard's first authenticated read now degrades any store failure to an
empty project set (200 []) — a new org genuinely has zero projects — logging the
real cause for operators (never swallowed), written in-band so no outer error
filter can reflatten it. Also guards a stray nil row from nil-derefing into a
500. The store-level 503 guard + its three unit tests are unchanged.
Repro + regression gate: TestListProjects_NilIAMStore_ServesEmpty200 (real
iamProjects over a nil in-process IAM engine — the deployed condition) and
TestListProjects_StoreError_ServesEmpty200.
RED found the /v1/paas auth broadening handed every brand-org ("hanzo") OrgAdmin
fleet-wide rolling-restart of the platform's OWN tier (the only namespaces the board
scans are hanzo{,-testnet,-devnet}, where iam/kms/gateway/cloud/… run) — a live DoS
lever, partially re-opening the 2026-07-08 admin-org P0.
H1: the MUTATING POST /v1/paas/apps/:app/deploy now uses operatorGuard (principal.
IsSuperAdmin ONLY), not the broad read guard. Restarting a shared platform service is
a platform-operator action; a customer-org admin — even of the brand org — is refused
403. The READ board (list/get) stays SuperAdmin||OrgAdmin (observe, audit-logged,
bounded). Confinement (scopedNamespaces) unchanged.
L1: deploy REQUIRES ?env=main|test|dev (nsForEnv-validated) — a bare deploy no longer
silently targets production; the CLI requires --env before the call.
Tests: TestDeploy_OrgAdmin_403_Platform (the H1 regression), _NonAdmin_403,
_SuperAdmin_RollingRestart, _RequiresExplicitEnv, _SuperAdmin_EnvSelectsNamespace;
CLI TestDeployRequiresEnv. All green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The embedded ai per-tier SKU gate (family_tier.go) was fail-open in-cluster: it
resolved the caller's tier with an authed HTTP self-call to the cloud edge, which
401/403s a service token on /v1/billing/*, so the gate saw "" and admitted every
tier — enso/enso-ultra were open to free callers.
Mirror wireFinance's SetBalanceReader: install aiobject.SetTierReader so ai reads
the subscription tier DIRECTLY over the co-resident commerce client the metering
gate already bills over (commerceinproc in-process, with the service token commerce
itself accepts) — never the cloud edge. Add metering.Client.Tier to decode tier.name
from GET /v1/billing/tier. Fail-safe preserved: a commerce error or unknown tier
folds to "" (allow), so a commerce blip never locks out a paying caller.
Bumps ai v1.824.2 -> v1.825.2 (the object.TierReader seam).
managedMachines unioned Visor's registry (/v1/get-machines) + live droplet list
(/v1/machines); a DOKS cluster's worker NODES appeared in neither (their droplet
carries a k8s tag, not a hanzo-org droplet tag), so world.hanzo.ai showed
standalone droplets but never cluster nodes.
Add GET /v1/kubernetes-nodes as the THIRD source (Visor unions the house-account
hanzo-org-tagged clusters + BYOC Provider.ClusterID clusters and returns each
worker node as a Machine keyed by droplet id). It is processed after registry and
live, so a DOKS node whose droplet is ALSO in the live list dedupes by droplet id
and never lists twice; a cluster-only node surfaces. Independently resilient like
the other two — a kubernetes-nodes outage is logged and skipped, never hiding the
registry/live/BYO sources.
Test: TestMachinesMergeDOKSNodes — a DOKS-only node appears, and a node whose
droplet is already live collapses BY ID (the node row carries a different name, so
only id-dedup can merge it). Full clients/visor suite green (31 subtests).
Claude-Session: https://claude.ai/code/session_01SpMZ69ur3tjAXCiwaa7Wv2
POST /v1/event is the ONE ingestion door: body is a single Event or a JSON-array
batch (no /v1/event/batch), org resolved IAM-only and fail-closed (eventTenant),
funneled through the ONE write core (ingestEvents) into hanzo.events. The
Segment/beacon (/v1/analytics,/v1/tracker) and PostHog (/v1/insights/e) wires
become thin DEPRECATED adapters over the same core. Org is never read from body.
capture resolves a presented project/API key to its owner org through the single
IAM key resolver (sharedKeys, 60s cache incl. miss-cache). A presented-but-
unresolvable key is refused (403) and NEVER falls through to the brand-host
fallback, so a keyed request can never cross-tenant write. Anonymous marketing
traffic still resolves to the public brand org server-side from Host.
Brings the merged Enso router fixes into the deployed cloud binary:
- #107 (v1.825.0): auto never routes to a family SKU it can't serve + forward the
resolved model (withModel body rewrite) → model=auto serves 200 (was 404); grant-
aware known predicate; flywheel boots from the single shared Bootstrap.
- #108 (v1.825.1): trainer fits EARLY (~90s after boot) then cadence → completed
retrain cycles survive frequent redeploys (churn-resilient).
./apps (ai.Mount site) compiles clean against v1.825.1 (API-compatible).
Co-authored-by: zeekay <zeekay@hanzo.ai>
The metering cap-gate authorize (and the SuperAdmin cap-oversight Forward) call the
in-proc /v1/billing/spend-alerts/authorize with the COMMERCE_SERVICE_TOKEN, but the
customer /v1/billing/* bridge required a validated IAM principal -> "sign in to view
billing" (403) -> the cap fails-open and never enforces. billingData now admits a
trusted S2S caller carrying the verified service token: org from the EdgeAuth-controlled
X-Org-Id, query forwarded VERBATIM (a trusted caller names its own subject), no
subject-pin. A public caller can never present it (the gateway 401s a Bearer that is
not an IAM JWT / hk-|pk-|sk- key), and an unauthenticated caller still gets 403 (tested).
Adds spend-alerts/authorize to the GET allowlist; constant-time token compare.
ATOMIC with the flag: bumps commerce to v1.49.2 (SPEND_CAP_ENFORCE, default OFF), so
the instant the cap can reach the handler the enforcement gate is fail-open in the
binary -> auto-deploy stays safe until an operator flips the flag after the canary proof.
Security invariants tested: public/unauth -> 403; wrong bearer -> 403; verified token +
X-Org-Id -> forwards verbatim; token без X-Org-Id -> 403.
The per-org Cloudflare asset plane (Pages/Workers/R2/KV/D1) is repointed from a
first-level /v1/cloudflare/* surface to /v1/integrations/cloudflare/*, so a
third-party provider is connected AND used under one unified namespace — matching
where the connector's connect/callback/verify/disconnect legs and the KMS token
coordinate already live. Pure path move: no auth, isolation, or handler logic
changes. Route registrations, doc comments, and tests repointed together.
No collision with the connector's parametric routes: the asset paths are all
3+ segments (/cloudflare/{pages,workers,r2,kv,d1}/...) while the connector's
/v1/integrations/:provider and /:provider/{connect,callback,disconnect,verify}
are 1- and 2-segment patterns whose literal second segment never equals an asset
group. Static-under-param co-registration is already proven in the integrations
plane (slack/link static beside :provider). Mount order unchanged: integrations
before cloudflare.
Assisted-by: Claude:claude-opus-4-8
The first rootless spec over-hardened (allowPrivilegeEscalation:false +
capabilities drop ALL), which breaks rootlesskit's setuid newuidmap/newgidmap
sub-uid mapping — proven by an on-cluster canary:
newuidmap ... failed: operation not permitted
Relax to the documented moby/buildkit k8s rootless posture: privileged:false,
runAsUser/Group 1000, runAsNonRoot, seccomp+AppArmor Unconfined, and leave
allowPrivilegeEscalation / default caps at k8s defaults (newuidmap needs them).
Still user-namespaced, no host root — the decisive win over privileged=true.
Re-canaried: rootless build + scoped push-hanzoai cred pushed to ghcr OK.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`hanzo apps list`, `hanzo deploy`, `hanzo clusters` targeted the OLD TS-Dokploy
contract (/v1/apps, /v1/org/{org}/cluster, /v1/org/.../redeploy) — all 404 on the
live Go cloud (ghcr.io/hanzoai/cloud). Repoint the CLI at the endpoints the Go
cloud actually serves, authorized off the SAME IAM login `hanzo build` now uses
(no --platform-token). Drift confirmed live as z@hanzo.ai:
/v1/apps → 404 /v1/paas/apps → 403 (was SuperAdmin-only)
/v1/org/*/cluster → 404 /v1/clusters → 200 (already org-scoped)
/v1/platform/projects → 500 (co-resident IAM off)
CLI (cli/platform.go, cli/commands.go):
apps list/get → GET /v1/paas/apps[/{app}] (no client org filter — the board is
confined to the caller's org SERVER-side by the validated identity)
deploy <app> → POST /v1/paas/apps/{app}/deploy (rolling restart; --env selects
the lifecycle namespace; org from identity, not the path)
clusters list/get → GET /v1/clusters (Visor-managed + BYO; org from identity)
Removed the TS-contract vestiges with NO Go backend: `apps sync` (the board is
live-computed), `clusters create/select/install-baseline/target` and `k8s target`
(DOKS provisioning + deploy-target selection are not implemented on the Go cloud).
Reshaped the Cluster DTO to the live visor clusterView (dropped dead Phase/Active/
operator/baseline fields). Platform client doc corrected: it CAN validate IAM tokens.
Backend (clients/paas): authorize the fleet board off ONE IAM identity, exactly like
/v1/runner (clients/platform/runner.go). guard now admits a validated principal who
is a SuperAdmin OR an OrgAdmin (principal.Validated + IsSuperAdmin || IsOrgAdmin —
the ONE verifier, unforgeable off-gateway), and each handler CONFINES a non-super
caller to the platform namespaces its own validated org owns (scopedNamespaces, keyed
on principal.Org — never a client header): a SuperAdmin sees the whole fleet, an
OrgAdmin only its own org (empty board / clean 404 otherwise), so a tenant admin can
never observe — or restart — another org's, or a platform, app. `?org=` cannot widen
the view (confinement is at the namespace scan, before the filter).
deploy is now a real zero-downtime ROLLING RESTART (the kubectl-rollout-restart
mechanism: stamp the pod-template hanzo.ai/restartedAt annotation) instead of the
409-refuse. It never changes the declared TAG (that stays a git commit, the one thing
Hanzo CD's selfHeal reconciles), so there is no drift to revert — the honest,
GitOps-compatible "redeploy this app". resolveTarget split so the machine release
path (release.go) keeps the full scan; the identity-scoped deploy/getApp use the
caller's authorized namespaces.
SECURITY: this broadens auth on a control-plane surface (new mutating restart path).
Mirrors blue's IAM-admin pattern; flag for red. TDD: 12 new paas cases (role gate,
tenant confinement on list/get/deploy, forged-org cannot widen, rolling-restart lands
the annotation, foreign-org 404 + no mutation) + the CLI path/DTO tests, all green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RED re-review of the unify-infra PaaS-auth flip found 2 HIGH + 1 MED on the
privileged build endpoint. Fixes:
H1 (borders CRITICAL) — cross-org supply-chain push. imageAllowed() permitted
ghcr.io/{hanzoai,luxfi,zooai}/ regardless of the caller's validated org, so any
org-admin could overwrite another brand's prod image via the shared push cred.
Bind the image's registry-org to the caller's org (orgRegistryNamespaces map);
only a real SuperAdmin may cross. The machine (fabric) token keeps full
owned-registry latitude. Cross-org now 403; same-org 202.
H2 — privileged rootful buildkit in the main platform namespace with the shared
3-org push cred. buildJobSpec is now ROOTLESS (moby/buildkit:*-rootless, uid 1000,
no privileged, no privilege-escalation, caps dropped, --oci-worker-no-process-
sandbox), runs in a DEDICATED isolated namespace (CLOUD_PLATFORM_BUILD_NS default
→ hanzo-build, off the platform ns), and mounts ONLY the target org's push
credential (push-<namespace>), never the shared kaniko-ghcr. Node-pool taint +
automountServiceAccountToken:false retained.
M1 — `image` bypassed validateBuildInputs → buildkit --output attribute
injection (ghcr.io/hanzoai/x,registry.insecure=true). Added validateImageRef
(strict single OCI ref, rejects comma/space/quote/newline/'='), folded into
validateBuildInputs and enforced early at the handler.
I2 — the identity validator now fail-closes on an empty resolved issuer OR
audience set instead of silently disabling that axis; issuerAllowed denies on
an empty trusted set.
TDD: cross-org 403 + same-org 202 + SuperAdmin cross-org + orgless-403 +
image-injection-reject + rootless/scoped-cred spec + empty-trust-set-deny all
green (pure-Go, as prod ships).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The prior prefix guard checked fasthttp's URI().Path(), which decodes only ONE
layer of percent-encoding. A DOUBLE-encoded traversal survives that one decode as
a literal %2e/%2f that still KEEPS the /v1/dns/ prefix -- so the prefix check
passes, cloud forwards base + /v1/dns/%2e%2e/admin, and the upstream decodes the
second layer to /v1/admin. Proven bypasses: /v1/dns/%252e%252e/admin,
/v1/dns/%252e%252e%252fadmin, /v1/dns/..%252fadmin.
After the prefix check, also refuse any once-decoded path that still carries a `%`
(a still-encoded byte => the client double-encoded) or `..` (residual traversal).
Neither appears in a legitimate DNS-API path -- zone labels are DNS names /
punycode xn--, and the query string (checked separately) is unaffected. Fail
closed 400 before a byte leaves cloud.
Also relay the upstream Location header so a 3xx -- never followed, per
CheckRedirect -- passes back verbatim (status + Location) as the comment claims,
rather than being silently dropped.
Regression: the escaped-path test gains the 3 double-encoded vectors (each refused
400 with 0 upstream bytes), plus a redirect test proving an upstream 302 is not
followed and its Location relays verbatim. 9 tests green.
Assisted-by: Claude:claude-opus-4-8
The forward head built the upstream target from uri.Path(), which is NORMALIZED
and percent-decoded. Fiber matches the /v1/dns/* wildcard on the RAW path, so a
dot-segment or encoded-dot traversal (/v1/dns/../../admin/secrets,
/v1/dns/..%2f..%2fadmin, /v1/dns/../../../metrics) still routed to the handler
while the normalized path escaped the prefix -- letting the caller drive the
WHOLE path on the DNS host. Contained today only because the upstream 404s
unknown paths; a latent path-scope escape the moment :8443 serves anything else.
Guard the normalized path fail-closed BEFORE building the target: require it to
be exactly /v1/dns or under /v1/dns/, else 400 and forward nothing. Because the
path is already normalized, every traversal/encoded-dot escape fails this check.
Correct the comment that wrongly claimed the path was locked by the route match
(the host-pinning claim was, and stays, true).
Also stop the shared http.Client from following upstream 3xx (CheckRedirect =>
http.ErrUseLastResponse) so redirect responses pass through verbatim and a 3xx
can never silently re-target the request onto another host or path.
Regression test proves fail-closed: each escaped path is refused (400) and 0
bytes reach the upstream. Existing 7 tests stay green (8 total).
Assisted-by: Claude:claude-opus-4-8
Addresses Red's FIX-THEN-SHIP findings:
- HIGH: stamp X-Hanzo-Org (the served org) on every /v1/cloudflare response so a
per-org caller can detect a pinned/comingled org. The platform Pages client asserts
it equals the requested org and fails LOUD if a non-org-switch-capable service token
made the identity boundary pin X-Org-Id to the token's own owner — no silent
cross-tenant read/write.
- MEDIUM: gate mutations (POST/PUT/DELETE) on principal.IsOrgAdmin via a new authWrite
front door (reads stay validated-org-only), parity with the AdminOnly connector. A
non-admin is refused before any KMS token read and never reaches Cloudflare.
- LOW: resolveAccount now prefers the account captured at connect time
(integrations.ConnectionFor ExternalID), falling back to live /accounts discovery
only when none is stored — no per-call round-trip, deterministic for multi-account
tokens.
Tests: +TestResponseStampsActingOrg, +TestMutationRequiresOrgAdmin,
+TestStoredAccountSkipsDiscovery; existing mutation tests drive as org admin. 12/12
pass, -race clean.
Assisted-by: Claude:claude-opus-4-8
New cloud subsystem clients/cloudflare exposing /v1/cloudflare/{pages,workers,r2,kv,d1}/*,
sibling to hanzodns's /v1/dns. It reads each org's KMS-sealed Cloudflare token in-process
through the integrations custody seam (integrations.TokenFor) and proxies to the Cloudflare
API v4 with the cfDo shape reused verbatim from hanzodns — no global env token, no
bearer-relay hop (that is only hanzodns's separate-process need).
Tenant isolation: org is derived ONLY from the validated principal (principal.Org); the KMS
token path is keyed on that org, so cross-org token reach is structurally impossible and an
unvalidated request fails closed (403). Pages (project CRUD, deploy, custom-domain add/delete)
and Workers (script put/list/delete via multipart module upload, workers.dev subdomain, zone
route bind/list/delete) are wired; R2/KV/D1 ship typed provider methods + routes that answer
an honest 501 (never a fake success).
Appends the Workers connector scopes (Account:Workers Scripts:Edit, Zone:Workers Routes:Edit)
for the now-callable capabilities and wires the subsystem into apps.Wire after integrations.
Assisted-by: Claude:claude-opus-4-8
House rule: no extraneous /api/, just /v1/. The projection API moves from
/v1/deploy/api/v1/* → /v1/deploy/<resource> (settings, session/userinfo, version,
account/can-i, applications, applications/{name}/resource-tree, .../{sync,rollback}).
This IS the deploy API now — the superseded native /v1/deploy/{applications,:name/*}
routes are removed (their readers stay, reused by the projection). health +
reconcile unchanged. Guard test updated.
console.hanzo.ai serves the DnsModule but cloud held no /v1/dns head, so
console.hanzo.ai/v1/dns/* 404'd and the dashboard showed empty zones. Add a thin
forward head (clients/dns) that relays each /v1/dns/* request to the DNS control
plane (HANZO_DNS_URL, default the in-cluster coredns-hanzodns service), preserving
verb, path, query, body, status codes and error bodies.
Isolation is bearer-relay: the head forwards the caller's OWN validated bearer
(cloud.CallerBearer) plus the server-validated X-Org-Id, substituting NO service
credential, so the DNS plane's own per-org authorization still holds and a caller
in org A can reach only org A's zones. Fail-closed: no validated principal => 403,
before any byte leaves cloud. It builds a fresh upstream request, so no inbound
header is blindly relayed; the upstream host comes only from env (no SSRF).
Decomplect the token resolution the identity boundary and this relay both need
into one callerToken helper (validatedPrincipal now delegates to it) and expose
CallerBearer for the relay; an opaque API key is never relayed as a bearer.
Assisted-by: Claude:claude-opus-4-8
The monochrome dashboard now serves at cd.hanzo.ai/ (root, base-href /) from the
cd-ui App CR (hanzoai/spa); cloud keeps ONLY the IAM-gated projection API at
/v1/deploy/api/*. Drops the go:embed FE + the deploy-ui-embed Dockerfile stage.
The FE is no longer /v1/-prefixed and no longer baked into the money binary.
The monochrome dashboard SPA now ships as the hanzoai/spa-based cd-ui App CR
served at cd.hanzo.ai/ (base-href /); cloud keeps ONLY the IAM-gated projection
API, moved from /v1/deploy/ui/api/* to /v1/deploy/api/* (same-origin with the
SPA). Removes the go:embed dashFS + static serve (dashStatic/serveDashIndex) +
webui/dist + the deploy-ui-embed Dockerfile COPY stage. Guard test updated to the
/v1/deploy/api/* routes (still 403 without SuperAdmin). RED invariant holds — the
API terminates in cloud behind IdentityMiddleware + the guard.
The platform build muscle (launchDirectBuild) clones an https git URL; the CLI
sent the bare `owner/name` positional verbatim, so `hanzo build luxfi/wallet`
failed server-side with "repo.url must use https". normalizeRepoURL expands a
bare owner/name to https://github.com/owner/name (the host for every
hanzoai/luxfi/zooai repo) and passes an explicit URL / scp-style remote through
untouched. Test: TestNormalizeRepoURL. Completes the IAM-login dogfood:
`hanzo build luxfi/wallet ... --sha <full> ...` now 202s + launches the build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Publish routing is now org-scoped: a project publishes to
<slug>.<org>.<apex> (e.g. myapp.maxpower.hanzo.app) instead of the flat
global <slug>.<apex>. The slug namespace becomes per-org — two orgs can
own the same slug and their sites can never collide or shadow one another.
- deploy.go: siteHost(org,slug)=<slug>.<org> is the ONE bound/resolved key;
onPublish binds it; siteURL renders https://<slug>.<org>.<apex>.
- sites.go siteSlug: accept the two-label host <slug>.<org>.<apex>, validate
both labels (slug non-reserved), return <slug>.<org> as the resolve key so
bind and resolve agree. Org isolation is now STRUCTURAL in the hostname.
- store unchanged: site_hosts already keys on arbitrary full host strings
(custom-domain path proves exact full-host ResolveHost match).
containment.yml: add a required "zen streaming-fix floor" check that fails
any PR/push whose effective github.com/hanzoai/zen is below v1.4.1 (the
first release carrying the SSE body-close fix, commit 50328b8). A stale
branch that reverts go.mod's zen pin to v1.4.0 can no longer silently
re-break streaming (empty SSE completions) — the durable root-cause guard.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The one admin mint surface. SuperAdmin-only (core.Guard); forwards verbatim to
commerce's already-mint-gated POST /v1/billing/credit-grants (middleware.Mint →
PlatformOnly) via COMMERCE_SERVICE_TOKEN, scoped to the target org, and writes one
tamper-evident audit record. Commerce stays the sole credit-grant ledger — no
in-process mint. NOT deployed; for red review (mint surface).
Assisted-by: neo:claude-opus-4-8
A plain `hanzo login` (IAM) now authorizes every PaaS control-plane op with no
separate --build-token / --platform-token. ONE identity, org+role scoped.
CLI (cli/cli.go): Env.buildToken() and Env.platformToken() fall back to the IAM
access token as the FINAL resort (precedence unchanged above it: flag > env >
credential-store service token > IAM login). So after `hanzo login` the CLI sends
the IAM JWT as the platform bearer. "No token" errors now point at `hanzo login`
and fire only when there is ALSO no IAM login. Tests: added
TestBuildTokenFallsBackToIAM / TestPlatformTokenFallsBackToIAM (precedence
preserved — a dedicated token still wins).
Platform (clients/platform/runner.go): /v1/runner (build-enqueue) — the one
control-plane endpoint that ignored identity — now accepts EITHER the shared
build-callback token (machine path: git-push, self-release, operator; constant-time,
unchanged) OR a validated IAM principal who is an admin (principal.IsSuperAdmin ||
principal.IsOrgAdmin over principal.Validated). Both bounded by the SAME
owned-registry allowlist, so identity never widens the image boundary. Release
self-publish stays machine-token-only. IAM builds are org-attributed to the caller's
VALIDATED org and refuse a foreign organizationId unless SuperAdmin. Reuses the ONE
identity verifier (SanitizeIdentity mints unforgeable X-User-* from the verified JWT)
— no parallel JWT crypto. deploy/apps were already IAM-authorized via the tenant()
boundary. Tests: 7 new IAM cases (admin launches, non-admin 403, forged-no-user 403,
disallowed-image 403, foreign-org 403, release 403) + corrected fail-closed 403.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The argo/gitops merge d3f60be (v1.801.77) resolved the go.mod conflict to its
stale second parent (zen v1.4.0), silently reverting the v1.4.1 bump landed at
v1.801.76. v1.4.0 still carries the serve() `defer resp.Body.Close()` race that
empties every streamed completion (200 with a 0-byte body) — which broke every
hanzo.app builder stream. Re-pin to zen v1.4.2 (the body-close fix + reasoning_content
passthrough regression pin) and thinking v0.1.1 (reasoning default). Streaming
now forwards every chunk, including delta.reasoning_content.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The deploy-ui-embed image is published; cloud's Dockerfile now COPYs its /dist
into clients/deploy/webui/dist (go:embed). release.yml builds a cloud image that
serves the REAL monochrome dashboard at /v1/deploy/ui instead of the fallback.
Mirrors the console-embed stage: FROM the prebuilt deploy-ui-embed image, COPY
/dist -> clients/deploy/webui/dist (go:embed source for /v1/deploy/ui). GATE: do
NOT merge until ghcr.io/hanzoai/deploy-ui-embed:latest is published (else the
build cannot pull the base). Until merged, the money binary serves the fallback.
LOW-2: TestDeployRoutesRequireAdmin asserts every /v1/deploy(/ui) route 403s
without X-User-IsAdmin and passes with it (health stays public) — guards against
a future unguarded-route refactor.
INFO-1: the unauthenticated /v1/deploy/health path reports booleans only; the raw
k8s error (apiserver/RBAC detail) is logged server-side, not returned.
LOW-1 (CSRF): verified no-op — the IAM session cookie is SameSite=Lax AND the
ambient cookie->JWT bridge is same-origin-gated (sessionBridgeSameOrigin), so a
cross-origin CSRF POST gets no identity and the deploy guard 403s.
Serves the full ArgoCD React UI fed a read-projection of operator App CRs shaped
as v1alpha1 Applications — no argocd api-server/repo-server/redis/stored CRD.
SuperAdmin-gated; IAM owns identity at the edge. Money binary builds green;
projection render tests pass. Real monochrome bundle is a CI artifact (make
deploy-ui / deploy-ui-embed image); fallback shell until that lands.
CI story for the dashboard bundle, mirroring make webui: DEPLOY_DIR=<hanzoai/deploy
rebrand/hanzo-monochrome> yarn build -> clients/deploy/webui/dist (go:embed). Only
the fallback index.html + .gitignore are tracked; the real 43MB bundle is
build-time-only. Money binary builds green with the real bundle embedded.
Serves the full ArgoCD React UI fed a READ-PROJECTION of operator App CRs shaped
as v1alpha1 Applications — NO argocd api-server, NO repo-server, NO redis, NO
stored Application/AppProject CRD. projection.go maps App CR -> Application +
resource-tree (reusing the native readers/engine health). dashboard.go
reimplements the UI's api-server subset (settings/userinfo/version/can-i +
applications list/get/resource-tree + sync/rollback->App-CR reconcile) + serves
the go:embed'd monochrome bundle with base-href rewrite. SuperAdmin-gated; argocd
auth disabled (IAM owns identity at the edge). Projection render tests green.
UI bundle is a CI artifact (committed fallback shell; make deploy-ui overwrites).
v1.824.2 unmasks the router-stats model ids (arm-N → real names like zen5-coder /
opus-4.8) for the PLATFORM scope when the caller is a super-admin of the own brand;
every other caller keeps the arm-N privacy masking. So the world.hanzo.ai admin view
(Routing Throughput / Enso arms) shows actual models instead of "Enso arm 2".
Claude-Session: https://claude.ai/code/session_01SpMZ69ur3tjAXCiwaa7Wv2
world's admin fleet (GET /v1/machines -> listMachines) sourced machines only
from Visor's registry (/v1/get-machines), so DigitalOcean droplets that were
provisioned but not (yet) in the registry never appeared -- DO nodes were
entirely missing from the fleet.
Source the managed-machine set as the deduped UNION of the registry AND
Visor's LIVE DO reseller list (GET /v1/machines -> ListComputeMachines ->
service.ListOrgMachines, the live Droplets.ListByTag(orgTag)). Dedup is by
provider id OR name; the registry entry wins a collision so its enrichment/
masking is preserved. One helper (managedMachines) now feeds listMachines,
listGPUs and the /v1/fleet board so all three agree on which machines exist,
not just how they normalize. BYO fold unchanged; only machines Visor actually
returns are surfaced (nothing fabricated).
Claude-Session: https://claude.ai/code/session_01SpMZ69ur3tjAXCiwaa7Wv2
buildMeteringClient ignored the documented METERING_TEST env, so the metering client
was ALWAYS live (c.test=false) — a staging/canary could not route debits to the
sandbox books, and the usage-cap smoke would have moved real money. Now METERING_TEST=true
sets Config.Test, so fin.RecordUsage writes the TEST finance books and the cap read
(org.TestMode via SQUARE_ENVIRONMENT=sandbox) sees the SAME test books. Unset in prod
= live, unchanged.
The spend cap read commerce's transaction store, which the co-resident cloud binary
leaves EMPTY (usage is recorded via fin.RecordUsage on the finance ledger) — so in
prod the cap summed 0 and never enforced, and the alert never fired. This wires the
cap onto the ledger prod actually writes, ORG-WIDE (the finance Entry carries no
scope; per-scope is a follow-up):
- sqlstore.SumByKindSince: additive read-only aggregate (kind + created_at index,
18-decimal TEXT folded in Go) — no Entry schema change.
- finance.SumUsageSince: the org's metered usage (cents) since a cutoff, deposits
excluded, sandbox books for a test org.
- finance.SetUsageHook: dependency-inverted post-debit seam (finance never imports
commerce) the cap alert fires through.
- apps/commerce.go: SetPeriodSpendReader(financePeriodSpend) so AuthorizeSpendCap
reads finance spend since the UTC month start, and SetUsageHook(fireCapAlert) so a
finance debit fires the org's spend-alerts on the same crossing.
Composes the commerce policy/CRUD/promo/admin/ancestor-fix (commerce
v1.49.1->v1.49.2 injection seam) — a targeted host re-wire, not a redo.
Completes the Enso per-tier gate: ai v1.824.1 already enforces min_tier at the
family pipe + auto-router; this bumps the co-resident commerce so /v1/billing/tier
returns the caller's REAL plan (was stubbed always-Free). Fail-open on uncertainty.
v1.824.1 boots StartRouterTrainer + StartRouterProbe from ai.Mount, so the flywheel
runs in the deployed (embedded-in-cloud) service, not just the standalone aid binary.
Both still self-gate on their env flags; universe sets ROUTER_TRAIN_ENABLED=1 to turn
training on. Also carries the retrain-timeline fix (retrains now count).
Claude-Session: https://claude.ai/code/session_01SpMZ69ur3tjAXCiwaa7Wv2
Five guards before any deletion: (i) refuse an empty desired set; (ii) dry-run
sizes the prune set + a count/ratio fuse (DEPLOY_ENGINE_PRUNE_MAX default 10,
_RATIO default 0.20) refuses a mass prune; (iii) WithPruneConfirmed gates prune
on the fuse passing; (iv) PVC + KMSSecret are excluded from prune entirely (data
anchors, irreversible); (v) parseManifestDir walks recursively so a nested
manifest is never silently dropped (which prune would read as a deletion).
prune stays off by default (DEPLOY_ENGINE_PRUNE).
Drops the filesystem replace => ../deploy/gitops-engine. The fork's engine module
was renamed to its real repo path (github.com/hanzoai/deploy/gitops-engine, tag
gitops-engine/v0.7.2) so cloud requires it as a normal pinned version — CI builds
the money binary with NO sibling checkout, NO argoproj alias. tidy + scoped build
green over SSH.
Every merge to main builds + smoke-tests + tags a proven image, but nothing
recorded that tag as the desired state Hanzo CD deploys, so api.hanzo.ai sat on
a stale pin (v1.801.71) while proven images (…72-…75) never rolled. The old
image-update.yml deploy hub was deleted in the Hanzo CD cutover; a direct CR
patch is reverted by ArgoCD selfHeal.
Add a promote job that, after the tag receipt, bumps spec.image.tag in
hanzoai/universe crs/cloud.yaml and commits deploy(cloud): <tag> — the SAME
yq-bump the hanzoai/ci reusable does for every other service. The universe-crs
ArgoCD Application (automated sync + selfHeal) then reconciles it to the cluster.
No hand-dispatch, no hand-edit.
Fixes empty streaming completions for zen5* (hanzo.app builder P0):
- zen v1.4.1 stops serve() from closing the upstream body before fiber's lazy
SendStreamWriter drains it (every SSE completion was truncated to a 200 + empty body).
- thinking v0.1.1 makes glm-5.2/deepseek Off send reasoning_effort:none, so zen5-coder
streams the answer immediately instead of a long silent content:null reasoning preamble.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the /v1/admin control plane admin.hanzo.ai drives, twinning /v1/admin/flags:
- /v1/admin/promos (GET/PUT, core.Guard SuperAdmin) → commerce /v1/platform/promo:
configure the admin-controlled plan promo (percentOff/start/end/plans/active).
- /v1/admin/spend-caps (GET/POST/PATCH/DELETE, core.GuardScoped) → commerce
/v1/billing/spend-alerts with X-Org-Id: oversee/override ANY org’s usage caps
(SuperAdmin via ?org=; a scoped admin hard-pinned to their own — the escalation
line). Reuses the customer’s OWN spend-alert rows, no parallel model.
commerce.Forward is the ONE service-token seam these ride, relaying commerce’s own
status so a 400/403/404 surfaces honestly instead of masking as success.
Fixes clients/metering/middleware.go defaultOnDenied: a FUNDED caller over a
per-scope spend cap now maps to a DISTINCT 402 spend_cap_exceeded (errors.Is), not
the 503 it fell through to — parity with the zip-native denyVerdict, so any product
on the net/http middleware surfaces the same honest verdict.
Brings the ACTIVE Hanzo KMS custody plane (api.hanzo.ai/v1/kms/*, served by the
cloud-embedded luxfi/kms per HIP-0106) to the latest v1.x. keys (v1.4.1) and
crypto (v1.20.2) already latest; luxfi/mpc stays out of the graph (threshold
signing is a wire-coupled external daemon, not a linked module). v1.12.4 verifies
against the public sumdb; clients/kms + cmd/cloud build green.
toMachineView's memGB fallback read the integer before "gb" out of any
size slug. On a DO GPU droplet the slug's gb is VRAM (gpu-h100x8-640gb ->
640 GB VRAM), not system RAM, so a GPU node missing its upstream memSize
would render 640 GB of "system memory" -- a misleading number.
Guard the fallback with the GPU check already needed for v.GPU: reuse the
single gpuSpecOf(slug) call (spec, isGpu) and apply the slug's gb figure
only when !isGpu. Real m.MemSize still takes precedence for every provider,
GPU nodes included, so a GPU machine that reports its true RAM is
unaffected -- only the VRAM-as-RAM fallback is suppressed.
Table tests: a gpu-h100x8-640gb slug with empty MemSize yields Mem=="" (not
"640 GB") while still resolving GPU=="H100", and the same slug with a real
MemSize=="1920gb" reports "1920 GB".
Claude-Session: https://claude.ai/code/session_01SpMZ69ur3tjAXCiwaa7Wv2
The fleet view (world.hanzo.ai cloud variant) renders machines from
cloud's /v1/machines -> listMachines -> toMachineView. Two honest-data
gaps left system RAM and DigitalOcean vCPU counts blank:
1. System memory was never surfaced: machineView had no memory field and
toMachineView never read the upstream memSize, so every provider's RAM
column rendered empty.
2. DO vCPU was dropped: toMachineView filled vcpu only when CpuSize parsed
as a bare integer, but DigitalOcean reports size SLUGS (s-4vcpu-8gb),
so strconv.Atoi failed and vCPU showed nothing.
Fix:
- Add MemSize to visorMachine (upstream already sends it; it was simply
unmapped) and Mem to machineView.
- parseSizeSlug pulls the integer before "vcpu" and the integer before
"gb" out of a size slug (s-4vcpu-8gb -> 4,8; g-8vcpu-32gb -> 8,32).
- normalizeMem renders "N GB" only for trustworthy inputs (explicit
gb/gib, explicit mb converted with rounding, a bare integer as MB when
>=1024 else GB) and returns "" for anything ambiguous -- never a
fabricated number.
- toMachineView keeps the Atoi(CpuSize) path and falls back to the slug's
vcpu; sets Mem from normalizeMem(MemSize) and falls back to the slug's
gb figure. The GPU-spec logic is unchanged.
Table tests cover the slug parser, the mem rounding, and the mapper
precedence (explicit values win, honest omission when neither yields one).
Claude-Session: https://claude.ai/code/session_01SpMZ69ur3tjAXCiwaa7Wv2
collectOutputs read only the 'images' key, but SaveGLB publishes under '3d', so a
generated .glb never mirrored to the org library. Gather every saver's outputs
(images + 3d), so the studio 3D lane's mesh lands like an image or video render.
The cloudflare provider now offers a browser OAuth path in addition to the
shipped apikey path. /connect dispatches by request: a "token" key in the body
seals via apikey (verify-before-store, unchanged); its absence starts the
Authorization Code flow (confidential client, client_secret, no PKCE — the
framework's OAuth pattern) and the exchanged access token is sealed to the SAME
KMS coordinate (/orgs/{org}/integrations/cloudflare/api_token), so the DNS
provider layer is auth-method-agnostic.
Framework: connect dispatch is now capability-based (Verify and/or Authorize)
rather than Kind-only; Mount validates RedirectPath for any OAuth-capable
provider; bodyHasCredential picks the path. The OAuth leg gates on its own app
creds (Creds().ClientID) so a missing Cloudflare OAuth app degrades to an honest
503 without breaking the always-available apikey path.
Requires a registered Cloudflare OAuth app: CLOUDFLARE_OAUTH_CLIENT_ID/SECRET in
env, redirect https://api.hanzo.ai/v1/integrations/cloudflare/callback.
No Go change — this rebuild re-resolves the freshly-republished console-embed:latest
(console main bd8a81651) into the go:embed console served at console.hanzo.ai, and
ships builtin.go's auto '/v1 route → MCP tool' plane at /v1/tools/mcp (already in main,
newer than the deployed v1.801.69).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The value routes registered the optional-greedy wildcard `/secrets/*`, which
fiber also matches with an empty tail — so the bare `GET .../secrets` list path
was answered by getSecret (400 "secret name is required") and listSecrets was
unreachable. Switch the getSecret/deleteSecret value routes to the required-
greedy `+` (one-or-more), so `/secrets` falls through to the exact list route
while `/secrets/<path>/<name>` still reads/deletes. reqWildcard reads the `+`
param.
Regression test list_route_test.go asserts the bare list path returns 200 with
a secrets array (was 400) and that value reads still work.
Register Cloudflare as an apikey-kind connector on the /v1/integrations plane.
A customer-supplied scoped API token is verified live against Cloudflare's
GET /user/tokens/verify (must be status:active) before it is sealed into the
org's KMS namespace (/orgs/{org}/integrations/cloudflare/api_token); the
connection row holds only non-secret account metadata. connect/verify/disconnect
are org-admin gated from the validated principal (principal.IsOrgAdmin).
Extends the connector framework with the apikey credential seam shared by future
customer-credential providers: Provider.Kind/AdminOnly/Verify, VerifyInput,
connectByCredential (verify-before-seal, fail-closed), and the verify route.
Serves POST /v1/integrations/cloudflare/{connect,verify,disconnect} and
GET /v1/integrations.
A routed run's machine pushes with its own credential and streams into the
session, but cloud still owns the completion — the integrity gate, the PR row, and
the session's terminal state (the machine never closes the session, so it was
staying "running" forever). Give a routed run the SAME cloud-side completion the
local keystone path runs after a sandbox push.
- completeChanged: the shared terminal for a run that reported changes — VerifyRef
the pushed branch LANDED (fail-closed to a session error + no PR if absent), file
the native PR, mirror done, close the session done. The local path (Run) now calls
it too, so the two paths cannot drift.
- finalizeRouted: maps a machine's terminal report onto that completion — reported
failure closes the session error (no PR), no-changes closes done (no PR), a changed
push runs completeChanged. No secret crosses; cloud only reads the ref it can see.
- DeliverRoutedRunActivity runs the completion once, after a real report, on a
cancel-immune bounded context, so a completed run is never re-executed by a retry.
The completion seam is injected at the composition root (NewDispatcher), the same
injected-seam shape index_on_push uses, so the free-function activity reaches the
dispatcher's git/tracker/session seams without a global Dispatcher.
- RoutedRun carries Actor + AgentRef (cloud-side only, never sent to the machine) so
the completion attributes the session close and files the PR with the right
assignee.
Also document the mailbox's single-replica dependency at its definition (accepted,
inherited from cloud's KMS-lock replicas:1) with a future replica-aware note.
Tests: routed changed+verify -> PR filed + session done; verify fails -> no PR +
session error; no changes -> done no PR; reported error -> error no PR (verify never
runs); NewDispatcher wires the seam; the durable type bridge preserves attribution.
The cloud already trusts hanzo-admin-guard (the admin surface) but not admin-console
(the admin console's own OIDC client), so a SuperAdmin token minted via admin-console
was rejected on /v1/admin with 'invalid audience' — forcing an awkward hanzo-admin-guard
detour. Add admin-console so the admin console's tokens work directly, matching
GATEWAY_ALLOWED_AUDIENCES which already lists it.
Co-authored-by: zeekay <z@hanzo.ai>
Plane map with honest tiers in LLM.md: /v1/connectors custody (in flight), /v1/channels transport (planned, branch reserved), shipped planes named by package. Spec home HIP-0129; roadmap P1-P15 lives there.
"gatewaypolicy" is a compound (gateway+policy) and read as a second gateway
package. It is the ONE Gateway concern with clients/gateway — the per-org edge
policy STORE (OrgRPM ceiling + CORS + cache) that the /v1/gateway/config plane and
the package-cloud edge middleware both read.
They are two packages only to break a Go import cycle: clients/gateway imports root
cloud (cloud.Deps), and middleware_edge.go IS package cloud — so the store must be a
LEAF both can import. A flat merge cycles. Fix per the no-compound law: nest the leaf
UNDER gateway as clients/gateway/edge (edge.Policy/Store/New). One gateway namespace;
/v1/gateway/config surface unchanged; edge-middleware logic unchanged.
NOT redundant with the external hanzoai/gateway (KrakenD): that does coarse per-route
edge rate-limit + auth at ingress; this is per-AUTHENTICATED-org RPM (needs the decoded
token org), an app-level ceiling the edge proxy cannot compute. Different layer.
Pure rename (49/49, no logic change); full cmd/cloud binary links; gateway + gateway/edge
tests pass; gofmt/vet clean.
waitlist_* / public_signup / gateway_* are runtime flags; strip their Env: fallbacks so
they resolve from the /v1/flags DB engine → Default (single source of truth, flipped live,
no redeploy). Boot-time ReadOnly rows (subsystem_*, network_id_*) keep Env — that IS their
boot mechanism. Nothing read these env vars outside the flags engine (verified).
Co-authored-by: zeekay <z@hanzo.ai>
Wire the load-bearing trigger so a coding run can be dispatched to a chosen
linked machine end-to-end. Extend the Slack coding grammar with an optional
routing prefix — code: <repo> on <machine> <task> — resolve <machine> org-scoped
(id or friendly label) and set Req.TargetID. An unknown or foreign machine is an
honest error, never a silent local fallback; an untargeted request is byte-
unchanged (repo <task>).
- agents.ResolveTarget: the ONE org-scoped id-or-label resolver, fail-closed, so a
trigger surface turns 'on evo' into a target id without leaking another tenant's
inventory.
- slack_coding: parseCoding yields (repo, target, task); a routed run skips the KMS
agent-credential fetch (the machine authenticates with its own credential); the
ack + result card report a routed run as queued-on-<machine>, followed live in
mission-control, not a premature branch-pushed verdict.
Tests: ResolveTarget id/label precedence + cross-org not-found + unmounted fail-
closed; parseCoding on-prefix grammar (routing only when 'on' is the token after
the repo; 'on-call'/'only' untouched); routed result card is queued not done.
cloud transitively pulled github.com/WqyJh/{go-cosyvoice,go-openai-realtime}
through ai v1.822.2 (the go-openai-fork release, which predated ai's TTS
switch to the hanzo-owned forks). ai v1.822.3 wires ai/tts onto
github.com/hanzoai/go-cosyvoice + go-openai-realtime, so tidy drops both WqyJh
edges from cloud's graph. cloud now pulls ZERO third-party OpenAI-lineage:
WqyJh 0, sashabaranov 0, ClickHouse 0. Single datastore sql registrant
(hanzo-ds/go). Builds clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
admin.hanzo.ai's SaaS-metrics/Invoices/Subscriptions pages were placeholders
awaiting cross-org /v1/admin/* endpoints. Add them as super-admin (core.Guard)
fleet-aggregate readers over the existing commerce billing engine:
GET /v1/admin/metrics — SaaS god-view: MRR/ARR/net-new/churn/active-subs/
paying-customers/plan-mix/top-customers/recent.
Single S2S proxy — commerce /v1/metrics/saas is
already a cross-org aggregate (same gate finance
Costs uses: RequirePlatformAdmin→IsServiceToken).
GET /v1/admin/invoices — cross-org invoice list; fan core.ListOrgs out
GET /v1/admin/subscriptions — cross-org subscription list; per-tenant reads
merged (identical to revenue.go's fan-out).
Honest degradation: a failed per-org read contributes no rows, never fabricated.
go build ./clients/admin/... green, gofmt clean. Pairs with admin operator UI
(feat/admin-billing-fleet-ui).
cloud/clients{,/agent} used sashabaranov/go-openai only via a 'replace =>
hanzoai/go-openai' — which does not propagate to cloud's own consumers, so
gateway/iam/etc. each had to copy it. Now the fork declares its own module
path (github.com/hanzoai/go-openai v1.41.0): require it directly. Bumps the
lockstep fork adopters — ai v1.822.2, agent v0.1.3 — so the hz.Mount Completer
boundary shares ONE openai type set (was a hanzoai-vs-sashabaranov type
mismatch). No replace anywhere; upstream sashabaranov remains only as an
indirect dep of the go-cosyvoice TTS chain (owned next). cmd/cloud keeps its
single datastore sql registrant (hanzo-ds/go). Builds clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Embeds the fleet KMS into cloud by re-sealing the ~125 KMSSecret-referenced
secrets from the legacy standalone (unsealed at rest) into cloud's embedded
/v1/kms (AES-256-GCM sealed per secret). Driven by the KMSSecret CRs — the
authoritative (org,path,env,key) manifest — not a raw store copy.
- inventory: pure CR parse/validate/dedup; handles explicit keys, folder-sync
(empty keys[]), the env-default divergence (cloud refuses empty env), and
malformed CRs (fail-loud, never silently dropped). Reuses cloud/clients/kms
ValidSegment/ValidSubpath so a coordinate the store would reject is never built.
- reseal: per-CR org-bound auth (owner==projectSlug), GET standalone -> POST cloud
(cloud seals). Idempotent upserts, re-runnable. Plaintext transits memory only,
wiped after write; results carry coordinates + status, never values.
- verify: read-only SHA-256 hash-compare standalone-vs-cloud + org-isolation matrix
(cross-org 403, no-principal 403) against the real cloud guard.
- preflight: cloud /v1/kms reachability + JWT-validation probes + offline G1 delta.
- runbook: the ordered, rollback-safe cutover (standalone stays read-only).
Tests: round-trip against cloud's REAL embedded /v1/kms in-process (real seal,
real guard) via a zip app.Fiber().Test transport adapter; seal-proof (no plaintext
on disk); wrong-org refusal; folder-sync via LIST; hash-mismatch detection;
isolation matrix. go test ./cmd/kmsreseal green.
featuregate READ like a synonym for flags — the source of the "isn't this the
same thing?" confusion. It is not: flags is the Policy decide-ENGINE (/v1/flags);
this package is the request-ADMISSION gate that composes it one-way (host→service
registry + waitlist.<svc> mode read + Enforce middleware + IAM approval check).
Renamed to `admission` — the precise systems term for policy-gating requests
(k8s-style admission control) — which also dodges the gate/gateway/gatewaypolicy
name cluster. flags stays THE engine; admission is a thin one-way consumer.
Also kills the /v1/featuregate/mode compat alias entirely (route + Enforce exempt
entry + test): one route, /v1/flags/waitlist. No shim, no adaptor, no backwards
compat — per the one-and-only-one-way law.
flags engine surface unchanged. Build/vet green; admission tests + apps frozen-Wire
order test pass (admission holds featuregate's slot). Deeper Policy collapse
(authz/entitlements/gatewaypolicy → one engine) is a separate staged HIP-0127 pass.
When a coding run carries a targetId (a registered /v1/agents/targets
machine), enqueue it as a durable task addressed to that target on the ONE
embedded tasks engine instead of running it in the cloud sandbox. No target
keeps the local sandbox path byte-unchanged.
- mailbox: an in-process rendezvous between the durable RoutedRunWorkflow and
the external machine that claims a run over HTTP; tenant + machine isolation
is a property of the (org,target) key, not a check a caller can skip.
- routing: per-target claim key (a capability, stored only as a SHA-256 hash,
constant-time verified) is the machine identity; a fail-closed liveness gate
(online + a live runner) is the dispatch admission.
- claim/report HTTP surface (org bearer + X-Target-Key) lets a machine claim
and complete only runs addressed to it.
- coding.Dispatcher gains a routed branch: open the session on the target,
enqueue the durable RoutedRunWorkflow (no secret in the payload — the machine
authenticates with its own credential), return queued; fail closed on an
unavailable target or a failed enqueue, never fall back to local.
Tests: dispatch-to-target, no-target-local-unchanged, dead-target-fail-closed,
cross-machine/cross-org claim denied, mailbox isolation + claim race.
Swap the S3 client in the 7 direct importers from github.com/minio/minio-go/v7
to github.com/hanzoai/s3-go (package minio; a minio-go v7.0.98 fork). Drop-in:
same package name and same New/Client/Options, {Get,Put,List,MakeBucket,
RemoveObject,RemoveObjects}Options, ObjectInfo/Object/ErrorResponse surface and
credentials.NewStaticV4; conditional-CAS (SetMatchETag/SetMatchETagExcept) and
presign paths unchanged.
minio-go leaves the direct requires and stays indirect (luxfi/zapdb via
clients/kms). Run go mod tidy after the s3-go v1.0.0 tag is published to
populate go.sum.
The supervisor recycles at queue-idle, but a freshly CLAIMED job is invisible
to the engine queue until its graph is submitted — so recycles fired over the
claim-to-submit window and staging failed on a dead engine, consuming the job
(observed twice in prod, seconds apart). Two invariants close it: a staging
latch the supervisor honors before recycling, and waitEngine() so a job
claimed while a recycle is already mid-flight waits out the restart instead
of dying on connection-refused.
Add the execution half of route.go's redundancy seam: a Router that routes a
signed-in caller's inference through one of their OWN linked provider accounts,
resolves that account's KMS-sealed credential, and cycles to the next in-org
account on a live 429 — never falling back to a platform key or crossing the
tenant boundary. Meters each served call per (org, provider, profile) and
exposes the per-account breakdown at /v1/billing/usage/accounts (+ the canonical
/v1/links/usage/accounts).
- resolver.go: the credential-fetch contract consumed (Resolver seam) + a
KMS-backed impl keyed at orgs/<org>/providers/<provider>/<profile>; Credential
redacts under every fmt verb, resolves only within one org's namespace, and
never falls back to a platform key.
- select.go: the non-secret account selector (openclaw Model@provider:profile,
X-Provider-Account header, session pin) — carries no org/subject.
- router.go: in-org candidate selection (Links.ListLinked only), cycle-on-429,
fail-secure, per-account cooldown; PolicyPlan|MostRemaining|RoundRobin.
- routed.go + meter.go: a summing per-account usage counter beside the Links +
the meter that bills api-key accounts via commerce and leaves subscriptions
plan-paid (BillingMode), so a call is metered once per meaning.
- carrier.go + wire.go: a process-local credential carrier (never serialized) +
the Deps composition root and the AIClient upstream adapter. Inert until called.
Tests prove routing through a linked account, in-org cycling on 429, the
cross-org isolation boundary (against the real store), fail-secure with no
platform-key fallback, and that a credential never reaches a log or error.
The account-usage plane (7456318) wrongly opened a SECOND usage surface inside
clients/link (/v1/links/usage). Move it into clients/usage so usage owns ALL
usage and link owns links and nothing usage — one surface, orthogonal, one window.
Moves (package link -> usage): sample.go (the Sample value + Sanitize), datastore.go
(the hanzo.account_usage warehouse series + reads, now behind a `warehouse` type that
holds only the DDL latch over aiobject's shared datastore — no handle, so usage keeps
NO Shutdown), and the record/samples handlers (account.go). Reconciled with the usage
subsystem: cloudUsageTable -> the existing llmTable, dsTime -> the existing tsLiteral,
duplicate aString -> dsString.
Route table (was /v1/links/usage*):
POST /v1/usage record account-usage samples (the collector)
GET /v1/usage/samples one provider account's own lane dash (time series)
GET /v1/usage/summary THE one summary — merged (see below)
GET /v1/usage/analytics{,/access} unchanged
Summary collision resolved by MERGE, not two endpoints: the account-usage global view
folds into the existing /v1/usage/summary as a labelled `accounts` block beside spend +
LLM, over ONE window (aiobject.ResolveCloudUsageWindow drives both). Nothing dropped —
the caller's own linked-account rows AND the org Hanzo-routed rows both ride the one
summary, each side reporting its own availability, never summed.
Decomplected the Link-refresh: reportUsage braided a warehouse write with a Link
upsert, and since POST /v1/links already sets an account's usage snapshot, the sample
-> snapshot path was a SECOND way to do that. record now records usage only; the link
registry stays link's own concern. Drops the 3 Link-registry tests (they exercised
/v1/links, unreachable in a usage-only mount) and the Link half of 2 more; the warehouse
+ value coverage moves intact. No back-compat alias (the route was hours old).
Wire guard unchanged: link keeps its Shutdown (SQLite store), usage keeps none.
console.hanzo.ai is served one-binary off cloud, so its /v1/iam/* calls hit the
iam_edge — which required a validated org for EVERY route. That 401'd
'sign in to continue' on the sign-in routes themselves (get-app-login, login,
oauth token exchange), a chicken-and-egg that bricked console login (the
'unknown iam route' / 'sign in to continue' users saw). Forward the
unauthenticated-by-design sign-in surface (login-page config, credential submit,
signin/signup, captcha/verification aids, the OAuth token endpoint, OIDC
discovery) straight to IAM BEFORE the org gate. Tenant CRUD + org metadata stay
fully gated — no tenant-data route is opened. Test: TestIamEdgePublic.
v0.15.4 closes the red-team CRITICAL in the federation broker: authorize
Application.Organization on write + reserved-org guard in federation
link/provision (was: social login could mint a SuperAdmin / take over a
cross-tenant account) + SSRF IP filter. Required before the hanzo.id social
cutover. Build pipeline healthy (consensus v1.36.3).
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
A read-only subsystem_iam2_active switch on the platform panel, mirroring
subsystem_iam_active, makes the clean-room iam2 selection visible in the /v1/flags
cockpit. The selector stays ONE thing — CLOUD_IAM_IMPL=iam2 at boot, applied on
the next reconcile — this switch reflects it, it does not add a second control.
Its description names the gate: the IAM cutover parity suite
(universe e2e/50-iam-cutover-parity) must be green against the iam2 shadow before
the canary is flipped.
The account-usage plane over clients/link: a Sample value (one metering lane of
one provider account at one instant), a ReplacingMergeTree warehouse projection
(hanzo.account_usage + a dedup-preserving daily rollup MV) read back with explicit
read-time argMax dedup, and the /v1/links/usage surface — report samples, a
per-provider dash, and a global summary that sets a user's own linked-account plan
usage beside the org's Hanzo-routed cost of record, every row labelled by
source/scope/confidence and never summed together.
A windowless sample (a valid window class with no meter-reported duration or
reset) keys its class's nominal bucket, never the zero instant: every ranged read
filters window_start into [from,to) and the TTL drops epoch rows on arrival, so a
zero-keyed row would be written-but-never-read and would silently drop out of the
summary. Re-polls of a windowless counter collapse onto that one nominal instance
(ReplacingMergeTree by ts), so it is one row per lane, never summed across polls —
reconciling the two window-instance tests the rescued WIP left in contradiction.
The header was captured from client input and re-injected verbatim for any
validated principal. That was defensible while it was a mere attribution hint
no debit ever read — the comment said as much. It is not one anymore:
ai/object.Payer now resolves the PAYING account from it, so forwarding the
client's copy would let a caller name its own payer, which is the whole thing
the claim exists to prevent. A signup-org member could have sent
`X-Billing-Account-Id: org:hanzo` and pointed their spend at the shared pool.
It is now minted from the validated `billing_account` claim
(idClaims.mintedBillingAccount), mirroring iamauth.Claims.MintedBillingAccount
byte-for-byte, so the in-binary path binds what the gateway would and both
resolve one payer. The raw client copy is deleted on ingress and not restored.
The console read and the top-up now hand Payer that same claim, so the balance a
member SEES, the account a top-up FUNDS, and the account the ai gate DEBITS are
one wallet. Feeding Payer a different credential per call site is the modern
shape of the old org-vs-"org/user" split: a funded balance the gate refuses.
Tests drive real signed tokens through the boundary: the claim reaches the
header for person/org/project, a forged copy never survives (even on a token
that carries no claim, where a restored copy would be the only value present),
and an anonymous caller carries no payer at all.
flags is now the PURE (Principal,context)->verdict engine: Register/Bool/Int/
String/Board/SetPlatformSwitch/Defs + /v1/flags/* + native evaluator + the
platform-switch seed. ZERO host->service / ModeForHost / waitlist.<svc> /
mode-route knowledge.
The complete launch waitlist-gate feature moves to clients/featuregate, which
COMPOSES flags one-way (flags.Bool/Register/SetPlatformSwitch/Def/Defs; flags
never imports featuregate):
- flags/waitlist_store.go -> featuregate/registry.go (host->service map)
- flags/waitlist.go -> featuregate/waitlist.go (mode decide + admin funcs +
seed + waitlist.<svc> Def registration + Mount/Shutdown + the mode route)
- flags/waitlist_store_test.go -> featuregate/registry_test.go
- the registry OrgStore handle (was flags.Client.registry) is now featuregate
package state, opened in featuregate.Mount, closed in Shutdown
- Enforce default gate is now the LOCAL WaitlistModeForHost
- /v1/flags/waitlist AND /v1/featuregate/mode compat alias served by
featuregate (route name unchanged)
- apps.Wire re-adds featuregate after admin (after flags); wire_test frozen row
- admin/services.go swaps the flags import to featuregate for the board funcs
The namespace collapse (75d6f36) renamed the live waitlist-mode read to
/v1/flags/waitlist and made /v1/featuregate/mode a 404 — correct per the one-namespace
Policy primitive, but a BREAKING change to a public route whose external callers cannot
be fully enumerated from the monorepo (a deployed frontend could still call the old
path). Per the hard "never goes down for any customer" constraint, ship the collapse
WITHOUT the break: /v1/flags/waitlist is canonical; /v1/featuregate/mode is a temporary
alias to the same handler; both exempt from the Enforce gate.
Delete the alias (this route + its exempt entry in featuregate/middleware.go) once every
caller is confirmed on /v1/flags/waitlist — a one-line follow-up, gated on the owner.
Verified: gofmt clean, go build/vet green, exempt-path test asserts BOTH routes ungated.
services.hanzo.ai is dead (0 Service CRs cluster-wide; the fleet is 100% App).
clients/paas, clients/deploy, and clients/platform drop the two-kind read shim
and read apps.hanzo.ai only. The paas deploy endpoint (and release seam) now
always refuse a git-declared App with 409, naming the universe git path to
commit the tag to.
The services.hanzo.ai kind is dead: zero Service CRs exist cluster-wide and
the whole fleet is apps.hanzo.ai (kind App). These three cluster-facing planes
carried a two-kind read shim (App first, Service fallback) that is no longer
reachable, so strip it and read one kind — App.
- clients/paas: drop servicesGVR + crGVRs(); listApps/getApp/observeFleet read
appsGVR directly (no cross-kind dedup). The deploy endpoint now always refuses
(409): every App CR in the platform namespaces is git-declared and reconciled
by Hanzo CD with selfHeal, so a patch here is reverted — the response names the
git path to commit the tag to. releaseService refuses on the same grounds.
- clients/deploy: drop servicesCRGVR + appCRGVRs() and the "hanzo.ai/Service"
registry entry; health/getAppCR/listAppCRs read appsCRGVR directly. coreSvcGVR
(the core/v1 Service child object) is unchanged.
- clients/platform: drop servicesGVR + crGVRs(); resolveCR/getCR/deleteService
read and delete appsGVR only. Tenant apps are still written and patched as App
CRs in tenant-<org>.
Tests updated to the one-kind reality. Builds/vets/gofmt clean; go.mod untouched.
The guard's public waitlist-mode read now lives under /v1/flags (the flags engine
owns it) — there is NO /v1/featuregate HTTP endpoint. The route, the Enforce
exempt prefix, and the doc/prose comments move; the featuregate Go PACKAGE (native
Enforce middleware) is NOT renamed, and the /v1/admin/services board is unchanged.
- clients/flags/routes.go GET /v1/featuregate/mode -> GET /v1/flags/waitlist
- clients/flags/waitlist.go doc comments repointed
- clients/featuregate/middleware.go defaultExemptPrefixes /v1/featuregate/ -> /v1/flags/waitlist
- clients/featuregate/middleware_test.go exempt-path assertion updated
- apps/apps.go stale prose comment repointed
Verified green: go build ./clients/flags/... ./clients/featuregate/... ./apps/...,
go vet, and CGO_ENABLED=0 go test ./clients/featuregate/...
Three coordinate-hygiene fixes so the pipeline resolves deterministically:
- ai v1.820.0 -> v1.821.1. v1.820.0 pinned iam at an orphaned pseudo-version
(a commit rebased out of existence); v1.821.1 pins the real iam tag v1.31.28.
- iam -> v1.31.28, the real published tag; the pseudo-version and its replace
are gone.
- luxfi go.sum re-recorded from the immutable sum.golang.org via go mod tidy,
so consensus/vm can no longer carry the hashes a force-moved git tag served.
Nothing but coordinates changed; ai v1.821.1 is v1.820.0's tree with one dep line
repinned, so the compiled result is identical to the shipped v1.801.49. Real
public semver only: no pseudo-versions, no replaces, no force-moved tags.
The v1.801.50 tag collision: a run pushed :v1.801.50 then was cancelled after
imagetools-create but before its git tag (orphaned container tag). The Tag steps
container-tag floor (cont_max) was fail-OPEN — `gh api ... 2>/dev/null || true`
yields "" on any API error — so a later run did NOT see :v1.801.50, recomputed
the same number, and REASSIGNED :v1.801.50 to a different image: an ambiguous
mutable prod tag (silent flip on any fresh-node reschedule).
Fail-CLOSED: if the container-tag lookup ERRORS (vs legitimately empty), retry the
whole attempt instead of proceeding on a git-only floor that cannot see the orphan.
A version that already has a pushed image is now never reused.
NOT reordering git-tag before imagetools-create (the other candidate fix): that
reintroduces the phantom "tag exists, image does not" this workflow was built to
prevent. Pairs with the crane-mirror timeout (ed4d372) that stops the hang→cancel
which orphans tags in the first place. Compute-step cont_max left as-is (hint only).
[skip ci]
Recycling on each completed render killed long renders mid-sample when short
jobs shared the engine (observed: every direct render died within ~6 minutes
while probe jobs cycled). The recycle now defers until the queue is empty.
Health: an engine that answers /queue with work in it is alive however slowly
it answers /system_stats; restarts require three consecutive silent probes
with an idle or unreadable queue.
The 'Mirror credential' step fail-safed only on a missing KMS token, not on the
docker-login to registry.hanzo.ai itself. A transient 502 from the mirror registry
(ingress blip; the registry was healthy 6m before and after) killed the whole
serialized release — no image, no tag — even though ghcr (the PRIMARY) was fine.
Both login paths now skip the mirror (MIRROR_OK unset) on failure and continue.
Complements ed4d372 (the crane-copy timeout): the mirror is now best-effort end to end.
Claude-Session: https://claude.ai/code/session_015Z1iLf7QBrq1LhignJrzDw
An unbounded `crane copy` to registry.hanzo.ai can HANG (not just fail) — the
best-effort mirror once livelocked the Tag step and held the entire serialized
release lane (concurrency: release-cloud, cancel-in-progress:false), so no queued
release could run. A best-effort mirror must never be able to block the git-tag
receipt that follows it. `timeout 120` makes it truly best-effort.
[skip ci]
v0.15.0 adds the OIDC/OAuth2 social-federation broker (Google/GitHub);
v0.15.1 mints signing keys for keyless reserved-org certs so the embedded
iam2 publishes a JWKS and can sign tokens (shadow-canary finding). Carries
the full parity + RFC surface into the cloud image for the hanzo.id cutover.
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
luxfi/consensus v1.36.2 was force-repushed with different go.mod content, so
cloud's committed go.sum no longer matches and 'go mod download' aborts with a
SECURITY ERROR — breaking EVERY release. Same recurring luxfi force-move pattern
as c93ddf9 (keys). Bump to latest stable v1.36.9; clients/controlplane (only
importer) compiles clean, go mod verify passes.
Claude-Session: https://claude.ai/code/session_015Z1iLf7QBrq1LhignJrzDw
* render: poll window matches the dispatch cap; engine recycles after each render
The 10m local history poll undercut the 4h startToCloseTimeout the dispatcher
grants — live renders (observed 8-70m) were marked failed while still sampling;
only the mirror later delivered them. renderWindow now matches the cap.
The engine leaks ~58GB per render. The handler signals a recycle after each
COMPLETED render (never on timeout — the engine may still be sampling and the
mirror rescues late finishes); the supervisor restarts on the signal.
* mirror: skip hidden files — AppleDouble forks pass the extension check
._foo.png is a mac resource fork, not a render; 700+ of them poisoned a
library within an hour of the mirror going live.
* deps: luxfi/consensus v1.36.2 -> v1.36.3 — the v1.36.2 tag was re-pushed
Cold builds fail sumdb verification against the moved tag (downloaded
eKzasq4O... vs sealed IbeWQF1w...). v1.36.3 is the immutable successor;
never re-tag a published version.
Ships to prod: router.enabled=true (model=auto routes for every org by default),
per-request RoutingEvent recording for auto AND explicit models (up/down feedback
works on all models), per-org + global fit-gate-deploy-publish training, and the
self-scoped routing-data export/delete (data ownership). Pairs with the universe
CR ROUTER_ENDPOINT removal (heuristic 300ns is the live path).
Claude-Session: https://claude.ai/code/session_018PmFAHZvbBSTsuWyebwMra
referrals/affiliates/authors each carried a byte-identical commerce.go (their own
doc-comments said so): the same commerce interface, httpCommerce, newCommerceClient,
deposit(), spendCents(), errUnconfigured — the S2S COMMERCE_SERVICE_TOKEN money-in
path (POST /v1/billing/deposit) + usage-rollup, triplicated.
Extract ONE clients/payout (attributed-credit -> commerce via commerceinproc): the
exported Commerce/Client/NewClient/ErrUnconfigured. Each program keeps a THIN
adapter — its own lowercase commerce interface + a commerceSeam that delegates to
payout.Client — so the program store/handler code AND their fakeCommerce test doubles
are untouched, and each program still names its own grant tag (grant:referral /
grant:affiliate / grant:author). ~330 duplicated lines collapse to one binding.
Zero behaviour change: identical HTTP contract, headers (X-Org-Id, Bearer), body,
fail-soft (ErrUnconfigured on deposit / 0 on spend when unwired), and errors.Is
sentinel. Adds payout unit tests (httptest) that give the extracted HTTP path REAL
coverage the fakes never did — ok clients/payout 0.010s.
gojabase was the read-WRITE-Base sibling of goja: it wrapped a goja.Host and
added per-tenant Base/SQLite persistence, but duplicated the Host/Config/Request/
Response/New surface. Fold it into the ONE goja package as the Base-binding
CONSTRUCTOR — the persistence layer is now opted into via NewBase (vs New for a
read-only catalog bundle):
goja.New / goja.Host / goja.Config / goja.Request read-only engine (plans/pricing)
goja.NewBase / goja.BaseHost / goja.BaseConfig / goja.BaseRequest + per-tenant Base
Moves gojabase.go -> clients/goja/base.go (renamed types, no goja. self-import),
store.go -> basestore.go, and both test files, all into package goja (zero
identifier collisions, coverage preserved). Repoints every importer —
dataroom/captable/sign (RW) to goja.Base*; plan/pricing already used goja and are
unchanged; base uses goja.TenantSegment. clients/gojabase deleted.
Behaviour is byte-identical: the engine, the per-request transaction commit-on-
<400, the injective TenantSegment, and the __db/__blob/__newId/__now host globals
are unchanged; only the package + exported names moved. No routes (both are
libraries). The gojabase[...] error prefix is kept as the RW-layer diagnostic label.
clients/connectorruntime mounts exactly ONE route —
POST /v1/automations/connectors/:id/run — the in-process goja runner paired with
automations own GET /v1/automations/connectors catalogue. It was a separate Wire
entry solely for that route. Fold connectorruntime.Mount in as a terminal
sub-mount of automations.Mount and drop its Wire entry + import -> ONE
automations subsystem.
The route is DISTINCT from every automations route and automations mounts no
/v1/automations/* wildcard, so there is no shadow; the runner still resolves the
shared engine lazily. clients/connectorruntime stays a focused package
(composition); its internal bundlecmd tool is untouched. Frozen wire row
removed.
clients/cron mounts NO routes — its Mount only launches a background starter
that registers durable schedules on the SAME shared engine (cloud.EmbeddedTasks)
that clients/tasks fronts. It was a separate Wire entry purely to get its
goroutine launched. Fold it in as a terminal sub-mount of tasks.Mount and drop
the cron Wire entry + import -> ONE tasks subsystem.
clients/cron stays a focused package (composition, not code-dumping): tasks
imports and invokes it. No routes change (cron never had any); the scheduler
still waits for the post-MountAll engine, so timing is unchanged. Frozen wire
row removed.
clients/plan.Mount was wired under the name "plans" while its package, and now
its generated standalone cmd, are "plan" — one subsystem, two names. Normalize
the Wire enable id (and cmd/plans -> cmd/plan, ServeSingle arg) to "plan".
Product routes are unchanged: the subsystem still serves /v1/plans/* (plural),
including its OwnsHealth /v1/plans/health probe — only the enable id / binary
name changes. No route drop; mount-all default still enables it (empty Enable =
all on). Updated the frozen wire row and the two cmd/cloud enable-id references;
TestMountAllAndServeHealth now maps plan to its real /v1/plans/health path
(enable id no longer equals route prefix for plan, as is already true for
account/runtime/agent).
cloud#321 landed the Go-only Dockerfile (FROM cloud-flags:latest) but the
release.yml integration wasn't in it, and the reusable could not publish
cloud-flags at all — so every release since has FAILED at
'FROM cloud-flags:latest: not found'. Two fixes:
1. native/flags/Dockerfile base ghcr.io/hanzoai/mirror/rust -> public.ecr.aws
(digest-identical). The hanzoai/ci reusable builds cloud-flags with the repo
GITHUB_TOKEN, which 403s pulling the cross-repo-linked private mirror package;
a public base is GITHUB_TOKEN-pullable, so cloud-flags finally publishes.
2. release.yml resolves console-embed/agent-skills/cloud-flags :latest to
IMMUTABLE digests at release time (crane) and passes them as CONSOLE_IMAGE/
SKILLS_IMAGE/FLAGS_IMAGE build-args to BOTH the smoke build and the push build,
replacing CONSOLE_CACHEBUST. Reproducible (pinned, not floating :latest) AND
fresh (a console/skills/flags change is a new digest). A MISSING artifact FAILS
the release BEFORE build/smoke/tag — the receipt invariant holds, never a
phantom tag on an image that could not embed the real console.
Preserves #322's functional + migration smoke gates (different sections).
Claude-Session: https://claude.ai/code/session_015Z1iLf7QBrq1LhignJrzDw
Co-authored-by: hanzo-dev <dev@hanzo.ai>
zip.Get[In, Out] is advertised as ONE op projected into three surfaces (REST ·
OpenAPI · MCP), and cloud's raw routes were slated to migrate onto it starting
with clients/agents. Measured first, against the route shapes cloud actually
has.
The registry works: registering one typed op populates /.well-known/openapi.json
and the /mcp tool surface, both of which are absent today only because cloud
registers zero typed ops (installOpenAPIRoutes/installMCP early-return on
len(a.ops)==0).
The binding does not. registerTyped's fiber handler passes c.Body() — nil for a
GET — into op.invoke and nothing else, and fiber's DefaultCtx.Context() returns
context.Background(). So a typed handler sees no path param, no query param, and
no header. 16 of clients/agents' 25 routes carry a path param and would receive a
zero In; the ?live/?host/?status/?agent filters would vanish; and the org, which
every agents handler reads via principal.Org(c) -> tenant(c) -> 403, is
unreachable. Migrating as-is would answer the wrong session for every :id route
and drop the authz gate on all of them.
MCP is the sharp edge: tool arguments arrive as the body, so MCP is the ONE
projection that DOES fill In, and mcpCall runs op.invoke with no identity at all.
A migrated org-scoped op would answer an anonymous caller, and the only way to
give it an org would be an org field in the typed In — caller-supplied, i.e. a
cross-tenant read. Both horns are unacceptable, so clients/agents does not
migrate at this zip version.
The org half needs no framework change: fiber's SetContext (already used by
TracingMiddleware) is honored by the ctx registerTyped hands to op.invoke, so
middleware can carry the validated org to a typed handler off the wire, over REST
and MCP alike, without an In field. TestPrincipalBridgeCarriesOrg proves it on
stock zip. Only URL binding is missing.
These are characterization tests: they pin the gap as the current contract and
fail with "UNBLOCKED: invert this test" the moment zip binds a URL, so the
migration restarts on a failing build rather than on someone remembering.
zenKeyResolver read the co-resident KMS store ONLY. The upstream provider
keys (DO_AI_API_KEY, ANTHROPIC_API_KEY) are provisioned as env, injected
from the KMS-synced cloud-api-llm-keys secret — they are NOT sealed in the
embedded KMS store. So GetSecret missed, the resolver returned an empty key,
and zen's call to DO GenAI answered 401 'Unable to authenticate you'. Every
zen chat failed at the upstream while ai (which reads the key from env) worked.
Try the sealed KMS value first (so completing sealed-store provisioning later
needs no code change), then fall back to env. Absent from both still returns
'' so the call fails fast — never silent free usage. Tests cover env-fallback,
sealed-precedence, and absent-everywhere.
Marketing sites emit anonymous pageviews (no session). captureTenant now falls
back — when there is no validated principal — to the PUBLIC brand org derived
SERVER-SIDE from the request Host via the white-label registry (BrandForHostOK),
never a client-claimed org. A forged X-Org-Id is still ignored, and an
unrecognized Host is refused (anonymous events are never dumped into a default
org). Gated by CLOUD_ANALYTICS_PUBLIC_CAPTURE (default on, matching the existing
public insights-capture posture). Verified live: an anonymous pageview to
Host hanzo.ai lands under tenant_id=hanzo.
The analytics subsystem served only read lenses over hanzo.events; nothing
wrote the table, so the web/commerce lenses were permanently honest-empty. This
adds the symmetric ingest: products POST batches to cloud (the ONE native front
door) and cloud writes org-scoped rows into the datastore warehouse the read
side already queries.
- POST /v1/analytics, /v1/analytics/batch, /v1/tracker (beacon alias) — all
tenant-gated in-handler; tenant_id is always principal.Org, never client input.
- Writes ride ai/object.DatastoreExec (the SAME pooled client the reads use).
- The writer owns the hanzo.events DDL (EnsureEventsTable, idempotent/latched).
- Privacy scrub: credential/PII-shaped property keys dropped, email values
redacted, before any row is built.
- Pure core (normalizeEvent/scrubProps/buildEventsInsert) unit-tested; HTTP
contract tests cover no-principal 403, forged-org 403, oversized 400,
datastore-down 503; a build-tagged live test proves the full round trip
against a real datastore.
2026-07-13 16:09:17 -07:00
815 changed files with 151671 additions and 7055 deletions
# semver-correct compare: the lowest of {V, FLOOR} under `sort -V` must be
# the FLOOR, i.e. V >= FLOOR. (sort -V orders v1.4.2 above v1.4.10 too.)
low="$(printf '%s\n%s\n' "$V" "$FLOOR" | sort -V | head -1)"
if [ "$low" != "$FLOOR" ]; then
echo "::error::github.com/hanzoai/zen is pinned at ${V}, below the streaming-fix floor ${FLOOR} — this re-breaks SSE streaming (empty completions). Re-pin zen to >= ${FLOOR} in go.mod before merging."
exit 1
fi
echo "OK: zen ${V} is at or above the streaming-fix floor ${FLOOR}"
- name:positive proof — clients/controlplane is unreachable from the default build
[ -n "$d" ] || { echo "::error::decomplection artifact ghcr.io/hanzoai/${repo}:latest is not published — refusing to cut a release that would embed a stale/placeholder ${repo}"; return 1; }
@@ -40,6 +52,17 @@ webui: ## Build the real console static bundle into webui/dist (go:embed source)
cp -r "$(CONSOLE_DIR)/out/." webui/dist/
@echo ">> embedded real console bundle into webui/dist (index.html $$(wc -c < webui/dist/index.html) bytes)"
deploy-ui:## Build the monochrome ArgoCD dashboard bundle into clients/deploy/webui/dist (go:embed source). DEPLOY_DIR=<path to hanzoai/deploy>.
@command -v yarn >/dev/null 2>&1||{echo"yarn is required to build the deploy dashboard bundle";exit 1;}
@test -f "$(DEPLOY_DIR)/ui/package.json"||{echo"deploy checkout not found at $(DEPLOY_DIR) — set DEPLOY_DIR=<path to hanzoai/deploy on rebrand/hanzo-monochrome>";exit 1;}
agentskills:## Regenerate the FULL agent-skills catalog into clients/agentskills/catalog (go:embed source) from the openapi SOT. OPENAPI_DIR=<path to openapi>.
@test -f "$(OPENAPI_DIR)/skills.py"||{echo"openapi checkout not found at $(OPENAPI_DIR) — set OPENAPI_DIR=<path> or clone hanzoai/openapi";exit 1;}
# skills.py rewrites the whole catalog dir; the .gitignore keeps only the tiny
@@ -53,9 +76,13 @@ build: ## Build the unified cloud binary into ./bin/cloud (embeds whatever webui
build-standalone:webuibuild## Build the REAL 1-binary console: console build:embed → webui/dist → go build.
hanzo:## Build the hanzo control-plane CLI into ./bin/hanzo (pure Go, same mode as cmd/cloud — registers the ONE "sqlite" driver exactly once; a plain CGO_ENABLED=1 `go build ./cmd/hanzo` links the fork's mattn backend alongside the embedded modernc importers and panics, see header).
t.Fatalf("GET /v1/catalog/entries fell through to the AI balance gate (402) — /v1/catalog must be a commercePrefix so the SuperAdmin CMS reaches commerce")
t.Fatalf("GET /v1/plans/entries fell through to the AI balance gate (402) — /v1/plans must be a commercePrefix so the plan authority CMS reaches commerce")
{"flags",true,true},// was order 9; native engine: /v1/flags health + store shutdown
{"kms",true,false},// was order 10
{"metrics",false,false},// was order 40
{"ingress",false,true},// was order 42
{"account",false,false},// was order 48
{"iam",false,false},// was order 50
{"base",true,true},// was order 60; per-org embed added Shutdown (#298)
{"o11y",false,true},// ONE observability subsystem (was co-owned orders 69+70): read plane + the hanzoai/o11y module wildcard folded in as MountO11y's terminal sub-mount. OwnsHealth=false keeps /v1/o11y/health the generic always-ok route the module co-entry used to trigger.
{"authz",false,false},// was order 70
{"commerce",false,false},// was order 100
{"licensing",false,false},// was order 110
{"plans",true,false},// was order 111
{"pricing",true,false},// was order 112
{"storage",true,false},// was order 118
{"provisioning",false,false},// was order 120
{"billing",false,false},// was order 121
{"account-bridge",false,false},// was order 122
{"do",false,false},// was order 123
{"platform",true,false},// was order 124
{"projects",false,false},// was order 125
{"prompts",false,false},// was order 126
{"agents",false,true},// was order 127
{"link",false,true},// new: unified AI login manager (/v1/links), after agents
{"zen",false,false},// zen* claim middleware before ai's catch-all (hip-00NN)
{"ai",false,false},// was order 150
{"plugins",false,false},// was order 900
{"pubsub",false,true},// was order 5
{"kafka",false,true},// was order 6
{"agentskills",false,false},// was order 8
{"flags",true,true},// was order 9; native engine: /v1/flags health + store shutdown
{"kms",true,false},// was order 10
{"metrics",false,false},// was order 40
{"ingress",false,true},// was order 42
{"account",false,false},// was order 48
{"iam",false,false},// was order 50
{"base",true,true},// was order 60; per-org embed added Shutdown (#298)
{"o11y",false,true},// ONE observability subsystem (was co-owned orders 69+70): read plane + the hanzoai/o11y module wildcard folded in as MountO11y's terminal sub-mount. OwnsHealth=false keeps /v1/o11y/health the generic always-ok route the module co-entry used to trigger.
{"authz",false,false},// was order 70
{"commerce",false,false},// was order 100
{"licensing",false,false},// was order 110
{"plan",true,false},// was order 111; enable id normalized plans->plan (routes stay /v1/plans/*)
{"pricing",true,false},// was order 112
{"storage",true,false},// was order 118
{"provisioning",false,false},// was order 120
{"billing",false,false},// was order 121
{"rollingcap",false,false},// rolling spend-cap gate (after billing); golden drifted — refrozen
{"account-bridge",false,false},// was order 122
{"do",false,false},// was order 123
{"platform",true,false},// was order 124
{"projects",false,false},// was order 125
{"dns",false,false},// new: /v1/dns zone plane (after projects)
{"domain",false,false},// new: Hanzo Domains registrar (/v1/domain), after dns
{"prompts",false,false},// was order 126
{"agents",false,true},// was order 127
{"link",false,true},// new: unified AI login manager (/v1/links), after agents
{"channels",false,true},// new: /v1/channels transport plane (after notify; must mount after integrations so RegisterIngress installs before webhooks emit)
{"gateway",false,false},// was order 139
{"entitlements",false,true},// was order 139
{"exec",false,false},// was order 140
{"websearch",false,false},// was order 141
{"world",false,true},// was order 142
{"runtime",false,false},// was order 143; was "bot" until the transport was named for what it is
{"authors",false,true},// was order 143
{"bots",false,false},// was order 143
{"audit",false,false},// was order 144
{"affiliates",false,false},// was order 144
{"sign",true,true},// was order 145
{"product",false,false},// was order 145
{"evals",false,false},// was order 145
{"benchmark",false,false},// benchmark plane (after evals, before treasury)
// A multi-replica deployment REQUIRES the durable plane: with >1 writer, a per-org
// store that is not hydrate-on-open + fenced is the outage this exists to fix.
// disabledDurability logs at the severity the replica count warrants, so a
// misconfigured prod deployment is never SILENTLY non-durable (Red L2).
multiReplica:=len(parsePeers(cfg.ShardPeers))>1
// Explicit opt-in: the object-store fence rests on the deployed SeaweedFS enforcing
// conditional-PUT (If-Match) atomically, which must be validated against the deployed
// version before it fences real tenant data (the takeover-fence staging gate). Until
// CLOUD_RESEARCH_DURABLE is set the store runs local-only — the shard router still
// pins each org to one writer, so this is not the rolling-deploy outage; only the
// cross-restart object-store snapshot waits for the opt-in.
if!cfg.ResearchDurable{
disabledDurability(log,multiReplica,"CLOUD_RESEARCH_DURABLE not set — HA object-store durability is opt-in pending the SeaweedFS conditional-PUT atomicity gate")
log.Error("DURABILITY DISABLED on a MULTI-REPLICA deployment — per-org stores survive only via shard routing + per-pod RWO PVC; a lost/rescheduled PVC loses committed data. Configure S3_ADMIN_* to enable the durable plane.","reason",why)
Short:"Bring this machine into the Hanzo cloud fleet as a node",
Long:"Link this machine into the Hanzo cloud as a node: join the fabric (start\n"+
"hanzod on hanzo.network) and register as a compute worker — advertising this\n"+
"host's CPU (cores + model), memory, and each GPU as its own resource, then\n"+
"heartbeating and claiming jobs from your org's queue. The node shows up in the\n"+
"console (Machines + GPUs) and on `hanzo status`. Works on a CPU-only box.\n"+
"Authentication reuses the `hanzo login` token; the org is taken from its claims.",
Args:cobra.NoArgs,
RunE:func(cmd*cobra.Command,_[]string)error{
ifdaemon{
returninstallDaemon(cmd,opts)
}
// 1. Fabric: start hanzod via the canonical `hanzo node up` (best-effort).
startFabric(cmd,noFabric)
// 2. Compute fleet: register inventory, heartbeat, claim jobs (foreground).
returnrunConnect(cmd,envOf(),opts)
},
}
f:=cmd.Flags()
f.StringVar(&opts.jobsNS,"jobs-namespace",defaultJobsNS,"tasks namespace to claim jobs from")
f.BoolVar(&noFabric,"no-fabric",false,"join the compute fleet only; do not start hanzod")
f.BoolVar(&daemon,"daemon",false,"install a systemd --user unit (Restart=always) instead of running in the foreground")
f.BoolVar(&opts.serveEngine,"serve-engine",false,"also advertise a hanzo-engine model server (OpenAI + Anthropic) running on this node")
f.StringVar(&opts.engineURL,"engine-url",defaultEngineURL,"local URL where hanzo-engine is probed (GET /v1/models)")
f.StringVar(&opts.engineEndpoint,"engine-endpoint","","public URL to advertise for gateway routing (defaults to --engine-url; a node behind NAT needs a reachable URL/tunnel)")
f.BoolVar(&opts.registerProvider,"register-provider",false,"auto-register the engine endpoint as an org model provider (POST /v1/add-provider)")
f.StringVar(&opts.studioDir,"studio-dir",os.Getenv("HANZO_STUDIO_DIR"),"local Hanzo Studio checkout; when set, link launches and supervises the render backend on 127.0.0.1:8188")
f.StringVar(&opts.studioURL,"studio-url",firstNonEmpty(os.Getenv("HANZO_STUDIO_UPLOAD_URL"),defaultStudioUploadURL),"studio base URL the render mirror uploads finished images to (POST /v1/library/upload)")
f.BoolVar(&opts.mirror,"mirror",true,"sweep local renders into the org studio library; --mirror=false serves jobs only")
returnfmt.Errorf("no platform token: pass --platform-token, set HANZO_PLATFORM_TOKEN, or run `hanzo login --platform-token <tok>`")
returnfmt.Errorf("not authenticated: run `hanzo login` (an IAM login now authorizes the platform; a --platform-token / HANZO_PLATFORM_TOKEN still works for machine automation)")
returnnil,fmt.Errorf("no build token: set HANZO_BUILD_TOKEN / PLATFORM_BUILD_CALLBACK_TOKEN or `hanzo login --build-token <tok>`")
returnnil,fmt.Errorf("not authenticated: run `hanzo login` (an IAM login now authorizes builds; HANZO_BUILD_TOKEN / --build-token still works for machine automation)")
log.Warn("route-work: the routed-run rendezvous is process-global and REQUIRES cloud to run single-replica, but CLOUD_REPLICAS>1 — routed /claim will silently fail on a replica that does not own the delivery. Make the rendezvous replica-aware (sticky target route or shared broker) before scaling out.",
"replicas",replicas)
return
}
log.Info("route-work: routed-run rendezvous is in-process; assumes cloud single-replica (inherited from the KMS exclusive lock)")
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.