fix(console): finetuning/kubeflow 403 + API-keys CORS crack — mint user Bearer, same-origin keys route (v8.4.89)

The Finetuning + ML Pipelines pages showed 'Not enabled for your account' for a
signed-in customer, and Organization-Settings API keys couldn't be listed/minted.
Both are real money-path cracks — verified live as Dave (org maxpower).

ROOT CAUSE (proven live, NOT the iss theory):
- /training proxy forwarded the raw session COOKIE to cloud-api /v1/train/*, which
  authorizes on a validated JWT principal => 403 'no validated principal'. With a
  Bearer it is 200. (The token iss is already https://hanzo.id — IAM folds the
  in-cluster host to originFrontend[0]; cloud-api accepts it. functions/app-platform
  already work via the /cloud bearer proxy, so they were not the break.)
- keys.ts called cloud.hanzo.ai/v1/console/keys — a DIFFERENT origin than
  console.hanzo.ai => browser CORS 'Failed to fetch' (and cloud-api 501s that
  handler anyway).

FIX (surgical, isolation-safe, DRY):
- /training now mints a short-lived user-bound Bearer via adminBearer (the ONE
  per-user cache the /cloud proxy uses) and forwards Bearer + X-Org-Id (orgFor pin);
  the cookie is dropped upstream. Fails closed (502) if the token can't be minted.
  Org stays server-authoritative (token owner claim) — no tenant-isolation change.
- New same-origin app/keys/route.ts uses identity.ts mintUserKey/getUserKey/
  revokeUserKey (IAM confidential-client, the WORKING key path); keys.ts addresses
  <origin>/keys. CSRF-guarded, honest 501 when unconfigured, secret shown once.
- Bundle the edge-503 hygiene: interpretPlatformError maps 503 -> 'unavailable'
  (clean card), so a fail-closed zt backend never leaks ZT_CLIENT_* env to a customer.

tsc clean; vitest green; next build ok (/keys + /training routes registered).

app/keys/route.ts

@@ -0,0 +1,96 @@
+/**
+ * Per-user `hk-` Cloud API key — the SAME-ORIGIN console route (the fix for the
+ * API-keys "sign in to manage API keys" / CORS crack).
+ *
+ * The browser calls this OWN-origin route (`/keys`) with just its first-party
+ * session cookie. This handler resolves the signed-in user from that cookie
+ * (`resolveUser`) and mints/reads/revokes the key through IAM as the confidential
+ * `hanzo-console` client (`identity.ts` `mintUserKey`/`getUserKey`/`revokeUserKey`,
+ * over IAM `mint-user-keys`/`get-user`/`revoke-user-keys` — the WORKING key path,
+ * verified live). No credential ever reaches the browser; the `hk-` secret is
+ * returned ONLY by POST (show once).
+ *
+ * Why not `cloud.hanzo.ai/v1/console/keys` (the old path): that is a DIFFERENT
+ * ORIGIN than console.hanzo.ai, so a browser `fetch` is blocked by CORS ("Failed to
+ * fetch") — and cloud-api's own keys handler 501s ("IAM client unset") on this
+ * deployment anyway. The IAM confidential-client mint the console already uses for
+ * `hk-` keys elsewhere (`app/ai` chat) is the ONE authoritative, same-origin,
+ * always-working path — so the Org-Settings API-keys surface uses it too (DRY: the
+ * exact primitives from `identity.ts`, no new IAM plumbing).
+ *
+ *   GET    → { hasKey, keyPrefix, createdAt }  (no secret)
+ *   POST   → { accessKey }                       (mint/rotate; full hk- shown ONCE)
+ *   DELETE → { ok: true }                        (revoke; the old key stops working)
+ */
+import { type NextRequest, NextResponse } from 'next/server'
+
+import { resolveUser, mintUserKey, getUserKey, revokeUserKey, mintConfigured } from '~/lib/server/identity'
+import { csrfRefusal } from '~/lib/server/bearer-proxy'
+
+export const runtime = 'nodejs'
+
+const msgOf = (e: unknown) => (e instanceof Error ? e.message : String(e))
+
+/** 401 (not signed in) — the honest state the UI shows to sign in. */
+function unauthorized() {
+  return NextResponse.json({ error: 'Sign in to manage API keys.' }, { status: 401 })
+}
+
+/** GET — the user's current key state (existence + public prefix, NEVER the secret). */
+export async function GET(req: NextRequest) {
+  const user = await resolveUser(req)
+  if (!user) return unauthorized()
+  if (!mintConfigured()) {
+    // Honest, non-leaking: the confidential client isn't wired on this deployment.
+    return NextResponse.json({ error: 'API key management is not configured on this deployment.' }, { status: 501 })
+  }
+  try {
+    const { accessKey, updatedAt } = await getUserKey(user)
+    return NextResponse.json({
+      hasKey: Boolean(accessKey),
+      keyPrefix: accessKey ? accessKey.slice(0, 11) : '',
+      createdAt: updatedAt || '',
+    })
+  } catch (e) {
+    console.error('keys: could not read key state:', msgOf(e))
+    return NextResponse.json({ error: 'Could not read the API key state.' }, { status: 502 })
+  }
+}
+
+/** POST — mint (or rotate) the key. Returns the full `hk-` secret ONCE. */
+export async function POST(req: NextRequest) {
+  // CSRF: minting mutates (and is billable-adjacent) from the auto-sent cookie —
+  // refuse a cross-origin request before any work.
+  const csrf = csrfRefusal(req)
+  if (csrf) return csrf
+  const user = await resolveUser(req)
+  if (!user) return unauthorized()
+  if (!mintConfigured()) {
+    return NextResponse.json({ error: 'API key management is not configured on this deployment.' }, { status: 501 })
+  }
+  try {
+    const accessKey = await mintUserKey(user)
+    return NextResponse.json({ accessKey })
+  } catch (e) {
+    console.error('keys: could not mint key:', msgOf(e))
+    return NextResponse.json({ error: 'Could not create the API key.' }, { status: 502 })
+  }
+}
+
+/** DELETE — revoke the key (the old key stops working; gateway cache ~5m). */
+export async function DELETE(req: NextRequest) {
+  const csrf = csrfRefusal(req)
+  if (csrf) return csrf
+  const user = await resolveUser(req)
+  if (!user) return unauthorized()
+  if (!mintConfigured()) {
+    return NextResponse.json({ error: 'API key management is not configured on this deployment.' }, { status: 501 })
+  }
+  try {
+    await revokeUserKey(user)
+    return NextResponse.json({ ok: true })
+  } catch (e) {
+    console.error('keys: could not revoke key:', msgOf(e))
+    return NextResponse.json({ error: 'Could not revoke the API key.' }, { status: 502 })
+  }
+}

app/training/[...path]/route.ts

@@ -4,15 +4,24 @@
  *
  * The console's Training page calls its OWN origin (`/training/...`) with just the
  * first-party session cookie; this server handler resolves the signed-in user from
- * that cookie and forwards to the cloud backend's `/v1/...` surface, passing the
- * cookie through (the proven `get-account` server-to-server pattern in
- * lib/server/identity.ts) plus the active `X-Org-Id`. Training is a TENANT action —
- * any signed-in org user may run it — so this is user-scoped (resolveUser), NOT the
- * control-plane admin gate the `/paas` proxy uses. The cloud backend scopes by org
- * (GetEffectiveOrg / the X-Org-Id the plain-REST train sub-service requires), so a
- * caller can only ever touch their own org's jobs. `POST /v1/train/jobs` is
- * billing-gated by the live ResourceMeter and returns 402 on an unfunded org — that
- * status flows straight back so the UI can surface it honestly.
+ * that cookie, mints a SHORT-LIVED, user-bound IAM Bearer (`adminBearer` — the ONE
+ * per-user cache shared with the `/cloud` bearer proxy), and forwards to the cloud
+ * backend's `/v1/...` surface with `Authorization: Bearer <token>` + the active
+ * `X-Org-Id`. Training is a TENANT action — any signed-in org user may run it — so
+ * this is user-scoped (resolveUser), NOT the control-plane admin gate the `/paas`
+ * proxy uses. The cloud backend resolves the org from the token's `owner` claim (and
+ * the X-Org-Id the plain-REST train sub-service reads), so a caller can only ever
+ * touch their own org's jobs. `POST /v1/train/jobs` is billing-gated by the live
+ * ResourceMeter and returns 402 on an unfunded org — that status flows straight back
+ * so the UI can surface it honestly.
+ *
+ * Why a Bearer and NOT the cookie (the fix for the "Not enabled" 403): cloud-api's
+ * `/v1/train/*` authorizes on a VALIDATED JWT principal and returns 403 "no validated
+ * principal" for a cookie-only call — the raw casibase session cookie is NOT a
+ * principal it accepts (only the sanitizer's cookie-token names or a Bearer). Minting
+ * the same user-bound token the `/cloud` proxy uses is the ONE way a signed-in tenant
+ * reaches the train surface; the cookie is deliberately dropped upstream (it can't
+ * authenticate, and a cookie + JWT together risks the public-gateway 431).
  *
  * Least privilege: only the explicit ML/training sub-paths are forwarded; anything
  * else 404s, so this is not a general backend tunnel. No secret ever reaches the
@@ -21,7 +30,7 @@
  */
 import { type NextRequest, NextResponse } from 'next/server'
 
-import { resolveUser } from '~/lib/server/identity'
+import { resolveUser, adminBearer } from '~/lib/server/identity'
 import { orgFor } from '~/lib/server/admin-policy'
 import { csrfRefusal } from '~/lib/server/bearer-proxy'
 import { fetchWithTimeout } from '~/lib/server/fetch-timeout'
@@ -29,6 +38,7 @@ import { fetchWithTimeout } from '~/lib/server/fetch-timeout'
 export const runtime = 'nodejs'
 
 const trim = (s: string) => s.replace(/\/+$/, '')
+const msgOf = (e: unknown) => (e instanceof Error ? e.message : String(e))
 /** Cloud `/v1` backend (hanzoai/ai) — same target lib/server/identity.ts resolves. */
 const CLOUD_API_URL = trim(process.env.CLOUD_API_URL ?? 'http://cloud.hanzo.svc.cluster.local:8000')
 
@@ -71,16 +81,34 @@ async function forward(req: NextRequest, path: string[]): Promise<NextResponse>
     )
   }
 
-  const cookie = req.headers.get('cookie') ?? ''
+  // Mint a short-lived, user-bound Bearer (the SAME per-user cache the `/cloud`
+  // proxy uses). cloud-api's `/v1/train/*` 403s a cookie-only call ("no validated
+  // principal"); a Bearer is the one credential it accepts. Fail CLOSED with 502 if
+  // the token can't be minted — never fall through to an unauthenticated forward.
+  let bearer: string
+  try {
+    bearer = await adminBearer(user)
+  } catch (e) {
+    // Redact — the exception carries the internal IAM host/port. Log server-side only.
+    console.error('training-proxy: could not mint user bearer:', msgOf(e))
+    return NextResponse.json(
+      { status: 'error', msg: 'Could not authorize the request.' },
+      { status: 502 },
+    )
+  }
+
   const url = `${CLOUD_API_URL}/v1/${rel}${req.nextUrl.search}`
   const headers: Record<string, string> = {
-    cookie,
+    Authorization: `Bearer ${bearer}`,
     Accept: 'application/json',
     'Content-Type': 'application/json',
     // Org is SERVER-RESOLVED, not the raw browser header: a global admin's switched
     // org (?/X-Org-Id) is honored, a non-global caller is PINNED to their own — so a
     // brand admin can't drive another tenant's training jobs even if the backend
-    // trusted the forwarded header. Matches the /paas + /admin/kms orgFor pin.
+    // trusted the forwarded header. For a non-global caller this equals the token
+    // owner (the Bearer's own claim), so header and token agree. Matches the /paas +
+    // /admin/kms orgFor pin. The raw session cookie is NOT forwarded (cloud-api can't
+    // validate it as a principal, and cookie + JWT together risks the gateway 431).
     'X-Org-Id': orgFor({ isGlobalAdmin: user.isGlobalAdmin, orgScope: user.owner }, req.headers.get('X-Org-Id')),
   }
   const projectId = req.headers.get('X-Project-Id')

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hanzo/console",
-  "version": "8.4.88",
+  "version": "8.4.89",
   "private": true,
   "license": "BSD-3-Clause",
   "author": "Hanzo AI <dev@hanzo.ai>",

src/components/products/platform/state.tsx

@@ -3,9 +3,10 @@
 /**
  * Honest platform states — ONE place that maps a `/paas` proxy / platform error
  * to a truthful explanation, shared by every platform module (Clusters,
- * Kubernetes). No fabricated data: a 501 means the proxy has no service token
- * yet, a 404 means the platform backend doesn't serve that surface yet, anything
- * else is the real error.
+ * Kubernetes, Edge). No fabricated data: a 501 means the proxy has no service
+ * token yet, a 503 means the route is mounted but its runtime/dependency isn't
+ * configured on this deployment, a 404 means the backend doesn't serve that
+ * surface yet, anything else is the real (transient) reach error.
  */
 import { Button, Card, Text, XStack } from '@hanzo/gui'
 import { CheckCircle2, TriangleAlert } from '@hanzogui/lucide-icons-2'
@@ -27,6 +28,12 @@ export function interpretPlatformError(e: unknown): PlatformError {
   if (status === 501) return { kind: 'not-configured', message }
   if (status === 401 || status === 403) return { kind: 'forbidden', message }
   if (status === 404) return { kind: 'unavailable', message }
+  // 503 = the route is mounted but its runtime/dependency is not configured on
+  // THIS deployment (e.g. zt networking fail-closed until ZT_CLIENT_* is set).
+  // That's a deployment-state truth, not a transient reach failure — show the
+  // clean "not available yet" card, NEVER the raw backend message (which can name
+  // internal env/config the customer should never see).
+  if (status === 503) return { kind: 'unavailable', message }
   return { kind: 'error', message }
 }
 

src/lib/api/keys.ts

@@ -1,15 +1,20 @@
 /**
- * Cloud API key client — the per-user `hk-` credential, via the unified backend's
- * same-origin `/v1/console/keys` route (the cloud console subsystem; task #41). The
- * server resolves the user from the VALIDATED principal (gateway-injected identity)
- * and mints/revokes through IAM as the confidential `hanzo-console` client; the
- * browser only ever sends its cookie. Same handler in both topologies — served
- * directly by the go:embed one-binary, and via the gateway in the split deploy.
+ * Cloud API key client — the per-user `hk-` credential, via the console's OWN
+ * same-origin `/keys` route (`app/keys/route.ts`). The server resolves the user
+ * from the first-party session cookie and mints/reads/revokes through IAM as the
+ * confidential `hanzo-console` client; the browser only ever sends its cookie.
+ *
+ * SAME-ORIGIN by construction (the fix for the money crack): the old client hit
+ * `config.cloudUrl/v1/console/keys` — a DIFFERENT origin than console.hanzo.ai — so
+ * the browser `fetch` was blocked by CORS ("Failed to fetch"), and cloud-api's own
+ * `/v1/console/keys` handler 501s on this deployment regardless. Addressing the
+ * console's own `/keys` route (which uses the working IAM `mint-user-keys` path)
+ * keeps the request same-origin and the credential entirely server-side.
  *
  * The secret is returned ONLY by `create()` (show once). `status()` reports
  * existence + the public prefix, never secret material.
  */
-import { ApiError, v1Url } from './client'
+import { ApiError } from './client'
 
 export type KeyStatus = {
   hasKey: boolean
@@ -18,10 +23,13 @@ export type KeyStatus = {
   createdAt?: string
 }
 
+/** The console's OWN same-origin key route (`<origin>/keys`); root-relative on the server. */
+const keysUrl = (): string => (typeof window !== 'undefined' ? `${window.location.origin}/keys` : '/keys')
+
 async function keysReq<T>(method: 'GET' | 'POST' | 'DELETE'): Promise<T> {
   let res: Response
   try {
-    res = await fetch(v1Url('console/keys'), {
+    res = await fetch(keysUrl(), {
       method,
       credentials: 'include',
       headers: { Accept: 'application/json' },