# hanzoai/k8s — deep notes The one Kubernetes owner in the fleet. Read this before changing anything here; the README is the product surface, this is the reasoning. ## Why the repo exists (and why a library would not have worked) Nine places in `hanzoai/cloud` built a Kubernetes client. Measured, not estimated: | Holder | Credential | Cluster it assumed | |---|---|---| | `apps/platform` ×2, `apps/deploy` ×2, `apps/provisioning`, `apps/validators`, `apps/cron` | in-cluster ServiceAccount → `KUBECONFIG` | one ambient | | `apps/ml` | in-cluster SA, re-scoped to a token file; federated via `fleet.DynForOrg` | ambient **or** the org's BYO | | `apps/fleet` | **KMS-sealed kubeconfig** per org+project | N registered BYO | | `apps/admin/infra` | **`DO_API_TOKEN` → a cluster-admin kubeconfig per cluster** | N DOKS clusters | | `hanzoai/ai/cluster` | **kubeconfig in a database row**, fed to an apply-any-YAML function | one global mutable singleton | Two of those are the reason this is a service and not a shared package: 1. **`ai/cluster/client.go:210 deployResource`** decoded arbitrary YAML, resolved the GVK through a discovery `RESTMapper`, and create-or-updated **any kind the cluster served**. With a DB-sourced kubeconfig, one edited row reached any object on any cluster. It is DELETED, not ported. Its callers go through `resources.apply` with a named kind. 2. **`admin/infra`** held cluster-admin on every cluster on the house account to perform nine operations. `nodes/proxy get` alone is arbitrary kubelet access. A library cannot fix either, because the defect is not the code — it is that N binaries hold cluster credentials. One owner, named ops, credentials in KMS, consumers ASK. The **deciding** argument, though, is neither of those. It is that **lux runs its own `lux-k8s`**: an org's resources may live on a different cluster from Hanzo's, plus BYO for customers. A single-cluster client, however cleanly factored, cannot express that and would have been rewritten. Hence multi-cluster from the first line. ## The model ``` Cluster{Name, Org, Provider, Endpoint, Nodes, NvidiaGPU, AmdGPU, Namespaces, Registered, Default} ``` - **No Kubeconfig field and no Credential field.** The ref is DERIVED — `orgs//k8s/clusters//kubeconfig` — because a ref stored as data could be edited to name another org's material. That is the same defect as letting a caller name its own org, one level down. - **`Org` is set from the validated caller**, never from an argument, and appears only on replies (where it is a property of the thing described). - **`Namespaces`** bounds a registration made with a scoped credential on a shared cluster. Empty means the whole cluster, which is what an org registering its own cluster gets. `registry.Reach(ctx, name) (*Bound, error)` is the only entry point: 1. `Tenant(ctx)` — zip's full rule (validated user claim, non-empty org, under `MaxOrgLen`) plus one thing zip cannot know: the org becomes a KMS ref segment, so a value containing `/` or `..` is refused here. 2. the name is looked up in **that org's index only** → `ErrNoCluster` (404) otherwise. 3. the credential is resolved from KMS over the call plane, carrying the caller, so KMS re-applies its own `orgs//` rule. 4. `SafeRESTConfig` → typed + dynamic + metrics clients, cached per `org/name`. `Bound.Scope(ns)` / `Bound.One(ns)` / `Bound.WholeCluster()` are the namespace gate, and every namespaced op goes through one of them. A named namespace outside the bound is REFUSED rather than silently narrowed — a caller that asked for the wrong thing must be told, or it will believe an empty answer. ## What a "grant" is An org may reach a cluster it **owns**. There is no grant table, and that is a decision rather than an omission: cross-org credential sharing is refused by KMS by construction (a call acting for B asking for `orgs/A/…` is forbidden), so a grant table would have been a second authority that could only ever disagree with the first. Granting org B access to a cluster A owns is therefore: seal a scoped ServiceAccount kubeconfig **under B**, and register it in B's own scope with a `namespaces` bound. That is one existing verb, no new concept, and it is honest — to reach a cluster you must hold a credential for it, sealed in your own org's KMS. The platform's own path for a tenant on the shared cluster is the same call, made by the provisioning job with `zip.WithCaller` stating the tenant it acts for. ## The 36 ops `plane/plane.go` is the whole contract. Groups: `clusters` (5), `nodes` (3), `namespaces` (4), `pods` (2), `volumes` (4), `workloads` (8), `jobs` (3), `resources` (5), `access` (1), `metrics` (1). Each is registered TWICE from ONE handler: on the app (HTTP, which projects OpenAPI + an MCP tool + a CLI command) and on `app.Peer()` (the fleet's 0600 socket, invoked by op token through `zip.Ask`). The registrations are written out as direct `zip.Get` / `zip.Post` calls with literal paths **deliberately** — zipdoc reads the AST for exactly that shape, so a registration wrapped in a helper is a registration whose doc comment reaches neither the OpenAPI description nor the MCP tool description. Design notes worth keeping: - **`resources.watch` is a bounded read, not a stream.** A stream cannot cross the call plane as a value; rendering one (SSE, a websocket) is the edge's concern. It returns the changes within `waitSeconds` and the cursor to resume from, and `expired` is the field that matters — a too-old cursor rendered as an empty page is how a watch-backed board silently stops updating. - **`nodes.cordon` is one op over three calls** (patch `spec.unschedulable`, list the node's pods, evict each). There is no generic node patch, because one would let a caller write any field of any node's spec. The drain evicts rather than deletes, so a PodDisruptionBudget refusal is the system working. - **`volumes.expandClaim` takes `gi`, not a patch.** Growing the claim is the one way that leaves claim, PV, device and filesystem agreeing. Nothing deletes a claim. - **`namespaces.delete` is separately gated** and refused for a bounded registration: it garbage-collects every object in the namespace, claims included. - **`nodes.volumeStats` is its own op** so `nodes/proxy` can be dropped from every other caller's grant. It is the only source of volume fill we have (no cloud provider exposes it; metrics-server is absent from most of our clusters) and it reports `nodesRead` / `nodesTotal`, because a claim with no row is UNMEASURED, which is a different fact from empty. - **`metrics.pods` reports `available`.** A cluster with no metrics-server rendered as zeroes tells an operator a busy namespace is idle. - **`clusters.status` probes the version endpoint**, not a namespace list: a registration bounded to one namespace would fail a namespace list and look dead while being healthy. ## Secrets — the decision There is **no secrets op**, and the absence is the answer to the question the measurement report left open. Three consumers wrote cluster Secrets (`platform/secrets.go`, `provisioning/addon_inject.go`, `ai/cluster.ensureHfTokenSecret`) and one read one back. An RPC that reads a tenant Secret is a credential-exfiltration endpoint however carefully it is gated, and the standing rule is that credentials live in KMS. The mechanism already exists in-cluster: **`KMSSecret`** (`secrets.lux.network/v1alpha1`) names a KMS ref, and the kms-operator resolves it into a mounted Secret. It is in the closed kind set, so `resources.apply` with `kind: KMSSecret` is the write-only projection the report asked for — no new op, nothing returned, and the material never enters this process. `platform/secrets.go`'s read-back must be re-sourced from KMS. ## Package weight, stated plainly ``` binary 630 packages zip 257 · k8s.io 276 · sigs.k8s.io 10 · own 4 · rest stdlib/shared plane contract 1 package stdlib only goja / esbuild 0 zip >= v1.18.7 makes the JS toolchain opt-in OTLP 0 telemetry rides middleware.Telemetry, nil-safe kustomize 0 rendering is a deploy concern; it never enters here ``` **630 is not ~260 and never could be.** client-go's typed clientset alone is ~279 packages, and the typed client is required: `EvictV1` (a PDB-respecting drain), `GetLogs`, and the kubelet `nodes/proxy` read have no dynamic-client equivalent. The measurement report predicted 560–600 and said to say so up front rather than discover it at the end; the measured answer is 630. What this repo does control is that **nobody else pays it**. The same ~286 packages were linked into four binaries; the consumer-facing cost is now one package. ## Operational notes - **RBAC**: `rbac.yaml` is the union ceiling, gated against `ops.Kinds()` by the suite. On a shared cluster do NOT bind it to a tenant — bind a namespaced Role and register the cluster with a `namespaces` bound. - **Migration from `apps/fleet`**: fleet sealed BYO kubeconfigs at `/fleet/clusters//kubeconfig` (no `orgs/` prefix, sharded by project). This repo derives `orgs//k8s/clusters//kubeconfig` and does not read the old shape — deriving the ref is what makes a row unable to name foreign material, so a compatibility branch would give that property up. Migration is therefore a one-time re-registration per BYO cluster (or a one-shot copy old-ref → new-ref inside KMS), and it must happen in the change that removes fleet's registry. - **Project sub-scope is gone on purpose.** fleet sharded by org+project with a backward-compatibility seam that kept default-project keys byte-identical. This is a new ref namespace with no existing key to stay identical with, and a cluster belongs to an org, so there is one key shape and no seam. - **`K8S_ALLOW_PRIVATE_HOSTS`** disables the non-routable-apiserver rejection. Tests only. - **`plane.KMS`/`KMSGet`/`KMSPut`** are the only tokens here this repo does not own. They are declared identically in cloud's own plane package; when HIP-0106 migration step 2 extracts `github.com/hanzoai/plane`, both collapse into that module and these three declarations are deleted in the same change. One place to move. ## Open, and needing z's call 1. **DOKS provision/destroy stays with visor.** `POST`/`DELETE /v1/k8s/clusters` in visor today provisions and destroys DigitalOcean clusters on the house account — a cloud-provider spend action, and providers are visor's job. So `/v1/k8s/clusters` here is the REGISTRY (list/get/register/deregister); visor keeps provider-side creation under `/v1/compute/…` and calls `clusters.register` when a cluster comes up. This is the one place "`/v1/k8s/clusters` survives and moves" and "cloud PROVIDERS are visor's job" pull in opposite directions, and it is flagged rather than guessed. 2. **`/v1/clusters` removal is six routes, not a rename.** visor registers `GET`, `POST` (BYO attach), `DELETE /:id` (BYO detach), `POST /:clusterId/pools`, `POST /:clusterId/pools/:poolId/scale`, `DELETE /:clusterId/pools/:poolId`. The node-pool trio is provider work and follows (1). `POST /v1/clusters` — the BYO kubeconfig attach — is the single most important operation in the whole set and lands here as `clusters.register`. Every caller (console, `hanzo clusters …`, SDKs, docs) moves in the same change; forward-only, no alias.