k8s/LLM.md

hanzoai/k8s — deep notes

The one Kubernetes owner in the fleet. Read this before changing anything here; the README is the product surface, this is the reasoning.

Why the repo exists (and why a library would not have worked)

Nine places in hanzoai/cloud built a Kubernetes client. Measured, not estimated:

Holder	Credential	Cluster it assumed
apps/platform ×2, apps/deploy ×2, apps/provisioning, apps/validators, apps/cron	in-cluster ServiceAccount → KUBECONFIG	one ambient
apps/ml	in-cluster SA, re-scoped to a token file; federated via fleet.DynForOrg	ambient or the org's BYO
apps/fleet	KMS-sealed kubeconfig per org+project	N registered BYO
apps/admin/infra	DO_API_TOKEN → a cluster-admin kubeconfig per cluster	N DOKS clusters
hanzoai/ai/cluster	kubeconfig in a database row, fed to an apply-any-YAML function	one global mutable singleton

Two of those are the reason this is a service and not a shared package:

1. ai/cluster/client.go:210 deployResource decoded arbitrary YAML, resolved the GVK through a discovery RESTMapper, and create-or-updated any kind the cluster served. With a DB-sourced kubeconfig, one edited row reached any object on any cluster. It is DELETED, not ported. Its callers go through resources.apply with a named kind.
2. admin/infra held cluster-admin on every cluster on the house account to perform nine operations. nodes/proxy get alone is arbitrary kubelet access.

A library cannot fix either, because the defect is not the code — it is that N binaries hold cluster credentials. One owner, named ops, credentials in KMS, consumers ASK.

The deciding argument, though, is neither of those. It is that lux runs its own lux-k8s: an org's resources may live on a different cluster from Hanzo's, plus BYO for customers. A single-cluster client, however cleanly factored, cannot express that and would have been rewritten. Hence multi-cluster from the first line.

The model

Cluster{Name, Org, Provider, Endpoint, Nodes, NvidiaGPU, AmdGPU, Namespaces, Registered, Default}

• No Kubeconfig field and no Credential field. The ref is DERIVED — orgs/<org>/k8s/clusters/<name>/kubeconfig — because a ref stored as data could be edited to name another org's material. That is the same defect as letting a caller name its own org, one level down.
• Org is set from the validated caller, never from an argument, and appears only on replies (where it is a property of the thing described).
• Namespaces bounds a registration made with a scoped credential on a shared cluster. Empty means the whole cluster, which is what an org registering its own cluster gets.

registry.Reach(ctx, name) (*Bound, error) is the only entry point:

1. Tenant(ctx) — zip's full rule (validated user claim, non-empty org, under MaxOrgLen) plus one thing zip cannot know: the org becomes a KMS ref segment, so a value containing / or .. is refused here.
2. the name is looked up in that org's index only → ErrNoCluster (404) otherwise.
3. the credential is resolved from KMS over the call plane, carrying the caller, so KMS re-applies its own orgs/<org>/ rule.
4. SafeRESTConfig → typed + dynamic + metrics clients, cached per org/name.

Bound.Scope(ns) / Bound.One(ns) / Bound.WholeCluster() are the namespace gate, and every namespaced op goes through one of them. A named namespace outside the bound is REFUSED rather than silently narrowed — a caller that asked for the wrong thing must be told, or it will believe an empty answer.

What a "grant" is

An org may reach a cluster it owns. There is no grant table, and that is a decision rather than an omission: cross-org credential sharing is refused by KMS by construction (a call acting for B asking for orgs/A/… is forbidden), so a grant table would have been a second authority that could only ever disagree with the first.

Granting org B access to a cluster A owns is therefore: seal a scoped ServiceAccount kubeconfig under B, and register it in B's own scope with a namespaces bound. That is one existing verb, no new concept, and it is honest — to reach a cluster you must hold a credential for it, sealed in your own org's KMS.

The platform's own path for a tenant on the shared cluster is the same call, made by the provisioning job with zip.WithCaller stating the tenant it acts for.

The 36 ops

plane/plane.go is the whole contract. Groups: clusters (5), nodes (3), namespaces (4), pods (2), volumes (4), workloads (8), jobs (3), resources (5), access (1), metrics (1).

Each is registered TWICE from ONE handler: on the app (HTTP, which projects OpenAPI + an MCP tool + a CLI command) and on app.Peer() (the fleet's 0600 socket, invoked by op token through zip.Ask). The registrations are written out as direct zip.Get / zip.Post calls with literal paths deliberately — zipdoc reads the AST for exactly that shape, so a registration wrapped in a helper is a registration whose doc comment reaches neither the OpenAPI description nor the MCP tool description.

Design notes worth keeping:

• resources.watch is a bounded read, not a stream. A stream cannot cross the call plane as a value; rendering one (SSE, a websocket) is the edge's concern. It returns the changes within waitSeconds and the cursor to resume from, and expired is the field that matters — a too-old cursor rendered as an empty page is how a watch-backed board silently stops updating.
• nodes.cordon is one op over three calls (patch spec.unschedulable, list the node's pods, evict each). There is no generic node patch, because one would let a caller write any field of any node's spec. The drain evicts rather than deletes, so a PodDisruptionBudget refusal is the system working.
• volumes.expandClaim takes gi, not a patch. Growing the claim is the one way that leaves claim, PV, device and filesystem agreeing. Nothing deletes a claim.
• namespaces.delete is separately gated and refused for a bounded registration: it garbage-collects every object in the namespace, claims included.
• nodes.volumeStats is its own op so nodes/proxy can be dropped from every other caller's grant. It is the only source of volume fill we have (no cloud provider exposes it; metrics-server is absent from most of our clusters) and it reports nodesRead / nodesTotal, because a claim with no row is UNMEASURED, which is a different fact from empty.
• metrics.pods reports available. A cluster with no metrics-server rendered as zeroes tells an operator a busy namespace is idle.
• clusters.status probes the version endpoint, not a namespace list: a registration bounded to one namespace would fail a namespace list and look dead while being healthy.

Secrets — the decision

There is no secrets op, and the absence is the answer to the question the measurement report left open.

Three consumers wrote cluster Secrets (platform/secrets.go, provisioning/addon_inject.go, ai/cluster.ensureHfTokenSecret) and one read one back. An RPC that reads a tenant Secret is a credential-exfiltration endpoint however carefully it is gated, and the standing rule is that credentials live in KMS.

The mechanism already exists in-cluster: KMSSecret (secrets.lux.network/v1alpha1) names a KMS ref, and the kms-operator resolves it into a mounted Secret. It is in the closed kind set, so resources.apply with kind: KMSSecret is the write-only projection the report asked for — no new op, nothing returned, and the material never enters this process. platform/secrets.go's read-back must be re-sourced from KMS.

Package weight, stated plainly

binary            630 packages   zip 257 · k8s.io 276 · sigs.k8s.io 10 · own 4 · rest stdlib/shared
plane contract      1 package    stdlib only
goja / esbuild      0            zip >= v1.18.7 makes the JS toolchain opt-in
OTLP                0            telemetry rides middleware.Telemetry, nil-safe
kustomize           0            rendering is a deploy concern; it never enters here

630 is not ~260 and never could be. client-go's typed clientset alone is ~279 packages, and the typed client is required: EvictV1 (a PDB-respecting drain), GetLogs, and the kubelet nodes/proxy read have no dynamic-client equivalent. The measurement report predicted 560–600 and said to say so up front rather than discover it at the end; the measured answer is 630.

What this repo does control is that nobody else pays it. The same ~286 packages were linked into four binaries; the consumer-facing cost is now one package.

Operational notes

• RBAC: rbac.yaml is the union ceiling, gated against ops.Kinds() by the suite. On a shared cluster do NOT bind it to a tenant — bind a namespaced Role and register the cluster with a namespaces bound.
• Migration from apps/fleet: fleet sealed BYO kubeconfigs at <org>/fleet/clusters/<name>/kubeconfig (no orgs/ prefix, sharded by project). This repo derives orgs/<org>/k8s/clusters/<name>/kubeconfig and does not read the old shape — deriving the ref is what makes a row unable to name foreign material, so a compatibility branch would give that property up. Migration is therefore a one-time re-registration per BYO cluster (or a one-shot copy old-ref → new-ref inside KMS), and it must happen in the change that removes fleet's registry.
• Project sub-scope is gone on purpose. fleet sharded by org+project with a backward-compatibility seam that kept default-project keys byte-identical. This is a new ref namespace with no existing key to stay identical with, and a cluster belongs to an org, so there is one key shape and no seam.
• K8S_ALLOW_PRIVATE_HOSTS disables the non-routable-apiserver rejection. Tests only.
• plane.KMS/KMSGet/KMSPut are the only tokens here this repo does not own. They are declared identically in cloud's own plane package; when HIP-0106 migration step 2 extracts github.com/hanzoai/plane, both collapse into that module and these three declarations are deleted in the same change. One place to move.

Open, and needing z's call

1. DOKS provision/destroy stays with visor. POST/DELETE /v1/k8s/clusters in visor today provisions and destroys DigitalOcean clusters on the house account — a cloud-provider spend action, and providers are visor's job. So /v1/k8s/clusters here is the REGISTRY (list/get/register/deregister); visor keeps provider-side creation under /v1/compute/… and calls clusters.register when a cluster comes up. This is the one place "/v1/k8s/clusters survives and moves" and "cloud PROVIDERS are visor's job" pull in opposite directions, and it is flagged rather than guessed.
2. /v1/clusters removal is six routes, not a rename. visor registers GET, POST (BYO attach), DELETE /:id (BYO detach), POST /:clusterId/pools, POST /:clusterId/pools/:poolId/scale, DELETE /:clusterId/pools/:poolId. The node-pool trio is provider work and follows (1). POST /v1/clusters — the BYO kubeconfig attach — is the single most important operation in the whole set and lands here as clusters.register. Every caller (console, hanzo clusters …, SDKs, docs) moves in the same change; forward-only, no alias.