zeekay 468afedb43 Delete the Node of a terminated EC2 instance
k3s here runs no cloud controller, so an instance an autoscaling group
terminates leaves its Node NotReady, and a StatefulSet pod bound to it
waits for a person (five Nodes were deleted by hand on 2026-10-08).

controllers::node watches Nodes and deletes one that is not Ready, has
been not Ready for two minutes, whose providerID is
aws:///<zone>/<instance>, and whose instance EC2 reports terminated or no
longer knows. The grace keeps a Node that has just registered, or an
instance EC2 has not yet heard of, from being taken for a dead one. Any
other answer, refusal or failure deletes nothing and is asked again in a
minute. The delete carries the Node's uid. The deletion is logged.

core::ec2 is the one AWS call, DescribeInstances signed with Signature
Version 4 (hmac + sha2), tested against AWS's documented example. The
controller runs only when AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are
set, and warns when only one is; the key pair is read through a lookup so
the gate is tested without the process environment. The operator
ClusterRole gains get/list/watch/delete on nodes.

Inert until hanzoai/universe gives the Deployment the key and the live
ClusterRole the nodes rule; LLM.md lists the three pieces.

Claude-Session: https://claude.ai/code/session_01SCPMGSQsLNF2YtHFL4bwyp
2026-10-09 04:24:50 -07:00

operator

operator

One Rust binary that runs the Hanzo platform on Kubernetes — N CRD Kinds, one reconcile loop, an API group you pick at install time. Declare a Service, Gateway, Datastore, Network, or Ingress (and 20+ more) and the operator materializes the Deployments, Services, StatefulSets, TLS, autoscaling, and KMS-synced secrets behind it. See LLM.md for the agent-friendly deep dive.

Canonical homes

Impl Repo Image
Rust hanzoai/operator (this repo) ghcr.io/hanzoai/operator
Go luxfi/operator ghcr.io/luxfi/operator

Both implementations target full feature parity against a shared CRD wire contract — each cluster deploys whichever language fits its operating context (Hanzo web2 ↔ Lux web3). Any change to the CRD wire shape lands in both impls.

What it manages

Kinds at <api-group>/v1. No compat aliases — the v1 Kinds are the one way.

Kind Purpose Materializes
Service Stateless service (most common) Deployment, Service, Ingress, HPA, PDB, NetworkPolicy, KMSSecret
Datastore Generic stateful service (dispatches on spec.type) StatefulSet, ClusterIP + headless Service, PVC, KMSSecret
SQL / KV / DocDB / S3 Thin facades over Datastore for each engine Same as Datastore
Gateway KrakenD-based API gateway Deployment, Service, ConfigMap (krakend.json), Ingress
MPC Multi-party computation threshold cluster StatefulSet, ClusterIP + headless Service
Network Blockchain validator network (mode derived from networkID + validators) StatefulSet (validators), Services, PVC
LuxRuntime Lux primary-network validators + tenant chain imports StatefulSet, Services, PVC, CronJob, Jobs
NodeFleet Pinned node fleet StatefulSet, Services
Ingress Multi-domain routing with cert-manager TLS Multiple Ingress resources
DNS Multi-tenant CoreDNS deployment Deployment, Service
Base hanzoai/base-ha cluster (Quasar-pinned writer) StatefulSet, headless + ClusterIP Services
IAM / KMS / LLM / Indexer / Explorer Thin facades over Service Same as Service
SPA / Static / Queue / Observability / Function App-shaped facades Service / Datastore facades
Chain / Validator Sub-resources of Network (NoOp stubs) —

Quick start

# Install CRDs + operator (namespace, RBAC, Deployment) — idempotent SSA.
operator install --image ghcr.io/hanzoai/operator:vX.Y.Z

# Install, then apply your platform's App CRs so the operator brings the stack up.
operator up --image ghcr.io/hanzoai/operator:vX.Y.Z --manifests ./crs

Then declare a workload:

apiVersion: hanzo.ai/v1
kind: Service
metadata:
  name: hello
  namespace: hanzo
spec:
  image: ghcr.io/hanzoai/hello:v1.0.0
  env:
    - name: PORT
      value: "8080"
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10

kubectl apply -f it and the operator reconciles the Deployment, Service, Ingress, HPA, PDB, NetworkPolicy, and KMSSecret behind it.

Critical invariant

spec.env, spec.volumes, spec.volumeMounts MUST be honored on every generated Deployment.

cargo test --lib controllers::service::tests
# env_is_carried_to_main_container ... ok
# volume_mounts_are_carried_to_main_container ... ok
# deployment_carries_volumes ... ok

When a Service CR sets autoscaling.enabled = true, the operator emits the Deployment with no spec.replicas so the HPA is the sole field manager for that field (server-side apply would otherwise fight the HPA on every reconcile cycle).

Build / Test

cargo build --release
cargo test --lib                      # unit tests
cargo clippy --all-targets -- -D warnings
cargo fmt --check

API group rebinding

kube-rs's CustomResource derive bakes the API group at compile time — the default is hanzo.ai. To target another universe's API group, regenerate the CRD YAML via the generate-crd-yaml binary:

generate-crd-yaml --api-group lux.cloud   --out k8s/crds/all-lux.cloud.yaml
generate-crd-yaml --api-group hanzo.ai    --out k8s/crds/all-hanzo.ai.yaml
generate-crd-yaml --api-group zoo.cloud   --out k8s/crds/all-zoo.cloud.yaml
generate-crd-yaml --api-group osage.cloud --out k8s/crds/all-osage.cloud.yaml

The running binary accepts --api-group X.Y / OPERATOR_API_GROUP=X.Y and uses the resolved group for owner references and dynamic KMSSecret references.

Layout

src/
  main.rs              clap args, leader election, controller spawn
  lib.rs               library facade (re-exports)
  crd.rs               CRD types
  crd_types.rs         JsonSchema wrappers for k8s-openapi types
  manifests.rs         pure K8s object builders
  apply.rs             server-side apply (typed + DynamicObject)
  api_group.rs         runtime API-group resolution
  controllers/         one module per Kind
  core/                shared reconciler primitives
    error.rs           OperatorError + Result
    leader.rs          coordination.k8s.io/v1 lease loop
    iam_admin.rs       POST /v1/iam/admin/applications/upsert
    secret.rs          KMSSecret hijack guard + NUL-byte rejection
    status.rs          status.conditions helpers
    reconciler.rs      Action requeue cadence + clamp_resync
  bin/
    generate_crd_yaml.rs   CRD YAML generator with --api-group rewriter
scripts/                migration scripts (v0.2.x → v0.3.0)
k8s/crds/               generated CRD YAMLs per universe

Rules

  • Never :latest, :main, :dev — semver tags only (vX.Y.Z).
  • amd64 only (arm64 paused per global LLM.md 2026-04-27).
  • Honor spec.env/volumes/volumeMounts — the load-bearing assertion.
  • Edition 2021, Rust 1.79+, kube 4, k8s-openapi 0.28 (v1_33), schemars 1, jiff (not chrono).

License

BSD-3-Clause (see LICENSE — header in source files).

Hanzo — the Open AI Cloud

Open source · every language · on-chain settlement. hanzo.ai · docs.hanzo.ai

SDKs in every language — Python (flagship) · TypeScript · Go · Rust · C++ · Swift · Kotlin · umbrella

S
Description
Kubernetes operator that reconciles Hanzo platform custom resources into running infrastructure.
Readme BSD-3-Clause
2.6 MiB
Languages
Rust 99.4%
Dockerfile 0.3%
Shell 0.2%
Makefile 0.1%