The host resolved and answered 404 on every path, so Lux had a secret store and no way to reach it by name. Three separate reasons, each silent: DNS pointed kms.lux.cloud at 129.212.164.5 — hanzo-k8s — which has no ingress for that host. The Lux KMS is on lux-k8s (134.199.138.27). Repointed. The Ingress named namespace lux-kms, the Node-era KMS. Its Service still exists and has ZERO endpoints, so even a matching route would have found an empty backend. The live store is the Go one in lux-kms-go. cert-manager.io/cluster-issuer said agnost-dns01, which is not a ClusterIssuer on this cluster — only letsencrypt-prod is. No certificate was ever going to be issued. Now serving with a real certificate (CN=kms.lux.cloud, Let's Encrypt, to 2026-11-03): /v1/kms/health 200, /v1/kms/secrets 401 missing bearer token. Publishing it is safe BECAUSE it is API-only — `/` and `/admin` are 404, there is no console. That is the difference from cloud's embedded KMS, whose console had no authentication of its own and is why its brand hosts sit behind an admin-guard ForwardAuth. The dead Node-era manifests are gone rather than left to be re-applied: their kustomization set `namespace: lux-kms`, which would have pushed this very ingress back into the empty namespace. The ingress now lives beside the StatefulSet it fronts, and the image pin records what the cluster runs (v1.12.13, not v1.12.11). IN-CLUSTER CALLERS STILL USE THE SERVICE, and that is not an oversight: from inside the cluster the public host answers 404 (the hairpin does not reach the ingress), which is exactly why the two KMSSecrets that already named https://kms.lux.cloud were failing to sync. 42 of 51 CRs already use http://kms.lux-kms-go.svc.cluster.local; the strays now do too. Co-authored-by: Hanzo Dev <[email protected]>
110 lines
4.8 KiB
YAML
110 lines
4.8 KiB
YAML
---
|
|
# Lux KMS — Go-native MPC-backed KMS server (~/work/lux/kms).
|
|
# Replaces the legacy KMS image (ghcr.io/hanzoai/kms) that ran in `lux-kms` ns.
|
|
#
|
|
# Image: ghcr.io/luxfi/kms:v1.12.5 — deletes the unauthenticated
|
|
# GET /v1/kms/secrets/{name} route that echoed os.Getenv(name) UNWRAPPED (it
|
|
# would return KMS_MASTER_KEY_B64, the root REK, to any caller who reached :8080).
|
|
# Runs SECRETS-ONLY: KMS_MASTER_KEY_B64 is intentionally NOT injected here.
|
|
# On the main line a loaded REK activates the /v1/sdk + ZAP enveloped-secrets
|
|
# plane, whose consensus authorizer fail-closes at boot without
|
|
# KMS_CONSENSUS_VALIDATORS/OPERATORS (cmd/kms/consensus.go). This deployment has
|
|
# no consensus authority and never served /v1/sdk (ZAP_PORT=0), so injecting the
|
|
# REK would only crashloop the pod. Dropping it keeps the org-scoped secrets
|
|
# surface fully working (it uses Badger at-rest via KMS_ENCRYPTION_KEY_B64, not
|
|
# the REK) and removes the crown-jewel from the process env — completing the leak
|
|
# closure. The kms-secrets/master-key-b64 Secret is retained UNTOUCHED for a
|
|
# future MPC-rooted or consensus-wired /v1/sdk enablement. Patch-line forward,
|
|
# NEVER a major bump.
|
|
# Routes ALL on /v1/kms/<path> per Hanzo /v1/<service>/<path> standard.
|
|
# Secrets-only mode (MPC_VAULT_ID="") — secrets surface (HTTP :8080) is the
|
|
# active path the kms-operator + Go callers use.
|
|
# Auth: /v1/kms/auth/login forwards to IAM client_credentials at
|
|
# https://hanzo.id/v1/iam/oauth/token; org JWTs validated against
|
|
# https://hanzo.id/.well-known/jwks. White-label: KMS_EXPECTED_ISSUER
|
|
# accepts BOTH https://hanzo.id and https://lux.id — lux.cloud users carry
|
|
# iss=https://lux.id and both brands share the same IAM signing keys.
|
|
# ZAP secrets-server DISABLED (ZAP_PORT=0): enabling it requires
|
|
# KMS_CONSENSUS_VALIDATORS/OPERATORS authority or kmsd fail-closes.
|
|
#
|
|
# Bootstrap secrets (kms-secrets, in-cluster only — do NOT lose encryption-key-b64;
|
|
# the ZapDB at-rest store is unreadable without it):
|
|
# master-key-b64 random 32B (SecretStore/REK envelope; unused on the HTTP
|
|
# org-scoped path — RETAINED in the Secret but NOT injected
|
|
# as env in v1.12.5, see header for why)
|
|
# encryption-key-b64 random 32B (ZapDB Badger at-rest encryption)
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: lux-kms-go
|
|
---
|
|
apiVersion: apps/v1
|
|
kind: StatefulSet
|
|
metadata:
|
|
name: kms
|
|
namespace: lux-kms-go
|
|
labels: { app: kms, brand: lux }
|
|
spec:
|
|
serviceName: kms
|
|
replicas: 1
|
|
selector: { matchLabels: { app: kms } }
|
|
template:
|
|
metadata: { labels: { app: kms, brand: lux } }
|
|
spec:
|
|
imagePullSecrets:
|
|
- name: ghcr-luxfi
|
|
containers:
|
|
- name: kms
|
|
image: ghcr.io/luxfi/kms:v1.12.13
|
|
imagePullPolicy: IfNotPresent
|
|
ports:
|
|
- { name: http, containerPort: 8080, protocol: TCP }
|
|
env:
|
|
- { name: KMS_DATA_DIR, value: /data/kms }
|
|
- { name: KMS_LISTEN, value: ":8080" }
|
|
- { name: ZAP_PORT, value: "0" }
|
|
- { name: MPC_VAULT_ID, value: "" }
|
|
- { name: IAM_ENDPOINT, value: "https://hanzo.id" }
|
|
- { name: IAM_TOKEN_PATH, value: "/v1/iam/oauth/token" }
|
|
- { name: KMS_IAM_URL, value: "https://hanzo.id" }
|
|
- { name: KMS_EXPECTED_ISSUER, value: "https://hanzo.id,https://lux.id" }
|
|
# KMS_MASTER_KEY_B64 (REK) intentionally NOT injected — see header. It would
|
|
# activate the /v1/sdk + ZAP plane whose consensus authorizer fail-closes
|
|
# (crashloop) without KMS_CONSENSUS_VALIDATORS/OPERATORS. Org-scoped secrets
|
|
# do not use the REK; they rely on KMS_ENCRYPTION_KEY_B64 (Badger at-rest).
|
|
- name: KMS_ENCRYPTION_KEY_B64
|
|
valueFrom: { secretKeyRef: { name: kms-secrets, key: encryption-key-b64 } }
|
|
livenessProbe:
|
|
httpGet: { path: /healthz, port: http }
|
|
initialDelaySeconds: 10
|
|
periodSeconds: 10
|
|
readinessProbe:
|
|
httpGet: { path: /healthz, port: http }
|
|
initialDelaySeconds: 5
|
|
periodSeconds: 5
|
|
resources:
|
|
requests: { cpu: 50m, memory: 64Mi }
|
|
limits: { cpu: 500m, memory: 512Mi }
|
|
volumeMounts:
|
|
- { name: data, mountPath: /data/kms }
|
|
volumeClaimTemplates:
|
|
- metadata: { name: data }
|
|
spec:
|
|
accessModes: [ReadWriteOnce]
|
|
# Retain so an accidental PVC delete never nukes the KMS store again
|
|
# (the 2026-04 outage was loss of the lux-kms-go PVC).
|
|
storageClassName: do-block-storage-retain
|
|
resources: { requests: { storage: 5Gi } }
|
|
---
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: kms
|
|
namespace: lux-kms-go
|
|
labels: { app: kms, brand: lux }
|
|
spec:
|
|
type: ClusterIP
|
|
selector: { app: kms }
|
|
ports:
|
|
- { name: http, port: 80, targetPort: http, protocol: TCP }
|