The host resolved and answered 404 on every path, so Lux had a secret store and
no way to reach it by name. Three separate reasons, each silent:
DNS pointed kms.lux.cloud at 129.212.164.5 — hanzo-k8s — which has no ingress
for that host. The Lux KMS is on lux-k8s (134.199.138.27). Repointed.
The Ingress named namespace lux-kms, the Node-era KMS. Its Service still
exists and has ZERO endpoints, so even a matching route would have found an
empty backend. The live store is the Go one in lux-kms-go.
cert-manager.io/cluster-issuer said agnost-dns01, which is not a ClusterIssuer
on this cluster — only letsencrypt-prod is. No certificate was ever going to
be issued.
Now serving with a real certificate (CN=kms.lux.cloud, Let's Encrypt, to
2026-11-03): /v1/kms/health 200, /v1/kms/secrets 401 missing bearer token.
Publishing it is safe BECAUSE it is API-only — `/` and `/admin` are 404, there
is no console. That is the difference from cloud's embedded KMS, whose console
had no authentication of its own and is why its brand hosts sit behind an
admin-guard ForwardAuth.
The dead Node-era manifests are gone rather than left to be re-applied: their
kustomization set `namespace: lux-kms`, which would have pushed this very
ingress back into the empty namespace. The ingress now lives beside the
StatefulSet it fronts, and the image pin records what the cluster runs
(v1.12.13, not v1.12.11).
IN-CLUSTER CALLERS STILL USE THE SERVICE, and that is not an oversight: from
inside the cluster the public host answers 404 (the hairpin does not reach the
ingress), which is exactly why the two KMSSecrets that already named
https://kms.lux.cloud were failing to sync. 42 of 51 CRs already use
http://kms.lux-kms-go.svc.cluster.local; the strays now do too.
Co-authored-by: Hanzo Dev <[email protected]>
zeekayandHanzo Dev
53deab0d3b