mirror of
https://github.com/luxfi/monitoring.git
synced 2026-10-30 15:24:48 +00:00
zeekayandHanzo Dev
ac1543af69
Every lux_* series was missing from the metrics store and all 29 Lux validator alert rules had been evaluating against an empty database — inactive, health=ok, lastError empty. That is the worst shape a monitoring failure can take: it looks identical to "nothing is wrong" on every dashboard. Cause: metrics.hanzo.ai was withdrawn because it published all of vmsingle:8428 unauthenticated (anonymous reads of every series, plus /-/reload and a reachable /api/v1/admin/tsdb/delete_series). Closing it was right, but it was the only path lux telemetry had into the stack. The host has since been restored as ingest-only behind basicAuth; this side was never updated to send a credential, so vmagent POSTed into a 401 on a 60s retry loop — retry #1770 by the time it was found. Scraping was healthy throughout: 20/20 targets up, 179,691 successful scrapes, packets_dropped_total=0. The data was being collected correctly and thrown away at the last hop. The credential already existed as Secret vm-remote-write in lux-system and was never referenced. This wires it in: basicAuth.username plus a passwordFile from that Secret, mounted read-only. Nothing new is minted. Verified after rollout: vmagent_remotewrite_errors_total 44 -> 0 status_code 401 791 -> none since restart metric names in the store 322 -> 718 lux_* names 0 -> 25 kube_* names 0 -> 26 cluster label values [hanzo-k8s] -> [hanzo-k8s, lux-k8s] lux rules 29 inactive -> 13 inactive, 11 pending, 3 firing Apply with `kubectl apply -k`, never `-f` on this file alone: the scrape config is a kustomize configMapGenerator, so the bare manifest names a ConfigMap that does not exist and the new pod hangs in ContainerCreating on FailedMount while the old one keeps serving. I did that first and it looked like a stuck rollout. Co-authored-by: Hanzo Dev <[email protected]>
80 lines
3.4 KiB
YAML
80 lines
3.4 KiB
YAML
# vmagent in the lux cluster: scrapes the validator-health exporter and
|
|
# remote-writes to the central Hanzo VictoriaMetrics (the o11y stack behind
|
|
# metrics.hanzo.ai). Hub-and-spoke — no new monitoring TSDB in the lux cluster.
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: lux-validator-vmagent
|
|
namespace: lux-system
|
|
labels: {app: lux-validator-vmagent, brand: lux}
|
|
spec:
|
|
replicas: 1
|
|
selector: {matchLabels: {app: lux-validator-vmagent}}
|
|
template:
|
|
metadata:
|
|
labels: {app: lux-validator-vmagent, brand: lux}
|
|
spec:
|
|
serviceAccountName: lux-validator-vmagent
|
|
# Prefer roomy non-luxd nodes; this shipper was previously scheduled on a
|
|
# luxd node (3cghsl) — move it off so it never competes with a validator.
|
|
affinity:
|
|
nodeAffinity:
|
|
preferredDuringSchedulingIgnoredDuringExecution:
|
|
- weight: 100
|
|
preference:
|
|
matchExpressions:
|
|
- {key: kubernetes.io/hostname, operator: In, values: [lux-nodes-3cgkgx, lux-nodes-3cr32q, lux-nodes-3csr33]}
|
|
securityContext: {runAsNonRoot: true, runAsUser: 65532, fsGroup: 65532}
|
|
containers:
|
|
- name: vmagent
|
|
image: victoriametrics/vmagent:v1.144.0
|
|
args:
|
|
- -promscrape.config=/config/scrape.yaml
|
|
- -remoteWrite.url=https://metrics.hanzo.ai/api/v1/write
|
|
# metrics.hanzo.ai is ingest-only and authenticated: the host serves
|
|
# POST /api/v1/write and nothing else. Without these two flags every
|
|
# POST comes back 404/401 and the whole lux_* dataset silently never
|
|
# arrives — which is exactly what happened between the host being
|
|
# withdrawn and 2026-07-28.
|
|
- -remoteWrite.basicAuth.username=lux-vmagent
|
|
- -remoteWrite.basicAuth.passwordFile=/secrets/vm-remote-write/password
|
|
- -remoteWrite.tmpDataPath=/tmp/vmagent
|
|
- -remoteWrite.label=cluster=lux-k8s
|
|
- -promscrape.suppressDuplicateScrapeTargetErrors
|
|
- -memory.allowedPercent=60
|
|
ports:
|
|
- {name: http, containerPort: 8429}
|
|
readinessProbe: {httpGet: {path: /health, port: 8429}, initialDelaySeconds: 5, periodSeconds: 15}
|
|
livenessProbe: {httpGet: {path: /health, port: 8429}, initialDelaySeconds: 10, periodSeconds: 30}
|
|
resources:
|
|
requests: {cpu: 50m, memory: 96Mi}
|
|
limits: {cpu: 300m, memory: 256Mi}
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities: {drop: [ALL]}
|
|
volumeMounts:
|
|
- {name: config, mountPath: /config}
|
|
- {name: tmp, mountPath: /tmp/vmagent}
|
|
- {name: rw-auth, mountPath: /secrets/vm-remote-write, readOnly: true}
|
|
volumes:
|
|
- name: config
|
|
configMap: {name: lux-validator-vmagent-config}
|
|
- name: tmp
|
|
emptyDir: {}
|
|
# Ingest credential. The token lives in KMS (lux-infra /o11y-ingest,
|
|
# VM_REMOTE_WRITE_TOKEN); this Secret is its delivery target.
|
|
- name: rw-auth
|
|
secret: {secretName: vm-remote-write}
|
|
---
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: lux-validator-vmagent
|
|
namespace: lux-system
|
|
labels: {app: lux-validator-vmagent, brand: lux}
|
|
spec:
|
|
selector: {app: lux-validator-vmagent}
|
|
ports:
|
|
- {name: http, port: 8429, targetPort: 8429}
|