mirror of
https://github.com/luxfi/monitoring.git
synced 2026-10-30 15:24:48 +00:00
zeekayandHanzo Dev
9b0dfa0794
Every liveness signal we had asked the validators whether they agreed with each
other. That question has one answer a uniformly-frozen fleet always passes: yes.
Five nodes stopped at the same height, on the same hash, all Ready, all peered,
score full marks on every relative check — which is exactly what the two-day
freeze looked like from the outside, and why it was never alerted.
The exporter now also measures against clocks the fleet does not control:
lux_validator_c_block_timestamp the accepted tip block's OWN timestamp
lux_network_c_tip_age_seconds wall clock minus the freshest tip in the set
lux_network_tip_hash_variants distinct hashes at one height (>1 is a fork,
a count rather than a boolean so the page can
state how wide the split is)
lux_network_ready_but_rpc_dead pods reporting Ready while their RPC is dead
lux_validator_health_check_failed the named checks behind healthy=0
Tip age cannot be satisfied by consensus among stopped nodes; it climbs while the
chain sits still. On first scrape it read 29.8h on mainnet, 3.7d on devnet and
7.0d on testnet — three facts nothing was previously able to state.
/v1/health answers 503 WITH a JSON body when a check fails, so catching only the
transport error discarded the diagnosis and left every node reading a reasonless
healthy=0. The body is now read off the HTTPError too.
The scrape keep-list was silently dropping kube-state-metrics' cronjob/job
families, which are what make the chain-heartbeat CANARY observable. These
C-Chains are demand-driven (post-Etna, etnaTimestamp=0) and seal a block only
when a transaction is pending, so a flat tip proves nothing alone — the canary
supplies the missing half by submitting a real transaction every 5 minutes. Two
facts were invisible for want of five metric names: devnet's heartbeat had not
succeeded in 3.7 days, and its tip had not moved in over 24h. The tip stopped in
the same hour the canary did.
Co-authored-by: Hanzo Dev <[email protected]>
94 lines
4.4 KiB
YAML
94 lines
4.4 KiB
YAML
global:
|
|
scrape_interval: 30s
|
|
scrape_timeout: 10s
|
|
external_labels:
|
|
cluster: lux-k8s
|
|
|
|
scrape_configs:
|
|
# Validator-health exporter — carries network/instance/brand labels per series.
|
|
- job_name: lux-validators
|
|
honor_labels: true
|
|
metrics_path: /metrics
|
|
static_configs:
|
|
- targets: ["lux-validator-exporter.lux-system.svc:9101"]
|
|
labels:
|
|
brand: lux
|
|
|
|
# vmagent self-monitoring (alert if the shipper itself dies).
|
|
- job_name: lux-vmagent
|
|
static_configs:
|
|
- targets: ["127.0.0.1:8429"]
|
|
labels:
|
|
brand: lux
|
|
|
|
# Per-NODE host memory/CPU/disk — node-exporter DaemonSet (one endpoint/node,
|
|
# incl. luxd nodes). Incident-priority signal: which node is about to OOM.
|
|
- job_name: lux-node-exporter
|
|
kubernetes_sd_configs:
|
|
- role: endpoints
|
|
namespaces: {names: [lux-system]}
|
|
relabel_configs:
|
|
- source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
|
|
action: keep
|
|
regex: node-exporter;metrics
|
|
- source_labels: [__meta_kubernetes_endpoint_node_name]
|
|
target_label: node
|
|
- target_label: brand
|
|
replacement: lux
|
|
|
|
# Per-POD memory usage — kubelet cAdvisor via the apiserver proxy (reliable,
|
|
# no direct kubelet TLS). Drop-filtered to memory series only to keep the
|
|
# cardinality low for the remote-write hub.
|
|
- job_name: lux-cadvisor
|
|
scheme: https
|
|
metrics_path: /metrics/cadvisor
|
|
tls_config:
|
|
insecure_skip_verify: true
|
|
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
|
|
kubernetes_sd_configs:
|
|
- role: node
|
|
relabel_configs:
|
|
- target_label: __address__
|
|
replacement: kubernetes.default.svc:443
|
|
- source_labels: [__meta_kubernetes_node_name]
|
|
regex: (.+)
|
|
target_label: __metrics_path__
|
|
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
|
|
- source_labels: [__meta_kubernetes_node_name]
|
|
target_label: node
|
|
- target_label: brand
|
|
replacement: lux
|
|
metric_relabel_configs:
|
|
- source_labels: [__name__]
|
|
action: keep
|
|
regex: (container_memory_working_set_bytes|container_memory_rss|container_spec_memory_limit_bytes|machine_memory_bytes)
|
|
- source_labels: [container]
|
|
regex: (POD|)
|
|
action: drop
|
|
|
|
# kube-state-metrics — pod phase/restarts/OOMKilled/Evicted reason, node
|
|
# MemoryPressure condition, resource requests/limits. Drop-filtered.
|
|
#
|
|
# The cronjob/job families are what make the chain-heartbeat CANARY observable.
|
|
# These C-Chains are demand-driven (post-Etna, etnaTimestamp=0): they seal a block
|
|
# only when a transaction is pending, so a flat tip proves nothing on its own. The
|
|
# heartbeat CronJob supplies that missing half by submitting a real transaction
|
|
# every 5 minutes — it is the independent evidence of demand that lets a flat tip
|
|
# be called a STALL rather than an idle chain.
|
|
#
|
|
# kube-state-metrics already published these series; this keep-list was silently
|
|
# discarding them, so the canary's own health never reached VictoriaMetrics. Two
|
|
# facts were therefore invisible: devnet's heartbeat had not succeeded in ~3.7
|
|
# days, and its tip had not moved in over 24h. Both are now first-class signals
|
|
# (kube_cronjob_status_last_successful_time is an absolute wall-clock stamp — it
|
|
# ages on its own and needs no comparison to any other job or node).
|
|
- job_name: lux-kube-state-metrics
|
|
static_configs:
|
|
- targets: ["kube-state-metrics.lux-system.svc:8080"]
|
|
labels:
|
|
brand: lux
|
|
metric_relabel_configs:
|
|
- source_labels: [__name__]
|
|
action: keep
|
|
regex: (kube_pod_status_phase|kube_pod_status_reason|kube_pod_container_status_restarts_total|kube_pod_container_status_last_terminated_reason|kube_pod_container_status_waiting_reason|kube_node_status_condition|kube_node_status_capacity|kube_node_status_allocatable|kube_pod_info|kube_pod_container_resource_requests|kube_pod_container_resource_limits|kube_node_info|kube_deployment_status_replicas|kube_deployment_status_replicas_available|kube_deployment_status_replicas_unavailable|kube_deployment_spec_replicas|kube_statefulset_status_replicas|kube_statefulset_status_replicas_ready|kube_statefulset_replicas|kube_daemonset_status_number_ready|kube_daemonset_status_desired_number_scheduled|kube_cronjob_status_last_successful_time|kube_cronjob_status_last_schedule_time|kube_cronjob_spec_suspend|kube_job_status_failed|kube_job_status_succeeded)
|