mirror of https://github.com/luxfi/monitoring.git synced 2026-10-30 15:24:48 +00:00

Files

T

zeekayandHanzo Dev ac3fda1b38 docs: publish /v1/bc/C/rpc, drop dead /ext/ URLs

/ext/ is gone from api.lux.network, api.zoo.network and api.hanzo.network
(all 404 today), so every published /ext/ URL was already broken. Convert
to the one canonical path and fix the URL forms that were dead for other
reasons at the same time:

  /mainnet|/testnet|/devnet path prefix -> the per-network host (404 today)
  api.testnet.lux.network, testnet-api.lux.network -> api.lux-test.network
  /ext/C/rpc (missing bc/) -> /v1/bc/C/rpc
  chain aliases + blockchain IDs under api.lux.network -> the L1s own host

Third-party RPC hosts (Avalanche, Flare, blastapi) are left untouched.

Co-authored-by: Hanzo Dev <[email protected]>

2026-07-27 18:22:03 -07:00

4.7 KiB

Raw Permalink Blame History

Lux monitoring — AI knowledge base

Single source of truth for Lux observability. Symlinked as CLAUDE.md / AGENTS.md.

Validator-health alerting (the canonical path)

Stood up after a mainnet incident where 3/5 validators crash-looped ~5h UNSEEN (no alerting existed). Now there is end-to-end paging.

Why an exporter (not luxd's own metrics)

luxd v1.31.2 serves an empty /v1/metrics (HTTP 200, 0 bytes — a node bug; /v1/health shows the matching "health reply encode failed"). So consensus metrics are sourced from the JSON-RPC + info + health APIs + the k8s API, which also yields block hashes (for fork detection) and pod restart counts — things luxd's Prometheus registry never exposed anyway.

Topology (hub-and-spoke; reuses the Hanzo o11y stack — NO new TSDB)

lux cluster (do-sfo3-lux-k8s, ns lux-system)
  lux-validator-exporter   stdlib python, polls luxd-0..4 in lux-{mainnet,testnet,devnet}
                           - C-Chain RPC: eth_getBlockByNumber(latest|finalized) -> height+hash
                           - info.peers / info.isBootstrapped ; /v1/health
                           - k8s API: pod restartCount + Ready ; pod logs -> pattern counters
                           exposes :9101/metrics  (metric prefix lux_*, label brand="lux")
  lux-validator-vmagent    scrapes the exporter, remote-writes ->
                              https://metrics.hanzo.ai/api/v1/write   (cluster=lux-k8s)
        |
        v
hanzo cluster (do-sfo3-hanzo-k8s, ns hanzo) — the existing VictoriaMetrics o11y stack
  vmsingle ............ stores the lux_* series
  vmalert ............. evaluates lux-validator-alert-rules -> Alertmanager
  alertmanager ........ routes page="true" -> lux-pager ; everything else -> lux-slack
  alert-sink .......... webhook receiver = the live, observable ops channel (proves a page)
  grafana ............. dashboard "Lux — Validator & Consensus Health" (uid lux-validators)

luxd HTTP ports per net: mainnet 9630, testnet 9640, devnet 9650 (metrics endpoint metrics:9090 is declared on the STS but luxd does not listen on it).

Source of truth

lux side (this repo): k8s/validator-health/ (exporter.py + manifests + kustomization)
- kubectl --context do-sfo3-lux-k8s apply -k k8s/validator-health
hanzo side (hanzoai/universe): infra/k8s/monitoring/
- lux-validator-alerts.yaml vmalert rule ConfigMap (all required alerts)
- vmalert.yaml alertmanager.yaml alert-sink.yaml
- grafana-lux-validators-dashboard.yaml (+ Grafana sidecar RBAC fix)
- grafana.yaml gained a VictoriaMetrics datasource (uid: victoriametrics)

Key metrics (brand="lux")

per-validator: lux_validator_up|healthy|bootstrapped|peers|ready, lux_validator_c_block_height, lux_validator_c_finalized_height, lux_validator_restarts_total, lux_validator_log_matches_total{pattern=fatal|equivocation|cert_rejected}. per-network: lux_network_validators_up|healthy|ready, lux_network_finalized_fork, lux_network_preferred_fork, lux_network_{finalized,block}_height_spread.

Alerts -> paging policy (5-validator sets, quorum 4 of 5)

page="true" alerts page immediately (Alertmanager lux-pager): LuxQuorumAtRisk (up<=3), LuxNetworkDown, LuxFinalizedTipMismatch, LuxPreferredTipMismatch(>5m), LuxNoFinalizedBlocksMainnet(5m), LuxFatalConsensusError, LuxEquivocationDetected, LuxValidatorNoPeers, LuxValidatorExporterDown. Warnings (Slack only): LuxValidatorDown, LuxValidatorsDegraded, LuxValidatorCrashLooping, LuxValidatorNotReady, LuxValidatorHeightLag, LuxNoAcceptedBlocks, LuxCertRejectionSpike, LuxValidatorLowPeers.

Activate real Slack paging (one KMS write, no redeploy)

Store the ops webhook in KMS hanzo:/o11y-alerting key SLACK_WEBHOOK_URL (KMSSecret alertmanager-slack-kms-sync syncs it to secret alertmanager-slack; Alertmanager reads api_url_file per-notification). Until then the alert-sink webhook is the live ops receiver (kubectl -n hanzo logs deploy/alert-sink).

Log-pattern accuracy (validated against live luxd logs)

fatal=\bfatal\b|\bpanic\b, equivocation=equivocat, cert_rejected requires the literal cert token next to reject/invalid/conflict/expired/untrusted. Verified 0 matches on healthy nodes; catches the real fatal EQUIVOCATION: conflicting finality cert ... crash on mainnet.

Open follow-ups

luxd /v1/metrics empty in v1.31.2 — fix in node so the native registry is scrapeable (then the exporter can be slimmed to RPC-only/hash signals).
metrics.hanzo.ai remote-write ingest is currently unauthenticated — add a bearer/basic-auth token (KMS) on vmsingle ingest + vmagent.
Optionally migrate the dead prometheus-alerts.yaml PrometheusRule (no operator installed) onto vmalert too.

4.7 KiB Raw Permalink Blame History