mirror of
https://github.com/hanzoai/o11y.git
synced 2026-10-15 12:58:33 +00:00
zeekay
9870d086b4
o11y receives pushed telemetry — a service emits, o11y stores. That is right for traces, metrics and logs and it has one blind spot: a crashed service pushes nothing, and nothing is indistinguishable from idle. Availability is the one question that must be asked, not answered. That gap is why health checks lived in a separate metrics stack outside o11y, probed by blackbox scrape jobs, with their own alert rules. The split was not a design choice; it was a missing capability. The prober closes it. Each target is probed on an interval and the result is an ordinary OTel gauge, so it lands in the same store as every other metric and pkg/ruler evaluates alerts on it with no special path. One system, one store, one place alert rules live. Two behaviours are deliberate and pinned by tests. An unreachable target records 0, never silence: a gauge that stops being written looks exactly like a prober that died, and telling those apart is the entire point. And Start probes immediately rather than after one interval, so a service already down at boot reads as down instead of unknown. Defaults are applied to targets in New, not Start. They were in Start first, which left a zero Timeout on any target reached another way — and a zero timeout is an already-expired context, so every probe failed and every service read as down. Caught by the healthy-target test.