mirror of
https://github.com/hanzoai/traces.git
synced 2026-10-30 18:20:41 +00:00
Suraj NathandGitHub
3666bb7e86
`prometheus.MustNewConstMetric` will panic when you try to create an metric with invalid label name. in prometheus world any labels that start with `__` are invalid because they are reserved. registering same labels will also cause prometheus.MustNewConstMetric to panic. you can configure overrides to map to invalid label names or use label names that are sanitized into `__` and it will cause distributors to panic when you scrape `/usage_metrics` to collect usage tracker metrics. there are also ways to cause panic by pushing data into tempo that results in an invalid metric. `prometheus.MustNewConstMetric` is designed to panic and is expected to panic when you have invalid config. in this case, we should use `prometheus.NewConstMetric` (it's what prometheus.MustNewConstMetric uses internally) and handle errors when we can't guarantee valid label names. This PR replaces usage of `prometheus.MustNewConstMetric` in usage tracker with `prometheus.NewConstMetric` so never panic. Along with that, this PR does few more things: - validate per tenant runtime overrides and fail if overrides contains a usage tracker config that results in invalid label names (note: this will cause tempo to fail at startup with errors like `level=error ts=2025-11-19T18:23:43.440930939Z caller=app.go:231 msg="module failed" module=overrides-api err="failed to start overrides-api, because it depends on module overrides, which has failed: invalid service state: Failed, expected: Running, failure: starting module overrides: invalid service state: Failed, expected: Running, failure: failed to start subservices: not healthy, 0 terminated, 1 failed: [failed to load runtime config: load file: validating overrides for * failed: cost_attribution.dimensions has duplicate label name: 'name', both 'service.name' and 'span.http.target' map to it]"` - validate usage tracker config in user configurable overrides API so we don't allow invalid config into tempo. this will reject al - log the errors with a rate limited logger from `prometheus.NewConstMetric` in case of an invalid label name in usage tracker config. - Add `tempo_distributor_usage_tracker_errors_total` counter to capture the errors with reason for the failures in the the usage tracker