Files
8b7d03cbf0 [Metrics generator] Initial implementation (#1282)
* Add metrics-generator component

* Add some initial logic to make the generator module optional

* Rename tempo_distributor_metrics_generator_clients metric

* Periodical check in: add remote-write setup

* jsonnet: rename generator -> metrics_generator

* Periodical check in: add instance, copy servicegraphprocessor over, add scrape endpoint

* Fix make test

* Support spanmetrics via OTel collector

Generator is now able to run an embedded OTel collector.
This allows to run any number of processors, which includes spanmetrics.

This collector uses a custom metrics exporter and a remote write client.
Metrics generated by the spanmetrics processor are converted to Prometheus
and appended to the remote write client's WAL, which will export the metric
data after commit.

* Implement internal spanmetrics processor

* Add latency metrics

* Only loop one for latency metrics

* Improve how metrics are collected, some bug fixes

Changes:
- Remote write was constantly failing with “out of order samples”. Fix: apparently the timestamp has to be in milliseconds, not in seconds. There is not a lot of documentation on the Prometheus interfaces, but the cortex proto has a field timestampMs.
- After this remote write was failing with label name \"le\" is not unique: invalid sample. Fix: there was a bug in the append logic of the labels array. Since you were reusing the labels variable in a for-loop, the le label was added multiple times.

Initially I thought pushing metrics on every push request was causing the out-of-order issues, so I refactored the processor a bit to have both a PushSpans and a CollectMetrics method. CollectMetrics is called every 15s with a storage.Appender. This way we can also use a single appender for multiple processors,
This wasn’t necessary to get it working, but I think this is the logic we want to end up with (i.e. don’t send samples too often to keep DPM low).

* make vendor-check

* Add spanmetrics_test.go

* jsonnet: add metrics-generator service

* Add distributor.enable_metrics_generator_ring; some bug fixes

* Remove unused Registerer and scrape endpoint code; sprinkle some more TODO's around

* Add crude mechanism to trim active series in span metrics processor

* span metrics: add delete_after_last_update

* Admit config parameters in spanmetrics processor

Histogram buckets and additional dimensions can be configured

* Initial implementation of service graphs processor

* Hook up service graph processor

Store
- make sure tests pass
- fix copy-paste error in shouldEvictHead

Service graphs
- ensure metric names align with agent
- server latencies were overwriting client latencies
- expose operational metrics

* Rename metrics from generator_processor_... to metrics_generator_processor_...

* Set up BasicLifecycler, load balance spans across metrics-generator

* Replace processor.CollectMetrics with a custom prometheus.Registerer

Changes:

Service graphs processor
- replaced code to manually build metrics with CounterVec and HistogramVec
- aligned metric names with the agent
- commented out code around collectCh, this was leaking memory

Span metrics processor
- replaced code to manually build metrics with CounterVec and HistogramVec

Introduced Registry to bridge prometheus.Registerer and storage.Appender

Add support for configuring external labels

Fix generator_test.go

* make vendor-check

* Refactor remote-write structures, add tests

* Add add_instance_id_label config

* Split write requests up to a max msg size

Builds write requests up to a max size and sends them sequentially.

* Service graphs: collect edges when completed

When an edge is completed, it's sent to be collected by a number of
goroutines that call collectEdge.

Edges can also be collected when expired.

* Store: interface: give paramaters names

* Service graphs: Close closech

* Simplify evict method name

* Move remote_write to its own method

* Add metrics_generator_processors to overrides, dynamically create/remove processors

* Add tests for split requests in remote write appender

* Refactor distributor code a bit

* Make collection interval configurable

* Add concurrency test for instance

* Minor tweaks

* Add metrics to track failed active processors updates

* Tweak remote write metrics

* Add exemplars support

* Fix latency measurement in spanmetrics processor

* Fix typo

* Change metrics generator enabled config var

* Return ready when generator is registered in the ring

* Add read-only mode during generator shutdown

* Configure docker-compose distributed to work with metrics generator

* Enable pushing bytes to metrics-generator

* Remove cortex from distributed example

* Replace cortexpb -> prompb, move all remote-write stuff into remotewrite package

* Set context timeout when collecting metrics, remove hola

* Protect readOnly from concurrency

* Update vendor

* Rename metrics prefix span_metrics to spanmetrics

This matches the naming from OTel spanmetrics processor

* Always add le="+Inf" bucket when not set already

* Fix lint errors

* Fix compactor test

Test was writing tempodb.Trace to the WAL, instead of tempodb.TraceBytes

* Move expire goroutine to collector workers

* Move metrics_generator_enabled to the top-level config

* Update docker-compose distributor example

* Use snappy from klauspost/compress

* Fix distributor test after moving metrics_generator_enabled flag

* Reorganize how processors are updated, simplify use of locks

* Stub time in processor tests

* Fix how label slices are handled in registry

* jsonnet: disable metrics-generator by default

* Regenerate kube manifests

* Pin time.Now in test with registry

* Update CHANGELOG.md

Co-authored-by: Koenraad Verheyden <[email protected]>
2022-02-21 15:34:24 +01:00

109 lines
3.4 KiB
Go

// Copyright 2016 Michal Witkowski. All Rights Reserved.
// See LICENSE for licensing terms.
package conntrack
import (
"context"
"net"
"os"
"syscall"
prom "github.com/prometheus/client_golang/prometheus"
)
type failureReason string
const (
failedResolution = "resolution"
failedConnRefused = "refused"
failedTimeout = "timeout"
failedUnknown = "unknown"
)
var (
dialerAttemptedTotal = prom.NewCounterVec(
prom.CounterOpts{
Namespace: "net",
Subsystem: "conntrack",
Name: "dialer_conn_attempted_total",
Help: "Total number of connections attempted by the given dialer a given name.",
}, []string{"dialer_name"})
dialerConnEstablishedTotal = prom.NewCounterVec(
prom.CounterOpts{
Namespace: "net",
Subsystem: "conntrack",
Name: "dialer_conn_established_total",
Help: "Total number of connections successfully established by the given dialer a given name.",
}, []string{"dialer_name"})
dialerConnFailedTotal = prom.NewCounterVec(
prom.CounterOpts{
Namespace: "net",
Subsystem: "conntrack",
Name: "dialer_conn_failed_total",
Help: "Total number of connections failed to dial by the dialer a given name.",
}, []string{"dialer_name", "reason"})
dialerConnClosedTotal = prom.NewCounterVec(
prom.CounterOpts{
Namespace: "net",
Subsystem: "conntrack",
Name: "dialer_conn_closed_total",
Help: "Total number of connections closed which originated from the dialer of a given name.",
}, []string{"dialer_name"})
)
func init() {
prom.MustRegister(dialerAttemptedTotal)
prom.MustRegister(dialerConnEstablishedTotal)
prom.MustRegister(dialerConnFailedTotal)
prom.MustRegister(dialerConnClosedTotal)
}
// preRegisterDialerMetrics pre-populates Prometheus labels for the given dialer name, to avoid Prometheus missing labels issue.
func PreRegisterDialerMetrics(dialerName string) {
dialerAttemptedTotal.WithLabelValues(dialerName)
dialerConnEstablishedTotal.WithLabelValues(dialerName)
for _, reason := range []failureReason{failedTimeout, failedResolution, failedConnRefused, failedUnknown} {
dialerConnFailedTotal.WithLabelValues(dialerName, string(reason))
}
dialerConnClosedTotal.WithLabelValues(dialerName)
}
func reportDialerConnAttempt(dialerName string) {
dialerAttemptedTotal.WithLabelValues(dialerName).Inc()
}
func reportDialerConnEstablished(dialerName string) {
dialerConnEstablishedTotal.WithLabelValues(dialerName).Inc()
}
func reportDialerConnClosed(dialerName string) {
dialerConnClosedTotal.WithLabelValues(dialerName).Inc()
}
func reportDialerConnFailed(dialerName string, err error) {
if netErr, ok := err.(*net.OpError); ok {
switch nestErr := netErr.Err.(type) {
case *net.DNSError:
dialerConnFailedTotal.WithLabelValues(dialerName, string(failedResolution)).Inc()
return
case *os.SyscallError:
if nestErr.Err == syscall.ECONNREFUSED {
dialerConnFailedTotal.WithLabelValues(dialerName, string(failedConnRefused)).Inc()
}
dialerConnFailedTotal.WithLabelValues(dialerName, string(failedUnknown)).Inc()
return
}
if netErr.Timeout() {
dialerConnFailedTotal.WithLabelValues(dialerName, string(failedTimeout)).Inc()
}
} else if err == context.Canceled || err == context.DeadlineExceeded {
dialerConnFailedTotal.WithLabelValues(dialerName, string(failedTimeout)).Inc()
return
}
dialerConnFailedTotal.WithLabelValues(dialerName, string(failedUnknown)).Inc()
}