otel-collector/processor/o11yspanmetricsprocessor

Hanzo O11y Span Metrics Processor

Aggregates Request, Error and Duration (R.E.D) metrics from span data.

Metrics generated by this processor:

• o11y_calls_total
• o11y_latency_sum
• o11y_latency_count
• o11y_latency_bucket
• o11y_db_latency_sum
• o11y_db_latency_count
• o11y_external_call_latency_sum
• o11y_external_call_latency_count

Request counts are computed as the number of spans seen per unique set of dimensions, including Errors. For example, the following metric shows 142 calls:

o11y_calls_total{http_status_code="200",operation="/Address",service_name="shippingservice",span_kind="SPAN_KIND_SERVER",status_code="STATUS_CODE_UNSET"} 142

Multiple metrics can be aggregated if, for instance, a user wishes to view call counts just on service_name and operation.

Error counts are computed from the Request counts which have an "Error" Status Code metric dimension. For example, the following metric indicates 220 errors:

o11y_calls_total{http_status_code="503",operation="/checkout",service_name="frontend",span_kind="SPAN_KIND_CLIENT",status_code="STATUS_CODE_ERROR"} 220

Duration is computed from the difference between the span start and end times and inserted into the relevant latency histogram time bucket for each unique set dimensions. For example, the following latency buckets indicate the vast majority of spans (9K) have a 100ms latency:

o11y_latency_bucket{http_status_code="200",label1="value1",operation="/Address",service_name="shippingservice",span_kind="SPAN_KIND_SERVER",status_code="STATUS_CODE_UNSET",le="2"} 327
o11y_latency_bucket{http_status_code="200",label1="value1",operation="/Address",service_name="shippingservice",span_kind="SPAN_KIND_SERVER",status_code="STATUS_CODE_UNSET",le="6"} 751
o11y_latency_bucket{http_status_code="200",label1="value1",operation="/Address",service_name="shippingservice",span_kind="SPAN_KIND_SERVER",status_code="STATUS_CODE_UNSET",le="10"} 1195
o11y_latency_bucket{http_status_code="200",label1="value1",operation="/Address",service_name="shippingservice",span_kind="SPAN_KIND_SERVER",status_code="STATUS_CODE_UNSET",le="100"} 10180
o11y_latency_bucket{http_status_code="200",label1="value1",operation="/Address",service_name="shippingservice",span_kind="SPAN_KIND_SERVER",status_code="STATUS_CODE_UNSET",le="250"} 10180
...

Each metric will have at least the following dimensions because they are common across all spans:

• Service name
• Operation
• Span kind
• Status code

This processor lets traces to continue through the pipeline unmodified.

Time-Bucketed Keys (Timestamp-Aware Metrics)

The processor supports time-bucketed keys that enable timestamp-aware metrics. This feature is only applied when using DELTA temporality to avoid memory issues with CUMULATIVE temporality. When enabled, spans are grouped into time buckets based on their start timestamp rather than when they are processed. This ensures that metrics align with the actual time when spans occurred, providing more accurate temporal correlation.

How it works

1. Conditional Time Bucketing: Time bucketing is only applied for DELTA temporality. CUMULATIVE temporality uses traditional keys without time bucketing to prevent memory growth.
2. Time Bucketing: For DELTA temporality, spans are grouped into time buckets based on their StartTimestamp() using the configured time_bucket_interval
3. Key Structure: For DELTA temporality, time bucket information is prepended to metric keys: <bucketStartUnix> | service | operation | span.kind | status.code | [dimensions...]
4. Timestamp Alignment: Metrics are exported with timestamps that reflect the actual time bucket when spans occurred
5. Backward Compatibility: Legacy keys without time bucket prefixes are handled gracefully using current time

Benefits

• Temporal Accuracy: Metrics appear at the correct time in dashboards, not when they were processed
• Late-Arriving Spans: Spans that arrive minutes after generation are correctly attributed to their original time
• Correlation: Better alignment between traces, metrics, and logs for debugging time-sensitive issues
• Memory Efficiency: DELTA temporality with bounded memory usage per flush cycle, CUMULATIVE temporality avoids memory growth

Example Configuration

# Delta temporality with time bucketing (recommended for timestamp-aware metrics)
processors:
  o11yspanmetrics/delta:
    time_bucket_interval: 1m  # 1 minute buckets
    metrics_exporter: o11yclickhousemetrics
    aggregation_temporality: AGGREGATION_TEMPORALITY_DELTA
    metrics_flush_interval: 60s

# Cumulative temporality (no time bucketing to prevent memory issues)
processors:
  o11yspanmetrics/cumulative:
    metrics_exporter: o11yclickhousemetrics
    aggregation_temporality: AGGREGATION_TEMPORALITY_CUMULATIVE
    metrics_flush_interval: 60s

Behavioral Notes

• Delta Temporality: Uses time-bucketed keys for timestamp-aware metrics with bounded memory usage
• Cumulative Temporality: Uses traditional keys without time bucketing to prevent memory growth
• Start-Time Bucketing: Spans are attributed to the bucket of their start time, even if they cross minute boundaries
• No Lateness Window: All encountered buckets in the current cycle are exported immediately
• Memory Usage: Bounded per cycle - one key per (series × minute) encountered, then reset
• Out-of-Order: Older minute spans arriving in later cycles may cause out-of-order writes (acceptable for DELTA)

The following settings are required:

• metrics_exporter: the name of the exporter that this processor will write metrics to. This exporter must be present in a pipeline.

The following settings can be optionally configured:

• exclude_patterns: the list of regex patterns to exclude while aggregating the span data.

  • Default: []
  • Example:

    ...
      exclude_patterns:
        - name: operation
        - pattern: *health*
    ...
    

• latency_histogram_buckets: the list of durations defining the latency histogram buckets.

  • Default: [2ms, 4ms, 6ms, 8ms, 10ms, 50ms, 100ms, 200ms, 400ms, 800ms, 1s, 1400ms, 2s, 5s, 10s, 15s]

• dimensions: the list of dimensions to add together with the default dimensions defined above.

  Each additional dimension is defined with a name which is looked up in the span's collection of attributes or resource attributes (AKA process tags) such as ip, host.name or region.

  If the named attribute is missing in the span, the optional provided default is used.

  If no default is provided, this dimension will be omitted from the metric.

• dimensions_cache_size: the max items number of metric_key_to_dimensions_cache. If not provided, will use default value size 100000.

• aggregation_temporality: Defines the aggregation temporality of the generated metrics. One of either AGGREGATION_TEMPORALITY_CUMULATIVE or AGGREGATION_TEMPORALITY_DELTA.

  • Default: AGGREGATION_TEMPORALITY_CUMULATIVE

• time_bucket_interval: Defines the time interval for bucketing spans based on their start timestamp. Spans are grouped into time buckets based on when they started, not when they are processed. This enables timestamp-aware metrics that align with the actual time when spans occurred.

  • Default: 1m (1 minute)
  • Example: 30s, 2m, 5m
  • Note: This feature is only applied for DELTA temporality to prevent memory issues with CUMULATIVE temporality.