Files
skills/public/observability-cost-optimization.md

15 KiB
Raw Permalink Blame History

name, description
name description
observability-cost-optimization Optimizing observability costs through sampling, cardinality management, and retention strategies

Observability Cost Optimization

Scope: Cardinality management, sampling strategies, log reduction, metrics aggregation, retention policies, OTel Collector cost reduction

Lines: 420

Last Updated: 2025-10-26


When to Use This Skill

Use this skill when:

  • Observability costs are growing faster than infrastructure costs
  • Experiencing cardinality explosion in metrics or traces
  • Need to reduce log ingestion volume without losing critical data
  • Implementing sampling strategies for high-traffic systems
  • Optimizing retention policies for traces, metrics, and logs
  • Configuring OTel Collector processors for cost reduction
  • Balancing observability coverage with budget constraints

Don't use for:

  • Initial observability setup (implement first, optimize later)
  • Low-traffic systems (<1000 req/sec)
  • Systems where observability budget is unconstrained

Context (2024-2025):

  • 50%+ of teams cite cost as biggest observability challenge (CNCF survey)
  • Average 30-40% cost reduction achievable with proper sampling
  • Cardinality is #1 cost driver for metrics-based systems
  • Logs account for 60-70% of total observability costs

Core Concepts

Cost Drivers

1. Cardinality (Metrics/Traces)

  • Number of unique time series (metric + unique label combinations)
  • Example: http_requests_total{method="GET", endpoint="/api/users", status="200"}
  • Impact: 10x cardinality = 10x storage + 10x query cost

2. Ingestion Volume (Logs/Traces)

  • Number of events per second
  • Impact: 1M logs/day vs 1B logs/day = 1000x cost difference

3. Retention Period

  • How long data is stored
  • Impact: 30d vs 90d retention = 3x storage cost

4. Query Complexity

  • Aggregations, joins, scans
  • Impact: High-cardinality queries can timeout or spike costs

Cost Optimization Strategy

1. Identify: What's costing the most? (cardinality, volume, retention)
2. Reduce: Apply sampling, filtering, aggregation
3. Retain: Short retention for raw data, long retention for aggregates
4. Monitor: Track cost metrics (bytes ingested, cardinality growth)

Patterns

Pattern 1: Cardinality Management (Metrics)

from prometheus_client import Counter, Histogram

# ❌ HIGH CARDINALITY: user_id has 1M+ unique values
requests_bad = Counter(
    'http_requests_total',
    'HTTP requests',
    ['method', 'endpoint', 'status', 'user_id']  # 10 × 50 × 5 × 1M = 2.5M time series!
)

# ✅ LOW CARDINALITY: bounded label values
requests_good = Counter(
    'http_requests_total',
    'HTTP requests',
    ['method', 'endpoint', 'status']  # 10 × 50 × 5 = 2,500 time series
)

# ✅ ALTERNATIVE: Use histograms to reduce cardinality
request_duration = Histogram(
    'http_request_duration_seconds',
    'Request duration',
    ['method', 'endpoint'],  # No status code (use buckets instead)
    buckets=[0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0]
)

# Usage
def handle_request(method, endpoint, user_id):
    # Track request count (low cardinality)
    requests_good.labels(method=method, endpoint=endpoint, status="200").inc()

    # Track user_id in logs/traces (not metrics)
    logger.info("Request processed", extra={"user_id": user_id})

    # Track duration in histogram (bounded cardinality)
    with request_duration.labels(method=method, endpoint=endpoint).time():
        process_request()

Cardinality Reduction Techniques:

  1. Remove high-cardinality labels: user_id, email, trace_id, session_id
  2. Group labels: Instead of 1000 endpoints, group into 10 categories
  3. Use histograms: Instead of per-bucket metrics, use histogram buckets
  4. Relabel in Prometheus: Drop or aggregate labels at scrape time

Pattern 2: Prometheus Relabeling

# prometheus.yml
scrape_configs:
  - job_name: 'api-servers'
    static_configs:
      - targets: ['api-1:8080', 'api-2:8080']

    metric_relabel_configs:
      # Drop high-cardinality labels
      - source_labels: [user_id]
        action: labeldrop

      # Group endpoints into categories
      - source_labels: [endpoint]
        regex: '/api/users/.*'
        replacement: '/api/users/:id'
        target_label: endpoint

      # Drop low-value metrics entirely
      - source_labels: [__name__]
        regex: 'go_gc_.*'
        action: drop

      # Limit label values
      - source_labels: [status_code]
        regex: '2..'
        replacement: '2xx'
        target_label: status_code

Pattern 3: Tail-Based Sampling (Traces)

# otel-collector-config.yaml
processors:
  tail_sampling:
    # Wait 10s for trace to complete before sampling decision
    decision_wait: 10s

    # Limit in-memory traces
    num_traces: 100000

    policies:
      # ALWAYS sample errors (0% sampling = keep all)
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]

      # ALWAYS sample slow requests (>1s)
      - name: slow-requests
        type: latency
        latency:
          threshold_ms: 1000

      # Sample 100% of critical endpoints
      - name: critical-endpoints
        type: string_attribute
        string_attribute:
          key: http.target
          values: [/api/checkout, /api/payment]
          enabled_regex_matching: true
          invert_match: false

      # Sample 5% of all other requests
      - name: probabilistic
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlp/jaeger]

Tail-Based Sampling Benefits:

  • Smart sampling: Keep errors and slow requests, sample rest
  • Cost reduction: 95% reduction in trace volume → 95% cost savings
  • No data loss: Critical traces always retained

Pattern 4: Probabilistic Sampling (Head-Based)

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import ParentBasedTraceIdRatioBased

# Sample 10% of traces (decided at trace creation)
sampler = ParentBasedTraceIdRatioBased(0.1)

tracer_provider = TracerProvider(sampler=sampler)

Head-Based vs Tail-Based:

Feature Head-Based Tail-Based
Decision Point Trace start Trace end
Context No visibility into trace outcome Full trace visibility
Latency Low Adds 10s buffer
Cost Lower (drops early) Higher (buffers traces)
Use Case High-volume, uniform traffic Error/latency-aware sampling

Pattern 5: Log Sampling and Filtering

import logging
import random

class SamplingFilter(logging.Filter):
    """Sample 10% of INFO logs, keep all WARNING+."""

    def filter(self, record):
        if record.levelno >= logging.WARNING:
            return True  # Keep all warnings/errors
        return random.random() < 0.1  # Sample 10% of info logs

logger = logging.getLogger(__name__)
logger.addFilter(SamplingFilter())

OTel Collector Log Filtering:

processors:
  filter:
    logs:
      include:
        # Keep errors
        match_type: regexp
        record_attributes:
          - key: level
            value: ERROR|FATAL

      exclude:
        # Drop noisy debug logs
        match_type: regexp
        record_attributes:
          - key: message
            value: '.*health check.*'

  # Rate limiting (max 1000 logs/sec)
  groupbytrace:
    wait_duration: 1s
    num_traces: 1000

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [filter, groupbytrace, batch]
      exporters: [loki]

Pattern 6: Adaptive Sampling

from opentelemetry.sdk.trace.sampling import Sampler, Decision, SamplingResult
import time

class AdaptiveSampler(Sampler):
    """Dynamically adjust sampling based on request rate."""

    def __init__(self, target_rps=100):
        self.target_rps = target_rps
        self.request_count = 0
        self.last_reset = time.time()
        self.current_rate = 1.0

    def should_sample(self, *args, **kwargs):
        now = time.time()

        # Reset counter every second
        if now - self.last_reset > 1.0:
            current_rps = self.request_count
            self.request_count = 0
            self.last_reset = now

            # Adjust sampling rate
            if current_rps > 0:
                self.current_rate = min(1.0, self.target_rps / current_rps)

        self.request_count += 1

        # Sample based on current rate
        if random.random() < self.current_rate:
            return SamplingResult(Decision.RECORD_AND_SAMPLE)
        return SamplingResult(Decision.DROP)

    def get_description(self):
        return f"AdaptiveSampler(target_rps={self.target_rps})"

Pattern 7: Metrics Aggregation and Recording Rules

# prometheus-rules.yml
groups:
  - name: aggregation
    interval: 1m
    rules:
      # Pre-aggregate high-cardinality metrics
      - record: http_requests:rate5m
        expr: |
          sum by (method, status) (
            rate(http_requests_total[5m])
          )

      # Drop detailed labels, keep aggregates
      - record: http_errors:rate5m
        expr: |
          sum by (service) (
            rate(http_requests_total{status=~"5.."}[5m])
          )

      # Compute p95 latency (cheaper to query pre-computed)
      - record: http_request_duration:p95
        expr: |
          histogram_quantile(0.95,
            sum by (le, service) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )

Retention Strategy:

Raw metrics: 7d retention
Aggregated metrics (5m): 30d retention
Aggregated metrics (1h): 90d retention

Pattern 8: Attribute Processor (OTel Collector)

processors:
  # Remove high-cardinality attributes
  attributes:
    actions:
      # Delete PII
      - key: email
        action: delete
      - key: user_id
        action: delete

      # Hash high-cardinality IDs
      - key: session_id
        action: hash

      # Limit attribute value length
      - key: error_message
        action: extract
        pattern: '^(.{100}).*'
        extracted_attributes: {short_message: '\1'}

  # Group spans by endpoint pattern
  spanmetrics:
    metrics_exporter: prometheus
    latency_histogram_buckets: [0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0]
    dimensions:
      - name: http.method
      - name: http.status_code

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [attributes, spanmetrics, batch]
      exporters: [otlp/jaeger]

Cost Optimization Checklist

Metrics

[ ] Identify high-cardinality metrics (>10k time series)
[ ] Remove user_id, email, trace_id from labels
[ ] Group endpoints into categories (e.g., /api/users/:id)
[ ] Use histograms instead of per-value metrics
[ ] Set up Prometheus relabeling rules
[ ] Create recording rules for expensive queries
[ ] Reduce scrape interval for low-priority metrics (30s → 1m)
[ ] Set retention: 7d raw, 30d aggregated

Traces

[ ] Implement tail-based sampling (sample 5-10% of normal traffic)
[ ] ALWAYS sample errors and slow requests
[ ] ALWAYS sample critical endpoints
[ ] Use adaptive sampling for high-traffic services
[ ] Remove PII from span attributes
[ ] Set trace retention: 7d for sampled traces
[ ] Enable span metrics for aggregate view

Logs

[ ] Sample INFO logs (10-20%), keep WARNING/ERROR (100%)
[ ] Filter out health checks, heartbeats
[ ] Limit log message length (max 1KB)
[ ] Use structured logging (cheaper to query)
[ ] Set retention: 3d raw, 30d errors only
[ ] Rate limit noisy services (max 1k logs/sec)

General

[ ] Monitor cost metrics (bytes ingested, cardinality growth)
[ ] Set up alerts for cost spikes
[ ] Review and prune unused metrics/logs
[ ] Use OTel Collector for preprocessing (cheaper than backend)
[ ] Negotiate volume discounts with observability vendor
[ ] Consider open-source alternatives (Grafana stack)

Quick Reference

Cardinality Limits (Best Practices)

Metric label cardinality: <100 unique values per label
Total time series per metric: <10,000
Total time series per service: <100,000
Prometheus instance limit: <1M active time series

Sampling Rates by Traffic Volume

< 100 req/sec: 100% sampling
100-1k req/sec: 50% sampling
1k-10k req/sec: 10% sampling
10k+ req/sec: 1-5% sampling (with tail-based for errors)

Retention Guidelines

Traces:
  - Raw: 7 days
  - Aggregated (span metrics): 30 days

Metrics:
  - Raw (15s): 7 days
  - 5m aggregates: 30 days
  - 1h aggregates: 90 days

Logs:
  - INFO: 3 days
  - WARNING: 14 days
  - ERROR: 30 days

Cost Estimation Formula

Monthly Cost = (Ingestion Volume × Ingestion Rate) + (Storage Volume × Retention Days × Storage Rate)

Example:
Ingestion: 1TB/day × $0.50/GB = $500/day
Storage: 30TB × $0.02/GB/month = $600/month
Total: ~$15,600/month

After optimization (90% reduction):
Ingestion: 100GB/day × $0.50/GB = $50/day
Storage: 3TB × $0.02/GB/month = $60/month
Total: ~$1,560/month (90% savings)

Anti-Patterns

❌ High-Cardinality Labels

# WRONG: Unbounded label values
requests = Counter('requests', 'Requests', ['user_id', 'email'])

# CORRECT: Bounded labels
requests = Counter('requests', 'Requests', ['method', 'status'])

❌ No Sampling Strategy

# WRONG: 100% sampling for high-traffic service
processors:
  probabilistic_sampler:
    sampling_percentage: 100  # 10k req/sec = 10k traces/sec!

# CORRECT: Tail-based sampling
processors:
  tail_sampling:
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: sample
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

❌ Same Retention for All Data

# WRONG: 90d retention for all logs (expensive)
retention_days: 90

# CORRECT: Tiered retention
retention:
  info_logs: 3d
  error_logs: 30d
  raw_metrics: 7d
  aggregated_metrics: 90d

❌ Not Monitoring Cost Metrics

# WRONG: No visibility into observability costs

# CORRECT: Track cost metrics
from prometheus_client import Counter, Gauge

logs_ingested_bytes = Counter('logs_ingested_bytes_total', 'Logs ingested')
metrics_cardinality = Gauge('metrics_active_series', 'Active time series')
traces_sampled = Counter('traces_sampled_total', 'Traces sampled', ['sampled'])

  • opentelemetry-integration.md - OTel Collector processors for cost reduction
  • metrics-instrumentation.md - Cardinality management best practices
  • distributed-tracing.md - Sampling strategies for traces
  • structured-logging.md - Efficient log formatting

Last Updated: 2025-10-26 Format Version: 1.0 (Atomic)