mirror of
https://github.com/hanzoai/skills.git
synced 2026-11-19 00:47:36 +00:00
15 KiB
15 KiB
name, description
| name | description |
|---|---|
| observability-cost-optimization | Optimizing observability costs through sampling, cardinality management, and retention strategies |
Observability Cost Optimization
Scope: Cardinality management, sampling strategies, log reduction, metrics aggregation, retention policies, OTel Collector cost reduction
Lines: 420
Last Updated: 2025-10-26
When to Use This Skill
Use this skill when:
- Observability costs are growing faster than infrastructure costs
- Experiencing cardinality explosion in metrics or traces
- Need to reduce log ingestion volume without losing critical data
- Implementing sampling strategies for high-traffic systems
- Optimizing retention policies for traces, metrics, and logs
- Configuring OTel Collector processors for cost reduction
- Balancing observability coverage with budget constraints
Don't use for:
- Initial observability setup (implement first, optimize later)
- Low-traffic systems (<1000 req/sec)
- Systems where observability budget is unconstrained
Context (2024-2025):
- 50%+ of teams cite cost as biggest observability challenge (CNCF survey)
- Average 30-40% cost reduction achievable with proper sampling
- Cardinality is #1 cost driver for metrics-based systems
- Logs account for 60-70% of total observability costs
Core Concepts
Cost Drivers
1. Cardinality (Metrics/Traces)
- Number of unique time series (metric + unique label combinations)
- Example:
http_requests_total{method="GET", endpoint="/api/users", status="200"} - Impact: 10x cardinality = 10x storage + 10x query cost
2. Ingestion Volume (Logs/Traces)
- Number of events per second
- Impact: 1M logs/day vs 1B logs/day = 1000x cost difference
3. Retention Period
- How long data is stored
- Impact: 30d vs 90d retention = 3x storage cost
4. Query Complexity
- Aggregations, joins, scans
- Impact: High-cardinality queries can timeout or spike costs
Cost Optimization Strategy
1. Identify: What's costing the most? (cardinality, volume, retention)
2. Reduce: Apply sampling, filtering, aggregation
3. Retain: Short retention for raw data, long retention for aggregates
4. Monitor: Track cost metrics (bytes ingested, cardinality growth)
Patterns
Pattern 1: Cardinality Management (Metrics)
from prometheus_client import Counter, Histogram
# ❌ HIGH CARDINALITY: user_id has 1M+ unique values
requests_bad = Counter(
'http_requests_total',
'HTTP requests',
['method', 'endpoint', 'status', 'user_id'] # 10 × 50 × 5 × 1M = 2.5M time series!
)
# ✅ LOW CARDINALITY: bounded label values
requests_good = Counter(
'http_requests_total',
'HTTP requests',
['method', 'endpoint', 'status'] # 10 × 50 × 5 = 2,500 time series
)
# ✅ ALTERNATIVE: Use histograms to reduce cardinality
request_duration = Histogram(
'http_request_duration_seconds',
'Request duration',
['method', 'endpoint'], # No status code (use buckets instead)
buckets=[0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0]
)
# Usage
def handle_request(method, endpoint, user_id):
# Track request count (low cardinality)
requests_good.labels(method=method, endpoint=endpoint, status="200").inc()
# Track user_id in logs/traces (not metrics)
logger.info("Request processed", extra={"user_id": user_id})
# Track duration in histogram (bounded cardinality)
with request_duration.labels(method=method, endpoint=endpoint).time():
process_request()
Cardinality Reduction Techniques:
- Remove high-cardinality labels: user_id, email, trace_id, session_id
- Group labels: Instead of 1000 endpoints, group into 10 categories
- Use histograms: Instead of per-bucket metrics, use histogram buckets
- Relabel in Prometheus: Drop or aggregate labels at scrape time
Pattern 2: Prometheus Relabeling
# prometheus.yml
scrape_configs:
- job_name: 'api-servers'
static_configs:
- targets: ['api-1:8080', 'api-2:8080']
metric_relabel_configs:
# Drop high-cardinality labels
- source_labels: [user_id]
action: labeldrop
# Group endpoints into categories
- source_labels: [endpoint]
regex: '/api/users/.*'
replacement: '/api/users/:id'
target_label: endpoint
# Drop low-value metrics entirely
- source_labels: [__name__]
regex: 'go_gc_.*'
action: drop
# Limit label values
- source_labels: [status_code]
regex: '2..'
replacement: '2xx'
target_label: status_code
Pattern 3: Tail-Based Sampling (Traces)
# otel-collector-config.yaml
processors:
tail_sampling:
# Wait 10s for trace to complete before sampling decision
decision_wait: 10s
# Limit in-memory traces
num_traces: 100000
policies:
# ALWAYS sample errors (0% sampling = keep all)
- name: errors
type: status_code
status_code:
status_codes: [ERROR]
# ALWAYS sample slow requests (>1s)
- name: slow-requests
type: latency
latency:
threshold_ms: 1000
# Sample 100% of critical endpoints
- name: critical-endpoints
type: string_attribute
string_attribute:
key: http.target
values: [/api/checkout, /api/payment]
enabled_regex_matching: true
invert_match: false
# Sample 5% of all other requests
- name: probabilistic
type: probabilistic
probabilistic:
sampling_percentage: 5
service:
pipelines:
traces:
receivers: [otlp]
processors: [tail_sampling, batch]
exporters: [otlp/jaeger]
Tail-Based Sampling Benefits:
- Smart sampling: Keep errors and slow requests, sample rest
- Cost reduction: 95% reduction in trace volume → 95% cost savings
- No data loss: Critical traces always retained
Pattern 4: Probabilistic Sampling (Head-Based)
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import ParentBasedTraceIdRatioBased
# Sample 10% of traces (decided at trace creation)
sampler = ParentBasedTraceIdRatioBased(0.1)
tracer_provider = TracerProvider(sampler=sampler)
Head-Based vs Tail-Based:
| Feature | Head-Based | Tail-Based |
|---|---|---|
| Decision Point | Trace start | Trace end |
| Context | No visibility into trace outcome | Full trace visibility |
| Latency | Low | Adds 10s buffer |
| Cost | Lower (drops early) | Higher (buffers traces) |
| Use Case | High-volume, uniform traffic | Error/latency-aware sampling |
Pattern 5: Log Sampling and Filtering
import logging
import random
class SamplingFilter(logging.Filter):
"""Sample 10% of INFO logs, keep all WARNING+."""
def filter(self, record):
if record.levelno >= logging.WARNING:
return True # Keep all warnings/errors
return random.random() < 0.1 # Sample 10% of info logs
logger = logging.getLogger(__name__)
logger.addFilter(SamplingFilter())
OTel Collector Log Filtering:
processors:
filter:
logs:
include:
# Keep errors
match_type: regexp
record_attributes:
- key: level
value: ERROR|FATAL
exclude:
# Drop noisy debug logs
match_type: regexp
record_attributes:
- key: message
value: '.*health check.*'
# Rate limiting (max 1000 logs/sec)
groupbytrace:
wait_duration: 1s
num_traces: 1000
service:
pipelines:
logs:
receivers: [otlp]
processors: [filter, groupbytrace, batch]
exporters: [loki]
Pattern 6: Adaptive Sampling
from opentelemetry.sdk.trace.sampling import Sampler, Decision, SamplingResult
import time
class AdaptiveSampler(Sampler):
"""Dynamically adjust sampling based on request rate."""
def __init__(self, target_rps=100):
self.target_rps = target_rps
self.request_count = 0
self.last_reset = time.time()
self.current_rate = 1.0
def should_sample(self, *args, **kwargs):
now = time.time()
# Reset counter every second
if now - self.last_reset > 1.0:
current_rps = self.request_count
self.request_count = 0
self.last_reset = now
# Adjust sampling rate
if current_rps > 0:
self.current_rate = min(1.0, self.target_rps / current_rps)
self.request_count += 1
# Sample based on current rate
if random.random() < self.current_rate:
return SamplingResult(Decision.RECORD_AND_SAMPLE)
return SamplingResult(Decision.DROP)
def get_description(self):
return f"AdaptiveSampler(target_rps={self.target_rps})"
Pattern 7: Metrics Aggregation and Recording Rules
# prometheus-rules.yml
groups:
- name: aggregation
interval: 1m
rules:
# Pre-aggregate high-cardinality metrics
- record: http_requests:rate5m
expr: |
sum by (method, status) (
rate(http_requests_total[5m])
)
# Drop detailed labels, keep aggregates
- record: http_errors:rate5m
expr: |
sum by (service) (
rate(http_requests_total{status=~"5.."}[5m])
)
# Compute p95 latency (cheaper to query pre-computed)
- record: http_request_duration:p95
expr: |
histogram_quantile(0.95,
sum by (le, service) (
rate(http_request_duration_seconds_bucket[5m])
)
)
Retention Strategy:
Raw metrics: 7d retention
Aggregated metrics (5m): 30d retention
Aggregated metrics (1h): 90d retention
Pattern 8: Attribute Processor (OTel Collector)
processors:
# Remove high-cardinality attributes
attributes:
actions:
# Delete PII
- key: email
action: delete
- key: user_id
action: delete
# Hash high-cardinality IDs
- key: session_id
action: hash
# Limit attribute value length
- key: error_message
action: extract
pattern: '^(.{100}).*'
extracted_attributes: {short_message: '\1'}
# Group spans by endpoint pattern
spanmetrics:
metrics_exporter: prometheus
latency_histogram_buckets: [0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0]
dimensions:
- name: http.method
- name: http.status_code
service:
pipelines:
traces:
receivers: [otlp]
processors: [attributes, spanmetrics, batch]
exporters: [otlp/jaeger]
Cost Optimization Checklist
Metrics
[ ] Identify high-cardinality metrics (>10k time series)
[ ] Remove user_id, email, trace_id from labels
[ ] Group endpoints into categories (e.g., /api/users/:id)
[ ] Use histograms instead of per-value metrics
[ ] Set up Prometheus relabeling rules
[ ] Create recording rules for expensive queries
[ ] Reduce scrape interval for low-priority metrics (30s → 1m)
[ ] Set retention: 7d raw, 30d aggregated
Traces
[ ] Implement tail-based sampling (sample 5-10% of normal traffic)
[ ] ALWAYS sample errors and slow requests
[ ] ALWAYS sample critical endpoints
[ ] Use adaptive sampling for high-traffic services
[ ] Remove PII from span attributes
[ ] Set trace retention: 7d for sampled traces
[ ] Enable span metrics for aggregate view
Logs
[ ] Sample INFO logs (10-20%), keep WARNING/ERROR (100%)
[ ] Filter out health checks, heartbeats
[ ] Limit log message length (max 1KB)
[ ] Use structured logging (cheaper to query)
[ ] Set retention: 3d raw, 30d errors only
[ ] Rate limit noisy services (max 1k logs/sec)
General
[ ] Monitor cost metrics (bytes ingested, cardinality growth)
[ ] Set up alerts for cost spikes
[ ] Review and prune unused metrics/logs
[ ] Use OTel Collector for preprocessing (cheaper than backend)
[ ] Negotiate volume discounts with observability vendor
[ ] Consider open-source alternatives (Grafana stack)
Quick Reference
Cardinality Limits (Best Practices)
Metric label cardinality: <100 unique values per label
Total time series per metric: <10,000
Total time series per service: <100,000
Prometheus instance limit: <1M active time series
Sampling Rates by Traffic Volume
< 100 req/sec: 100% sampling
100-1k req/sec: 50% sampling
1k-10k req/sec: 10% sampling
10k+ req/sec: 1-5% sampling (with tail-based for errors)
Retention Guidelines
Traces:
- Raw: 7 days
- Aggregated (span metrics): 30 days
Metrics:
- Raw (15s): 7 days
- 5m aggregates: 30 days
- 1h aggregates: 90 days
Logs:
- INFO: 3 days
- WARNING: 14 days
- ERROR: 30 days
Cost Estimation Formula
Monthly Cost = (Ingestion Volume × Ingestion Rate) + (Storage Volume × Retention Days × Storage Rate)
Example:
Ingestion: 1TB/day × $0.50/GB = $500/day
Storage: 30TB × $0.02/GB/month = $600/month
Total: ~$15,600/month
After optimization (90% reduction):
Ingestion: 100GB/day × $0.50/GB = $50/day
Storage: 3TB × $0.02/GB/month = $60/month
Total: ~$1,560/month (90% savings)
Anti-Patterns
❌ High-Cardinality Labels
# WRONG: Unbounded label values
requests = Counter('requests', 'Requests', ['user_id', 'email'])
# CORRECT: Bounded labels
requests = Counter('requests', 'Requests', ['method', 'status'])
❌ No Sampling Strategy
# WRONG: 100% sampling for high-traffic service
processors:
probabilistic_sampler:
sampling_percentage: 100 # 10k req/sec = 10k traces/sec!
# CORRECT: Tail-based sampling
processors:
tail_sampling:
policies:
- name: errors
type: status_code
status_code: {status_codes: [ERROR]}
- name: sample
type: probabilistic
probabilistic: {sampling_percentage: 5}
❌ Same Retention for All Data
# WRONG: 90d retention for all logs (expensive)
retention_days: 90
# CORRECT: Tiered retention
retention:
info_logs: 3d
error_logs: 30d
raw_metrics: 7d
aggregated_metrics: 90d
❌ Not Monitoring Cost Metrics
# WRONG: No visibility into observability costs
# CORRECT: Track cost metrics
from prometheus_client import Counter, Gauge
logs_ingested_bytes = Counter('logs_ingested_bytes_total', 'Logs ingested')
metrics_cardinality = Gauge('metrics_active_series', 'Active time series')
traces_sampled = Counter('traces_sampled_total', 'Traces sampled', ['sampled'])
Related Skills
- opentelemetry-integration.md - OTel Collector processors for cost reduction
- metrics-instrumentation.md - Cardinality management best practices
- distributed-tracing.md - Sampling strategies for traces
- structured-logging.md - Efficient log formatting
Last Updated: 2025-10-26 Format Version: 1.0 (Atomic)