mirror of https://github.com/hanzoai/insights.git synced 2026-11-03 00:52:32 +00:00

Files

T

zeekayandHanzo Dev 5ac678fd84 no hogs: the person services, the mock's customer, and one enum member

PersonHog was the last of it, and it was not only a name. The package is
personinsights and the types in it are already Person* -- the shipped
descriptor under personinsights_client/proto/generated says PersonService
and the tests say TestFakePersonClient -- but the .proto still declared
PersonHogService. A client dialling one service name and a server built
from a .proto declaring another is not a naming problem, it is a wire
that does not connect; the only reason nobody hit it is that none of
these services runs here yet. Five service declarations now say what the
generated client has said all along.

StreamHog was a second fictional company, invented so a Stripe mock had
someone to bill. This fork already has Hedgebox, in five hundred
fixtures, so the mock bills Hedgebox and there is one made-up company
again instead of two.

REVIEW_HOG was an enum member in three enums that had to agree, whose
value has been `review` since the app was renamed. Only the member said
otherwise.

Every rule is in bin/debrand and every spelling is refused there, so a
fresh upstream import lands on these names and none of them can come
back. Two are named as outstanding rather than refused, with the reason
each is still sayable: hogdesk survives inside applied migrations'
choices lists, where Django compares it in Python and never writes it --
the member is Desk = "desk" and all three tables that store it hold no
rows; sherlockhog keys a service in another repository that still calls
itself that, and renaming our half would classify its traffic as a
product that does not exist.

Co-authored-by: Hanzo Dev <[email protected]>

2026-08-13 13:57:50 -07:00

36 KiB

Raw Permalink Blame History

name, description

name	description
monitoring-ingestion-pipeline	Guide for using the Grafana MCP to monitor and diagnose the Node.js ingestion pipeline workers in production. Use when investigating event lag, drops, pipeline errors, person/group processing, Kafka consumer health, Redis, Postgres, Datastore downstream health, or any ingestion worker question. Covers prod-us and prod-eu environments.

Monitoring the ingestion pipeline with Grafana MCP

The ingestion pipeline (nodejs/) is Insights's Node.js event processing layer. It consumes events from Kafka (produced by the capture service), runs them through processing steps (person resolution, group assignment, property overrides, etc.), and produces enriched events to Datastore-bound Kafka topics.

A single codebase is deployed as many K8s Deployments via the insights-app Helm chart (golden-chart migration). Each deployment sets PLUGIN_SERVER_MODE and is distinguished in metrics by two default Prometheus labels:

ingestion_pipeline — values: analytics, heatmaps, clientwarnings, errortracking
ingestion_lane — values: main, overflow, historical, async, turbo

The app label (set by K8s pod labels) matches the deployment name and is the most universal scope filter across all telemetry domains.

This skill teaches how to discover live metrics using the Grafana MCP tools rather than memorizing metric names that change as the code evolves.

Environment context

The Grafana MCP is connected to a single Grafana instance scoped to one environment. If the user hasn't specified, ask which environment they want to investigate:

prod-us — US production (us-east-1)
prod-eu — EU production (eu-central-1)

Most ingestion app metrics are environment-specific by virtue of which Grafana you're connected to — they don't carry an environment label. CloudWatch metrics are also scoped to the connected Grafana's AWS account.

Cross-environment comparison requires switching Grafana instances (not possible in one session).

All datasource UIDs, dashboard UIDs, ingestion_pipeline/ingestion_lane label values, and Datastore type label values are identical across prod-us and prod-eu.

Key differences:

ingestion-analytics-turbo deployment exists only in prod-us.
Both envs use a dedicated ingestion MSK cluster separate from the events cluster — consumers point at msk-ingestion not events (prod-us: c21; prod-eu: insights-prod-eu-ingestion-2026-05-04).
CloudWatch cluster IDs differ by region suffix (see topology sections below).

Observability landscape

Six telemetry domains, all validated identical across prod-us and prod-eu:

Domain	Datasource UID	Discovery tool	Scope filter
App metrics (VictoriaMetrics)	`victoriametrics`	`list_prometheus_metric_names`	See metric prefixes below
App metrics (realtime)	`victoriametrics-realtime`	same	same (lower retention, higher resolution)
Logs	`P44D702D3E93867EC` (Loki-logs)	`list_loki_label_names`	`app=~"ingestion-.*"`
Profiling	`pyroscope`	`list_pyroscope_profile_types`	See Pyroscope services below
CloudWatch (ElastiCache, MSK, RDS)	`P034F075C744B399F`	`query_prometheus`	env-specific cluster IDs
Dashboards	n/a	`search_dashboards`	query `"ingestion"` or deployment name

Datasource notes:

Do NOT use primary Loki (P8E80F9AEF21F6940) — it returns 502 intermittently in both envs. Always use Loki-logs (P44D702D3E93867EC).
Do NOT use CloudWatch Root (PAAE47F430CFD1449) — it exists only in prod-us.

Stable waypoints

These facts change infrequently and are hard to discover dynamically.

Deployment roles

All PLUGIN_SERVER_MODE=ingestion-v2 deployments (the "analytics ingestion" family), plus specialized modes. Each golden-chart deployment runs in its own namespace matching the deployment name.

Deployment name	Mode	Pipeline	Lane	Consumer group	Consume topic (EU example)
`ingestion-analytics-main`	`ingestion-v2`	`analytics`	`main`	`ingestion-analytics-main`	`ingestion-analytics-main-512`
`ingestion-analytics-overflow`	`ingestion-v2`	`analytics`	`overflow`	`ingestion-analytics-overflow`	`ingestion-analytics-overflow-128`
`ingestion-analytics-historical`	`ingestion-v2`	`analytics`	`historical`	`ingestion-analytics-historical`	`ingestion-analytics-historical-128`
`ingestion-analytics-async`	`ingestion-v2`	`analytics`	`async`	`ingestion-analytics-async`	`ingestion-analytics-async-8`
`ingestion-analytics-turbo`	`ingestion-v2`	`analytics`	`turbo`	`ingestion-analytics-turbo`	`ingestion-analytics-turbo-1024`
`ingestion-clientwarnings-main`	`ingestion-v2`	`clientwarnings`	—	`ingestion-clientwarnings-main`	`ingestion-clientwarnings-main-32`
`ingestion-heatmaps-main`	`ingestion-v2`	`heatmaps`	—	`ingestion-heatmaps-main`	`ingestion-heatmaps-main-128`
`ingestion-errortracking-main`	`ingestion-errortracking`	`errortracking`	—	`ingestion-errortracking-main`	`ingestion-errortracking-main-128`
`ingestion-errortracking-overflow`	`ingestion-errortracking`	`errortracking`	—	`ingestion-errortracking-overflow`	`ingestion-errortracking-overflow-32`
`logs-ingestion`	`ingestion-logs`	—	—	`logs-ingestion`	`ingestion-logs`
`traces-ingestion`	(traces)	—	—	`traces-ingestion`	`ingestion-traces`
`recordings-blob-ingestion-v2`	`recordings-blob-ingestion-v2`	—	—	`session-recordings-blob-v2`	`ingestion-sessionreplay-main-256`
`recordings-blob-ingestion-v2-overflow`	`recordings-blob-ingestion-v2`	—	—	`session-recordings-blob-v2-overflow`	`ingestion-sessionreplay-overflow-32`

Notes:

ingestion-analytics-turbo exists only in prod-us.
Topic partition counts differ by env (e.g., main is 1024 in US, 512 in EU).
Each deployment also has a DLQ topic (ingestion-analytics-main-dlq, etc.).
KEDA autoscaling queries reference both old and new consumer group names during migration (e.g., groupId=~"ingestion-events|ingestion-analytics-main").

Metric prefixes

Every prefix here can be discovered live with list_prometheus_metric_names using datasourceUid: "victoriametrics" and regex: "<prefix>.*".

Prefix	Domain	Key scope labels
`ingestion_*`	Core ingestion app metrics (~80 metrics)	`app`, `ingestion_pipeline`, `ingestion_lane`
`ingestion_lag_ms*`	Per-partition lag (primary lag signal)	`groupId`, `partition`
`consumed_batch_*`	Kafka consumer batch processing	`topic`, `groupId`
`consumer_batch_` / `consumer_background_`	Consumer loop health	`topic`, `groupId`
`kafka_broker_*`	librdkafka broker stats	`broker_id`, `broker_name`, `consumer_group`
`kafka_consumer_*`	Consumer rebalance, assignment	`groupId`, `type`
`events_pipeline_*`	Legacy pipeline step metrics	`step_name`
`person_*`	Person processing (~30 metrics)	`db_write_mode`, `operation`, `method`
`group_*` (non-AWS)	Group processing	`operation`
`personinsights_*`	personinsights gRPC client + service	`method`, `source`, `client`
`overflow_redirect_*`	Stateful overflow routing	`type`, `result`, `decision`, `operation`
`cookieless_*`	Cookieless mode	—
`http_request_duration_seconds`	HTTP health/readiness server	`method`, `route`, `status_code`
`recording_blob_ingestion_v2_*`	Session replay ingestion	`app`
`logs_ingestion_*`	Logs ingestion pipeline	`app`
`error_tracking_` / `cymbal_`	Error tracking pipeline	`app`
`kminion_kafka_*`	KMinion consumer group lag & topic offsets	`group_id`, `topic_name`, `partition_id`
`aws_msk_kafka_*`	MSK broker-side JMX metrics	`environment`
`warpstream_agent_*`	WarpStream agent metrics (~10 metrics)	`virtual_cluster_id`, `agent_group`, `operation`
`kube_` / `container_`	K8s resources	`namespace=~"ingestion-."`, `container=~"ingestion-."`
`pg_` / `pgbouncer_`	Postgres exporter	varies
`DatastoreMetrics_` / `DatastoreProfileEvents_` / `DatastoreAsyncMetrics_*`	Datastore cluster health	`type` (=cluster role)
`kafka_connect_*`	Kafka Connect bridge to Datastore	`namespace`, `connector`
`insights_celery_datastore_*`	CH health monitors from Django celery	`scenario`

Redis topology

Ingestion workers depend on up to five Redis instances. Redis health is inferred from ingestion-side metrics and CloudWatch ElastiCache metrics.

Redis instance	ElastiCache cluster (prod-us)	Env var	Use
Ingestion Redis	`ingestion-prod-redis`	`INGESTION_REDIS_HOST`	Overflow state, pub/sub coordination
Insights/Primary Redis	`insights-solo`	`INSIGHTS_REDIS_HOST`	Billing/quota, restrictions, general
Cookieless Redis	`cookieless-prod-redis`	`COOKIELESS_REDIS_HOST`	Cookieless server hash mode
CDP Redis	`cdp-delivery-prod-redis`	`CDP_REDIS_HOST`	CDP Script function delivery
Dedup Redis	`ingestion-duplicates-prod-redis`	`DEDUPLICATION_REDIS_HOST`	Event deduplication

Ingestion-side Redis metrics: overflow_redirect_redis_*, cookieless_redis_error. Infrastructure-side: CloudWatch datasource P034F075C744B399F.

prod-eu uses the same logical cluster names but different endpoint suffixes. The prod-eu primary Redis is insights-prod-redis-encripted (sic — the typo is in the actual cluster name). The event restrictions Redis (ingestion-prod-redis) is a separate writable cluster from the primary — capture-analytics and ingestion workers both use it via EVENT_RESTRICTIONS_REDIS_URL.

Kafka topology

Ingestion workers interact with three Kafka systems via separate producer/consumer configs:

MSK ingestion cluster (consume side) — KAFKA_CONSUMER_METADATA_BROKER_LIST. Capture produces here; ingestion workers consume. Carries all ingestion-analytics-*, ingestion-errortracking-*, ingestion-heatmaps-*, ingestion-clientwarnings-* topics. Both envs have a dedicated cluster (prod-us: c21, 12 brokers; prod-eu: insights-prod-eu-ingestion-2026-05-04). KMinion: kminion-msk-ingestion.
WarpStream ingestion VC (output side) — KAFKA_WARPSTREAM_PRODUCER_METADATA_BROKER_LIST. Ingestion workers produce ALL Datastore-bound output here (datastore_events_json, datastore_person, datastore_groups, datastore_heatmap_events, datastore_ai_events_json, etc.). In-cluster WarpStream agents (warpstream-ingestion-v2) with multi-AZ pools; plaintext, no TLS. KMinion: kminion-warpstream-ingestion.
MSK ingestion cluster (feedback/DLQ producer) — same physical cluster as item 1, different producer config via KAFKA_INGESTION_PRODUCER_METADATA_BROKER_LIST. Used for overflow/DLQ/async topics that route events BACK to the ingestion system.

MSK (events/analytics) — the original events cluster. Still carries some legacy topics and Datastore consumer groups during migration. prod-us: insights-prod-us-events-2026-03-08 (12 brokers, kafka.m7g.8xlarge); prod-eu: insights-prod-eu-events-2025-10-16 (15 brokers). KMinion: kminion-msk-analytics.

WarpStream virtual clusters — each VC is a separate logical cluster backed by S3, with its own agent pool, KMinion instance, and topic namespace:

VC name	Topics carried	KMinion instance
`warpstream-ingestion-v2`	`datastore_events_json`, `datastore_person`, `datastore_person_distinct_id`, `datastore_groups`, `datastore_ai_events_json`, `datastore_heatmap_events`, `datastore_app_metrics2`, `datastore_topfn`, `datastore_ingestion_warnings`, `distinct_id_usage_events_json`, `log_entries`, `team_event_partitioned_events_json`	`kminion-warpstream-ingestion`
`warpstream-replay-v2`	`ingestion-sessionreplay-main-*`, `datastore_session_replay_events`, `datastore_session_replay_features`	`kminion-warpstream-replay`
`warpstream-logs`	`ingestion-logs`, `datastore_logs`	`kminion-warpstream-logs`
`warpstream-traces`	`ingestion-traces`, `datastore_traces`	`kminion-warpstream-traces`
`warpstream-shared`	`datastore_document_embeddings`, error tracking fingerprint/issue topics, `document_embeddings_input`	`kminion-warpstream-shared`
`warpstream-calculated-events`	`datastore_precalculated_person_properties`, `datastore_prefiltered_events`, `cohort_membership_changed` (US only)	`kminion-warpstream-calculated-events`
`warpstream-cyclotron`	CDP topics (`cdp_cyclotron_hog*`, `cdp_internal_events`, etc.)	`kminion-warpstream-cyclotron`
`warpstream-warehouse-pipelines`	`data_warehouse_source_webhooks`, `data_warehouse_sources_jobs` (US only)	`kminion-warpstream-warehouse-pipelines`

WarpStream agent metrics: warpstream_agent_* prefix (~10 metrics). Key: control_plane_operation_counter (by operation), file_cache_client_fetch_local_or_remote_counter (cache hit/miss). Labels: virtual_cluster_id, agent_group (default, general, multi-az). Dashboards: warpstream (Agent Overview), dbfj5c31spa1ogf (MSK vs WarpStream — Active Produce Topics). US-only personal dashboards (not synced to EU): 8e93b023-… (CH Consumer Lag), ws-coarse-lag-explore (Coarse Lag exploration).

Postgres topology

DB	Aurora cluster (prod-us)	Ingestion PgBouncer
Main app DB	`insights-cloud-prod-us-east-1` (2x `db.r8g.16xlarge`)	`ingestion-default-pgbouncer.insights.svc.cluster.local`
Persons DB	`insights-cloud-persons-prod-us-east-1` (3x `db.r8g.24xlarge`)	`ingestion-events-pgbouncer.insights.svc.cluster.local`

Postgres metrics via prometheus-postgres-exporter and prometheus-postgres-persons-exporter.

prod-eu: insights-cloud-prod-eu-central-1 and insights-cloud-persons-prod-eu-central-1.

Datastore topology

Datastore is the ultimate downstream consumer of events the ingestion pipeline produces. Ingestion workers never talk to CH directly — they publish to Kafka topics which CH consumes via its built-in Kafka engine and Kafka Connect (DuckLake). CH health directly impacts perceived ingestion quality: if CH falls behind on consumption, users see stale data.

Cluster roles (discovered via type label on DatastoreMetrics_*):

`type` label	Role	Notes
`events`	Main analytics events cluster	Consumes `datastore_events_json`
`online`	Online/fast queries cluster	Replicated from events
`offline`	Offline/batch queries cluster	Replicated from events
`medium`	Medium-sized tables	Persons, groups
`small`	Small/config tables	Infrequent writes
`sessions`	Session replay data	Consumes session recording topics
`logs`	Logs cluster	Consumes logs topics
`logs-new-schema`	Logs new schema migration	Migration target
`ai-events`	AI/LLM events	Consumes AI events topics
`endpoints`	API endpoints cluster	Lightweight
`migrations`	Migration-specific	Schema changes
`aux` / `ops`	Auxiliary/operations	Maintenance
`batch-exports`	Batch exports	prod-us has this; may not exist in prod-eu
`test`	Testing cluster	May not exist in all envs

Most type label values are identical across prod-us and prod-eu. Minor differences like batch-exports or test may exist only in one env.

Two consumption paths from Kafka to Datastore:

Datastore now reads primarily from the WarpStream ingestion VC (where ingestion workers produce their output), not directly from MSK.

Datastore Kafka Engine — native CH feature. Metrics prefixed DatastoreProfileEvents_Kafka* (e.g., KafkaMessagesPolled, KafkaRowsRead, KafkaRowsRejected, KafkaCommitFailures). Consumer groups: datastore_events_json (prod-us), group1 / group1_recent (prod-eu). These groups exist on BOTH MSK analytics and WarpStream ingestion — use the correct KMinion instance to distinguish: kminion-warpstream-ingestion for WarpStream, kminion-msk-analytics for MSK.
Kafka Connect — runs in kafka-connect namespace, uses DuckLake sink connector. Metrics prefixed kafka_connect_* and kafka_connect_ducklake_sink_task_metrics_*. Consumer groups: connect-events-ducklake*.

Key health signals for ingestion operators:

kminion_kafka_consumer_group_topic_lag{app_kubernetes_io_instance="kminion-warpstream-ingestion", group_id=~"datastore_events_json|group1|group1_recent", topic_name="datastore_events_json"} — lag between ingestion output and CH consumption on the WarpStream path (the primary path)
kminion_kafka_consumer_group_topic_lag_seconds with same group filter — same in seconds
DatastoreProfileEvents_KafkaRowsRejected — rows CH couldn't parse/insert
DatastoreProfileEvents_FailedInsertQuery — insert failures (schema issues, too many parts, etc.)
DatastoreAsyncMetrics_MaxPartCountForPartition — rising part count = merge pressure
DatastoreMetrics_ReadonlyReplica — replicas that fell behind and went read-only
DatastoreAsyncMetrics_ReplicasMaxAbsoluteDelay — max replication delay
insights_celery_datastore_table_parts_count / _table_row_count — Django-side CH health monitors

Pyroscope services

Both old (ingestion/{name}) and new ({namespace}/{name}) formats coexist in Pyroscope during the golden-chart migration. Prefer the new {namespace}/{name} format.

Service name (new format)	Deployment
`ingestion-analytics-main/ingestion-analytics-main`	Main analytics
`ingestion-analytics-overflow/ingestion-analytics-overflow`	Overflow lane
`ingestion-analytics-historical/ingestion-analytics-historical`	Historical lane
`ingestion-analytics-async/ingestion-analytics-async`	Async lane
`ingestion-analytics-turbo/ingestion-analytics-turbo`	Turbo lane (prod-us only)
`ingestion-heatmaps-main/ingestion-heatmaps-main`	Heatmaps
`ingestion-clientwarnings-main/ingestion-clientwarnings-main`	Client warnings
`ingestion-errortracking-main/ingestion-errortracking-main`	Error tracking
`ingestion-errortracking-overflow/ingestion-errortracking-overflow`	Error tracking overflow
`logs-ingestion/logs-ingestion`	Logs ingestion
`traces-ingestion/traces-ingestion`	Traces ingestion
`recordings/recordings-blob-ingestion-v2`	Session replay
`recordings/recordings-blob-ingestion-v2-overflow`	Session replay overflow

Profile types: process_cpu:cpu:nanoseconds:cpu:nanoseconds, wall:wall:nanoseconds:wall:nanoseconds, memory:inuse_space:bytes:inuse_space:bytes, memory:inuse_objects:count:inuse_space:bytes.

Grafana dashboards

UID	Title	Focus
`ingestion-analytics`	Ingestion - Analytics	Per-pipeline analytics breakdown
`ingestion-health`	Ingestion - Health	Health overview across all ingestion services
`ingestion-pipelines`	Ingestion - Pipelines	Per-lane pipeline step breakdown
`ingestion-reliability`	Ingestion - Reliability	Event trends, processing-lag SLOs, restarts, E2E lag
`ingestion-person-processing`	Ingestion -- Person Processing	Person store, merge, cache
`ingestion-group-processing`	Ingestion -- Group Processing	Group store
`ingestion-sessionreplay`	Ingestion - Session Replay	Replay blob pipeline
`capture`	Capture	Capture-side ingestion metrics (overview)
`bexr9fnja75z4f`	Kafka Deduplicator	Event deduplication pipeline
`pl-ingestion-slas`	Ingestion — SLIs / SLOs / SLAs	SLI/SLO view from `ingestion_sli_*` metrics (US only)
`warpstream`	Warpstream Agent Overview	Agent health, control plane ops, file cache
`dbfj5c31spa1ogf`	MSK vs Warpstream — Active Produce Topics	Side-by-side produce volume comparison
`8e93b023-a544-4a3b-8fac-123459d4eb84`	WarpStream: Datastore Consumer Lag	CH consumer lag on WarpStream topics (US only)
`ws-coarse-lag-explore`	WarpStream Coarse Lag — Explore	Agent-reported lag (US only, personal dashboard)
`personinsights-service`	personinsights service	personinsights latency decomposition
`dbfgkwxs3gw8owd`	KMinion Consumer Group Lag	Consumer lag by group (including CH groups)
`logs`	Logs (product)	Logs ingestion
`vm-datastore-cluster-overview`	Datastore (cluster overview)	QPS, memory, disk, replication, parts, merges
`datastore-ingestion-overview-20260615`	Datastore Ingestion Layer Overview	Ingestion-layer CH cluster health and resources
`datastore-keeper`	Datastore Keeper	ZooKeeper replacement health
`fe469e59-e10a-465a-9dac-ac8f6f82dc9a`	Datastore Stuck Merge Loop Investigation	Merges that stop making progress
`cdzv7o1635n9ca`	Kafka Connect	Kafka Connect tasks, lag, DuckLake sink
`fdrcd04np3hfkf`	Datastore - Kafka consumption	CH Kafka engine consumption stats (EU only)

Discovery workflows

Prometheus / VictoriaMetrics

list_prometheus_metric_names — datasourceUid: "victoriametrics", regex: "ingestion_.*" to enumerate app metrics
Pick a metric, then list_prometheus_label_names scoped to it — see available dimensions
list_prometheus_label_values — discover actual values for a label (e.g. labelName: "cause" on ingestion_event_dropped_total)
query_prometheus with PromQL — always scope by app or ingestion_pipeline and set a time range

Repeat with other prefixes: consumed_batch_*, person_*, personinsights_*, overflow_redirect_*, DatastoreMetrics_*, kafka_connect_*, etc.

Loki (logs)

list_loki_label_names — datasourceUid: "P44D702D3E93867EC"
list_loki_label_values for app — find ingestion containers (values like ingestion-analytics-main, logs-ingestion, etc.)
query_loki_logs — e.g. {app=~"ingestion-.*"} |= "error" or {namespace="datastore"} |= "Exception"

Pyroscope (profiling)

list_pyroscope_profile_types — data_source_uid: "pyroscope"
fetch_pyroscope_profile — matchers: '{service_name="ingestion-analytics-main/ingestion-analytics-main"}', profile_type: "process_cpu:cpu:nanoseconds:cpu:nanoseconds"

Dashboards

search_dashboards — query "ingestion" or "datastore" or a specific deployment name
get_dashboard_by_uid — use a known UID (e.g. "ingestion-health") to get panel details
get_dashboard_panel_queries — extract PromQL from existing panels

Redis / ElastiCache

Ingestion-side: discover overflow_redirect_redis_*, cookieless_redis_* metrics in VictoriaMetrics
Infrastructure: CloudWatch datasource P034F075C744B399F for ElastiCache (CPU, memory, connections, latency). Cluster IDs: ingestion-prod-redis, insights-solo (prod-us primary), ingestion-duplicates-prod-redis, cookieless-prod-redis

Postgres / Aurora

Ingestion-side: discover postgres_error_total, person_*, group_* metrics
Infrastructure: prometheus-postgres-exporter and prometheus-postgres-persons-exporter metrics
CloudWatch RDS: datasource P034F075C744B399F with cluster IDs insights-cloud-prod-us-east-1 / insights-cloud-persons-prod-us-east-1

Datastore

Cluster health: list_prometheus_metric_names with regex "DatastoreMetrics_.*" or "DatastoreProfileEvents_.*" Scope with type label for cluster role (e.g. type="events")
Kafka engine health: regex "DatastoreProfileEvents_Kafka.*"
Kafka Connect: regex "kafka_connect_.*" — scope with namespace="kafka-connect"
Consumer lag (the bridge): kminion_kafka_consumer_group_topic_lag with group_id=~"datastore_events_json|group1|group1_recent|connect-events-ducklake.*" and topic_name="datastore_events_json". Important: scope with app_kubernetes_io_instance="kminion-warpstream-ingestion" for the WarpStream path (primary) or "kminion-msk-analytics" for the MSK path.
Logs: {namespace="datastore"} |= "Exception" or {namespace="kafka-connect"}
Dashboards: vm-datastore-cluster-overview, datastore-ingestion-overview-20260615, cdzv7o1635n9ca, 8e93b023-a544-4a3b-8fac-123459d4eb84 (WarpStream CH consumer lag)

Key metric domains

Categories of what to look for. Discover specific metrics live using the prefixes above.

Kafka consumer health — batch duration, messages consumed per batch, consumer group assignment/rebalance events, consumer lag (via KMinion). Metrics: consumed_batch_*, kafka_consumer_*, kminion_kafka_consumer_group_topic_lag* scoped by group_id.

Pipeline processing — step-level latency and error rates, pipeline result distribution (ingested, filtered, dropped, DLQ'd). Metrics: events_pipeline_step_ms, events_pipeline_step_error_total, ingestion_pipeline_results by result.

Person/group stores — person flush latency, cache hit rates, Postgres write latency, merge failures, properties size. Metrics: person_*, group_*, personinsights_*.

Outputs — Kafka production to Datastore-bound topics. Message size, latency, errors. Metrics: ingestion_outputs_* by topic.

Overflow routing — stateful overflow decisions, Redis operations for overflow state. Metrics: overflow_redirect_* by type, result, decision.

Datastore downstream health — CH cluster QPS, memory, disk, merge pressure, replication lag, Kafka engine consumption (rows read/rejected/failed), Kafka Connect task health and consumer lag. This tells you whether events are actually making it to the query layer. Metrics: DatastoreMetrics_*, DatastoreProfileEvents_*, DatastoreAsyncMetrics_* scoped by type; kafka_connect_* scoped by namespace.

K8s resources — container_* and kube_* for CPU, memory, restarts, HPA state. Scope: namespace=~"ingestion-.*", pod=~"ingestion-.*" (or namespace="datastore", pod=~"chi-ingestion-.*" for CH ingestion pods).

Investigation playbooks

See references/investigation-playbooks.md for step-by-step workflows covering: health checks, event drops, latency, consumer lag, person processing (including the pganalyze query-level view of the persons DB), Kafka/MSK issues, Redis, Postgres, session replay, Datastore downstream health, single-partition lag (keyed-cost diagnosis, handing off to the querying-topfn skill for actor identification), and cross-environment comparison.

36 KiB Raw Permalink Blame History