insights/.agents/skills/adding-product-alerting/references/architecture.md

Shared alerting architecture

Use this reference to decide where code belongs before editing it.

Layer map

Layer	Location	Owns
Pure lifecycle decisions	products/alerts/backend/state_machine.py	State transitions, policy decisions, and notification actions
Shared alert infrastructure	products/alerts/backend/	Scheduling math, destination configuration and persistence, internal-event delivery, email transport, insight alert models/API, and insight evaluation
Product adapter	products/<name>/backend/	Domain evaluation, model snapshots, the single mutator, event payloads, allowed destinations, due queries, history, and orchestration
Shared alert creation UI	frontend/src/lib/components/Alerting/AlertWizard/	Reusable InsightsFunction destination, trigger, and configuration flow
Shared product alert UI	products/alerts/frontend/components/	Container-agnostic editor layout, definition primitives, advanced options, destination editor, schedule presentation, and evaluation chart
Product UI	products/<name>/frontend/ or frontend/src/scenes/<name>/	Form logic, API calls, product fields, normalized adapters, entry points, detail tables, and wizard configuration

products/logs is the reference adopter for fixed-cadence scheduling, InsightsFunction destinations, delivery rollback, product-owned Temporal orchestration, and the shared product alert editor components.

Logs and insight alerts both adapt their product state to the shared lifecycle engine. Insight alerts are also the reference adopter for shared calendar anchors, schedule restrictions, weekend skipping, and email delivery. Their model, API, query evaluation, and Django scheduling adapters live in products/alerts/backend/ and insights/tasks/alerts/. The evaluation package is shared across insight query kinds, but it is not a generic evaluator for unrelated products.

Frontend contract

Shared product alert UI lives in products/alerts/frontend/components/. It is presentation infrastructure, not a frontend product registry.

• AlertEditor, AlertEditorFormDetails, and AlertEditorSection provide the container-agnostic form shell.
• AlertDefinition* components provide composable definition, schedule, next-evaluation, and timezone presentation.
• AlertAdvancedOptions owns shared collapse and enabled-count behavior.
• AlertNotificationDestinationEditor renders normalized saved and pending destinations.
• AlertEvaluationHistoryChart renders normalized evaluation points and current thresholds.

Products own keyed kea logic, API calls, form schemas, source/filter controls, supported destination types, InsightsFunction payloads, threshold conversion, enabled-count calculation, and history tables. Keep modals, scene widths, and embedded-section sizing in the product surface.

AlertWizard remains the shared InsightsFunction creation flow. Adopters provide keyed alertWizardLogic props with supported sub-template IDs, WizardTrigger[], WizardDestination[], and optional URL or preset behavior. Backend destination support alone does not make an option available in the wizard.

Read frontend-alerting.md before adding or extending a product alert frontend. Follow frontend/src/AGENTS.md, use generated API types, and guard network submissions from double clicks.

Lifecycle contract

products/alerts/backend/state_machine.py is pure Python.

• CheckInput normalizes one product evaluation.
• AlertSnapshot contains only fields needed for lifecycle decisions.
• AlertPolicy represents deliberate differences between adopters. Its defaults are existing behavior, not a menu of speculative options.
• evaluate_alert_check(...) and evaluate_alert_failure(...) return AlertCheckOutcome without persistence.
• Control-plane helpers such as apply_enable, apply_disable, apply_snooze, and apply_threshold_change return ControlPlaneOutcome.
• NotificationAction tells the product which event to deliver.

The product converts its model to a snapshot and applies the outcome through one local apply_outcome function. Extend .semgrep/rules/security/alert-state-must-go-through-state-machine.yaml when another product adopts this contract.

Error behavior is load-bearing:

• Failed checks do not clear an already firing alert.
• Inconclusive checks preserve state and failure count.
• Transient errors are silent unless the policy opts into counting them.
• Error notification normally occurs on the first failure edge.
• Delivery failure must not consume a lifecycle or failure-counter edge that should be retried.

Destination contract

Product-facing destination setup is exported from products.alerts.backend.facade.api:

• validate_destination_data
• build_alert_destination_config
• create_alert_destination_insights_functions
• soft_delete_alert_destinations
• soft_delete_all_alert_destinations
• send_alert_email

EventKindSpec describes destination-neutral content for one event kind. The shared builder converts it into Slack, Discord, webhook, or Microsoft Teams InsightsFunction payloads. Products own event IDs, event properties, wording, actions, and their allowed destination list.

Deletion is fail-closed. Always scope it with team_id, alert_id, and the product's allowed event IDs.

Delivery contract

InsightsFunction notification workers use products.alerts.backend.destinations directly:

1. produce_alert_internal_event(...) returns a ProduceResult or None.
2. flush_alert_internal_events(...) flushes the shared producer. Batch workers should flush once per produced batch.
3. alert_internal_event_delivered(...) checks whether the producer acknowledged each internal event after the flush.
4. The product persists notification-dependent lifecycle changes only for acknowledged internal events.

This acknowledgement confirms production to the internal-event transport, not downstream InsightsFunction execution or final Slack, Discord, webhook, or Microsoft Teams delivery. The helpers log and capture producer failures. The product owns rollback, retry timing, schedule advancement, and check-history semantics.

Email callers use send_alert_email(...) through the facade. The caller owns recipients, authorization, subject, template, context, error handling, and a stable campaign_key for the required retry and deduplication behavior.

Scheduling contract

products/alerts/backend/scheduling.py is pure Python and owns reusable scheduling math:

• compute_shard_offset_seconds(...) deterministically assigns a UUID-keyed alert to scheduler ticks.
• advance_next_check_at(...) advances from the prior schedule, skips missed intervals, snaps to the midnight-anchored cadence grid, and applies the shard offset.
• CalendarInterval and to_calendar_interval(...) define the product-facing interval contract.
• next_calendar_check_time(...) computes fixed-cadence or team-local daily, weekly, and monthly anchors while preserving wall-clock behavior across DST changes.
• validate_and_normalize_schedule_restriction(...) and parse_blocked_windows_tuples(...) validate product payloads and produce the pure blocked-window contract.
• scan_next_unblocked_utc(...), is_utc_datetime_blocked(...), and is_weekend(...) apply timezone-aware restrictions without importing Django models.

Use the scheduler's real interval when computing shards. Use the same shard function when creating, updating, and advancing an alert.

Products translate their enums and stored payloads into the shared contract. Due eligibility, schedule persistence, retry behavior, and orchestration stay product-specific because model fields, states, snooze behavior, and tenant constraints differ.