hanzoai/insights
2
0
Fork 0
mirror of https://github.com/hanzoai/insights.git synced 2026-11-03 00:52:32 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
main
insights/.depot/workflows/ci-backend.yml
T
zeekay 168c1136b4 Ship the funnel UDF config in the image 
The funnel queries call the unversioned UDFs (aggregate_funnel_array and
kin), declared only in docker/datastore/user_defined_function.xml, which
no image carries. The binaries already ship in insights/user_scripts, so
the config moves beside them: a datastore takes both from one image.
2026-09-30 07:34:07 -07:00

2156 lines
130 KiB
YAML
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Depot CI shadow of .github/workflows/ci-backend.yml. Lives under .depot/ so
# GitHub Actions ignores it; only Depot CI parses this file. Statuses posted
# from here are non-blocking; canonical is the authoritative signal.
#
# Apples-to-apples is the whole point — keep action pins, env vars, runner
# sizes, and pytest flags in sync with the canonical workflow. A drift-check
# job in .github/workflows/ci-backend-shadow-drift.yml fails when canonical
# changes without a matching depot bump.
#
# Intentional deltas vs canonical (everything else is kept apples-to-apples):
# - runs-on: depot-* labels instead of ubuntu-latest
# - action paths: ./.depot/actions/... instead of ./.github/actions/... (pnpm-install
# and setup-sqlx-cli are byte-equivalent mirrors of the canonical composites)
# - permissions: pull-requests:read (no comment/auto-commit side effects)
# - Stripped side effects: artifact uploads, CI report section posters (canonical
# additionally skips these on trunk-merge/** branches to save GITHUB_TOKEN budget;
# the shadow strips them entirely, so that gating has nothing to mirror), snapshot
# auto-commit (handle-snapshots), insights-github-action telemetry (including the
# capture-test-selection job — select-tests still emits its telemetry outputs to
# stay apples-to-apples, they just have no consumer here),
# --store-durations write, test-selection verdict, Trunk flaky-test quarantine
# gate (canonical runs per-shard analytics-uploader steps that post to the
# external Trunk service; the upload is best-effort and a local "Fail on test
# failure" step owns the pass/fail verdict. The shadow strips all of it — no
# Trunk posting from a non-blocking timing trial — and so also omits the test
# steps' continue-on-error and that verdict step, letting each shard's test
# exit code be the verdict directly. Canonical gates each quarantine gate on the
# TRUNK_UPLOAD_ENABLED repo variable as an upload kill-switch, and its verdict step
# on TRUNK_QUARANTINE_ENABLED to decide whether known flakes are masked; since the
# shadow strips both the gate and the verdict step entirely, both variables are
# no-ops here and there is nothing to mirror)
# - Per-test failure rollup: canonical's django_tests gate uploads product JUnit and, when a
# test job fails, downloads that shard's JUnit to list the actual failing tests. The shadow
# strips artifact uploads (above), so it has nothing to download and keeps the plain
# dependency-result table — depot-exempt for the same reason as the other uploads.
# - report-test-timings: canonical emits per-test OTLP spans from the uploaded JUnit. The shadow
# strips artifact uploads (above) and posts no telemetry, so the whole job is depot-exempt —
# changes confined to it (script deps, the reporter's availability guard, emission secret
# plumbing such as dual-project tokens, the JUnit-artifact download pattern, re-run attempt
# handling — including the attempt-suffixed junit artifact names canonical uploads so the
# reporter can pair re-run recoveries with attempt-1 failures) need no depot bump.
# - OpenAPI types: the check-openapi-types job is folded into check-migrations as a
# check-only step (drift still fails the run); only its auto-commit side effect is dropped
# - events_json variants: the shadow never doubles the matrix (no RUN_NEW_EVENTS_SCHEMA
# plumbing), so canonical changes confined to that gating — e.g. skipping the
# events_json suite on master — need no depot bump
# - actions/cache reads/writes route to Depot Cache, not GitHub Actions Cache,
# so shared keys with canonical (schema, uv, pnpm) do not collide
# - Three data-modeling tests deselected (Depot-only MinIO 403 quarantine)
# - GITHUB_TOKEN / OTEL_SERVICE_NAME neutralized (ambient on Depot, not on GHA)
# - Docker Hub login reads Depot's secret store (secrets.DOCKERHUB_*) rather than
# canonical's env.DOCKERHUB_USERNAME / vars.DOCKERHUB_USER — depot never indirects
# through env:, so canonical's per-job env-scoping (workflow-level -> job-level,
# for the gha-workflow-env-secret Semgrep fix) is depot-exempt
# - Service launch via `nohup ... --down` (survives Depot's step-shell teardown) and
# waits on a superset (objectstorage, Temporal) with an explicit 900s timeout
# - Path-filter persons-SQL folded into migrations. check-migrations.if also gates
# on openapi_types so an OpenAPI-only PR still runs the folded check
# - Snapshot mode matches canonical, but snapshot patch upload/commit is stripped
# - Test durations restored from Depot Cache (insights-test-durations- prefix), refreshed by
# .depot/workflows/ci-backend-update-test-timing.yml, so shard counts match GitHub Actions.
# Committed .test_durations is the fallback.
# - Sharding-plan pinning: canonical snapshots the durations files as a run-scoped
# artifact and every test job applies it, so re-runs shard like attempt 1. The shadow
# strips artifact uploads (above), so it keeps floating restores; no depot bump needed
# for changes confined to that pinning.
# - master (push) runs only check-migrations to prime the shared schema cache; the
# test matrices are skipped on master (canonical runs them). PRs and dispatches
# run the full graph — that is where the apples-to-apples comparison happens
# - check-migrations timeout widened to 60m because it also runs the folded OpenAPI generation
# This workflow runs all of our backend django tests.
#
# Shards derive from .test_durations (Amdahl), restored from Depot Cache (else committed);
# .depot/workflows/ci-backend-update-test-timing.yml keeps that cache fresh.
name: '[optional, does not block merge] Backend CI'
# OPTIONAL, non-blocking shadow of canonical Backend CI — its statuses must never gate
# merging (hence the "[optional, does not block merge]" name prefix). The
# CI_DEPOT_SHADOW_PERCENT Depot CI variable (0-100; 0 or unset = fully off) is both the
# master switch and the pull_request sample rate: when >0, pull_request runs are sampled at
# that percent; master (push) runs only check-migrations to prime the schema cache (test
# matrices are skipped there); workflow_dispatch always runs. The `sample` job rolls a deterministic
# per-run dice; `changes` (root of the needs graph) only runs when sampled in, so the
# whole workflow cascades off on a miss. This is a Depot variable, NOT a GitHub one —
# Depot CI reads its own `vars` store. Set it with:
# depot ci vars add CI_DEPOT_SHADOW_PERCENT=<n> --repo Insights/insights (n=0 disables)
# The drift check (.github/workflows/ci-backend-shadow-drift.yml) keeps this file in sync
# with canonical; it runs whenever either file changes, independent of this variable.
on:
push:
branches:
- master
workflow_dispatch:
inputs:
datastoreServerVersion:
description: Datastore server version. Leave blank for default
type: string
pull_request:
# Draft PRs run the snob-selected Django subset (≤3 shards); turbo-tests
# still skip drafts. Ready PRs run the full matrices — the merge gate —
# and ready_for_review re-triggers them when a PR leaves draft. Add the
# `run-ci-backend` label to force the full matrices on a draft; the
# `no-ci` label silences the workflow on a draft entirely. No
# labeled/unlabeled triggers, so a label takes effect from the next push
# or from ready_for_review. Mirrors canonical.
types: [opened, synchronize, reopened, ready_for_review]
concurrency:
# PRs: one active run per branch, cancel stale. Push: per-SHA so master
# pushes never cancel each other (check-migrations always completes).
group: ${{ github.workflow }}-${{ github.event_name == 'push' && github.sha || github.head_ref || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
env:
# Schema cache epoch. Bump to abandon every shared schema cache entry at once
# (key is insights-schema-mig-<epoch>-<hash>). Single-sourced so the merge-base (PR)
# and HEAD (push) key computations below can't drift.
# ci-e2e-playwright.yml, ci-dagster.yml, and ci-mcp.yml restore by the same key; keep their copies in sync when bumping.
SCHEMA_CACHE_EPOCH: v2
SECRET_KEY: '6b01eee4f945ca25045b5aab440b953461faf08693a9abbf1166dc7c6b9772da' # unsafe - for testing only
DATABASE_URL: 'postgres://insights:insights@localhost:5432/insights'
REDIS_URL: 'redis://localhost'
DATASTORE_HOST: 'localhost'
DATASTORE_SECURE: 'False'
DATASTORE_VERIFY: 'False'
# Restricted `autoresearch` user defined in docker/datastore/users-dev.xml,
# pointed at the CI test DB (insights_test, not `default` as in regular dev).
DATASTORE_TEST_CLUSTER_HOST: 'localhost'
DATASTORE_TEST_CLUSTER_DATABASE: 'insights_test'
DATASTORE_TEST_CLUSTER_USER: 'autoresearch'
DATASTORE_TEST_CLUSTER_PASSWORD: 'autoresearchpass'
DATASTORE_TEST_CLUSTER_SECURE: 'False'
DATASTORE_TEST_CLUSTER_VERIFY: 'False'
TEST: 1
# Depot CI runners expose OTEL_SERVICE_NAME=ci-worker ambient; GitHub Actions test steps do not.
# Left set, it leaks into settings.OTEL_SERVICE_NAME (personinsights metric client_name label) and
# diverges from test assertions. Blank it to match the GHA environment. (GITHUB_TOKEN is also
# injected ambient and leaks into plugin download auth headers, but GITHUB_-prefixed vars are
# reserved and cannot be overridden via env: — it is unset inside the test run scripts instead.)
OTEL_SERVICE_NAME: ''
DATASTORE_SERVER_IMAGE_VERSION: ${{ github.event.inputs.datastoreServerVersion || '' }}
DATASTORE_COMPAT_PYTEST_TARGETS: 'insights/datastore ee/datastore'
OBJECT_STORAGE_ENABLED: 'True'
OBJECT_STORAGE_ENDPOINT: 'http://localhost:19000'
OBJECT_STORAGE_ACCESS_KEY_ID: 'object_storage_root_user'
OBJECT_STORAGE_SECRET_ACCESS_KEY: 'object_storage_root_password'
UV_HTTP_TIMEOUT: 120
# tests would intermittently fail in GH actions
# with exit code 134 _after passing_ all tests
# this appears to fix it
# absolute wild tbh https://stackoverflow.com/a/75503402
DISPLAY: ':99.0'
# this is a fake key so this workflow can run for external contributors as they do not have access to secrets (that we don't need here)
OIDC_RSA_PRIVATE_KEY: ${{ vars.OIDC_RSA_FAKE_PRIVATE_KEY }}
RUNS_ON_INTERNAL_PR: ${{ github.event_name != 'pull_request' || github.event.pull_request.head.repo.fork == false }}
SANDBOX_JWT_PRIVATE_KEY: ${{ vars.OIDC_RSA_FAKE_PRIVATE_KEY }}
permissions:
contents: read
pull-requests: read
jobs:
# Sampling gate (full design in the file header). The `if` keeps the whole workflow
# off when the percent is 0/unset; when >0 this job computes `sampled`, and `changes`
# — hence the rest of the graph — only runs when it is true.
sample:
if: vars.CI_DEPOT_SHADOW_PERCENT != '' && vars.CI_DEPOT_SHADOW_PERCENT != '0'
runs-on: depot-ubuntu-24.04
timeout-minutes: 5
name: '[optional, does not block merge] Sample run'
outputs:
sampled: ${{ steps.roll.outputs.sampled }}
steps:
- name: Roll sampling dice
id: roll
env:
PERCENT: ${{ vars.CI_DEPOT_SHADOW_PERCENT }}
RUN_ID: ${{ github.run_id }}
EVENT: ${{ github.event_name }}
run: |
set -euo pipefail
# Only pull_request is sampled; push and workflow_dispatch always run
# when enabled (sampling them would skip cache priming or no-op a dispatch).
if [ "$EVENT" != "pull_request" ]; then
echo "sampled=true" >> "$GITHUB_OUTPUT"
echo "::notice::Optional Backend CI: $EVENT always runs when enabled."
exit 0
fi
# Deterministic per-run dice. Non-integer percent → 0 (off).
case "$PERCENT" in ''|*[!0-9]*) PERCENT=0 ;; esac
bucket=$(( RUN_ID % 100 ))
if [ "$bucket" -lt "$PERCENT" ]; then
echo "sampled=true" >> "$GITHUB_OUTPUT"
echo "::notice::Optional Backend CI: PR sampled IN (bucket $bucket < ${PERCENT}%)."
else
echo "sampled=false" >> "$GITHUB_OUTPUT"
echo "::notice::Optional Backend CI: PR sampled OUT (bucket $bucket >= ${PERCENT}%) — skipped."
fi
# Job to decide if we should run backend ci
# See .github/actions/paths-filter/README.md for filter semantics
changes:
# Root of the `needs` graph: skipped when `sample` says skip (or is itself skipped
# at percent 0/unset), which cascades the whole workflow off. Also skipped on
# draft PRs labeled `no-ci`, mirroring canonical.
needs: sample
if: >-
needs.sample.outputs.sampled == 'true'
&& (github.event_name != 'pull_request'
|| github.event.pull_request.draft != true
|| startsWith(github.head_ref, 'trunk-merge/')
|| !contains(github.event.pull_request.labels.*.name, 'no-ci'))
runs-on: depot-ubuntu-24.04
timeout-minutes: 5
name: Determine need to run backend and migration checks
# Set job outputs to values from filter step
outputs:
backend: ${{ steps.filter.outputs.backend || 'true' }}
backend_files: ${{ steps.filter.outputs.backend_files }}
migrations: ${{ steps.filter.outputs.migrations || 'true' }}
migrations_files: ${{ steps.filter.outputs.migrations_files }}
migrations_any: ${{ steps.filter.outputs.migrations_any || 'true' }}
tasks_temporal: ${{ steps.filter.outputs.tasks_temporal || 'true' }}
openapi_types: ${{ steps.filter.outputs.openapi_types || 'true' }}
legacy: ${{ steps.filter.outputs.legacy || 'true' }}
schema: ${{ steps.filter.outputs.schema || 'true' }}
product_yamls: ${{ steps.filter.outputs.product_yamls || 'false' }}
product_yamls_files: ${{ steps.filter.outputs.product_yamls_files }}
steps:
# For pull requests it's not necessary to checkout the code, but we
# also want this to run on master so we need to checkout
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
clean: false
- name: Clean up data directories with container permissions
run: |
# Use docker to clean up files created by containers
[ -d "data" ] && docker run --rm -v "$(pwd)/data:/data" alpine sh -c "rm -rf /data/seaweedfs /data/minio" || true
continue-on-error: true
- uses: ./.github/actions/paths-filter
id: filter
if: github.event_name != 'push' # Run all tests on master push
with:
list-files: 'escape'
filters: |
backend:
# Avoid running backend tests for irrelevant changes
# NOTE: we are at risk of missing a dependency here. We could make
# the dependencies more clear if we separated the backend/frontend
# code completely
- 'ee/**/*'
- '!ee/frontend/**' # frontend code under ee/ must not trigger backend
- 'common/__init__.py'
- 'common/insightsql_parser/**'
- 'common/scriptvm/**'
- 'common/ingestion/**'
- 'common/migration_utils/**'
- 'common/plugin_transpiler/**'
- 'insights/**/*'
- 'products/**/backend/**/*'
# Python outside backend/ — products like logs and
# insights_ai point `backend:test` at extra roots
# (skills/, scripts/). Catch any .py change so the
# backend job triggers when only those files move.
- 'products/**/*.py'
# products/desktop is the imported desktop app; it has its own
# desktop-* CI and must not drag this suite onto its PRs
- '!products/desktop/**'
# Each product's package.json declares the per-product
# backend:test script that turbo-discover reads to
# compose the Product matrix. Without this entry,
# widening or fixing the script doesn't trigger
# tests it controls.
- 'products/*/package.json'
- bin/build-schema-latest-versions.py
- bin/build-taxonomy-json.py
- bin/check_uv_python_compatibility.py
- bin/find_python_dependencies.py
- bin/unit_metrics.py
- pyproject.toml
- uv.lock
- requirements.txt
- requirements-dev.txt
- mypy.ini
- pytest.ini
- conftest.py # Root conftest loads for every pytest run
- .test_durations # Used for pytest-split sharding
- .test_quarantine.json # Un-quarantining must re-run the affected tests
# turbo-discover reads the module graph for the dependent
# cascade, and `tach check` validates it — a graph-only PR
# must still dispatch both.
- tach.toml
# generates insights/schema.py
- frontend/src/queries/schema.json
# Operator parity check
- rust/feature-flags/src/properties/property_models.rs
- frontend/src/products.json # Loaded at runtime by insights/products.py
- 'products/*/manifest.tsx' # Generates products.json
# insightscli setup commands this workflow runs: db:restore-test-db
# and product:bootstrap/lint. Scoped to the modules those
# commands actually import — framework, manifest, boot
# modules, db_schema, product — so unrelated insightscli commands
# (doctor, devbox, ...) don't trigger this suite.
- 'tools/insightscli/**'
- insightscli.yaml
- 'tools/insightscli-commands/insightscli_commands/db_schema.py'
- 'tools/insightscli-commands/insightscli_commands/prechecks.py'
- 'tools/insightscli-commands/insightscli_commands/telemetry_props.py'
- 'tools/insightscli-commands/insightscli_commands/hint_hook.py'
- 'tools/insightscli-commands/insightscli_commands/hints.py'
- 'tools/insightscli-commands/insightscli_commands/product/**'
- 'tools/insightscli-commands/insightscli_commands/product_structure.yaml'
# Make sure we run if someone is explicitly changing the workflow
- .depot/workflows/ci-backend.yml
- .github/datastore-versions.json
# We use docker compose for tests, make sure we rerun on
# changes to docker-compose.dev.yml e.g. dependency
# version changes
- docker-compose.dev.yml
- docker-compose.profiles.yml
- docker-compose.base.yml
- bin/wait-for-docker
- bin/ci-wait-for-docker
- frontend/public/email/*
- 'docker/datastore/**'
# Markdown-only changes (e.g. a README under insights/) match the
# broad globs above but exercise no backend code — exclude them so
# a docs-only PR doesn't run the whole backend suite.
# These excludes are load-bearing: an over-broad exclude silently
# skips the suite rather than over-running it, so only exclude
# patterns that truly never affect backend. They also drop matched
# files from `backend_files`.
- '!**/*.md'
- '!**/*.mdx'
legacy:
# Non-product backend code — when only products/ change,
# turbo-discover uses Turbo query affectedness to detect changed
# products and decide whether Django runs.
# Everything from backend: EXCEPT products/**/backend/**/*
- 'ee/**/*'
- '!ee/frontend/**' # frontend code under ee/ must not trigger backend
- 'common/__init__.py'
- 'common/insightsql_parser/**'
- 'common/scriptvm/**'
- 'common/ingestion/**'
- 'common/migration_utils/**'
- 'common/plugin_transpiler/**'
- 'insights/**/*'
- bin/build-schema-latest-versions.py
- bin/build-taxonomy-json.py
- bin/check_uv_python_compatibility.py
- bin/find_python_dependencies.py
- bin/unit_metrics.py
- pyproject.toml
- uv.lock
- requirements.txt
- requirements-dev.txt
- mypy.ini
- pytest.ini
- conftest.py
- .test_durations
- .test_quarantine.json
- frontend/src/products.json
- 'products/*/manifest.tsx'
- rust/feature-flags/src/properties/property_models.rs
- .depot/workflows/ci-backend.yml
- .github/datastore-versions.json
- docker-compose.dev.yml
- docker-compose.profiles.yml
- docker-compose.base.yml
- bin/wait-for-docker
- bin/ci-wait-for-docker
- frontend/public/email/*
- 'docker/datastore/**'
# Same rationale as the backend filter's excludes above: docs-only
# changes match the broad globs but exercise no legacy backend code,
# so they shouldn't trigger the Django/turbo-discover path either.
- '!**/*.md'
- '!**/*.mdx'
schema:
# Tracked separately from `legacy` so turbo-discover can
# diff schema.json and narrow the product matrix to
# products that import the changed types.
- frontend/src/queries/schema.json
migrations:
- 'docker/datastore/**'
- 'insights/migrations/*.py'
- 'products/*/backend/migrations/*.py'
- 'products/*/migrations/*.py' # Legacy structure
# persons-SQL folded into migrations here (see header deltas):
# depot strips canonical's separate persons-migration job
- 'rust/persons_migrations/*.sql'
- 'rust/bin/migrate-persons'
# Any migration file in any app (broader than `migrations` above,
# which omits ee/ and insights sub-package dirs). Gates the
# deleted-migration check in repo-checks.
migrations_any:
- '**/migrations/*.py'
tasks_temporal:
- 'products/tasks/backend/temporal/**/*'
product_yamls:
- 'products/*/product.yaml'
- '**/owners.yaml'
# Resolver changes must re-run the owners.yaml lint too.
- 'tools/owners/**'
openapi_types:
# Generated OpenAPI types - validate they match schema
- 'frontend/src/generated/**/*'
- 'frontend/src/lib/agentScopes.generated.ts'
- 'products/*/frontend/generated/**/*'
- 'services/mcp/src/generated/**/*'
- 'services/mcp/src/api/generated.ts'
# Generation tooling - changes here could affect output
- 'tools/openapi-codegen/**/*'
- 'services/mcp/scripts/lib/**'
- 'frontend/bin/generate-openapi-types.mjs'
- 'frontend/src/lib/api-orval-mutator.ts'
- 'services/mcp/scripts/**/*'
# MCP tool config sources - feed generate-tools.ts
- 'services/mcp/definitions/**/*.yaml'
- 'services/mcp/definitions/prompts/**/*.md'
- 'products/*/mcp/**/*.yaml'
- 'products/*/mcp/prompts/**/*.md'
- 'services/mcp/schema/tool-definitions.json'
- 'services/mcp/schema/tool-definitions-v2.json'
- 'services/mcp/src/tools/generated/**/*'
- 'services/mcp/schema/generated-tool-definitions.json'
- 'services/mcp/schema/tool-definitions-all.json'
detect-snapshot-mode:
name: Detect snapshot mode
runs-on: depot-ubuntu-24.04
timeout-minutes: 5
needs: [changes]
if: needs.changes.outputs.backend == 'true'
outputs:
mode: ${{ steps.detect.outputs.mode }}
steps:
- name: Detect mode
id: detect
env:
PR_REPO: ${{ github.event.pull_request.head.repo.full_name }}
REPO: ${{ github.repository }}
HAS_NO_SNAPSHOT_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'no-snapshot-update') }}
AUTHOR: ${{ github.actor }}
run: |
if [ "$PR_REPO" != "$REPO" ] && [ -n "$PR_REPO" ]; then
echo "mode=check" >> "$GITHUB_OUTPUT"
echo "Fork detected - running in check mode"
elif [ "$HAS_NO_SNAPSHOT_LABEL" == "true" ]; then
echo "mode=check" >> "$GITHUB_OUTPUT"
echo "No-snapshot label detected - running in check mode"
else
echo "Workflow triggered by: $AUTHOR"
if [[ "$AUTHOR" != "dependabot[bot]" ]] && \
{ [[ "$AUTHOR" == *"github-actions"* ]] || [[ "$AUTHOR" == *"[bot]"* ]] || [[ "$AUTHOR" == "insights-bot" ]]; }; then
echo "mode=check" >> "$GITHUB_OUTPUT"
echo "Bot actor detected - running in check mode"
else
echo "mode=update" >> "$GITHUB_OUTPUT"
echo "Running in update mode"
fi
fi
# Fast pre-job: determines which products need testing and if Django should run
# Only needs pnpm + node — no Python, Docker, or services
# Runs on depot for turbo remote cache access (cache status drives Django gating)
turbo-discover:
needs: changes
# Not on master push: master only primes the schema cache; this cascades the matrices off.
if: needs.changes.outputs.backend == 'true' && github.event_name != 'push'
runs-on: depot-ubuntu-24.04
timeout-minutes: 20
name: Discover product tests
outputs:
run_legacy: ${{ steps.discover.outputs.run_legacy }}
matrix: ${{ steps.discover.outputs.matrix }}
schema_cache_key: ${{ steps.schema-key.outputs.key }}
schema_migrations_key: ${{ steps.schema-key.outputs.migrations_key }}
django_shards: ${{ steps.discover.outputs.django_shards }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
fetch-depth: 1000
filter: blob:none
- name: Fetch current PR base for Turbo affected diff
if: github.event_name == 'pull_request'
env:
BASE_REF: ${{ github.event.pull_request.base.ref }}
run: git fetch --no-tags --depth=1000 --filter=blob:none origin "$BASE_REF:refs/remotes/origin/$BASE_REF"
- name: Compute schema cache key from merge-base
id: schema-key
if: github.event_name == 'pull_request'
env:
BASE_REF: ${{ github.event.pull_request.base.ref }}
run: |
# HEAD is the synthetic merge commit; HEAD^2 is the PR branch tip.
# The fetch-depth:1000 checkout + base-ref fetch above ensure the
# full ancestry needed to find the divergence point is available.
MERGE_BASE=$(git merge-base HEAD^2 "origin/${BASE_REF}" 2>/dev/null || echo "")
if [ -z "$MERGE_BASE" ]; then
echo "key=" >> $GITHUB_OUTPUT
echo "migrations_key=" >> $GITHUB_OUTPUT
echo "::notice::merge-base not found (branch too stale?) — schema cache will be skipped"
exit 0
fi
# Routing decides which app labels reach the dump, and the consumers replay them
# from the checked-out copy, so these inputs belong in the key below. A PR that
# edits any of them matches no master dump, the merge-base one included: drop
# both keys and let migrations run from scratch.
DB_ROUTING=$(git ls-tree -r --format='%(objectname) %(path)' "$MERGE_BASE" -- products/db_routing.yaml insights/product_db_config.py insights/product_db_router.py)
DB_ROUTING_HEAD=$(git ls-tree -r --format='%(objectname) %(path)' HEAD -- products/db_routing.yaml insights/product_db_config.py insights/product_db_router.py)
if [ "$DB_ROUTING" != "$DB_ROUTING_HEAD" ]; then
echo "key=" >> $GITHUB_OUTPUT
echo "migrations_key=" >> $GITHUB_OUTPUT
echo "::notice::db routing config or router differs from the merge base, so the schema cache is skipped"
exit 0
fi
# Exact branch-point key, kept as the fallback restore key below.
echo "key=insights-schema-master-${MERGE_BASE}" >> $GITHUB_OUTPUT
# Shared key: hash merge-base schema inputs, not PR head, so PR migrations still
# apply as the pytest delta. uv.lock covers third-party migration changes; the
# postgres image is folded in so a server-version bump invalidates the key.
# Non-Django migration frameworks are excluded from this Postgres schema dump.
# SCHEMA_CACHE_EPOCH (workflow env) prefixes the key as a manual cache-bust knob.
MIG_FILES=$(git ls-tree -r --format='%(objectname) %(path)' "$MERGE_BASE" \
| grep -E '/migrations/[^/]+\.py$|^[0-9a-f]+ uv\.lock$' \
| grep -vE '/(datastore|async_migrations)/migrations/' \
| LC_ALL=C sort) || true
PG_IMAGE=$(git show "${MERGE_BASE}:docker-compose.base.yml" 2>/dev/null \
| grep -m1 -E '^[[:space:]]*image:[[:space:]]*postgres:' | tr -d '[:space:]')
if [ -z "$MIG_FILES" ]; then
echo "migrations_key=" >> $GITHUB_OUTPUT
echo "::notice::no migration files matched — restore will fall back to the SHA schema key"
else
MIG_HASH=$(printf '%s\n%s\n%s' "$MIG_FILES" "$PG_IMAGE" "$DB_ROUTING" | sha256sum | cut -c1-40)
echo "migrations_key=insights-schema-mig-${SCHEMA_CACHE_EPOCH}-${MIG_HASH}" >> $GITHUB_OUTPUT
fi
# Shadow trial: canonical threads a setup-action app token into
# pnpm-install; mint stripped here (secret not available on Depot).
- name: Install pnpm dependencies
uses: ./.depot/actions/pnpm-install
with:
install-command: pnpm install --frozen-lockfile --filter=@hanzo/root
- name: Restore test durations from cache
# Restore fresh timings so shard counts match GitHub Actions; miss falls back to committed.
uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: .test_durations
key: insights-test-durations-${{ github.run_id }}
restore-keys: |
insights-test-durations-
- name: Discover products to test
id: discover
env:
# On pushes to master, always run everything.
# On PRs, use the path filter to detect legacy changes.
LEGACY_CHANGED: ${{ github.event_name != 'pull_request' || needs.changes.outputs.legacy }}
SCHEMA_CHANGED: ${{ github.event_name != 'pull_request' || needs.changes.outputs.schema }}
TURBO_SCM_BASE: ${{ github.event_name == 'pull_request' && format('origin/{0}', github.event.pull_request.base.ref) || '' }}
TURBO_SCM_HEAD: ${{ github.sha }}
# Kill switch — drop comma-separated products from the matrix; empty = run all.
SKIP_PRODUCT_TESTS: ${{ vars.SKIP_PRODUCT_TESTS || '' }}
run: |
# turbo-discover.js uses Turbo's Git affectedness to detect
# changed products. Non-isolated product changes trigger the
# full suite (all products + Django).
RESULT=$(node .github/scripts/turbo-discover.js)
echo "Result: $RESULT"
echo "matrix=$(echo "$RESULT" | jq -c '.matrix')" >> $GITHUB_OUTPUT
echo "run_legacy=$(echo "$RESULT" | jq -r '.run_legacy')" >> $GITHUB_OUTPUT
echo "django_shards=$(echo "$RESULT" | jq -c '.django_shards // empty')" >> $GITHUB_OUTPUT
# Pick which Django tests to run on draft PRs. Drafts get the snob-selected
# subset for fast feedback; the full matrix runs once the PR is marked ready
# for review, and that ready run is the merge gate. When selection can't be
# trusted on a draft (legacy graph impact, turbo-discover or selector failure,
# a selector full-run signal), the draft skips the heavy matrices entirely —
# the pre-selection draft behavior — and defers to the ready full run.
# insightscli-lint: not-a-required-gate - selects draft coverage and emits no required check.
#
# Depot shadow note: canonical uploads /tmp/selection.json for its
# test-selection-verdict job. The depot shadow strips that job (and all
# artifact uploads), so this job has no Upload selection artifact step.
select-tests:
name: Select tests
needs: [changes, turbo-discover]
# Only draft PRs do selective runs; ready PRs and pushes always run full,
# which build_django_matrix falls back to
# when select-tests is skipped (empty MODE). The run-ci-backend label
# forces the full matrices on a draft.
# Trunk's merge-queue branches open as draft PRs, but their run IS the
# merge gate, so they must never get the narrowed selection.
if: |
github.event_name == 'pull_request' &&
github.event.pull_request.draft == true &&
!startsWith(github.head_ref, 'trunk-merge/') &&
!contains(github.event.pull_request.labels.*.name, 'run-ci-backend') &&
needs.changes.outputs.backend == 'true'
runs-on: depot-ubuntu-24.04
timeout-minutes: 5
outputs:
mode: ${{ steps.classify.outputs.mode }}
core_files: ${{ steps.classify.outputs.core_files }}
poe_files: ${{ steps.classify.outputs.poe_files }}
temporal_files: ${{ steps.classify.outputs.temporal_files }}
run_poe: ${{ steps.classify.outputs.run_poe }}
run_temporal: ${{ steps.classify.outputs.run_temporal }}
# Kept apples-to-apples with canonical, though the shadow strips the
# capture job that consumes them (insights-github-action telemetry).
narrowed: ${{ steps.classify.outputs.narrowed }}
skip_reason: ${{ steps.classify.outputs.skip_reason }}
changed_file_count: ${{ steps.classify.outputs.changed_file_count }}
selected_test_count: ${{ steps.classify.outputs.selected_test_count }}
full_run_reasons_count: ${{ steps.classify.outputs.full_run_reasons_count }}
selected_test_seconds: ${{ steps.classify.outputs.selected_test_seconds }}
skipped_test_seconds: ${{ steps.classify.outputs.skipped_test_seconds }}
run_legacy: ${{ steps.fallback.outputs.run_legacy }}
turbo_result: ${{ steps.fallback.outputs.turbo_result }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
fetch-depth: 1000
filter: blob:none
- name: Install uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
with:
version: '0.11.28'
- name: Decide whether selection can be trusted
id: fallback
env:
RUN_LEGACY: ${{ needs.turbo-discover.outputs.run_legacy }}
TURBO_RESULT: ${{ needs.turbo-discover.result }}
shell: bash
run: |
set -euo pipefail
skip=false
if [[ "$RUN_LEGACY" == "true" ]]; then
# turbo-discover detected product->legacy graph impact; the
# diff-based selector can't see this, so its subset would be
# incomplete. Skip the draft matrices; the ready run is full.
skip=true
elif [[ "$TURBO_RESULT" != "success" && "$TURBO_RESULT" != "skipped" ]]; then
# Conservative — turbo-discover failed.
skip=true
fi
echo "skip=$skip" >> "$GITHUB_OUTPUT"
# Surface the inputs behind an untrusted skip so the telemetry can
# tell legacy-graph impact apart from a turbo-discover failure.
echo "run_legacy=${RUN_LEGACY:-}" >> "$GITHUB_OUTPUT"
echo "turbo_result=${TURBO_RESULT:-}" >> "$GITHUB_OUTPUT"
- name: Run shadow selector
id: select
# continue-on-error so a selector or git fetch failure doesn't fail this
# job — a failed select-tests would leave MODE empty and build_django_matrix
# would fall back to an expensive full matrix on a draft. Classify reads
# steps.select.outcome (pre-continue-on-error) and emits mode=skip instead.
continue-on-error: true
if: steps.fallback.outputs.skip == 'false'
env:
BASE_REF: ${{ github.event.pull_request.base.ref }}
shell: bash
run: |
set -euo pipefail
git fetch --no-tags --depth=1000 --filter=blob:none origin "$BASE_REF:refs/remotes/origin/$BASE_REF"
uv run tools/snob_backend_test_selection_shadow.py \
--base-ref "origin/$BASE_REF" \
> /tmp/selection.json
- name: Classify selection into segments
id: classify
env:
FALLBACK_SKIP: ${{ steps.fallback.outputs.skip }}
SELECT_OUTCOME: ${{ steps.select.outcome }}
shell: bash
run: |
set -euo pipefail
# Selection telemetry, kept apples-to-apples with canonical (the shadow
# strips the capture job that consumes these, per the header).
emit_metrics() {
local changed="" selected="" reasons="" selected_seconds="" skipped_seconds=""
if [[ -s /tmp/selection.json ]]; then
changed=$(jq -r '.changed_file_count // empty' /tmp/selection.json)
selected=$(jq -r '.combined.count // empty' /tmp/selection.json)
reasons=$(jq -r '.ast.full_run_reasons | length' /tmp/selection.json)
selected_seconds=$(jq -r '.durations.selected_seconds // empty' /tmp/selection.json)
skipped_seconds=$(jq -r '.durations.skipped_seconds // empty' /tmp/selection.json)
fi
{
echo "narrowed=$1"
echo "skip_reason=$2"
echo "changed_file_count=$changed"
echo "selected_test_count=$selected"
echo "full_run_reasons_count=$reasons"
echo "selected_test_seconds=$selected_seconds"
echo "skipped_test_seconds=$skipped_seconds"
} >> "$GITHUB_OUTPUT"
}
# Untrusted selection on a draft skips the heavy matrices (the
# pre-selection draft behavior); the ready-for-review run is full.
fall_back_to_skip() {
echo "mode=skip" >> "$GITHUB_OUTPUT"
echo "core_files=" >> "$GITHUB_OUTPUT"
echo "poe_files=" >> "$GITHUB_OUTPUT"
echo "temporal_files=" >> "$GITHUB_OUTPUT"
echo "run_poe=false" >> "$GITHUB_OUTPUT"
echo "run_temporal=false" >> "$GITHUB_OUTPUT"
emit_metrics false "$1"
}
if [[ "$FALLBACK_SKIP" == "true" ]]; then
fall_back_to_skip untrusted
exit 0
fi
if [[ "$SELECT_OUTCOME" != "success" ]] || [[ ! -s /tmp/selection.json ]]; then
echo "::warning::shadow selector did not produce output; draft skips heavy matrices (full run happens on ready for review)"
fall_back_to_skip selector_error
exit 0
fi
full_run_reasons=$(jq -r '.ast.full_run_reasons | length' /tmp/selection.json)
if [[ "$full_run_reasons" -gt 0 ]]; then
echo "Selector requested a full run; draft defers it to ready for review:"
jq -r '.ast.full_run_reasons[]' /tmp/selection.json
fall_back_to_skip full_run_requested
exit 0
fi
# Partition selected files by the Django segment that owns them.
# Mirrors the positional path arguments in the Run Core / Run Temporal pytest invocations:
# Core (POE off): insights ee/ (minus insights/temporal, insights/dags)
# Core POE on: insights/datastore, insights/queries, insights/api/test/test_insight*,
# insights/api/test/dashboards/test_dashboard.py, ee/datastore/
# Temporal: insights/temporal, products/{batch_exports,tasks}/backend/temporal,
# products/signals/backend/emission
# Files outside insights/ and ee/ (e.g. products/foo/backend/test_*) are not in this
# matrix. turbo-tests handles them; warehouse_sources and managed_warehouse run
# their own temporal suites there.
core=()
poe=()
temporal=()
while IFS= read -r f; do
[[ -z "$f" ]] && continue
case "$f" in
insights/temporal/*|products/batch_exports/backend/tests/temporal/*|products/tasks/backend/temporal/*|products/signals/backend/emission/*)
temporal+=("$f")
;;
insights/datastore/*|insights/queries/*|ee/datastore/*|products/product_analytics/backend/api/test/*|insights/api/test/test_insight*|insights/api/test/dashboards/test_dashboard.py)
poe+=("$f")
core+=("$f")
;;
insights/dags/*|common/scriptvm/python/test/*)
# Explicitly --ignore'd by the Core invocation; nothing to do.
;;
insights/*|ee/*)
core+=("$f")
;;
esac
done < <(jq -r '.combined.tests[]?' /tmp/selection.json)
echo "mode=selected" >> "$GITHUB_OUTPUT"
echo "core_files=${core[*]:-}" >> "$GITHUB_OUTPUT"
echo "poe_files=${poe[*]:-}" >> "$GITHUB_OUTPUT"
echo "temporal_files=${temporal[*]:-}" >> "$GITHUB_OUTPUT"
echo "run_poe=$([[ ${#poe[@]} -gt 0 ]] && echo true || echo false)" >> "$GITHUB_OUTPUT"
echo "run_temporal=$([[ ${#temporal[@]} -gt 0 ]] && echo true || echo false)" >> "$GITHUB_OUTPUT"
emit_metrics true ""
echo "Selected: ${#core[@]} core, ${#poe[@]} POE-eligible, ${#temporal[@]} temporal"
build-product-test-matrix:
name: Build product test matrix
needs: [turbo-discover]
if: >-
!cancelled() && needs.turbo-discover.result == 'success' && needs.turbo-discover.outputs.matrix != '[]' && needs.turbo-discover.outputs.matrix != ''
runs-on: depot-ubuntu-24.04
timeout-minutes: 5
outputs:
include: ${{ steps.build.outputs.include }}
steps:
- name: Build product matrix include list
id: build
env:
PRODUCT_MATRIX_JSON: ${{ needs.turbo-discover.outputs.matrix }}
run: |
include=$(jq -cn --argjson matrix "$PRODUCT_MATRIX_JSON" '
[
$matrix[] as $entry
| [false, true][] as $new_events_schema
| $entry + {"new-events-schema": $new_events_schema}
]
')
echo "include=$include" >> "$GITHUB_OUTPUT"
echo "Product test matrix size: $(jq -r 'length' <<< "$include")"
# Runs product tests in parallel — one matrix job per group
# Each job gets its own runner + Docker stack, so no shared DB conflicts
# Small products (< 50 tests) are grouped into a single job to avoid setup overhead
# A status function is required to inspect dependency results after failures.
# Use !cancelled(), not always(), so superseded matrix jobs still receive cancellation.
turbo-tests:
needs: [changes, turbo-discover, detect-snapshot-mode, build-product-test-matrix, get_datastore_versions]
if: >-
!cancelled() && (github.event.pull_request.draft != true || startsWith(github.head_ref, 'trunk-merge/') || contains(github.event.pull_request.labels.*.name, 'run-ci-backend')) && needs.turbo-discover.result == 'success' && needs.build-product-test-matrix.result == 'success' && needs.build-product-test-matrix.outputs.include != '[]' && needs.build-product-test-matrix.outputs.include != ''
runs-on: depot-ubuntu-24.04
timeout-minutes: 40
name: Product tests (${{ matrix.group }}, events schema ${{ matrix.new-events-schema && 'json' || 'legacy' }})
strategy:
fail-fast: false
matrix:
include: ${{ fromJson(needs.build-product-test-matrix.outputs.include) }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
# Start Docker early (before dependency installs) so containers can pull
# images and initialize while we install deps. This matches the pattern
# used by core-tests/django which achieves ~1s wait times.
- name: Clean up data directories
run: |
[ -d "data" ] && docker run --rm -v "$(pwd)/data:/data" alpine sh -c "rm -rf /data/seaweedfs /data/minio" || true
continue-on-error: true
- name: Log in to Docker Hub
if: ${{ secrets.DOCKERHUB_USERNAME != '' && secrets.DOCKERHUB_TOKEN != '' }}
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Start services
env:
COMPOSE_FILE: docker-compose.dev.yml:docker-compose.profiles.yml
COMPOSE_PROFILES: temporal,azure
DATASTORE_SERVER_IMAGE: ${{ needs.get_datastore_versions.outputs.local_image }}
run: |
cp insights/user_scripts/latest_user_defined_function.xml insights/user_scripts/user_defined_function.xml
# nohup detaches the launch so it survives Depot's step-shell teardown
# (which SIGHUPs a plain `&` subshell). Isolated here; bin/ci-wait-for-docker stays == master.
nohup bin/ci-wait-for-docker launch --background --down \
db redis7 datastore zookeeper kafka objectstorage \
temporal elasticsearch objectstorage-azure \
</dev/null >"${RUNNER_TEMP:-/tmp}/compose-launch.log" 2>&1
# Shadow trial: canonical threads a setup-action app token into both
# pnpm-install and setup-python; mint stripped here (secret not
# available on Depot), so setup-python keeps the bot PAT.
- parallel:
- name: Install pnpm dependencies
uses: ./.depot/actions/pnpm-install
with:
install-command: pnpm install --frozen-lockfile --filter=@hanzo/root
- name: Set up Python
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: 3.13.13
token: ${{ secrets.INSIGHTS_BOT_PAT }}
- name: Install Rust
uses: dtolnay/rust-toolchain@3c5f7ea28cd621ae0bf5283f0e981fb97b8a7af9
with:
toolchain: 1.91.1
components: cargo
- name: Install uv
id: setup-uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
with:
version: '0.11.28' # pinned: unpinned setup-uv calls GH API on every job, exhausts rate limit
enable-cache: true
cache-dependency-glob: uv.lock
save-cache: ${{ github.ref == 'refs/heads/master' }}
- name: Install SAML dependencies
if: steps.setup-uv.outputs.cache-hit != 'true'
run: sudo rm -f /etc/apt/sources.list.d/*twingate* && sudo apt-get update && sudo apt-get install -y libxml2-dev libxmlsec1-dev libxmlsec1-openssl
- name: Install sqlx-cli
uses: ./.depot/actions/setup-sqlx-cli
- name: Install Python dependencies
run: UV_PROJECT_ENVIRONMENT=$pythonLocation uv sync --frozen --dev
- name: Add service hostnames to /etc/hosts
run: echo "127.0.0.1 db redis7 kafka datastore datastore-coordinator objectstorage temporal" | sudo tee -a /etc/hosts
- name: Set up needed files
run: |
mkdir -p frontend/dist
touch frontend/dist/index.html frontend/dist/layout.html frontend/dist/exporter.html
./bin/download-mmdb
- name: Wait for Docker services
env:
COMPOSE_FILE: docker-compose.dev.yml:docker-compose.profiles.yml
COMPOSE_PROFILES: temporal,azure
DATASTORE_SERVER_IMAGE: ${{ needs.get_datastore_versions.outputs.local_image }}
WAIT_FOR_DOCKER_TIMEOUT: 900
# objectstorage answers InvalidAccessKeyId until its bootstrap registers identities.
run: |
bin/ci-wait-for-docker wait objectstorage \
|| { echo "::group::compose launch log"; cat "${RUNNER_TEMP:-/tmp}/compose-launch.log" 2>/dev/null || true; echo "::endgroup::"; exit 1; }
- name: Register Temporal search attributes
env:
WAIT_FOR_DOCKER_TIMEOUT: 900
run: |
bin/ci-wait-for-docker wait temporal
python manage.py register_temporal_search_attributes
- name: Restore schema cache from master
if: ${{ github.event_name == 'pull_request' && needs.turbo-discover.outputs.schema_cache_key != '' }}
uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: schema.sql.gz
# Prefer the shared content key; fall back to the exact merge-base SHA key.
key: ${{ needs.turbo-discover.outputs.schema_migrations_key || needs.turbo-discover.outputs.schema_cache_key }}
restore-keys: |
${{ needs.turbo-discover.outputs.schema_cache_key }}
- name: Prime test_insights from cached schema
# No-op on cache miss; pytest --reuse-db falls back to a full migrate.
if: ${{ github.event_name == 'pull_request' }}
run: |
if [ ! -f schema.sql.gz ]; then
echo "::notice::Schema cache miss — pytest --reuse-db will run full migrations"
exit 0
fi
mkdir -p .postgres-backups
mv schema.sql.gz .postgres-backups/schema-latest.sql.gz
if ! ./bin/insightscli db:restore-test-db; then
echo "::warning::Schema restore failed (stale/incompatible cached dump?) — pytest --reuse-db will run a full migrate"
fi
- name: Run product tests
# --force: discover already decided this product needs testing, skip turbo cache
# --log-order=stream: stream pytest output live instead of buffering until completion
# pytest_args: optional pytest-split flags for sharded products (e.g. "-- --splits 3 --group 1")
env:
DATASTORE_INSIGHTSQL_USE_NEW_EVENTS_SCHEMA: ${{ matrix.new-events-schema && 'true' || 'false' }}
# --reuse-db: keep the test database between sequential product runs to avoid
# Datastore drop/create race conditions with ReplicatedMergeTree ZK metadata.
# --durations-path: turbo runs products from products/<name>/, so pytest-split
# needs an explicit path back to the repo-root durations file for balanced sharding.
PYTEST_ADDOPTS: >-
--reuse-db
--durations-path ../../.test_durations
${{ needs.detect-snapshot-mode.outputs.mode == 'update' && '--snapshot-update --snapshot-warn-unused' || '' }}
${{ github.event_name == 'pull_request' && '--cov=backend --cov-report=xml:coverage.xml' || '' }}
# turbo runs each product's backend:test with the product dir as CWD, so --cov=backend
# scopes coverage to that product and writes products/<name>/coverage.xml — works for
# bin-packed buckets (each product writes its own file) and split products (unioned later).
# sysmon: Python 3.12+'s low-overhead coverage backend, ~a few % vs ~20% for the C tracer.
COVERAGE_CORE: sysmon
run: |
set +e
pnpm turbo run backend:test ${{ matrix.filters }} --concurrency=1 --output-logs=full --force --log-order=stream ${{ matrix.pytest_args }}
exit_code=$?
set -e
if [ $exit_code -eq 5 ]; then
echo "No tests collected for this shard, this is expected when splitting tests"
exit 0
else
exit $exit_code
fi
# Lightweight repo-wide checks that only need Python + uv (no Docker/DB).
# Consolidates checks that previously each spun up their own runner.
repo-checks:
needs: [changes]
# product_yamls: ownership-only PRs still need the owners.yaml lint below.
if: (needs.changes.outputs.backend == 'true' || needs.changes.outputs.product_yamls == 'true') && github.event_name != 'push'
timeout-minutes: 10
permissions:
contents: read
pull-requests: read # list PR files for the deleted-migration check
name: Repo checks (depot-ubuntu-24.04)
runs-on: depot-ubuntu-24.04
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Set up Python
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: 3.13.13
- name: Install uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
with:
version: '0.11.28' # pinned: unpinned setup-uv calls GH API on every job, exhausts rate limit
- name: Install Python dependencies
run: UV_PROJECT_ENVIRONMENT=$pythonLocation uv sync --frozen --dev
- name: Bootstrap scaffold product
run: ./bin/insightscli product:bootstrap spline_reticulator --non-interactive
- name: Lint product structure
run: ./bin/insightscli product:lint --all
# Distributed owners.yaml tree: schema, product.yaml/owners.yaml
# conflicts, dead rule globs, coverage (warn-only). No --live, so no
# secrets — fork PRs get validated too (the --live slug check runs in
# the fork-gated validate-product-yamls job instead).
- parallel:
- name: Lint owners.yaml files
run: ./bin/insightscli owners:lint
- name: Check version specifiers
run: python .github/scripts/check-version-specifiers.py
- name: Check IDOR model coverage
run: python .github/scripts/check-idor-model-coverage.py
- name: Check DWH presentation layer stays source-agnostic
run: python .github/scripts/check-dwh-source-agnostic.py
- name: Check operator parity
run: python .github/scripts/check-operator-parity.py
- name: Check module boundaries (tach)
run: tach check --dependencies --interfaces
- name: Check product facade enforcement (import-linter)
run: lint-imports
# Block deleting historical migration files (see safe-django-migrations.md).
# GITHUB_ACTIONS=false suppresses the command's PR annotations on the shadow.
- name: Check deleted migrations
if: github.event_name == 'pull_request' && needs.changes.outputs.migrations_any == 'true'
env:
GITHUB_ACTIONS: 'false'
REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
run: .github/scripts/list-removed-renamed-paths.sh | ./bin/insightscli lint:migration-deletions
validate-product-yamls:
needs: [changes]
if: >
needs.changes.outputs.product_yamls == 'true' &&
github.event_name == 'pull_request' &&
github.event.pull_request.head.repo.full_name == github.repository
timeout-minutes: 5
name: Validate product.yaml owners (depot-ubuntu-24.04)
runs-on: depot-ubuntu-24.04
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Get app token
id: app-token
uses: actions/create-github-app-token@1b10c78c7865c340bc4f6099eb2f838309f1e8c3 # v3.1.1
with:
client-id: ${{ secrets.GH_APP_INSIGHTS_ASSIGN_REVIEWERS_APP_ID }}
private-key: ${{ secrets.GH_APP_INSIGHTS_ASSIGN_REVIEWERS_PRIVATE_KEY }}
owner: ${{ github.repository_owner }}
repositories: ${{ github.event.repository.name }}
- name: Set up Python
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: 3.13.13
token: ${{ steps.app-token.outputs.token || github.token }}
- name: Install uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
with:
version: '0.11.28'
- name: Install Python dependencies
run: UV_PROJECT_ENVIRONMENT=$pythonLocation uv sync --frozen --dev
- name: Validate changed product.yaml owners
env:
GH_TOKEN: ${{ steps.app-token.outputs.token }}
CHANGED_FILES: ${{ needs.changes.outputs.product_yamls_files }}
run: |
mapfile -t names < <(echo "$CHANGED_FILES" | tr ' ' '\n' | sed -n 's|^products/\([^/]*\)/product.yaml$|\1|p')
existing=()
for n in "${names[@]}"; do
[ -f "products/$n/product.yaml" ] && existing+=("$n")
done
if [ "${#existing[@]}" -eq 0 ]; then
echo "No existing product.yaml files in diff after filtering"
exit 0
fi
./bin/insightscli product:lint:owners "${existing[@]}"
- name: Validate owners.yaml identities against GitHub
env:
GH_TOKEN: ${{ steps.app-token.outputs.token }}
CHANGED_FILES: ${{ needs.changes.outputs.product_yamls_files }}
# Scoped to the diff: unscoped, one stale slug fails every owners.yaml PR.
# xargs re-parses the paths-filter escape format, keeping escaped paths whole.
run: printf '%s' "$CHANGED_FILES" | xargs -r ./bin/insightscli owners:lint --live
# Migration validation and OpenAPI type generation.
# This job needs Docker + DB — it checks out master first to run baseline
# migrations, then checks out the PR branch. All steps after the PR checkout
# require Django + DB. Lightweight checks belong in repo-checks above.
check-migrations:
needs: [changes]
if: needs.changes.outputs.backend == 'true' || needs.changes.outputs.migrations == 'true' || needs.changes.outputs.openapi_types == 'true'
timeout-minutes: 60
name: Validate migrations and OpenAPI types
runs-on: depot-ubuntu-24.04
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
clean: false
- name: Clean up data directories with container permissions
run: |
# Use docker to clean up files created by containers
[ -d "data" ] && docker run --rm -v "$(pwd)/data:/data" alpine sh -c "rm -rf /data/seaweedfs /data/minio" || true
continue-on-error: true
- name: Log in to Docker Hub
if: ${{ secrets.DOCKERHUB_USERNAME != '' && secrets.DOCKERHUB_TOKEN != '' }}
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Stop/Start stack with Docker Compose
env:
COMPOSE_FILE: docker-compose.dev.yml:docker-compose.profiles.yml
COMPOSE_PROFILES: temporal,azure
run: |
# nohup detaches the launch so it survives Depot's step-shell teardown
# (which SIGHUPs a plain `&` subshell). Isolated here; bin/ci-wait-for-docker stays == master.
nohup bin/ci-wait-for-docker launch --background --down \
</dev/null >"${RUNNER_TEMP:-/tmp}/compose-launch.log" 2>&1
- name: Set up Python
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: 3.13.13
- name: Install uv
id: setup-uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
with:
version: '0.11.28' # pinned: unpinned setup-uv calls GH API on every job, exhausts rate limit
enable-cache: true
cache-dependency-glob: uv.lock
save-cache: ${{ github.ref == 'refs/heads/master' }}
- name: Install SAML (python3-saml) dependencies
if: steps.setup-uv.outputs.cache-hit != 'true'
run: |
sudo rm -f /etc/apt/sources.list.d/*twingate*
sudo apt-get update
sudo apt-get install libxml2-dev libxmlsec1-dev libxmlsec1-openssl
- name: Install Rust
uses: dtolnay/rust-toolchain@3c5f7ea28cd621ae0bf5283f0e981fb97b8a7af9
with:
toolchain: 1.91.1
components: cargo
- name: Install sqlx-cli
uses: ./.depot/actions/setup-sqlx-cli
# First running migrations from master, to simulate the real-world scenario
- name: Checkout master
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
ref: master
clean: false
- name: Install python dependencies for master
run: |
UV_PROJECT_ENVIRONMENT=.venv-master uv sync --frozen --dev
- name: Compute master schema cache key
# Must match the migration-set computation in turbo-discover and the
# push-side save below (inputs, hash, and key prefix). Keyed off the
# master checkout (HEAD here) so a restored dump is exactly the state
# that "Run migrations up to master" would build from scratch; any
# difference in the migration set is a cache miss, never a stale hit.
if: github.event_name != 'push'
id: master-schema-key
run: |
MIG_FILES=$(git ls-tree -r --format='%(objectname) %(path)' HEAD \
| grep -E '/migrations/[^/]+\.py$|^[0-9a-f]+ uv\.lock$' \
| grep -vE '/(datastore|async_migrations)/migrations/' \
| LC_ALL=C sort) || true
# Routing decides which app labels are in the dump, so these inputs belong in the key.
DB_ROUTING=$(git ls-tree -r --format='%(objectname) %(path)' HEAD -- products/db_routing.yaml insights/product_db_config.py insights/product_db_router.py)
PG_IMAGE=$(git show "HEAD:docker-compose.base.yml" 2>/dev/null \
| grep -m1 -E '^[[:space:]]*image:[[:space:]]*postgres:' | tr -d '[:space:]')
if [ -z "$MIG_FILES" ]; then
echo "key=" >> $GITHUB_OUTPUT
else
MIG_HASH=$(printf '%s\n%s\n%s' "$MIG_FILES" "$PG_IMAGE" "$DB_ROUTING" | sha256sum | cut -c1-40)
echo "key=insights-schema-mig-${SCHEMA_CACHE_EPOCH}-${MIG_HASH}" >> $GITHUB_OUTPUT
fi
- name: Restore master schema dump from cache
if: github.event_name != 'push' && steps.master-schema-key.outputs.key != ''
uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: schema.sql.gz
key: ${{ steps.master-schema-key.outputs.key }}
- name: Wait for Docker services
env:
COMPOSE_FILE: docker-compose.dev.yml:docker-compose.profiles.yml
COMPOSE_PROFILES: temporal,azure
run: |
bin/ci-wait-for-docker wait objectstorage \
|| { echo "::group::compose launch log"; cat "${RUNNER_TEMP:-/tmp}/compose-launch.log" 2>/dev/null || true; echo "::endgroup::"; exit 1; }
- name: Prime insights DB from cached schema
# Turns "Run migrations up to master" below into a no-op instead
# of ~10 minutes of full migrations.
if: github.event_name != 'push'
run: |
if [ ! -f schema.sql.gz ]; then
echo "::notice::Master schema cache miss — the next step runs all migrations from scratch"
exit 0
fi
# The load is atomic: on failure the transaction rolls back and
# the fallback below recreates an empty DB for a full migrate.
# ensure_migration_defaults re-seeds data-migration rows the
# schema-only dump lacks.
if gunzip -c schema.sql.gz | docker compose -f docker-compose.dev.yml exec -T db psql -q -v ON_ERROR_STOP=1 --single-transaction -U insights insights > /dev/null \
&& .venv-master/bin/python manage.py ensure_migration_defaults; then
echo "Primed insights DB from cached master schema"
else
echo "::warning::Cached schema restore failed — recreating empty DB; the next step runs all migrations from scratch"
# Terminate lingering sessions first or DROP DATABASE fails
# (same guard as _recreate_database in insightscli's db_schema.py).
docker compose -f docker-compose.dev.yml exec -T db psql -U insights postgres -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'insights' AND pid <> pg_backend_pid();"
docker compose -f docker-compose.dev.yml exec -T db psql -U insights postgres -c "DROP DATABASE IF EXISTS insights;"
docker compose -f docker-compose.dev.yml exec -T db psql -U insights postgres -c "CREATE DATABASE insights;"
fi
rm -f schema.sql.gz
- name: Run migrations up to master
run: |
# Run Django migrations first (excluding managed=False models)
.venv-master/bin/python manage.py migrate
# Then run persons migrations using sqlx; comment out until we've merged
# DATABASE_URL="postgres://insights:insights@localhost:5432/insights_persons" \
# sqlx database create
# DATABASE_URL="postgres://insights:insights@localhost:5432/insights_persons" \
# sqlx migrate run --source rust/persons_migrations/
# Shadow trial: OpenAPI app-token step stripped (secret not available on Depot).
- name: Checkout this PR
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
ref: ${{ (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == 'Insights/insights') && github.event.pull_request.head.ref || github.ref }}
clean: false
token: ${{ github.token }}
- name: Install python dependencies for this PR
run: |
UV_PROJECT_ENVIRONMENT=$pythonLocation uv sync --frozen --dev
# Shadow trial: migration SQL-comment + risk-analysis-comment steps stripped.
# Keep the risk check itself so blocked migrations still fail the job.
- name: Run migration risk analysis
if: github.event_name == 'pull_request'
run: python manage.py analyze_migration_risk --fail-on-blocked
- name: Run migrations for this PR
run: |
# Run Django migrations first (excluding managed=False models)
python manage.py migrate
# Then run persons migrations using sqlx
DATABASE_URL="postgres://insights:insights@localhost:5432/insights_persons" \
sqlx migrate run --source rust/persons_migrations/
- name: Dump migrated schema
if: github.event_name == 'push'
run: |
set -e
set -o pipefail
# Dump schema + django_migrations data so Django knows which migrations are applied
# Run pg_dump inside container to ensure version match (host has pg_dump 16, container has 15)
(docker compose -f docker-compose.dev.yml exec -T db pg_dump --schema-only --clean --if-exists -U insights insights && \
docker compose -f docker-compose.dev.yml exec -T db pg_dump --data-only --table=django_migrations -U insights insights) | gzip > schema.sql.gz
# Verify the dump is valid
gunzip -t schema.sql.gz
- name: Save schema to Actions cache for PR shards
# Kept as the exact merge-base key that every consumer's restore-keys falls back to.
if: github.event_name == 'push'
uses: actions/cache/save@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: schema.sql.gz
key: insights-schema-master-${{ github.sha }}
- name: Compute migration-set schema key
# Must match the merge-base computation in turbo-discover and the
# master restore key above (inputs, hash, and key prefix).
if: github.event_name == 'push'
id: mig-schema-key
run: |
MIG_FILES=$(git ls-tree -r --format='%(objectname) %(path)' HEAD \
| grep -E '/migrations/[^/]+\.py$|^[0-9a-f]+ uv\.lock$' \
| grep -vE '/(datastore|async_migrations)/migrations/' \
| LC_ALL=C sort) || true
# Routing decides which app labels are in the dump, so these inputs belong in the key.
DB_ROUTING=$(git ls-tree -r --format='%(objectname) %(path)' HEAD -- products/db_routing.yaml insights/product_db_config.py insights/product_db_router.py)
PG_IMAGE=$(git show "HEAD:docker-compose.base.yml" 2>/dev/null \
| grep -m1 -E '^[[:space:]]*image:[[:space:]]*postgres:' | tr -d '[:space:]')
if [ -z "$MIG_FILES" ]; then
echo "key=" >> $GITHUB_OUTPUT
echo "::notice::no migration files matched — skipping migration-set schema save"
else
MIG_HASH=$(printf '%s\n%s\n%s' "$MIG_FILES" "$PG_IMAGE" "$DB_ROUTING" | sha256sum | cut -c1-40)
echo "key=insights-schema-mig-${SCHEMA_CACHE_EPOCH}-${MIG_HASH}" >> $GITHUB_OUTPUT
fi
- name: Save schema to Actions cache (migration-set key)
if: github.event_name == 'push' && steps.mig-schema-key.outputs.key != ''
uses: actions/cache/save@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: schema.sql.gz
key: ${{ steps.mig-schema-key.outputs.key }}
- name: Check migrations
# Skip migration safety check on master push (no migration_files from path filter)
if: github.event_name != 'push'
env:
MIGRATIONS_FILES: ${{ needs.changes.outputs.migrations_files }}
run: |
DATABASE_URL="postgres://insights:insights@localhost:5432/insights_persons" \
sqlx migrate info --source rust/persons_migrations/
python manage.py makemigrations --check --dry-run
- name: Check CH migrations
run: |
# Same as above, except now for CH looking at files that were added in insights/datastore/migrations/
git diff --name-status origin/master..HEAD | grep "A\sinsights/datastore/migrations/" | grep -v README | awk '{print $2}' | python manage.py test_ch_migrations_are_safe
# Shadow trial: canonical threads a setup-action app token into
# pnpm-install; mint stripped here (secret not available on Depot).
- name: Install package.json dependencies with pnpm
uses: ./.depot/actions/pnpm-install
with:
install-command: pnpm --filter=@hanzo/root --filter=@hanzo/frontend... --filter=@hanzo/mcp... install --frozen-lockfile
- name: Add OpenAPI Problem Matcher
run: echo "::add-matcher::.github/openapi-problem-matcher.json"
# Shadow trial: OpenAPI types are check-only — no auto-commit side effect.
- name: Check OpenAPI types
run: |
./bin/insightscli build:openapi
pnpm --filter=@hanzo/mcp run scaffold-yaml -- --sync-all
if git diff --exit-code; then
echo "OpenAPI types are up to date"
exit 0
fi
echo "::error::OpenAPI types are out of date — run 'insightscli build:openapi' locally and commit the result."
exit 1
build_django_matrix:
name: Build Django matrix
needs: [changes, get_datastore_versions, turbo-discover, select-tests]
if: |
!cancelled() &&
needs.changes.outputs.backend == 'true' &&
needs.get_datastore_versions.result == 'success'
runs-on: depot-ubuntu-24.04 # was: ubuntu-latest. Mapped standard GitHub runner to Depot equivalent.
timeout-minutes: 5
outputs:
include: ${{ steps.build.outputs.include }}
steps:
- name: Build matrix include list
id: build
env:
OLDEST_SUPPORTED_IMAGE: ${{ needs.get_datastore_versions.outputs.oldest_supported_image }}
COMPAT_MATRIX_JSON: ${{ needs.get_datastore_versions.outputs.compat_matrix }}
DJANGO_SHARDS_JSON: ${{ needs.turbo-discover.outputs.django_shards }}
MODE: ${{ needs.select-tests.outputs.mode }}
RUN_POE: ${{ needs.select-tests.outputs.run_poe }}
RUN_TEMPORAL: ${{ needs.select-tests.outputs.run_temporal }}
CORE_FILES: ${{ needs.select-tests.outputs.core_files }}
# A trunk-merge/** PR opens as a draft but is the merge gate, and
# select-tests deliberately does not run for it. Reporting it as a
# draft here would turn that empty MODE into "skip" and build an
# empty matrix, so the queue would merge having run no Django tests
# at all. Not-draft makes the empty MODE fall through to the full
# matrix, which is what the queue has to run.
IS_DRAFT: ${{ github.event.pull_request.draft && !startsWith(github.head_ref, 'trunk-merge/') }}
FORCE_FULL: ${{ contains(github.event.pull_request.labels.*.name, 'run-ci-backend') }}
run: |
# :NOTE: Keep shard counts/group ranges in sync with historical Django matrix tuning.
# Consult #team-devex before changing.
# A draft PR must never fall back to the full matrix. select-tests only
# runs on drafts and sets MODE to "skip"/"selected"; an empty MODE on a
# draft means select-tests was cancelled or failed (typically
# ready_for_review superseding it mid-flight). Skip here and defer to the
# ready run. The run-ci-backend label intentionally forces full on a draft.
if [[ -z "$MODE" && "$IS_DRAFT" == "true" && "$FORCE_FULL" != "true" ]]; then
echo "::notice::Draft PR with no trusted Django selection — skipping heavy matrices; the ready-for-review run is the full gate"
MODE="skip"
fi
if [[ "$MODE" == "skip" ]]; then
# Draft PR where selection couldn't be trusted: skip the heavy
# matrices and defer to the full run on ready for review.
echo "include=[]" >> "$GITHUB_OUTPUT"
echo "Django matrix size: 0 (mode=skip)"
exit 0
fi
if [[ "$MODE" == "selected" ]]; then
# Selected mode: collapse to one shard per active segment, skip
# segments with no selected files, drop compat entirely.
core_count=0
[[ -n "$CORE_FILES" ]] && core_count=1
core=$(jq -cn --arg image "$OLDEST_SUPPORTED_IMAGE" --argjson n "$core_count" '
[range(1; $n + 1) | {
segment: "Core",
"person-on-events": false,
"new-events-schema": false,
"python-version": "3.13.13",
"datastore-server-image": $image,
concurrency: 1,
group: 1,
artifact_key: "core-1",
compat: false
}]
')
core_new_events_schema=$(jq -cn --arg image "$OLDEST_SUPPORTED_IMAGE" --argjson n "$core_count" '
[range(1; $n + 1) | {
segment: "Core",
"person-on-events": false,
"new-events-schema": true,
"python-version": "3.13.13",
"datastore-server-image": $image,
concurrency: 1,
group: 1,
artifact_key: "core-events-json-1",
compat: false
}]
')
poe_count=0
[[ "$RUN_POE" == "true" ]] && poe_count=1
core_persons_on_events=$(jq -cn --arg image "$OLDEST_SUPPORTED_IMAGE" --argjson n "$poe_count" '
[range(1; $n + 1) | {
segment: "Core",
"person-on-events": true,
"new-events-schema": false,
"python-version": "3.13.13",
"datastore-server-image": $image,
concurrency: 1,
group: 1,
artifact_key: "core-poe-1",
compat: false
}]
')
core_persons_on_events_new_schema=$(jq -cn --arg image "$OLDEST_SUPPORTED_IMAGE" --argjson n "$poe_count" '
[range(1; $n + 1) | {
segment: "Core",
"person-on-events": true,
"new-events-schema": true,
"python-version": "3.13.13",
"datastore-server-image": $image,
concurrency: 1,
group: 1,
artifact_key: "core-poe-events-json-1",
compat: false
}]
')
temporal_count=0
[[ "$RUN_TEMPORAL" == "true" ]] && temporal_count=1
temporal=$(jq -cn --arg image "$OLDEST_SUPPORTED_IMAGE" --argjson n "$temporal_count" '
[range(1; $n + 1) | {
segment: "Temporal",
"person-on-events": false,
"new-events-schema": false,
"python-version": "3.13.13",
"datastore-server-image": $image,
concurrency: 1,
group: 1,
artifact_key: "temporal-1",
compat: false
}]
')
temporal_new_events_schema=$(jq -cn --arg image "$OLDEST_SUPPORTED_IMAGE" --argjson n "$temporal_count" '
[range(1; $n + 1) | {
segment: "Temporal",
"person-on-events": false,
"new-events-schema": true,
"python-version": "3.13.13",
"datastore-server-image": $image,
concurrency: 1,
group: 1,
artifact_key: "temporal-events-json-1",
compat: false
}]
')
compat="[]"
else
# Full run: auto-shard counts from turbo-discover (Amdahl's law on .test_durations).
# Falls back to hardcoded defaults if turbo-discover failed or data is missing.
CORE_SHARDS=38
CORE_POE_SHARDS=7
TEMPORAL_SHARDS=7
if [ -n "$DJANGO_SHARDS_JSON" ] && echo "$DJANGO_SHARDS_JSON" | jq -e '.Core.shards' > /dev/null 2>&1; then
CORE_SHARDS=$(echo "$DJANGO_SHARDS_JSON" | jq -r '.Core.shards')
CORE_POE_SHARDS=$(echo "$DJANGO_SHARDS_JSON" | jq -r '.CorePOE.shards')
TEMPORAL_SHARDS=$(echo "$DJANGO_SHARDS_JSON" | jq -r '.Temporal.shards')
echo "Auto-sharding: Core=$CORE_SHARDS, CorePOE=$CORE_POE_SHARDS, Temporal=$TEMPORAL_SHARDS"
else
echo "::warning::Django shard auto-sizing unavailable, using defaults: Core=$CORE_SHARDS, CorePOE=$CORE_POE_SHARDS, Temporal=$TEMPORAL_SHARDS"
fi
core=$(jq -cn --arg image "$OLDEST_SUPPORTED_IMAGE" --argjson shards "$CORE_SHARDS" '
[range(1; $shards + 1) | {
segment: "Core",
"person-on-events": false,
"new-events-schema": false,
"python-version": "3.13.13",
"datastore-server-image": $image,
concurrency: $shards,
group: .,
artifact_key: ("core-" + (.|tostring)),
compat: false
}]
')
core_new_events_schema=$(jq -cn --arg image "$OLDEST_SUPPORTED_IMAGE" --argjson shards "$CORE_SHARDS" '
[range(1; $shards + 1) | {
segment: "Core",
"person-on-events": false,
"new-events-schema": true,
"python-version": "3.13.13",
"datastore-server-image": $image,
concurrency: $shards,
group: .,
artifact_key: ("core-events-json-" + (.|tostring)),
compat: false
}]
')
core_persons_on_events=$(jq -cn --arg image "$OLDEST_SUPPORTED_IMAGE" --argjson shards "$CORE_POE_SHARDS" '
[range(1; $shards + 1) | {
segment: "Core",
"person-on-events": true,
"new-events-schema": false,
"python-version": "3.13.13",
"datastore-server-image": $image,
concurrency: $shards,
group: .,
artifact_key: ("core-poe-" + (.|tostring)),
compat: false
}]
')
core_persons_on_events_new_schema=$(jq -cn --arg image "$OLDEST_SUPPORTED_IMAGE" --argjson shards "$CORE_POE_SHARDS" '
[range(1; $shards + 1) | {
segment: "Core",
"person-on-events": true,
"new-events-schema": true,
"python-version": "3.13.13",
"datastore-server-image": $image,
concurrency: $shards,
group: .,
artifact_key: ("core-poe-events-json-" + (.|tostring)),
compat: false
}]
')
temporal=$(jq -cn --arg image "$OLDEST_SUPPORTED_IMAGE" --argjson shards "$TEMPORAL_SHARDS" '
[range(1; $shards + 1) | {
segment: "Temporal",
"person-on-events": false,
"new-events-schema": false,
"python-version": "3.13.13",
"datastore-server-image": $image,
concurrency: $shards,
group: .,
artifact_key: ("temporal-" + (.|tostring)),
compat: false
}]
')
temporal_new_events_schema=$(jq -cn --arg image "$OLDEST_SUPPORTED_IMAGE" --argjson shards "$TEMPORAL_SHARDS" '
[range(1; $shards + 1) | {
segment: "Temporal",
"person-on-events": false,
"new-events-schema": true,
"python-version": "3.13.13",
"datastore-server-image": $image,
concurrency: $shards,
group: .,
artifact_key: ("temporal-events-json-" + (.|tostring)),
compat: false
}]
')
compat_source="${COMPAT_MATRIX_JSON:-[]}"
compat=$(jq -cn --argjson compat "$compat_source" '
[
$compat
| to_entries[]
| .value + {
segment: "Core",
"person-on-events": false,
"new-events-schema": false,
"python-version": "3.13.13",
compat: true,
artifact_key: ("compat-" + ((.key + 1)|tostring))
}
]
')
fi
include=$(jq -cn \
--argjson core "$core" \
--argjson core_new_events_schema "$core_new_events_schema" \
--argjson core_persons_on_events "$core_persons_on_events" \
--argjson core_persons_on_events_new_schema "$core_persons_on_events_new_schema" \
--argjson temporal "$temporal" \
--argjson temporal_new_events_schema "$temporal_new_events_schema" \
--argjson compat "$compat" \
'$core + $core_new_events_schema + $core_persons_on_events + $core_persons_on_events_new_schema + $temporal + $temporal_new_events_schema + $compat')
echo "include=$include" >> "$GITHUB_OUTPUT"
echo "Django matrix size: $(jq -r 'length' <<< "$include") (mode=$MODE)"
django:
needs:
[changes, turbo-discover, detect-snapshot-mode, get_datastore_versions, build_django_matrix, select-tests]
# Run legacy pytest if:
# 1. Legacy code directly changed (ee/, insights/)
# 2. OR product changes affect legacy code (turbo-discover outputs run_legacy=true)
# 3. OR turbo-discover itself failed (conservative: run Django on detection failure)
# The non-empty include guard covers drafts in skip mode and selected runs
# that chose zero files — an empty matrix would otherwise fail the job.
# A status function is required to inspect dependency results after failures.
# Use !cancelled(), not always(), so superseded matrix jobs still receive cancellation.
if: |
!cancelled() &&
needs.changes.outputs.backend == 'true' &&
needs.build_django_matrix.result == 'success' &&
needs.build_django_matrix.outputs.include != '[]' &&
(needs.changes.outputs.legacy == 'true' ||
needs.turbo-discover.outputs.run_legacy == 'true' ||
(needs.turbo-discover.result != 'success' && needs.turbo-discover.result != 'skipped'))
# increase for tmate testing
# 45 to match canonical; the shadow is non-blocking so no deploy-gate coupling.
timeout-minutes: 45
name: Django tests – ${{ matrix.segment }}${{ matrix.compat && ' compat' || '' }} (persons-on-events ${{ matrix.person-on-events && 'on' || 'off' }}, events schema ${{ matrix.new-events-schema && 'json' || 'legacy' }}), Py ${{ matrix.python-version }}, ${{ matrix.datastore-server-image }} (${{matrix.group}}/${{ matrix.concurrency }})
# Match canonical's runner size. The shadow restores the same master-primed
# schema cache (turbo-discover key + check-migrations save), so shards skip
# migrations exactly like canonical once Depot's cache is warmed by a master run.
runs-on: depot-ubuntu-24.04
strategy:
fail-fast: false
matrix:
include: ${{ fromJson(needs.build_django_matrix.outputs.include) }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
fetch-depth: 1
repository: ${{ github.event.pull_request.head.repo.full_name }}
ref: ${{ github.event.pull_request.head.ref }}
lfs: true
clean: false
- name: Install signal-fanout
shell: bash
run: sudo install -m 0755 .github/scripts/signal-fanout /usr/local/bin/signal-fanout
- name: Clean up data directories with container permissions
run: |
# Use docker to clean up files created by containers
[ -d "data" ] && docker run --rm -v "$(pwd)/data:/data" alpine sh -c "rm -rf /data/seaweedfs /data/minio" || true
continue-on-error: true
- name: 'Safeguard: ensure no stray Python modules at product root'
run: |
echo "Checking that products/* only contain backend/, frontend/, or shared/ as Python code roots..."
BAD_FILES=$(find products -maxdepth 2 -type f -name "*.py" ! -path "*/backend/*" ! -name "__init__.py" ! -name "conftest.py" -o -maxdepth 2 -type d -name "migrations" ! -path "*/backend/*")
if [ -n "$BAD_FILES" ]; then
echo "❌ Found Python code or migrations outside backend/:"
echo "$BAD_FILES"
echo "Please move these into the appropriate backend/ folder."
exit 1
fi
echo "✅ No stray Python files or migrations found at product roots."
# Pre-tests
# Copies the fully versioned UDF xml file for use in CI testing
- name: Log in to Docker Hub
if: ${{ secrets.DOCKERHUB_USERNAME != '' && secrets.DOCKERHUB_TOKEN != '' }}
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Stop/Start stack with Docker Compose
shell: bash
env:
COMPOSE_FILE: docker-compose.dev.yml:docker-compose.profiles.yml
COMPOSE_PROFILES: temporal,azure
DATASTORE_SERVER_IMAGE: ${{ matrix.datastore-server-image || needs.get_datastore_versions.outputs.oldest_supported_image }}
WAIT_FOR_DOCKER_LAUNCH_RETRIES: 3
WAIT_FOR_DOCKER_LAUNCH_RETRY_DELAY: 5
run: |
cp insights/user_scripts/latest_user_defined_function.xml insights/user_scripts/user_defined_function.xml
# nohup detaches the launch so it survives Depot's step-shell teardown
# (which SIGHUPs a plain `&` subshell). Isolated here; bin/ci-wait-for-docker stays == master.
nohup bin/ci-wait-for-docker launch --background --down-all-profiles \
</dev/null >"${RUNNER_TEMP:-/tmp}/compose-launch.log" 2>&1
- name: Add service hostnames to /etc/hosts
shell: bash
run: echo "127.0.0.1 db redis7 kafka datastore datastore-coordinator objectstorage seaweedfs temporal" | sudo tee -a /etc/hosts
- parallel:
- name: Set up Python
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: ${{ matrix.python-version }}
- name: Install Rust
if: needs.changes.outputs.backend == 'true'
uses: dtolnay/rust-toolchain@3c5f7ea28cd621ae0bf5283f0e981fb97b8a7af9
with:
toolchain: 1.91.1
components: cargo
- name: Install uv
id: setup-uv-tests
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
with:
version: '0.11.28' # pinned: unpinned setup-uv calls GH API on every job, exhausts rate limit
enable-cache: true
cache-dependency-glob: uv.lock
save-cache: ${{ github.ref == 'refs/heads/master' }}
- name: Install SAML (python3-saml) dependencies
if: ${{ needs.changes.outputs.backend == 'true' && steps.setup-uv-tests.outputs.cache-hit != 'true' }}
shell: bash
run: |
sudo rm -f /etc/apt/sources.list.d/*twingate* && sudo apt-get update && sudo apt-get install libxml2-dev libxmlsec1-dev libxmlsec1-openssl
- name: Install sqlx-cli
if: needs.changes.outputs.backend == 'true'
uses: ./.depot/actions/setup-sqlx-cli
- name: Determine if insightsql-parser has changed compared to master
shell: bash
id: insightsql-parser-diff
run: |
git fetch --no-tags --prune --depth=1 origin master
changed=$(git diff --quiet HEAD origin/master -- common/insightsql_parser/ && echo "false" || echo "true")
echo "changed=$changed" >> $GITHUB_OUTPUT
# tests would intermittently fail in GH actions
# with exit code 134 _after passing_ all tests
# this appears to fix it
# absolute wild tbh https://stackoverflow.com/a/75503402
# Cache Qt library packages to reduce network dependency and flakiness
# Using cache-apt-pkgs-action instead of setup-qt-libs to avoid network issues
- name: Cache and install Qt libraries
uses: awalsh128/cache-apt-pkgs-action@acb598e5ddbc6f68a970c5da0688d2f3a9f04d05 # v1.6.0
with:
packages: libegl1 libdbus-1-3 libxkbcommon-x11-0 libxcb-icccm4 libxcb-image0 libxcb-keysyms1 libxcb-randr0 libxcb-render-util0 libxcb-xinerama0 libxcb-xinput0 libxcb-xfixes0 x11-utils libxcb-cursor0 libopengl0
version: 1.0
- name: Install Python dependencies
shell: bash
run: |
UV_PROJECT_ENVIRONMENT=$pythonLocation uv sync --frozen --dev
# Shadow trial: canonical precedes this with a setup-node step (.nvmrc,
# threading a setup-action app token). The shadow has never mirrored that
# step and runs npm on the runner's ambient Node, so the token threading
# has nothing to mirror here.
- name: Install canvas builder dependencies
if: ${{ needs.changes.outputs.backend == 'true' && matrix.segment == 'Core' }}
shell: bash
run: npm ci --ignore-scripts --omit=dev --prefix products/canvas/packages/canvas_builder
- name: Install the working version of insightsql-parser
if: ${{ needs.changes.outputs.backend == 'true' && steps.insightsql-parser-diff.outputs.changed == 'true' }}
shell: bash
# This is not cached currently, as it's important to build the current HEAD version of insightsql-parser if it has
# changed (requirements.txt has the already-published version)
run: |
sudo apt-get install unzip cmake curl uuid pkg-config
curl --fail --location https://www.antlr.org/download/antlr4-cpp-runtime-4.13.1-source.zip --output antlr4-source.zip || curl --fail --location https://raw.githubusercontent.com/antlr/website-antlr4/gh-pages/download/antlr4-cpp-runtime-4.13.1-source.zip --output antlr4-source.zip
# Check that the downloaded archive is the expected runtime - a security measure
anltr_known_md5sum="c875c148991aacd043f733827644a76f"
antlr_found_ms5sum="$(md5sum antlr4-source.zip | cut -d' ' -f1)"
if [[ "$anltr_known_md5sum" != "$antlr_found_ms5sum" ]]; then
echo "Unexpected MD5 sum of antlr4-source.zip!"
echo "Known: $anltr_known_md5sum"
echo "Found: $antlr_found_ms5sum"
exit 64
fi
unzip antlr4-source.zip -d antlr4-source && cd antlr4-source
cmake .
DESTDIR=out make install
sudo cp -r out/usr/local/include/antlr4-runtime /usr/include/
sudo cp out/usr/local/lib/libantlr4-runtime.so* /usr/lib/
sudo ldconfig
cd ..
pip install ./common/insightsql_parser
- name: Set up needed files
shell: bash
run: |
mkdir -p frontend/dist
touch frontend/dist/index.html
touch frontend/dist/layout.html
touch frontend/dist/exporter.html
./bin/download-mmdb
- name: Wait for Docker services
shell: bash
# Read-only health poll for core services only. Can't use --wait here
# because it blocks on ALL project containers including temporal, which
# boots slowly (auto-setup runs DB migrations). Temporal was started in
# background above and will be ready by the time temporal tests run.
env:
COMPOSE_FILE: docker-compose.dev.yml:docker-compose.profiles.yml
COMPOSE_PROFILES: temporal,azure
DATASTORE_SERVER_IMAGE: ${{ matrix.datastore-server-image || needs.get_datastore_versions.outputs.oldest_supported_image }}
WAIT_FOR_DOCKER_TIMEOUT: 900
run: |
bin/ci-wait-for-docker wait objectstorage \
|| { echo "::group::compose launch log"; cat "${RUNNER_TEMP:-/tmp}/compose-launch.log" 2>/dev/null || true; echo "::endgroup::"; exit 1; }
- name: Wait for Temporal
# Some Core tests reach Temporal indirectly (batch_exports, etc). On
# slower Depot runners Temporal is not ready when tests start, so
# block here instead of racing in pytest.
env:
WAIT_FOR_DOCKER_TIMEOUT: 900
run: bin/ci-wait-for-docker wait temporal
- name: Restore schema cache from master
if: ${{ github.event_name == 'pull_request' && needs.turbo-discover.outputs.schema_cache_key != '' }}
uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: schema.sql.gz
# Prefer the shared content key; fall back to the exact merge-base SHA key.
key: ${{ needs.turbo-discover.outputs.schema_migrations_key || needs.turbo-discover.outputs.schema_cache_key }}
restore-keys: |
${{ needs.turbo-discover.outputs.schema_cache_key }}
- name: Prime test_insights from cached schema
# No-op on cache miss; pytest --reuse-db falls back to a full migrate.
if: ${{ github.event_name == 'pull_request' }}
run: |
if [ ! -f schema.sql.gz ]; then
echo "::notice::Schema cache miss — pytest --reuse-db will run full migrations"
exit 0
fi
mkdir -p .postgres-backups
mv schema.sql.gz .postgres-backups/schema-latest.sql.gz
if ! ./bin/insightscli db:restore-test-db; then
echo "::warning::Schema restore failed (stale/incompatible cached dump?) — pytest --reuse-db will run a full migrate"
fi
# Tests
- name: Set snapshot update flags
if: ${{ needs.changes.outputs.backend == 'true' && (needs.detect-snapshot-mode.outputs.mode == 'update' || matrix.person-on-events) }}
shell: bash
run: echo "PYTEST_ARGS=--snapshot-update --snapshot-warn-unused" >> "$GITHUB_ENV"
- name: Set compat snapshot flags
if: ${{ needs.changes.outputs.backend == 'true' && matrix.compat }}
shell: bash
run: echo "PYTEST_ARGS=--snapshot-update --snapshot-warn-unused" >> "$GITHUB_ENV"
- name: Set check-mode snapshot flags
if: ${{ needs.changes.outputs.backend == 'true' && needs.detect-snapshot-mode.outputs.mode != 'update' && !matrix.person-on-events && !matrix.compat }}
shell: bash
run: echo "PYTEST_ARGS=--snapshot-warn-unused" >> "$GITHUB_ENV"
- name: Log test environment diagnostics
if: ${{ needs.changes.outputs.backend == 'true' && matrix.segment == 'Core' }}
shell: bash
env:
SHARD_GROUP: ${{ matrix.group }}
SHARD_CONCURRENCY: ${{ matrix.concurrency }}
DATASTORE_IMAGE: ${{ matrix.datastore-server-image }}
DATASTORE_INSIGHTSQL_USE_NEW_EVENTS_SCHEMA: ${{ matrix.new-events-schema && 'true' || 'false' }}
COMMIT_SHA: ${{ github.sha }}
BRANCH_NAME: ${{ github.head_ref || github.ref_name }}
run: |
echo "=== Test Environment Diagnostics ==="
echo "Shard: $SHARD_GROUP/$SHARD_CONCURRENCY"
echo "Python version: $(python --version)"
echo "Datastore version: $DATASTORE_IMAGE"
echo "Events schema: $DATASTORE_INSIGHTSQL_USE_NEW_EVENTS_SCHEMA"
echo "Commit: $COMMIT_SHA"
echo "Branch: $BRANCH_NAME"
echo "Runner: $(uname -a)"
echo "Memory: $(free -h | head -2)"
echo "Disk: $(df -h / | tail -1)"
echo "Docker containers:"
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" || true
echo "==================================="
- name: Run Core tests
id: run-core-tests
if: ${{ needs.changes.outputs.backend == 'true' && matrix.segment == 'Core' }}
env:
PERSON_ON_EVENTS_V2_ENABLED: ${{ matrix.person-on-events && 'true' || 'false' }}
DATASTORE_INSIGHTSQL_USE_NEW_EVENTS_SCHEMA: ${{ matrix.new-events-schema && 'true' || 'false' }}
MODE: ${{ needs.select-tests.outputs.mode }}
CORE_FILES: ${{ needs.select-tests.outputs.core_files }}
POE_FILES: ${{ needs.select-tests.outputs.poe_files }}
# devex: measure core (insights/ee) coverage on PRs to size the gap vs products. sysmon = low overhead.
COVERAGE_CORE: sysmon
COV_ARGS: ${{ github.event_name == 'pull_request' && '--cov=insights --cov=ee --cov-config=.github/coverage-core.cfg --cov-report=xml:coverage-core.xml' || '' }}
shell: 'signal-fanout bash --noprofile --norc -eo pipefail {0}'
run: |
unset GITHUB_TOKEN # reserved var injected by Depot CI; can't override via env: (see workflow env note)
set +e
# Unrebased branches predate .github/coverage-core.cfg; an explicit --cov-config
# pointing at a missing file is a hard error, so drop coverage rather than fail.
if [[ -n "$COV_ARGS" && ! -f .github/coverage-core.cfg ]]; then
COV_ARGS=""
fi
# Pick the split algorithm and granularity at runtime: optimal_chunks + file
# granularity only when the installed pytest-split offers them, else
# duration_based_chunks / item. A workflow edit runs merged with master, so an
# unrebased PR on older pytest-split must not hit an unknown flag.
SPLITTING_ALGORITHM=duration_based_chunks
SPLIT_GRANULARITY=()
if python -c "from pytest_split.algorithms import Algorithms; from pytest_split.plugin import PytestSplitFilePlugin; raise SystemExit(0 if 'optimal_chunks' in Algorithms.names() else 1)" 2>/dev/null; then
SPLITTING_ALGORITHM=optimal_chunks
SPLIT_GRANULARITY=(--split-granularity=file)
# Scope the file-mode plan to Core's clean per-segment durations when the
# pinned pytest-split supports it; only persons-on-events-off, non-compat Core
# matches the .core run-set, and a missing file falls back to the union.
if [ -f .test_durations.core ] && [[ "${{ matrix.person-on-events }}" != "true" ]] && [[ "${{ matrix.compat }}" != "true" ]] && python -c "from pytest_split.plugin import SUPPORTS_SPLIT_PLAN_PATH" 2>/dev/null; then
SPLIT_GRANULARITY+=(--split-plan-path .test_durations.core)
fi
fi
if [[ "$MODE" == "selected" ]]; then
# Selected mode: run only the test files chosen by the snob shadow selector.
# No --splits/--group (no sharding), and no --reruns anywhere, mirroring
# canonical: retries mask the raw flake signal Trunk needs (the shadow
# strips the quarantine gate itself).
if [[ "${{ matrix.person-on-events }}" == "true" ]]; then
targets="$POE_FILES"
else
targets="$CORE_FILES"
fi
# shellcheck disable=SC2086
pytest -v --tb=short --reuse-db -o junit_duration_report=call $targets -m "not async_migrations" \
-r fEsxX \
--junitxml=junit-core.xml \
$COV_ARGS \
$PYTEST_ARGS
else
full_targets="insights ee/"
full_ignores=()
if [[ "${{ matrix.compat }}" == "true" ]]; then
full_targets="$DATASTORE_COMPAT_PYTEST_TARGETS"
elif [[ "${{ matrix.person-on-events }}" == "true" ]]; then
full_targets="./insights/datastore/ ./insights/queries/ ./products/product_analytics/backend/api/test/ ./insights/api/test/dashboards/test_dashboard.py ee/datastore/"
full_ignores+=(--ignore=insights/insightsql_queries --ignore=insights/insightsql)
fi
# shellcheck disable=SC2086
pytest -v --tb=short --reuse-db -o junit_duration_report=call $full_targets -m "not async_migrations" \
--ignore=insights/temporal \
--ignore=insights/dags \
--ignore=common/scriptvm/python/test \
"${full_ignores[@]}" \
--splits ${{ matrix.concurrency }} --group ${{ matrix.group }} \
--durations=1000 --durations-min=1.0 \
--pytest-durations=100 \
--splitting-algorithm="$SPLITTING_ALGORITHM" \
"${SPLIT_GRANULARITY[@]}" \
-r fEsxX \
--junitxml=junit-core.xml \
$COV_ARGS \
$PYTEST_ARGS
fi
exit_code=$?
set -e
if [ $exit_code -eq 5 ]; then
echo "No tests collected for this shard, this is expected when splitting tests"
exit 0
else
exit $exit_code
fi
# Uncomment this code to create an ssh-able console so you can debug issues with github actions
# (Consider changing the timeout in ci-backend.yml to have more time)
# - name: Setup tmate session
# if: failure()
# uses: mxschmitt/action-tmate@v3
- name: Log test environment diagnostics
if: ${{ needs.changes.outputs.backend == 'true' && matrix.segment == 'Temporal' }}
shell: bash
env:
SHARD_GROUP: ${{ matrix.group }}
SHARD_CONCURRENCY: ${{ matrix.concurrency }}
DATASTORE_INSIGHTSQL_USE_NEW_EVENTS_SCHEMA: ${{ matrix.new-events-schema && 'true' || 'false' }}
COMMIT_SHA: ${{ github.sha }}
BRANCH_NAME: ${{ github.head_ref || github.ref_name }}
run: |
echo "=== Test Environment Diagnostics ==="
echo "Shard: $SHARD_GROUP/$SHARD_CONCURRENCY"
echo "Python version: $(python --version)"
echo "Events schema: $DATASTORE_INSIGHTSQL_USE_NEW_EVENTS_SCHEMA"
echo "Commit: $COMMIT_SHA"
echo "Branch: $BRANCH_NAME"
echo "Runner: $(uname -a)"
echo "Memory: $(free -h | head -2)"
echo "Disk: $(df -h / | tail -1)"
echo "Docker containers:"
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" || true
echo "==================================="
- name: Run Temporal tests
id: run-temporal-tests
if: ${{ needs.changes.outputs.backend == 'true' && matrix.segment == 'Temporal' }}
shell: 'signal-fanout bash --noprofile --norc -eo pipefail {0}'
env:
AWS_S3_ALLOW_UNSAFE_RENAME: 'true'
MODAL_TOKEN_ID: ${{ needs.changes.outputs.tasks_temporal == 'true' && secrets.MODAL_TOKEN_ID || '' }}
MODAL_TOKEN_SECRET: ${{ needs.changes.outputs.tasks_temporal == 'true' && secrets.MODAL_TOKEN_SECRET || '' }}
DATASTORE_INSIGHTSQL_USE_NEW_EVENTS_SCHEMA: ${{ matrix.new-events-schema && 'true' || 'false' }}
MODE: ${{ needs.select-tests.outputs.mode }}
TEMPORAL_FILES: ${{ needs.select-tests.outputs.temporal_files }}
run: |
unset GITHUB_TOKEN # reserved var injected by Depot CI; can't override via env: (see workflow env note)
set +e
# Pick the split algorithm and granularity at runtime (see Run Core tests).
SPLITTING_ALGORITHM=duration_based_chunks
SPLIT_GRANULARITY=()
if python -c "from pytest_split.algorithms import Algorithms; from pytest_split.plugin import PytestSplitFilePlugin; raise SystemExit(0 if 'optimal_chunks' in Algorithms.names() else 1)" 2>/dev/null; then
SPLITTING_ALGORITHM=optimal_chunks
SPLIT_GRANULARITY=(--split-granularity=file)
if [ -f .test_durations.temporal ] && python -c "from pytest_split.plugin import SUPPORTS_SPLIT_PLAN_PATH" 2>/dev/null; then
SPLIT_GRANULARITY+=(--split-plan-path .test_durations.temporal)
fi
fi
# Depot-only quarantine: the data-modeling materialize path (get_s3_client) 403s on
# MinIO on Depot-native CI only — MinIO itself is reachable with the standard test creds
# (verified via boto3 LIST/CREATE/HEAD), so root cause is env-specific and not yet
# isolated (tracked for follow-up). All three tests below run that materialize path; the
# two timeout tests fail downstream (insightsql_table never reached). Canonical GHA still runs them.
# The --deselect guards stay on both branches so a selected run can't re-enable them.
if [[ "$MODE" == "selected" ]]; then
# Selected mode: no sharding, no --reruns — see the core selected branch;
# the full run also skips --reruns, mirroring canonical.
# shellcheck disable=SC2086
pytest -v --tb=short --reuse-db -o junit_duration_report=call $TEMPORAL_FILES -m "not async_migrations" \
--deselect "insights/temporal/tests/data_modeling/test_run_workflow.py::test_run_workflow_with_minio_bucket" \
--deselect "insights/temporal/tests/data_modeling/test_run_workflow.py::test_run_workflow_timeout_does_not_pause_schedule_without_consecutive_failures" \
--deselect "insights/temporal/tests/data_modeling/test_run_workflow.py::test_run_workflow_timeout_pauses_schedule_after_5_consecutive_failures" \
-r fEsxX \
--junitxml=junit-temporal.xml \
$PYTEST_ARGS
else
# No per-test pytest --timeout; rely on the job-level timeout-minutes as the single
# safety net (matches canonical — the per-test guillotine added nothing over it,
# since --timeout-method=thread os._exit()s the whole process anyway).
pytest -v --tb=short --reuse-db -o junit_duration_report=call insights/temporal products/batch_exports/backend/tests/temporal products/tasks/backend/temporal products/signals/backend/emission -m "not async_migrations" \
--deselect "insights/temporal/tests/data_modeling/test_run_workflow.py::test_run_workflow_with_minio_bucket" \
--deselect "insights/temporal/tests/data_modeling/test_run_workflow.py::test_run_workflow_timeout_does_not_pause_schedule_without_consecutive_failures" \
--deselect "insights/temporal/tests/data_modeling/test_run_workflow.py::test_run_workflow_timeout_pauses_schedule_after_5_consecutive_failures" \
--splits ${{ matrix.concurrency }} --group ${{ matrix.group }} \
--durations=100 --durations-min=1.0 \
--pytest-durations=100 \
--splitting-algorithm="$SPLITTING_ALGORITHM" \
"${SPLIT_GRANULARITY[@]}" \
-r fEsxX \
--junitxml=junit-temporal.xml \
$PYTEST_ARGS
fi
exit_code=$?
set -e
if [ $exit_code -eq 5 ]; then
echo "No tests collected for this shard, this is expected when splitting tests"
exit 0
else
exit $exit_code
fi
# Post tests
- name: Show docker compose logs on failure
if: failure() && (needs.changes.outputs.backend == 'true' && steps.run-core-tests.outcome != 'failure' && steps.run-temporal-tests.outcome != 'failure')
shell: bash
run: docker compose -f docker-compose.dev.yml logs
# Read Datastore versions for the compat test matrix.
get_datastore_versions:
name: Get Datastore versions
needs: [changes]
if: needs.changes.outputs.backend == 'true' && github.event_name != 'push'
runs-on: depot-ubuntu-24.04 # was: ubuntu-latest. Mapped standard GitHub runner to Depot equivalent.
timeout-minutes: 5
outputs:
# Oldest supported version as plain string (for comparisons)
oldest_supported_image: ${{ steps.read-versions.outputs.oldest_supported_image }}
# Local dev / hobby image, mirrored from datastore-versions.json `local`
local_image: ${{ steps.read-versions.outputs.local_image }}
# Fully expanded compat matrix (version x shard group)
compat_matrix: ${{ steps.read-versions.outputs.compat_matrix }}
steps:
# Merge ref, for turbo-tests: it checks out the merge ref and mounts that config.xml.
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
sparse-checkout: .github/datastore-versions.json
sparse-checkout-cone-mode: false
# Head ref, for django: it checks out the PR head and mounts that config.xml. Reading
# its version from the merge ref instead would split an atomic version+config bump.
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
repository: ${{ github.event.pull_request.head.repo.full_name }}
ref: ${{ github.event.pull_request.head.ref }}
sparse-checkout: .github/datastore-versions.json
sparse-checkout-cone-mode: false
path: head-tree
- name: Read Datastore versions from JSON
id: read-versions
env:
DJANGO_VERSIONS: head-tree/.github/datastore-versions.json
run: |
compat_shards="2"
echo "Using $compat_shards shard(s) per compat version"
# Oldest supported version for main Django tests (for max compatibility)
oldest_supported=$(jq -r '.oldest_supported' "$DJANGO_VERSIONS")
if [ -z "$oldest_supported" ] || [ "$oldest_supported" = "null" ]; then
echo "::error::No oldest_supported version found in $DJANGO_VERSIONS"
exit 1
fi
echo "oldest_supported_image=$oldest_supported" >> $GITHUB_OUTPUT
echo "Oldest supported version for Django tests: $oldest_supported"
# Local dev / hobby image — the version product (turbo) tests run against.
local_image=$(jq -r '.local' .github/datastore-versions.json)
if [ -z "$local_image" ] || [ "$local_image" = "null" ]; then
echo "::error::No local version found in .github/datastore-versions.json"
exit 1
fi
echo "local_image=$local_image" >> $GITHUB_OUTPUT
echo "Local dev/hobby image for product tests: $local_image"
# Read all unique versions so we can derive compat coverage.
all=$(jq -c '[.[]] | unique' "$DJANGO_VERSIONS")
if [ "$all" = "[]" ] || [ -z "$all" ]; then
echo "::error::No versions found in $DJANGO_VERSIONS"
exit 1
fi
echo "All CH versions found: $all"
# Compat coverage is only needed for non-oldest versions.
compat_versions=$(jq -c --arg oldest "$oldest_supported" '[.[]] | unique | map(select(. != $oldest))' "$DJANGO_VERSIONS")
echo "Compat versions (excluding oldest): $compat_versions"
compat_count=$(jq -r 'length' <<< "$compat_versions")
echo "compat_count=$compat_count" >> $GITHUB_OUTPUT
if [ "$compat_count" -eq 0 ]; then
echo "compat_matrix=[]" >> $GITHUB_OUTPUT
echo "No non-oldest CH versions found — compat tests provide no additional coverage, skipping"
else
compat_matrix=$(jq -cn --argjson versions "$compat_versions" --argjson shards "$compat_shards" '
[
$versions[] as $version
| range(1; $shards + 1) as $group
| {
"datastore-server-image": $version,
"concurrency": $shards,
"group": $group
}
]
')
echo "compat_matrix=$compat_matrix" >> $GITHUB_OUTPUT
echo "Compat matrix: $compat_matrix"
fi
# Job just to collate the status of the matrix jobs for requiring passing status
django_tests:
needs:
[
sample,
changes,
get_datastore_versions,
build_django_matrix,
django,
check-migrations,
turbo-discover,
build-product-test-matrix,
turbo-tests,
repo-checks,
]
name: '[optional, does not block merge] Django Tests Pass'
runs-on: depot-ubuntu-24.04 # was: ubuntu-latest. Mapped standard GitHub runner to Depot equivalent.
timeout-minutes: 5
# always() collates skipped/failed matrix jobs; gated on the sampling result so it
# doesn't spin up a lone runner when this run was sampled out (or the shadow is off).
if: always() && needs.sample.outputs.sampled == 'true'
steps:
- name: Summarize dependency results
run: |
{
echo "### Backend CI dependency results"
echo "| Dependency | Result |"
echo "| --- | --- |"
echo "| changes | ${{ needs.changes.result }} |"
echo "| get_datastore_versions | ${{ needs.get_datastore_versions.result }} |"
echo "| build_django_matrix | ${{ needs.build_django_matrix.result }} |"
echo "| django | ${{ needs.django.result }} |"
echo "| check-migrations | ${{ needs.check-migrations.result }} |"
echo "| repo-checks | ${{ needs.repo-checks.result }} |"
echo "| turbo-discover | ${{ needs.turbo-discover.result }} |"
echo "| build-product-test-matrix | ${{ needs.build-product-test-matrix.result }} |"
echo "| turbo-tests | ${{ needs.turbo-tests.result }} |"
} | tee -a "$GITHUB_STEP_SUMMARY"
- name: Check dependency results
run: |
check_required_result() {
local label="$1"
local result="$2"
if [[ "$result" != "success" && "$result" != "skipped" ]]; then
echo "::error::$label did not succeed (result: '$result')."
exit 1
fi
}
# Upstream jobs whose failure would otherwise only cascade into django
# being *skipped*, which this gate reads as success — a green gate with
# zero backend tests run. Naming them makes the failure hit the gate
# directly; legitimate skips (frontend-only PRs) and plain successes still pass.
check_required_result "Change detection" "${{ needs.changes.result }}"
check_required_result "Datastore versions" "${{ needs.get_datastore_versions.result }}"
check_required_result "Django matrix build" "${{ needs.build_django_matrix.result }}"
check_required_result "Django test matrix" "${{ needs.django.result }}"
check_required_result "Migration checks" "${{ needs.check-migrations.result }}"
check_required_result "Repo checks" "${{ needs.repo-checks.result }}"
check_required_result "Turbo discover" "${{ needs.turbo-discover.result }}"
check_required_result "Product test matrix build" "${{ needs.build-product-test-matrix.result }}"
check_required_result "Product/Turbo tests" "${{ needs.turbo-tests.result }}"
echo "All backend and product checks passed."
# Shadow trial: calculate-running-time job stripped (would poison trial metrics).
Reference in New Issue View Git Blame Copy Permalink