zeekay
1b367e2b62
The tree was a debranded snapshot of upstream taken around 2026-02-20, and upstream has moved about five months since. This replaces it with upstream master, mechanically re-derived: bin/debrand does the rename, and the pipeline and squashed migrations are restored from the shipped tree. Not a merge. The previous import replaced the tree wholesale in a single commit, so there is no shared content history to merge against. Reproducing the rename on the new tree is what actually converges. What survives: .hanzo/workflows, the Dockerfiles and bin/docker* entrypoints, compose files, LLM.md, and insights/migrations. The squashed initial migration is load-bearing — the live database records it by name, so upstream's ~918 individual migrations cannot come back. Nine products are dropped as unreferenced. Twenty-one more that we do not run are kept anyway because the core app still imports them; unpicking those is a refactor and does not belong in a rebase. This base does not build yet, and the reasons are not in the rename: upstream now wants Python 3.13.13 against our 3.12.12 and Django 5.2 against our 4.2, and the dependency manifests still name upstream packages that we republish under our own names. Those are decisions, not renames.
4.3 KiB
Web analytics cache warming
Hourly Dagster job (web_analytics_cache_warming_job) that keeps web analytics
precompute buckets and result caches fresh for the fleet's hot query shapes.
Selection scans system.query_log for shapes real users ran, groups them by
normalized shape, and replays a representative per shape through the production
query runners.
Architecture
get_warmable_queries_op (fleet-wide selection, cached in object storage)
→ warm_queries_op (the split: scopes the selection and fans it into
team-disjoint shards) → mapped warm_queries_shard_op per shard.
Each shard runs as its own subprocess under the multiprocess executor. InsightsQL
compilation is CPU-bound Python, so parallelism comes from shard processes, not
threads; threads inside a shard overlap the IO-bound parts (Datastore reads
and inserts, cache lookups). Sharding is by team_id % shards, which keeps
every potential duplicate (team, cache_key) inside one shard so per-shard
dedupe needs no coordination. The run pod requests 6 CPUs / 12Gi via the
dagster-k8s/config tag on the job — the ceiling the 8-core dagster nodes can
actually schedule (~7.9 allocatable), so the default 8 shards deliberately
oversubscribe CPU slightly; that's fine because shards idle during their
IO-bound stretches. Don't size shards one-to-one with cores.
Live-tunable settings (instance settings, no redeploy)
| Setting | Default | Meaning |
|---|---|---|
WEB_ANALYTICS_WARMING_SHARDS |
8 | Shard subprocesses per run (clamp 1-16). Real CPU parallelism is capped by the pod's 6-CPU request (the node-schedulable ceiling), so shards beyond ~8 add little. |
WEB_ANALYTICS_WARMING_SHARD_THREADS |
6 | Worker threads inside each shard (clamp 1-64). Total Datastore-side concurrency ≈ shards × this. |
WEB_ANALYTICS_WARMING_DAYS |
14 | query_log lookback for demand selection. |
WEB_ANALYTICS_WARMING_MIN_QUERY_COUNT |
2 | Per-shape demand floor for selection. |
WEB_ANALYTICS_WARMING_MAX_SHAPES |
400000 | Fleet-wide selection cap. |
WEB_ANALYTICS_WARMING_SELECTION_TTL_SECONDS |
21600 | Selection cache TTL (the expensive fleet scan re-runs on this cadence). |
Settings are read at run start; changes apply to the next run.
Targeted runs (Launchpad)
Config binds under ops.warm_queries_op.config:
ops:
warm_queries_op:
config:
mode: backfill # full (default) | refresh | backfill
team_ids: [] # optional scoping
limit: 0 # hottest-first cap; bound manual backfills
full: everything selected — skip fresh, refresh stale, cold-build the rest.refresh: only shapes already warmed once (cache entry exists) — cheap freshness pass.backfill: only never-warmed shapes — coverage expansion without re-touching the warm set.
Launches share a concurrent-run guard with the hourly schedule, so bound manual
backfills with limit.
Observability
- Per-shape log line (
web_analytics_warming_shape): outcome, duration, team, kind, breakdown, requested and replayed date range. Slow shapes (≥15s full wall-clock) escalate to WARNING. - Per-shard heartbeat every ~2 minutes: progress, rate, ETA, outcome breakdown.
- Prometheus:
insights_web_analytics_warming_queries_total{outcome=...},insights_web_analytics_warming_shapes_selected.