zeekay
1b367e2b62
The tree was a debranded snapshot of upstream taken around 2026-02-20, and upstream has moved about five months since. This replaces it with upstream master, mechanically re-derived: bin/debrand does the rename, and the pipeline and squashed migrations are restored from the shipped tree. Not a merge. The previous import replaced the tree wholesale in a single commit, so there is no shared content history to merge against. Reproducing the rename on the new tree is what actually converges. What survives: .hanzo/workflows, the Dockerfiles and bin/docker* entrypoints, compose files, LLM.md, and insights/migrations. The squashed initial migration is load-bearing — the live database records it by name, so upstream's ~918 individual migrations cannot come back. Nine products are dropped as unreferenced. Twenty-one more that we do not run are kept anyway because the core app still imports them; unpicking those is a refactor and does not belong in a rebase. This base does not build yet, and the reasons are not in the rename: upstream now wants Python 3.13.13 against our 3.12.12 and Django 5.2 against our 4.2, and the dependency manifests still name upstream packages that we republish under our own names. Those are decisions, not renames.
3.0 KiB
title, sidebar, showTitle
| title | sidebar | showTitle |
|---|---|---|
| Working with Datastore materialized columns | Handbook | true |
This document outlines what materialized columns in Datastore are, how we're making use of them and how to manage them on cloud.
Background
We currently store JSON data in string columns in datastore, reading and parsing that data at query-time. This can be slow due to how "fat" these columns are.
Materialized columns allow us to "store" specific properties stored in JSON as separate columns that are there on disk, making reading these columns up to 25x faster than normal properties.
Also check out our Datastore manual and blog post for more information.
Materialized columns in practice
Materialized columns play a huge role in optimizing performance for large clients having difficulties with performance.
This is why we automatically materialize columns and have tooling for creating them manually as well.
Note that materialized columns also require backfilling the materialized columns to be effective - an operation best done on a weekend due to extra load it adds to the cluster.
Automatic materialization
We have a cron-job which analyzes slow queries ran last week and tries to find properties that are used in these slow queries, materializing some of these. Code for this can be found in ee/datastore/materialized_columns/analyze.py
Note that this cron can often be disabled due to cluster issues or ongoing data migrations.
See environment variables documentation + instance settings for toggles which control this.
Manual materialization via Dagster
We use Dagster to materialize columns in production. The job is create_materialized_column in the team-datastore location.
- EU: https://dagster.cloud/insights/prod-eu/locations/team-datastore/jobs/create_materialized_column/playground
- US: https://dagster.cloud/insights/prod-us/locations/team-datastore/jobs/create_materialized_column/playground
To materialize columns, go to the playground for the relevant region and configure the create_materialized_columns_op with the properties you want to materialize:
ops:
create_materialized_columns_op:
config:
backfill_period_days: 90
dry_run: false
properties:
- $browser_language_prefix
- $app_namespace
table: events
table_column: properties
Config options:
table: The Datastore table to materialize on (e.g.events,person)table_column: The JSON column containing the properties (e.g.properties,person_properties,group_properties)properties: List of property names to materialize as columnsbackfill_period_days: How many days of historical data to backfill (typically90)dry_run: Set totrueto preview what would be materialized without making changes
Note that backfilling adds extra load to the cluster, so it's best done on a weekend.