mirror of https://github.com/hanzoai/insights.git synced 2026-10-30 16:15:31 +00:00

Files

T

History

zeekayandHanzo Dev 841ea1c0da no hogs: products and insights

716 files rewritten, 30 renamed, same family mapping as the frontend pass.

Held back, and each for a reason rather than by oversight:

  db_table values          insights_hogfunction, insights_hogflow,
                           insights_hogfunctiontemplate, hogflow_templates.
                           Renaming a table is ALTER TABLE, so it ships as a
                           migration with the raw SQL in nodejs that names it.
  hog_invocation_results   a Datastore table with Kafka engine tables on it
  products/review_hog      the module path is the Django app path; moving it
                           moves app_label and every migration is recorded under
                           (app_label, name). That is the same history break as
                           renaming a migration file, one level up.
  */migrations/*           five files, untouched
  whogho                   an integration vendor

Checked: 395 of 397 changed Python files compile. The two that do not —
insightsql/database/database.py and temporal/schedule.py — fail identically at
origin/main under the same interpreter, because they use `type X = ...` and
`except*`, which need the 3.13 this repo pins and not the 3.9 doing the check.
Running the check against the original is what told those apart from a real
break.

Co-authored-by: Hanzo Dev <[email protected]>

2026-08-09 09:16:43 -07:00

views

no hogs: products and insights

2026-08-09 09:16:43 -07:00

__init__.py

…

managed_warehouse_data_status.py

chore: rebase the tree onto upstream master fa1a9a60

2026-08-06 01:25:36 -07:00

README.md

…

source_agnostic_baseline.txt

…

README.md

Data warehouse presentation layer

The presentation layer stays source-agnostic

This applies to both data warehouse presentation layers: the source/schema views (now in products/warehouse_sources/backend/presentation/views/: external_data_source.py, external_data_schema.py) and the table/query-side views here. Neither may branch on a concrete source type. No:

if source_type == ExternalDataSourceType.POSTGRES: ...
elif source.is_direct_snowflake: ...

Every such branch splits "how source X behaves" between the source class and the API layer, and grows the views without bound as new sources land. The API asks what a source supports and delegates the how.

Where the behaviour goes

Pick the mechanism by the kind of concern:

Source-domain behaviour (how to talk to the upstream system, whether it supports a feature) → the source class under products/warehouse_sources/backend/temporal/data_imports/sources:
- A boolean/value the API reads → a flag on _BaseSource with a safe default (like supports_column_selection, connection_host_fields, has_managed_insightsql_schema). Branch on the flag.
- Methods only some sources have (CDC, xmin, webhooks, custom manifests) → a capability mixin the source opts into; the API dispatches with isinstance(source, <Capability>).
Direct-query engine behaviour (how a SQL engine resolves a table location, builds and reprojects its DataWarehouseTable, maps columns) → an engine-keyed adapter, dispatched on source.direct_engine (via DIRECT_ENGINE_BY_SOURCE_TYPE / is_direct_capable), never on source_type. The engine, not the source type, is the axis of variation, and the same engine can back more than one source. Query-side engine behaviour already lives in the insights/insightsql/direct_sql/ adapter registry; the warehouse-side materialization (get_*_source_location, upsert_direct_*_table, reproject_direct_*_table, *_columns_to_dwh_columns) belongs behind the matching data_warehouse engine registry.

Whatever the mechanism, source-domain semantics live on the source and warehouse-domain work (DataWarehouseTable rows, managed viewsets, script functions) stays here, keyed off what the source / adapter returns. Source capabilities never import data_warehouse types.

Enforcement

.github/scripts/check-dwh-source-agnostic.py is a shrink-only CI guard (run in ci-backend.yml). It counts every ExternalDataSourceType.<NAME> / SourceType.<NAME> / is_direct_<engine> reference across both the data_warehouse and warehouse_sources presentation layers against a baseline (source_agnostic_baseline.txt, kept here). A new reference fails CI; migrating a family lets you shrink the baseline:

python .github/scripts/check-dwh-source-agnostic.py --regenerate-baseline

The baseline only shrinks. It is now empty: the presentation layer is fully source-agnostic, so the guard acts as a hard regression check — any new ExternalDataSourceType.<NAME> / is_direct_<engine> reference here fails CI. Keep it that way.

See implementing-warehouse-sources for how to add behaviour to a source.