zeekay
1b367e2b62
The tree was a debranded snapshot of upstream taken around 2026-02-20, and upstream has moved about five months since. This replaces it with upstream master, mechanically re-derived: bin/debrand does the rename, and the pipeline and squashed migrations are restored from the shipped tree. Not a merge. The previous import replaced the tree wholesale in a single commit, so there is no shared content history to merge against. Reproducing the rename on the new tree is what actually converges. What survives: .hanzo/workflows, the Dockerfiles and bin/docker* entrypoints, compose files, LLM.md, and insights/migrations. The squashed initial migration is load-bearing — the live database records it by name, so upstream's ~918 individual migrations cannot come back. Nine products are dropped as unreferenced. Twenty-one more that we do not run are kept anyway because the core app still imports them; unpicking those is a refactor and does not belong in a rebase. This base does not build yet, and the reasons are not in the rename: upstream now wants Python 3.13.13 against our 3.12.12 and Django 5.2 against our 4.2, and the dependency manifests still name upstream packages that we republish under our own names. Those are decisions, not renames.
12 KiB
Widget config contract and codegen
When to load: Pydantic config changes, insightscli build:openapi, Zod/OpenAPI drift, ENUM_NAME_OVERRIDES, or MCP config_schema updates.
Platform map (where files live): architecture.md. Ship checklist: checklist-new-widget-type.md §1–4. Update path: managing-existing-widgets.md § Config schema migration.
Config contract (widget_specs/)
Single source of truth for widget.config shape — backend Pydantic models drive runtime validation, REST/MCP OpenAPI, and frontend Zod codegen.
widget_specs/configs.py Pydantic *WidgetConfig per type (+ shared common.py)
│
├─► pydantic_openapi.py injects `model_json_schema()` into OpenAPI components (no DRF bridge)
├─► openapi.py polymorphic batch-add / PATCH / catalog OpenAPI (auto from WIDGET_SPECS)
├─► registry.py WIDGET_SPECS manifest + validate_widget_config() (config, catalog labels, run_*)
└─► widget_catalog.py config_schema = model_json_schema() (for agents)
bin/build-dashboard-widget-types.py (insightscli build:widget-types — step 1)
├─► widget-date-from-options.json date preset values + labels (from `constants.py`)
└─► widget-form-fields.json modal `.pick()` fields (from `WidgetSpec.form_fields`)
generate-widget-config-zod.mjs (insightscli build:widget-types — step 2)
├─► widget-config-property-keys.json per-type keys/trees via `discoverCatalogEntryConfigPropertyKeys()`
└─► Orval generateReusableSchemas (catalog slice → widget-config-schemas/*.zod.ts)
insightscli build:openapi
├─► frontend/generated/api.schemas.ts
├─► products/dashboards/frontend/generated/widget-configs.zod.ts (schemas, types, form picks)
└─► services/mcp/...
| File | Role |
|---|---|
configs.py |
Per-type Pydantic config models — edit here first for field changes |
common.py |
Shared dateRange, widgetFilters, filterTestAccounts |
registry.py |
WIDGET_SPECS + validate_widget_config() — per-type manifest (Pydantic model, run_*, scopes, RBAC, agent catalog labels/availability) |
widgets/config.py |
Query-time only — resolve_filter_test_accounts(config, team) (validation lives in Pydantic) |
openapi.py |
Polymorphic OpenAPI for batch-add, catalog config_schema, dashboard PATCH — built from WIDGET_SPECS (no per-type hand wiring) |
api/widget_openapi_serializers.py |
Stable re-export surface for dashboard.api imports (implementation in widget_specs/openapi.py) |
Frontend config layering (do not duplicate the full schema by hand):
| File | Role |
|---|---|
generated/widget-config-schemas/*.zod.ts |
Per-component Orval Zod (ErrorTrackingListWidgetConfig, shared WidgetDateRange, etc.) |
generated/widget-configs.zod.ts |
Friendly re-exports, inferred types, form .pick() schemas (insightscli build:widget-types) |
generated/widget-config-property-keys.json |
Per-type top-level config keys from catalog OpenAPI slice (generate-widget-config-zod.mjs) |
generated/widget-date-from-options.json |
Date preset value + label pairs from constants.py (build-dashboard-widget-types.py) |
generated/widget-form-fields.json |
Per-widget modal field manifest from WidgetSpec.form_fields (build-dashboard-widget-types.py) |
widgets/widgetConfigValidation.ts |
Shared InsightsQL filter helpers + parseWidgetConfigApiError — not the per-type schema |
widget_types/widgetConfigShared.ts |
Re-exports date select options from generated JSON + resolveWidgetFilterTestAccounts |
widgets/*/*WidgetConfigValidation.ts |
Import generated form schema; API error parsing only (colocated with validation) |
widget_types/catalog.ts |
Hand-written: labels, layouts, previews, defaultConfig via generated Zod |
Copy spine defaults for new types: widget-intake.md § Defaults.
Codegen & CI
One command — no separate widget codegen step:
insightscli build:openapi # openapi-schema → build:widget-types → openapi-types → MCP
Widget config Zod is product-scoped: products/dashboards/frontend/bin/generate-widget-config-zod.mjs slices the catalog op with filterSchemaByOperationIds (dashboards_widget_catalog_retrieve, includeResponseSchemas: true), then runs Orval 8.14+ (tools/openapi-codegen) with generateReusableSchemas: true into generated/widget-config-schemas/ and barrels friendly exports in widget-configs.zod.ts — separate from frontend/bin/generate-openapi-types.mjs. OpenAPI must expose a non-empty DashboardWidgetConfig oneOf (pydantic_openapi.py) or Orval emits an empty union type.
Local dev: Vite reads committed files under products/dashboards/frontend/generated/ — no regen on insightscli up / save. After widget_specs/ or serializer changes, run insightscli build:openapi and commit the generated diff. No pre-commit hook.
CI (check-openapi-types in ci-backend.yml): runs the same insightscli build:openapi, then diffs generated outputs. Same-repo PRs may auto-commit drift; fork PRs and unpushed fixes fail with “run insightscli build:openapi locally”. Triggers on products/**/backend/** (incl. widget_specs/) and products/*/frontend/generated/**.
New widget_type: add a Pydantic *WidgetConfig in widget_specs/configs.py following the *ListWidgetConfig → *WidgetConfig naming convention — build:widget-types auto-derives Orval export names and fails if the OpenAPI slice is missing the model.
Schema gen blocker: build:openapi-schema uses --fail-on-warn. Polymorphic per-type OpenAPI serializers each use a singleton ChoiceField for widget_type; dashboard.py uses the full EXPECTED_WIDGET_TYPES list — drf-spectacular enum names collide. When shipping a new type, add {YourWidgetTypeEnum: ["your_widget_type"]} to ENUM_NAME_OVERRIDES in insights/settings/web.py — insightscli build:widget-types and test_widget_openapi_enums.py fail if a registry type lacks an override; spectacular collision test fails if the override hash is wrong. Diagnostic: python manage.py find_enum_collisions (logic in insights/openapi/enum_collisions.py). See /improving-drf-endpoints.
Schema parity tests (cheap drift guards — run when touching widget_specs/):
insightscli test products/dashboards/backend/api/test/test_widget_config_schema_parity.py
insightscli test products/dashboards/frontend/widgets/widgetConfigSchemaParity.test.ts
Footguns (config / codegen)
| Mistake | Fix |
|---|---|
Slimming PatchedDashboardOpenApiSerializer when adding widget fields |
extend_schema(request=...) replaces the whole PATCH schema — extend the class, never rewrite from scratch; CI: test_dashboard_openapi.py compares runtime DashboardSerializer writables (minus api/test/dashboard_openapi_test_helpers.py exclusions) to serializer + spectacular output; MCP test chains dashboard-update schema to DashboardsPartialUpdateBody |
| Nested per-type widget config serializer on dashboard PATCH | Keep tile config as JSONField on runtime serializers — typed OpenAPI lives in widget_specs/openapi.py (PatchedDashboardOpenApiSerializer) only |
| Hand-writing FE Zod config schemas | widget-configs.zod.ts from codegen; add Pydantic *WidgetConfig + form_fields on WidgetSpec in registry.py |
Importing shared insights.schema models into widget config |
Prefer local Pydantic models in configs.py (e.g. WidgetAssigneeFilter) — avoids OpenAPI component name collisions in spectacular |
Duplicating catalog config_schema by hand |
BE catalog uses config_model.model_json_schema() — agents get bounds/choices/descriptions, not just default values |
| Editing generated Zod/TS without regen | Run insightscli build:openapi, commit products/dashboards/frontend/generated/* — CI check-openapi-types diffs and fails or auto-commits |
insightscli build:openapi-schema fails on warnings |
--fail-on-warn — use find_enum_collisions + ENUM_NAME_OVERRIDES; see § Codegen & CI |