zeekay
1b367e2b62
The tree was a debranded snapshot of upstream taken around 2026-02-20, and upstream has moved about five months since. This replaces it with upstream master, mechanically re-derived: bin/debrand does the rename, and the pipeline and squashed migrations are restored from the shipped tree. Not a merge. The previous import replaced the tree wholesale in a single commit, so there is no shared content history to merge against. Reproducing the rename on the new tree is what actually converges. What survives: .hanzo/workflows, the Dockerfiles and bin/docker* entrypoints, compose files, LLM.md, and insights/migrations. The squashed initial migration is load-bearing — the live database records it by name, so upstream's ~918 individual migrations cannot come back. Nine products are dropped as unreferenced. Twenty-one more that we do not run are kept anyway because the core app still imports them; unpicking those is a refactor and does not belong in a rebase. This base does not build yet, and the reasons are not in the rename: upstream now wants Python 3.13.13 against our 3.12.12 and Django 5.2 against our 4.2, and the dependency manifests still name upstream packages that we republish under our own names. Those are decisions, not renames.
3.5 KiB
title, sidebar, showTitle
| title | sidebar | showTitle |
|---|---|---|
| Tech stack | Docs | true |
Note: This page refers to our main product repository, not our website.
Frontend
- Web framework/library: React
- State management: Redux + Kea
- Layout/components: Ant Design
Backend
- Framework: Django
- High scale services: Rust
- Databases: PostgreSQL and Datastore
- Task queue/event streaming: Redis and Apache Kafka
- Task Worker: Celery, Temporal and Dagster
Testing
- Frontend E2E tests: Playwright
- Backend tests: Pytest and Django's built-in test suite
Additional tools
- CI/CD: GitHub Actions
- Containerization: Docker and Docker Compose
- Linter (frontend): ESLint
- Formatter (backend): Black
Workflow orchestration
We historically used Celery as our task worker. At Insights’s current scale Celery can be unreliable for larger or long-running workflows, but it remains a practical fit for small, low-latency background tasks (e.g. sending emails or other quick async side-effects). New medium- and large-scale jobs should use Temporal or Dagster instead.
We use both Temporal and Dagster for more complex orchestration, each chosen for their specific strengths.
When to use each tool
We tend to use Celery for lightweight ad-hoc tasks, Dagster for internal data jobs, and Temporal for user-facing workflows. Before choosing between Dagster and Temporal, check the cost note below.
- Is it a tiny, low-latency, fire-and-forget task (e.g. send email)? → Celery
- Is it mission-critical with complex failure scenarios? → Temporal
- Do you need exactly-once guarantees? → Temporal
- Do you need complex retry policies or long-running workflows? → Temporal
- Is it primarily about scheduled data transformation? → Dagster or Temporal
- Do you need rich data lineage and testing? → Dagster
Cost: Temporal vs Dagster
Temporal is dramatically cheaper than Dagster for recurring workloads. Dagster charges $0.02–0.04 per credit (1 credit = 1 asset materialization), while Temporal charges $0.00005 per action (1 action ≈ 1 activity execution). That's a ~300x difference per operation.
In practice, when we migrated experiment timeseries recalculation from Dagster to Temporal, the cost went from ~$6k/month to ~$20/month for the same workload. Keep this in mind before adding new Dagster jobs — if Temporal can do the job, it almost certainly should.
Where do we use each?
Celery: Small, fast background tasks (e.g. sending email, minor async operations) Temporal: Batch exports, data warehouse source syncing, AI platform task generation, experiment timeseries recalculation Dagster: Exchange rate tracking, one-off production management commands, web analytics data pre-processing