Insights Rust Workspace
The insights/rust directory serves as Insights's "Rust monorepo" hosting Rust libraries and service implementations. This is not the Rust client library for Insights.
Catalog
Some selected examples of subprojects homed in the Rust workspace.
capture
This is the microservice that receives HTTP event capture requests and extracts the event payloads, lightly preprocesses the events, and passes well-formed events along as Kafka messages for downstream validation, CDP processing, and ingestion into Postgres and Datastore.
feature-flags
The new feature flags backend service that handles flag-scoped Insights API requests. Currently, /flags endpoint is natively supported, and legacy /decide traffic is gradually being shifted to the Rust service.
cymbal
Processing inbound source map payloads into symbols for the Insights error tracking product.
property-defs-rs
Extracts event and property definitions from the post-processed event stream and infers data types for the same. Stores these in Postgres for use in event/property lookup widgets in the Insights product UI.
This service has some known data quality issues that are being addressed by a re-architecture effort. That effort may result in a change of role or even decomissioning of this service in the near future.
batch-import-worker
Backend service that handles Insights user requests to import bulk datasets from external sources (S3, Amplitude, Segment, etc.)
kafka-deduplicator
PoC service that will deduplicate events in the ingestion pipeline within a given time window.
cyclotron
Rust services to manage a job queuing service backed by Postgres. Includes NodeJS API bindings and backend state management functionality.
scriptvm
A Rust re-implementation of the ScriptVM stack machine for evaluating compiled InsightsQL bytecode.
common
Miscellaneous internal Rust libraries reused by service implementations.
Requirements
- Rust.
- Docker, or podman and podman-compose: To setup development stack.
Other useful links for those new to Rust:
Local Development
Start up and bootstrap the "top-level" insights repo dev environment, including the Docker-Compose support services. Ensure that bin/migrate has run and bin/start behaves as expected. Leave the Docker services running when developing in the Rust workspace. The bin/start processes are typically optional for running Rust tests or the inner dev loop.
You may optionally seed data using the "top-level" management console scripts:
# from repo root
$ ./manage.py generate_demo_data
$ ./manage.py setup_test_environment
# run tests for all workspace projects
$ cd rust
$ cargo test
Migrations for most Rust workspace subprojects are managed by the sqlx tool. When running cargo or sqlx commands from within the insights/rust workspace root directory, the top-level insights Docker Compose database is targeted as specified in the DATABASE_URL env var injected by the rust/.env file.