mirror of
https://github.com/hanzoai/insights.git
synced 2026-10-30 16:15:31 +00:00
zeekay
168c1136b4
The funnel queries call the unversioned UDFs (aggregate_funnel_array and kin), declared only in docker/datastore/user_defined_function.xml, which no image carries. The binaries already ship in insights/user_scripts, so the config moves beside them: a datastore takes both from one image.
Funnel UDFs
Development workflow
- Building the binaries:
- Exit flox:
exit - Switch to the subfolder:
cd funnel-udf - Execute the build script:
./build.sh - Restart the datastore docker container
- Exit flox:
Deployment workflow
The user_defined_function.xml deployed to Datastore is now pulled automatically from the master branch of this repo by ansible (insights-cloud-infra/ansible/roles/datastore/tasks/setup_udfs.yml).
It downloads insights/user_scripts/latest_user_defined_function.xml and deploys it as Datastore's user_defined_function.xml.
For revertible cloud deploys:
- Develop using the binary files at the top level of
user_scripts(see section above), with schema defined ininsights/user_scripts/user_defined_function.xml - When ready to deploy, increment the version in
insights/udf_versioner.pyand run it. This generates versioned binaries andlatest_user_defined_function.xml - Land a PR with the updated
user_scriptsfolder — include both the new version folder (e.g.v12/) and the regeneratedlatest_user_defined_function.xml. Do not include theUDF_VERSIONbump in this PR — that goes in a separate PR (step 5) - The binaries and XML should be automatically deployed to Datastore on the next config run. Verify in metabase by running
SELECT aggregate_funnel_vXX(). An "invalid arguments" error means the function is registered. An "unknown function" error means the deploy hasn't taken effect yet. Make sure to do this for both EU and US - Once the deploy is confirmed, land a separate PR that bumps
UDF_VERSIONininsights/udf_versioner.pyto the new version. This switches the insights query builder to use the new UDF
Design decisions
- We tried writing the UDFs in Python, but this cause Datastore nodes to crash as the garbage collector could not keep up. As such we're now writing the functions in Rust.
- We're using
cross-rsto compile the binaries - it's a cross platfrom rust compiler.
Troubleshooting
Running ./build.sh fails in flox environment
If you're using flox for development, you'll have to exit out of the environment first (nix os isn't supported).
Error:
0: could not determine os in target triplet
1: unsupported os in target, abi: "1.82.0", system: "rustc"