mirror of https://github.com/hanzoai/insights.git synced 2026-10-30 16:15:31 +00:00

Files

T

zeekay 168c1136b4 Ship the funnel UDF config in the image

The funnel queries call the unversioned UDFs (aggregate_funnel_array and
kin), declared only in docker/datastore/user_defined_function.xml, which
no image carries. The binaries already ship in insights/user_scripts, so
the config moves beside them: a datastore takes both from one image.

2026-09-30 07:34:07 -07:00

2.3 KiB

Raw Permalink Blame History

Funnel UDFs

Development workflow

Building the binaries:
- Exit flox: exit
- Switch to the subfolder: cd funnel-udf
- Execute the build script: ./build.sh
- Restart the datastore docker container

Deployment workflow

The user_defined_function.xml deployed to Datastore is now pulled automatically from the master branch of this repo by ansible (insights-cloud-infra/ansible/roles/datastore/tasks/setup_udfs.yml). It downloads insights/user_scripts/latest_user_defined_function.xml and deploys it as Datastore's user_defined_function.xml.

For revertible cloud deploys:

Develop using the binary files at the top level of user_scripts (see section above), with schema defined in insights/user_scripts/user_defined_function.xml
When ready to deploy, increment the version in insights/udf_versioner.py and run it. This generates versioned binaries and latest_user_defined_function.xml
Land a PR with the updated user_scripts folder — include both the new version folder (e.g. v12/) and the regenerated latest_user_defined_function.xml. Do not include the UDF_VERSION bump in this PR — that goes in a separate PR (step 5)
The binaries and XML should be automatically deployed to Datastore on the next config run. Verify in metabase by running SELECT aggregate_funnel_vXX(). An "invalid arguments" error means the function is registered. An "unknown function" error means the deploy hasn't taken effect yet. Make sure to do this for both EU and US
Once the deploy is confirmed, land a separate PR that bumps UDF_VERSION in insights/udf_versioner.py to the new version. This switches the insights query builder to use the new UDF

Design decisions

We tried writing the UDFs in Python, but this cause Datastore nodes to crash as the garbage collector could not keep up. As such we're now writing the functions in Rust.
We're using cross-rs to compile the binaries - it's a cross platfrom rust compiler.

Troubleshooting

Running `./build.sh` fails in flox environment

If you're using flox for development, you'll have to exit out of the environment first (nix os isn't supported).

Error:
   0: could not determine os in target triplet
   1: unsupported os in target, abi: "1.82.0", system: "rustc"

2.3 KiB Raw Permalink Blame History