Use `runsc --rootless --network=none do` for the `GVisorC` runtime command
instead of Docker's `--runtime=runsc`. The Docker runtime flag requires daemon
configuration and fails with `unknown or invalid runtime name: runsc` when
`runsc` is installed but not registered in Docker.
Keep `CLICKHOUSE_C_DRIVER_GVISOR_RUNTIME` as a compatibility fallback, and add
`CLICKHOUSE_C_DRIVER_GVISOR_BINARY` for the direct `runsc` executable path.
PR: https://github.com/ClickHouse/ClickHouse/pull/105131
Tests:
- `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_driver_common.py`
- `CLICKHOUSE_C_DRIVER_COMPILE_LOCAL=1 gvisor_c_create.sh --return UInt64 --args 'x UInt64, y UInt64'`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Rename the example `c_function_body` driver to `DockerC` and move the shared
C-body implementation into `c_driver_common.py`. Add `GVisorC` and `UnsafeC`
wrappers/configs that reuse the same generator.
`GVisorC` keeps the Docker command shape but adds `--runtime=runsc` for running
the compiled function. `UnsafeC` compiles locally and emits the compiled
`user_func` path directly without Docker or gVisor.
PR: https://github.com/ClickHouse/ClickHouse/pull/105131
Tests:
- `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_driver_common.py`
- `CLICKHOUSE_C_DRIVER_COMPILE_LOCAL=1 docker_c_create.sh --return UInt64 --args 'x UInt64, y UInt64'`
- `CLICKHOUSE_C_DRIVER_COMPILE_LOCAL=1 gvisor_c_create.sh --return UInt64 --args 'x UInt64, y UInt64'`
- `unsafe_c_create.sh --return UInt64 --args 'x UInt64, y UInt64'`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Call `loadUserDefinedExecutableFunctionDrivers` during initial
`clickhouse-server` startup before persisted user-defined functions are loaded.
This makes source-tree driver configs such as `c_function_body` available
without waiting for a later config reload.
Resolve driver config directories to absolute paths before applying relative
`create_command` and `drop_command` entries. Driver commands run from a
per-function work directory, so a relative config directory would otherwise make
source-tree example commands relative to the wrong directory.
Improve the unknown-driver exception so it distinguishes an empty driver
registry from a typo in the requested driver name.
Tests:
`ninja -C build programs/clickhouse`
`CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
`CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
`clickhouse-server` default-config smoke with `c_function_body`
Refs: https://github.com/ClickHouse/ClickHouse/pull/105131
Validate `CREATE FUNCTION ... ENGINE` against existing user-defined
function storage before invoking driver `create_command`, so failed creates
and `IF NOT EXISTS` no-ops do not run driver side effects. Persist the
normalized `ATTACH FUNCTION` query before reloading the generated executable
UDF configuration.
Drain driver stdout and stderr concurrently in
`UserDefinedExecutableFunctionDriverInvoker` to avoid pipe-buffer deadlocks,
and keep stderr attached to non-zero and abnormal child exits.
Resolve relative `create_command` and `drop_command` paths relative to the
driver config file, fix the example XML for `xmllint`, and clean up the
dynamic executable UDF config repository during context shutdown.
Tests:
`ninja -C build programs/clickhouse`
`./tests/clickhouse-test -b ./build/programs/clickhouse 04241_executable_udf_driver`
`./tests/clickhouse-test -b ./build/programs/clickhouse 04242_executable_udf_driver_errors`
Refs: https://github.com/ClickHouse/ClickHouse/pull/105131
A driver describes how to create executable user-defined functions; calling
the surrounding element `<function>` was misleading - it suggested the file
described a single function. Rename to `<driver>` everywhere (example config,
parsing in the registry, the doc comment, both stateless tests).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Introduce "drivers" for executable user-defined functions. A driver is a
small adapter program declared in a server config under
`<user_defined_executable_function_drivers_config>` that knows how to take a
user-supplied code snippet and produce a runnable executable user-defined
function configuration.
New CREATE FUNCTION syntax:
CREATE [OR REPLACE] FUNCTION [IF NOT EXISTS] name [ON CLUSTER c]
ARGUMENTS (a UInt8, b String) RETURNS UInt64
ENGINE = DriverName(key1 = 'value1', key2 = 42)
AS '...code body...'
On CREATE FUNCTION, the corresponding `create_command` of the driver is
invoked: arguments are passed via `--name`, `--return`, `--args` and the
declared engine arguments; the body is sent on stdin; the driver writes the
resulting executable UDF configuration (XML or YAML) to stdout, which is
saved into the new `<dynamic_user_defined_executable_functions_path>` and
picked up by the existing executable UDF loader. The corresponding SQL is
persisted as `ATTACH FUNCTION ...` in `<user_defined>` and re-evaluated on
restart: if the dynamic configuration is still present, the driver is not
invoked; otherwise the driver is re-run to materialize it.
On DROP FUNCTION, the `drop_command` (if any) is invoked, then the dynamic
configuration file and per-function working directory are removed.
Includes a proof-of-concept driver `c_function_body`, which compiles a user
C body into a binary (inside Docker by default for build-time isolation) and
runs it inside Docker at query time with strict sandboxing (`--network=none`,
`--read-only`, dropped capabilities, no-new-privileges, unprivileged user,
resource limits) in `executable_pool` mode for batched per-row evaluation.
See https://github.com/ClickHouse/ClickHouse/issues/71172
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Alexey Milovidov