Implements several reviewer-requested changes for driver-based executable
user-defined functions:
- Gate the feature behind a new experimental server setting
`allow_experimental_executable_udf_drivers` (disabled by default, mirroring
`allow_experimental_webassembly_udf`). When disabled, no drivers are loaded
into the registry and `CREATE FUNCTION ... ENGINE = DriverName(...)` throws
`SUPPORT_IS_DISABLED`.
- Reject driver-based `CREATE FUNCTION` when the user-defined SQL object storage
is replicated: driver artifacts (dynamic config and working directory) are
materialized only on the receiving replica, so the function would be
unavailable on the others and after `RESTORE`.
- Require `AS '...code...'` for driver-based `CREATE FUNCTION` in the parser, so
an accidental omission of `AS` becomes a syntax error instead of a
side-effectful create with an empty body.
- Honor `IF NOT EXISTS` in the executable-UDF existence check
(`UserDefinedExecutableFunctionFactory::has`) instead of throwing
`FUNCTION_ALREADY_EXISTS` unconditionally.
- Clear the driver registry when the driver config resolves to no files (or the
feature is disabled) on `SYSTEM RELOAD CONFIG`, so removed drivers stop being
usable immediately instead of lingering until restart.
- Add `ASTCreateFunctionWithDriverQuery` to `isUserDefinedFunctionQuery` in
`removeOnClusterClauseIfNeeded` so the new create path honors
`ignore_on_cluster_for_replicated_udf_queries`.
- Add `--security-opt=no-new-privileges` to the `c_function_body` driver's
compile and runtime `docker run` sandboxes.
- Enable `allow_experimental_executable_udf_drivers` in the stateless test
configs for `04241_executable_udf_driver` and `04242_executable_udf_driver_errors`.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Start each `executable_pool` worker through `docker run` with its own FIFO directory instead of sharing one long-lived container and using `docker exec`.
Keep `CLICKHOUSE_C_DRIVER_CPUS` as a per-worker limit and clean up any worker containers by the generated `DockerC` prefix on drop.
Tests: python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_driver_common.py
Tests: CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh
Tests: CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh
Tests: `DockerC` `numbers_mt(5000000)` with `max_threads = 64`
The persistent `DockerC` container hosts the whole `executable_pool`, so the previous default `--cpus=1.0` throttled all pooled workers to one core total.
Default the Docker CPU quota to `pool_size`, while preserving the explicit `CLICKHOUSE_C_DRIVER_CPUS` override.
Tests:
- `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_driver_common.py`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
- real `DockerC` `clickhouse-local` query with `numbers_mt(5000000)` and `max_threads = 64`
Keep each pooled `DockerC` process on its own FIFO directory created by the runner at process startup, and keep Docker CLI output out of the UDF result stream by redirecting it to a per-process log.
The runner now waits for the per-process `docker exec` first. If Docker cannot start the container-side function, the runner forwards the captured Docker error to stderr and terminates the FIFO proxy processes instead of letting Docker messages corrupt `Buffers` output.
Also size Docker's `pids-limit` from `pool_size` with enough headroom for concurrent `docker exec` startup. The previous limit was too close to the number of pooled functions and could fail under `max_threads = 64` with `OCI runtime exec failed`.
Tests:
- `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_driver_common.py`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
- real `DockerC` `clickhouse-local` query with `numbers_mt(5000000)` and `max_threads = 64`
Run `docker exec` attached in the background instead of detached so its exit status and stderr are observable, but redirect the Docker CLI stdin from `/dev/null` because the compiled function reads from the named pipe.
Also duplicate the runner stdin explicitly for the background writer. Some shells give background jobs `/dev/null` as stdin by default, which made `cat` send EOF to the input FIFO immediately and produced empty `Buffers` results.
Tests:
- `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_driver_common.py`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
- real `DockerC` `clickhouse-local` create/select/drop query
Resolve `execute_direct` commands for driver-generated executable UDFs relative to the function's UUID working directory from the `.workdir` sidecar instead of `user_scripts_path`.
This lets the C example drivers emit relative commands such as `user_func` and `docker_fifo_runner.sh`, while regular executable UDF configs continue to use `user_scripts_path`.
Tests:
- `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_driver_common.py`
- `ninja -C build programs/clickhouse`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Use `<execute_direct>0</execute_direct>` for the generated `DockerC` FIFO
runner command and prefix the command with shell `exec`. This avoids the
`user_scripts_path` validation that applies to direct executable commands,
because the runner lives in the dynamic UDF work directory, not in
`user_scripts`.
PR: https://github.com/ClickHouse/ClickHouse/pull/105131
Tests:
- `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_driver_common.py`
- `CLICKHOUSE_C_DRIVER_COMPILE_LOCAL=1 CLICKHOUSE_C_DRIVER_SKIP_DOCKER_CONTAINER=1 docker_c_create.sh --return UInt64 --args 'x UInt64, y UInt64'`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Default generated C-body driver configs to `command_pipe_capacity` `1048576`
for the `GVisorC` runtime, while keeping `DockerC` and `UnsafeC` on the OS
default unless `CLICKHOUSE_C_DRIVER_PIPE_CAPACITY` is explicitly set.
This preserves the throughput improvement measured with `GVisorC` without
reintroducing the slower enlarged pipe path for `DockerC`.
PR: https://github.com/ClickHouse/ClickHouse/pull/105131
Tests:
- `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_driver_common.py`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Change the generated `DockerC` runtime command from `docker exec -i` to a
per-function runner script that creates FIFO files inside the UDF work
directory. The script starts `/work/user_func` in the persistent Docker
container with its `stdin` and `stdout` redirected to those FIFOs, and then
relays ClickHouse input and output through host-side `cat` processes.
This keeps the hot data path out of Docker's `exec -i` attach stream while
leaving all temporary files inside the UDF work directory.
Also stop requesting `command_pipe_capacity` by default from generated C-body
driver configs, so this experiment is not mixed with the previous pipe-capacity
tuning.
PR: https://github.com/ClickHouse/ClickHouse/pull/105131
Tests:
- `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_driver_common.py`
- `CLICKHOUSE_C_DRIVER_COMPILE_LOCAL=1 CLICKHOUSE_C_DRIVER_SKIP_DOCKER_CONTAINER=1 docker_c_create.sh --return UInt64 --args 'x UInt64, y UInt64'`
- `sh -n tmp/pr105131_docker_fifo/create/docker_fifo_runner.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Add an optional `command_pipe_capacity` setting for executable commands and
pass it through `ShellCommandSource` to `ShellCommand`, where it uses the
existing `PipeFDs::tryIncreaseSize` helper on the command pipes before `exec`.
The generated C-body drivers request a 1 MiB pipe capacity and the wrapper also
tries to increase its own `stdin` and `stdout` pipe capacity. This can reduce
short-write wakeups in the `DockerC` path, while preserving the previous default
for other executable command users.
PR: https://github.com/ClickHouse/ClickHouse/pull/105131
Tests:
- `ninja -C build programs/clickhouse`
- `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_driver_common.py`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Start one idle Docker container per `DockerC` function at create time and
generate `docker exec -i <container> /user_func` for the `executable_pool`
command. This avoids creating and removing a Docker container for every pool
worker while preserving the container-level network, filesystem, capability,
memory, CPU, and pids restrictions.
The `drop` command removes the persistent container, and the stateless test
checks the generated `docker exec` command and container-name metadata without
requiring Docker.
PR: https://github.com/ClickHouse/ClickHouse/pull/105131
Tests:
- `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_driver_common.py`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Add `--log-driver=none` to Docker invocations generated by `DockerC`. The UDF
protocol streams binary `Buffers` data over stdout, and Docker's default log
driver can otherwise duplicate that stream through container logging while it is
also attached back to ClickHouse.
PR: https://github.com/ClickHouse/ClickHouse/pull/105131
Tests:
- `docker run --rm --log-driver=none alpine:3.20 sh -c 'printf hello'`
- `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_driver_common.py`
- `CLICKHOUSE_C_DRIVER_COMPILE_LOCAL=1 docker_c_create.sh --return UInt64 --args 'x UInt64, y UInt64'`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Use `runsc --rootless --network=none do` for the `GVisorC` runtime command
instead of Docker's `--runtime=runsc`. The Docker runtime flag requires daemon
configuration and fails with `unknown or invalid runtime name: runsc` when
`runsc` is installed but not registered in Docker.
Keep `CLICKHOUSE_C_DRIVER_GVISOR_RUNTIME` as a compatibility fallback, and add
`CLICKHOUSE_C_DRIVER_GVISOR_BINARY` for the direct `runsc` executable path.
PR: https://github.com/ClickHouse/ClickHouse/pull/105131
Tests:
- `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_driver_common.py`
- `CLICKHOUSE_C_DRIVER_COMPILE_LOCAL=1 gvisor_c_create.sh --return UInt64 --args 'x UInt64, y UInt64'`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Rename the example `c_function_body` driver to `DockerC` and move the shared
C-body implementation into `c_driver_common.py`. Add `GVisorC` and `UnsafeC`
wrappers/configs that reuse the same generator.
`GVisorC` keeps the Docker command shape but adds `--runtime=runsc` for running
the compiled function. `UnsafeC` compiles locally and emits the compiled
`user_func` path directly without Docker or gVisor.
PR: https://github.com/ClickHouse/ClickHouse/pull/105131
Tests:
- `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_driver_common.py`
- `CLICKHOUSE_C_DRIVER_COMPILE_LOCAL=1 docker_c_create.sh --return UInt64 --args 'x UInt64, y UInt64'`
- `CLICKHOUSE_C_DRIVER_COMPILE_LOCAL=1 gvisor_c_create.sh --return UInt64 --args 'x UInt64, y UInt64'`
- `unsafe_c_create.sh --return UInt64 --args 'x UInt64, y UInt64'`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Generated wrappers now read `Buffers` payloads directly into column buffers
and process numeric columns through typed `__restrict` views. The numeric
result path writes into the result buffer as a typed array and flushes it
directly, assuming little-endian host layout.
Move per-chunk processing into `ch_process_chunk` and mark it plus
`user_function` inline, so simple bodies such as `UInt64` addition can be
inlined and vectorized by the compiler.
PR: https://github.com/ClickHouse/ClickHouse/pull/105131
Tests:
- `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_function_body_driver.py`
- `CLICKHOUSE_C_DRIVER_FORCE_LOCAL=1 c_function_body_driver.py create --return String --args 'x String, y String'`
- `CLICKHOUSE_C_DRIVER_FORCE_LOCAL=1 c_function_body_driver.py create --return UInt64 --args 'x UInt64, y UInt64'`
- direct `Buffers` protocol checks for `String` and `UInt64`
- `cc -O3 -march=native -fopt-info-vec-optimized -c tmp/pr105131_zero_copy_check/wrapper.c`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Rewrite the `c_function_body` example driver from https://github.com/ClickHouse/ClickHouse/pull/105131 to use the `Buffers` format instead of row-wise `RowBinary`.
The generated wrapper now reads one `Buffers` block per `send_chunk_header` chunk, validates the column count and row count, exposes arguments row-by-row to the user body, and writes the result as a single-column `Buffers` block. `String` columns are parsed once per chunk into `struct buf` views over the serialized column buffer, while returned `String` values are serialized into the result column buffer after calls to `alloc`.
Tests:
* `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_function_body_driver.py`
* direct `Buffers` protocol check in `build/test_c_function_body_driver_buffers_protocol.log`
* `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
* `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
* `git diff --check`
Extend the `c_function_body` example driver from https://github.com/ClickHouse/ClickHouse/pull/105131 to support the `String` type in `RowBinary`.
`String` arguments and return values are exposed to the C body as `struct buf { const char * data; size_t size; }`. The generated wrapper also provides `alloc`, a simple chunk-scoped bump allocator for returning strings. The allocator keeps buffers stable for the whole chunk and resets after the chunk is flushed.
Also compile generated functions with `-O3 -march=native`, while keeping the Docker build static.
Tests:
* `python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_function_body_driver.py`
* direct `RowBinary` protocol check in `build/test_c_function_body_driver_string_protocol.log`
* `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
* `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
* `git diff --check`
Replace the generated wrapper's `stdio`-based `RowBinary` IO with a small
buffered `read` and `write` layer. Parse the text `send_chunk_header` row count
manually and serialize primitive values by copying fixed-width `RowBinary`
values directly to and from the in-memory buffers.
This keeps the example protocol-compatible while avoiding per-value C library
IO and formatting overhead in the generated executable.
Tests:
- generated-wrapper protocol check with one 3-row chunk
- generated-wrapper benchmark: 200000 rows, about 25M rows/sec in this run
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Refs: https://github.com/ClickHouse/ClickHouse/pull/105131
Update the source-tree `c_function_body` example driver to generate an
`executable_pool` function using `RowBinary` with `send_chunk_header` enabled.
The generated wrapper now reads the text chunk row count, consumes that many
`RowBinary` argument rows, writes the same number of `RowBinary` result rows,
and flushes once per chunk.
Remove the generated `setvbuf` call; chunk-level flushing is enough and matches
the executable UDF protocol better than per-row line buffering.
Tests:
- direct generated-wrapper protocol check with one 3-row chunk
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
- `CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
Refs: https://github.com/ClickHouse/ClickHouse/pull/105131
The source-tree `c_function_body` example driver used Docker with
`--security-opt=no-new-privileges`, but this option prevents containers from
executing on some developer hosts. Keep Docker for the example, but drop that
option from both the build container and the generated runtime command.
Run the Docker containers as the server OS user so they can read and write the
server-created per-function work directory without changing directory
permissions.
Tests:
`python3 -m py_compile programs/server/user_defined_executable_function_drivers/c_function_body_driver.py`
`git diff --check`
`CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04241_executable_udf_driver.sh`
`CLICKHOUSE_BINARY=./build/programs/clickhouse tests/queries/0_stateless/04242_executable_udf_driver_errors.sh`
`clickhouse-server` default-config smoke with Docker-backed `c_function_body`
Refs: https://github.com/ClickHouse/ClickHouse/pull/105131
Introduce "drivers" for executable user-defined functions. A driver is a
small adapter program declared in a server config under
`<user_defined_executable_function_drivers_config>` that knows how to take a
user-supplied code snippet and produce a runnable executable user-defined
function configuration.
New CREATE FUNCTION syntax:
CREATE [OR REPLACE] FUNCTION [IF NOT EXISTS] name [ON CLUSTER c]
ARGUMENTS (a UInt8, b String) RETURNS UInt64
ENGINE = DriverName(key1 = 'value1', key2 = 42)
AS '...code body...'
On CREATE FUNCTION, the corresponding `create_command` of the driver is
invoked: arguments are passed via `--name`, `--return`, `--args` and the
declared engine arguments; the body is sent on stdin; the driver writes the
resulting executable UDF configuration (XML or YAML) to stdout, which is
saved into the new `<dynamic_user_defined_executable_functions_path>` and
picked up by the existing executable UDF loader. The corresponding SQL is
persisted as `ATTACH FUNCTION ...` in `<user_defined>` and re-evaluated on
restart: if the dynamic configuration is still present, the driver is not
invoked; otherwise the driver is re-run to materialize it.
On DROP FUNCTION, the `drop_command` (if any) is invoked, then the dynamic
configuration file and per-function working directory are removed.
Includes a proof-of-concept driver `c_function_body`, which compiles a user
C body into a binary (inside Docker by default for build-time isolation) and
runs it inside Docker at query time with strict sandboxing (`--network=none`,
`--read-only`, dropped capabilities, no-new-privileges, unprivileged user,
resource limits) in `executable_pool` mode for batched per-row evaluation.
See https://github.com/ClickHouse/ClickHouse/issues/71172
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
